system

Abstract

With conventional countermeasures, it has been difficult to detect signs of fraud in advance and issue warnings in real time. For this reason, we provide an effective system to prevent fraud damage before it occurs. A means for receiving audio data; means for analyzing received audio data; means for calculating a risk score based on the analysis of the voice data; means for generating a warning message based on the risk score; means for transmitting the generated warning message to a user terminal; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Currently, financial crimes such as "I am your son" fraud and "transfer fraud" are on the rise, increasing the risk of people becoming victims of these scams. Socially vulnerable groups, such as the elderly, are particularly likely to be targeted, and effective countermeasures against this are needed. Conventional countermeasures have struggled to detect signs of fraud in advance and issue warnings in real time. Therefore, an effective system is needed to prevent fraud damage before it occurs. [Means for solving the problem]

[0005] The present invention provides a system that uses a user's smartphone or ATM to detect signs of fraud in real time and issue a warning. Specifically, the system includes a means for receiving and analyzing audio data, a means for calculating a risk score based on the analysis results, and a means for generating a warning message based on the risk score and sending it to the user's terminal. Furthermore, by including a means for receiving and analyzing video data, more detailed behavioral analysis can be performed and reflected in the risk score. This makes it possible to warn users of fraud risks in real time and prevent fraud damage.

[0006] "Voice data" refers to audio information uttered by a user recorded in digital format.

[0007] "Means for receiving" refers to means for obtaining data via a network or communication protocol.

[0008] "Means of analysis" are algorithms and software that process the received data and understand and evaluate its content.

[0009] A "risk score" is an evaluation value that quantifies or quantifies the likelihood of fraud.

[0010] A "warning message" is a notification that includes wording to alert users to the risk of fraud.

[0011] A "user terminal" is a device used by a user, and is an electronic device including a smartphone or ATM.

[0012] "Video Data" means video information recorded by a security camera or other video capture device.

[0013] A "behavioral pattern" is the result of analyzing a series of actions that indicate a user's movements and trading tendencies.

[0014] "Natural language processing" is a technology for analyzing voice and text data to understand and process human language.

[0015] A "machine learning model" is an algorithm that learns patterns and rules from data and makes predictions and classifications.

[0016] An "HTTP request" is a protocol for sending data to a web server.

[0017] "WebSocket" is a protocol that enables real-time, two-way communication between a browser and a server.

[0018] "Encryption" is the process of converting data into an unintelligible form in order to protect it. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9]1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0020] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0021] First, the terms used in the following description will be explained.

[0022] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0023] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0024] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0025] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0026] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0027] [First embodiment]

[0028] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0029] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0030] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0031] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0032] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0033] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0034] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0035] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0036] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0037] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0038] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0039] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0040] This invention is a system designed to prevent financial frauds such as "I'm your son" frauds and "transfer frauds." The system utilizes audio and video data to assess fraud risks in real time and issue warnings. The system is primarily composed of a server, terminals (smartphones and ATMs), and users.

[0041] Server-side processing

[0042] Data reception and initial analysis

[0043] The server receives voice and video data sent from the user's smartphone or ATM. For voice data, a speech recognition engine converts the speech into text, and a natural language processing (NLP) module analyzes keywords and context. For video data, a behavior recognition algorithm is applied to analyze the user's movements and behavioral patterns.

[0044] Risk Assessment

[0045] The server uses a machine learning model to calculate a risk score based on the results of voice and video data analysis. For example, keywords that indicate fraud, such as "transfer" and "secret," as well as behavioral patterns such as frequent large cash withdrawals, affect the risk score.

[0046] Alerting and Notifications

[0047] If the risk score exceeds a certain threshold, the server determines it to be a high risk and generates a warning message. This warning message is sent to the user's smartphone or ATM terminal. Security companies are also notified at the same time, enabling a prompt response.

[0048] Smartphone device processing

[0049] Audio recording and transmission

[0050] When a user makes a call through their smartphone, the device records the audio in the background, and the recorded audio data is encrypted and sent to a server using the HTTPS protocol.

[0051] Receiving and displaying alerts

[0052] When an alert is sent from the server, the smartphone receives a real-time warning message and displays it on the screen. This message may say something like, "This call may be fraudulent." By seeing this warning, the user can recognize the risk of fraud and take appropriate measures.

[0053] ATM terminal processing

[0054] Video recording and transmission

[0055] When a user operates an ATM, the security camera records the operation. This video data is encrypted and sent to a server. The server analyzes the video data to detect unusual behavior by the user or large cash withdrawals.

[0056] Receiving and displaying alerts

[0057] When an alert from the server is sent to the ATM, a warning message will appear on the ATM screen, specifically stating, "Please be aware that you may be a victim of fraud." In some cases, the ATM may also include the option to pause the transaction and offer direct assistance to the user.

[0058] Specific examples

[0059] Specific examples when using a smartphone

[0060] 1. When a user initiates a call with a scammer, the device records the audio and sends it to a server.

[0061] 2. The server analyzes the voice data and detects keywords such as "transfer."

[0062] 3. The server determines the risk to be high and sends a warning message to the smartphone.

[0063] 4. The user's smartphone displays a warning that the call may be fraudulent.

[0064] Examples of ATM usage

[0065] 1. When a user attempts to withdraw a large amount of cash from an ATM, the security camera records the footage and sends it from the ATM to a server.

[0066] 2. The server analyzes behavioral patterns and detects unusual transactions.

[0067] 3. The server determines the risk to be high and sends a warning message to the ATM.

[0068] 4. The ATM screen displays the message, "Please be aware that you may be a victim of fraud."

[0069] 5. If necessary, trading will be temporarily suspended and the user will be advised to contact the support center.

[0070] In this way, the system analyzes both audio and video for signs of fraud and assesses the risk, enabling users to receive real-time warnings and take prompt action to prevent fraud before it occurs.

[0071] The processing flow will be explained below.

[0072] Server-side processing

[0073] Step 1:

[0074] The server receives audio and video data sent from the user's smartphone or ATM via HTTP requests or WebSockets.

[0075] Step 2:

[0076] The server passes the received voice data to a voice recognition engine and converts it into text data.

[0077] Step 3:

[0078] The text data converted by the speech recognition engine is sent to a natural language processing (NLP) module, where it is analyzed for keywords and context.

[0079] Step 4:

[0080] The server passes the received video data to a behavior recognition algorithm to analyze the user's movements and behavior patterns.

[0081] Step 5:

[0082] The results of audio data analysis and video data analysis are integrated and input into a machine learning model to calculate a risk score.

[0083] Step 6:

[0084] If the risk score exceeds a certain threshold, the server generates a warning message.

[0085] Step 7:

[0086] The server sends the generated warning message to the user's smartphone or ATM terminal, and simultaneously notifies the security company.

[0087] Smartphone device processing

[0088] Step 1:

[0089] A user initiates a call on their smartphone, and an app on the smartphone automatically records the audio in the background.

[0090] Step 2:

[0091] The recorded audio data is encrypted and sent to the server using the HTTPS protocol.

[0092] Step 3:

[0093] When an alert is sent from the server, the smartphone receives a warning message in real time.

[0094] Step 4:

[0095] The app displays a warning message on the user's screen, providing specific warning information and recommended actions.

[0096] ATM terminal processing

[0097] Step 1:

[0098] When a user begins to operate the ATM, the security camera records the activity.

[0099] Step 2:

[0100] The recorded video data is encrypted and sent to the server.

[0101] Step 3:

[0102] When an alert is sent from the server, the ATM receives a warning message in real time.

[0103] Step 4:

[0104] A warning message appears on the ATM screen, stating, "You may be a victim of fraud."

[0105] Step 5:

[0106] If a transaction is deemed high risk, the ATM terminal will temporarily suspend the transaction and prompt the user to contact the support center.

[0107] Specific examples

[0108] Specific examples when using a smartphone

[0109] Step 1:

[0110] The user initiates a call with the scammer, and an app on their phone automatically records the audio in the background.

[0111] Step 2:

[0112] The recorded audio data is encrypted and sent to the server using the HTTPS protocol.

[0113] Step 3:

[0114] The server analyzes the audio data and detects keywords that indicate fraud, such as "transfer."

[0115] Step 4:

[0116] The server determines the risk to be high and sends a warning message to the user's smartphone.

[0117] Step 5:

[0118] The user's smartphone will display a warning that "This call may be fraudulent."

[0119] Examples of ATM usage

[0120] Step 1:

[0121] A user attempts to withdraw a large amount of cash from an ATM. The security camera records the footage, encrypts it, and sends it to a server.

[0122] Step 2:

[0123] The server analyzes the video data and detects unnatural trading behavior patterns.

[0124] Step 3:

[0125] The server determines the risk is high and sends a warning message to the ATM.

[0126] Step 4:

[0127] A warning message appears on the ATM screen saying, "Please be careful, you may be a victim of fraud."

[0128] Step 5:

[0129] If necessary, the ATM will temporarily halt the transaction and give the user the option to contact a support center.

[0130] Example 1

[0131] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0132] In recent years, financial frauds such as "I'm your son" scams and bank transfer scams have been increasing, and many people have fallen victim to them. These scams are becoming more sophisticated and cannot be prevented by simple warnings or alerts. Elderly people and users who are not tech-savvy are particularly at high risk of falling victim to fraud, so there is a need for a system that can efficiently and quickly detect fraud and issue warnings to users. Conventional prevention methods rely on analyzing audio and video separately, which has limitations in detecting complex fraudulent behavior.

[0133] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0134] In this invention, the server includes means for receiving voice data, means for converting the received voice data into text, means for analyzing the converted text and identifying specific keywords and contexts, means for receiving video data, means for analyzing the received video data and identifying behavioral patterns, means for calculating a risk score based on the analysis results of the voice data and video data, means for generating a warning message based on the risk score, and means for transmitting the generated warning message to a user terminal and a monitoring system. This enables integrated analysis of the voice and video data to assess and warn of fraud risks in real time.

[0135] "Voice data" refers to data in which voice information uttered by a user is recorded in digital format.

[0136] "Means for converting to text" refers to technology that analyzes voice data and converts it into digital data of the corresponding characters.

[0137] "Means for identifying specific keywords and contexts" refers to technology that analyzes and extracts the meaning of specific words, phrases, or sentences that indicate signs of fraud from the converted text.

[0138] "Video data" refers to visual information that is recorded in digital format and that captures the user's actions and behavior.

[0139] "Means for identifying behavioral patterns" refers to technology that analyzes specific user actions and behaviors from video data and detects unusual or unnatural behaviors or specific patterns.

[0140] The "means for calculating risk scores" is a technology that quantifies the risk of fraud based on the results of analyzing audio and video data.

[0141] The "means for generating a warning message" is a technology that automatically creates a warning message indicating suspected fraud based on the calculated risk score.

[0142] A "user terminal" is a terminal device that is directly operated or used by a user, such as a smartphone or ATM terminal.

[0143] A "monitoring system" is a system used by security companies, supervisory agencies, etc. to continuously monitor and manage signs of fraud.

[0144] The "server" is a central processing unit that receives and analyzes various data, and performs risk assessments and generates warnings.

[0145] This invention is a system designed to prevent financial frauds such as "I'm your son" frauds and "bank transfer" frauds. The system is mainly composed of a server, terminals (smartphones and ATMs), and users. Detailed embodiments of this system are described below.

[0146] Server-side processing

[0147] The server receives data sent from the user's smartphone or ATM terminal. Specifically, it receives voice and video data. The received voice data is converted into text using a speech recognition engine (e.g., Google® Cloud Speech-to-Text). The converted text is then analyzed by a natural language processing (NLP) module (e.g., Textrazor) to extract keywords and context.

[0148] The server also receives video data, which is then analyzed using behavior recognition algorithms (e.g., OpenPose) to detect user movements and patterns. For example, it can detect user behavior in front of an ATM.

[0149] Based on the analysis results of this audio and video data, the server calculates a risk score using a machine learning model (e.g., Scikit-learn, TENSORFLOW (registered trademark)). If the risk score exceeds a certain threshold, it is deemed to be high risk and a warning message is generated. This warning message is sent to the user's smartphone or ATM terminal. Security agencies are also notified at the same time.

[0150] Smartphone device processing

[0151] When a user makes a call through their smartphone, the device automatically records the audio in the background. This recorded audio data is encrypted and sent to a server using the HTTPS protocol. When a warning message is received from the server, a warning message is displayed in real time on the smartphone screen. For example, it may say, "This call may be fraudulent." This allows users to recognize the risk of fraud and take appropriate measures.

[0152] ATM terminal processing

[0153] When a user operates an ATM, the operation is recorded by a security camera. This video data is encoded in the H.264 standard and sent to a server using AES encryption. The server analyzes the video data and can detect abnormalities such as the user's unusual behavior or large cash withdrawals. When a warning message is sent from the server, the ATM screen displays the message, "Please be careful. You may be a victim of fraud." If necessary, the transaction is temporarily stopped and the user is prompted to contact the support center.

[0154] Specific examples

[0155] 1. Example when using a smartphone

[0156] When a user initiates a call with a scammer, the device records the audio and sends it to a server.

[0157] The server analyzes the voice data and detects keywords such as "transfer."

[0158] The server determines the risk is high and sends a warning message to the smartphone.

[0159] The user's smartphone will display a warning that "This call may be fraudulent."

[0160] 2. Example of using an ATM

[0161] When a user withdraws a large amount of cash from an ATM, the security camera records the footage and sends it from the ATM to a server.

[0162] The server analyzes behavioral patterns and detects unusual transactions.

[0163] The server determines the risk is high and sends a warning message to the ATM.

[0164] The ATM screen displays the message, "Please be aware that you may be a victim of fraud."

[0165] If necessary, trading will be temporarily suspended and users will be advised to contact support.

[0166] In this way, the system analyzes both audio and video for signs of fraud and assesses the risk, enabling users to receive real-time warnings and take prompt action, thereby preventing fraud before it occurs.

[0167] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0168] Step 1: Record and send audio data

[0169] When a user starts a call on their smartphone, the device records the audio in the background. This recorded audio data is encrypted with AES and sent to a server using the HTTPS protocol. Specifically, the smartphone's microphone captures the audio, temporarily stores it in local storage, and then uploads it to the server via an HTTP request. The server then receives the sent audio data.

[0170] Input: User's voice data

[0171] Output: Encrypted audio data sent to the server

[0172] Step 2: Convert audio data to text

[0173] The server passes the received voice data to a speech recognition engine (e.g., Google Cloud Speech-to-Text) to convert the voice into text, which is then sent to the next analysis step.

[0174] Input: Encrypted audio data

[0175] Output: Converted text data

[0176] Step 3: Analyzing the text data

[0177] The server analyzes the converted text using a natural language processing (NLP) module (e.g., Textrazor). This analysis extracts keywords and context. For example, keywords that indicate fraud, such as "transfer" or "secret," are detected. The analysis results are stored in an internal data store and used as input for calculating a risk score.

[0178] Input: Text data

[0179] Output: Keywords and context analysis results

[0180] Step 4: Record and transmit video data

[0181] When a user operates an ATM, the security camera records the video. The recorded video data is encoded in H.264, encrypted with AES, and then sent to a server using the HTTPS protocol. Specifically, the ATM camera captures the video and streams it to the server in real time.

[0182] Input: Video data including user actions

[0183] Output: Encrypted video data sent to the server

[0184] Step 5: Analyze the video data

[0185] The server passes the received video data to a behavioral recognition algorithm (e.g., OpenPose) to analyze the user's movements and behavioral patterns. As a result of the analysis, unnatural behaviors and specific behavioral patterns are extracted. These results are also stored in an internal data store and used to calculate the risk score.

[0186] Input: Encrypted video data

[0187] Output: Analysis results of movements and behavior patterns

[0188] Step 6: Calculate the risk score

[0189] The server calculates a risk score using a machine learning model (e.g., Scikit-learn, TensorFlow) based on the results of analyzing the audio and video data. The machine learning model uses pre-trained fraud behavior data and quantifies the risk of fraud from the analysis results. If this risk score exceeds a threshold, it is deemed to be high risk.

[0190] Input: Analysis results of audio and video data

[0191] Output: Risk score

[0192] Step 7: Generate and send a warning message

[0193] The server generates a warning message based on the calculated risk score if the call is deemed high risk. This warning message is sent to the user's smartphone, ATM terminal, and monitoring agency. Specific messages include "This call may be fraudulent" and "You may be a victim of fraud."

[0194] Input: Risk Score

[0195] Output: Generate and send warning messages

[0196] Step 8: Receive and view alerts

[0197] The user's smartphone or ATM terminal receives the warning message from the server and displays it to the user in real time. On smartphones, the screen displays "This call may be fraudulent." On ATMs, the screen displays "You may be a victim of fraud, please be careful."

[0198] Input: warning message

[0199] Output: A warning message that is displayed on the user's terminal.

[0200] (Application example 1)

[0201] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0202] Financial frauds such as "I'm your son" and "transfer fraud" are on the rise, causing serious economic damage to many people, especially the elderly. To address this issue, a system that can detect fraud risks in real time and issue warnings is needed. However, existing systems lack the ability to analyze audio and video data, making it difficult to detect fraud quickly and accurately. Therefore, there is a need for a system that can use both audio and video to provide users with warnings in real time.

[0203] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0204] In this invention, the server includes a means for receiving voice data and recording it in the background, a voice recognition means for analyzing the received voice data, a means for analyzing the received voice data and calculating a risk score by natural language processing, a means for generating a warning message in real time based on the risk score, and a means for displaying the generated warning message on a screen, thereby enabling fraud risks to be detected quickly and accurately and enabling users to take appropriate action immediately.

[0205] "Voice data" refers to data used to digitally record, store, and analyze words and sounds made by users.

[0206] "Background recording" is the process of automatically recording and saving audio during a call without the user being aware of it.

[0207] "Speech recognition means" is a technology that has the function of converting voice data into text data.

[0208] "Natural language processing" is a technology for analyzing text data and understanding the structure and meaning of language.

[0209] A "risk score" is a number that indicates the risk of fraud assessed based on the analysis results.

[0210] A "warning message" is a message consisting of text or audio that notifies the user of a risk.

[0211] "Generating in real time" refers to the process of generating an alert message immediately after receiving the analysis results.

[0212] "Display on screen" is the process of visually displaying the generated warning message on the screen of the user's terminal.

[0213] A "security camera" is an image capturing device installed for monitoring and recording purposes.

[0214] An "action recognition algorithm" is a technology that analyzes video data and identifies specific actions and behavior patterns.

[0215] "Encrypted data" is data that has been converted using a special code to prevent unauthorized access.

[0216] A "voice recognition engine" is software or hardware that analyzes voice data and identifies distinctive keywords and phrases.

[0217] The system based on this invention aims to analyze audio and video data, evaluate fraud risk in real time, and issue a warning. A specific implementation method of the invention will be described below.

[0218] Hardware and software used

[0219] Server: Use a server with high-performance processing capabilities. Examples include cloud servers from AWS (registered trademark) and Google Cloud.

[0220] Speech recognition engine: To convert voice data into text data, a speech recognition engine such as the Google Cloud Speech-to-Text API is used.

[0221] Natural Language Processing (NLP) module: NLP analysis could include the Japanese BERT model using the Hugging Face transformers library.

[0222] Action recognition algorithm: OpenCV and deep learning frameworks (e.g., TensorFlow, PyTorch) are used for video data analysis.

[0223] User devices: These include smartphones and ATM terminals. We assume that these devices are general devices equipped with recording and camera functions.

[0224] Specific examples of voice data processing

[0225] 1. Audio recording and transmission:

[0226] When a user initiates a call, the smartphone records the audio data in the background, encrypts it, and sends it to a server.

[0227] 2. Audio data analysis:

[0228] The server converts the received voice data into text using a speech recognition engine and analyzes it using an NLP module. Keywords such as "transfer" and "secret" are detected as a result of the analysis.

[0229] 3. Calculating the risk score:

[0230] A machine learning model calculates a risk score based on the detected keywords and other analysis results.

[0231] 4. Generate and display warning messages:

[0232] If the risk score exceeds a certain threshold, the server immediately generates a warning message and sends it to the user's smartphone, which displays a warning such as "This call may be fraudulent."

[0233] Specific examples of video data processing

[0234] 1. Video recording and transmission:

[0235] When a user operates an ATM, the security camera records the footage and encrypts and transmits the data to a server.

[0236] 2. Video data analysis:

[0237] The server analyzes the transmitted video data using behavioral recognition algorithms to detect behavioral patterns such as unusual operations or frequent large cash withdrawals.

[0238] 3. Calculating the risk score:

[0239] A risk score is calculated from the analysis of the video data, also using a machine learning model.

[0240] 4. Generate and display warning messages:

[0241] If the risk score exceeds a certain threshold, the server generates a warning message that appears on the ATM screen saying, "Please be careful, you may be a victim of fraud."

[0242] Examples of concrete examples and prompts

[0243] Specific scenarios

[0244] If a user makes a call using the word "transfer," a warning message will appear on their smartphone.

[0245] When a user attempts to withdraw a large amount of cash from an ATM, security camera footage is sent to a server and a warning message appears stating, "You may be a victim of fraud."

[0246] Prompt Sentence Examples

[0247] "Please assess the risk of fraud based on the following passage: 'To avoid falling victim to bank transfer fraud...'"

[0248] "Analyze the following conversation and assess the risk of fraud: 'I received a call asking me to transfer money, but it seems suspicious...'"

[0249] This allows for analysis of both audio and video to quickly and accurately detect the risk of users being scammed and provide necessary warnings in real time.

[0250] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0251] Step 1:

[0252] The smartphone records audio data in the background.

[0253] When a user starts a call, the device automatically records audio data in the background. This audio data is saved as a file containing the waveform information of the recorded sound. The input is the user's voice, and the output is the recorded audio data file.

[0254] Step 2:

[0255] Send the audio data to the server.

[0256] The audio data recorded on the device is encrypted and sent to the server using a secure protocol (e.g., HTTPS). The input is the recorded audio file, and the output is the encrypted audio data sent to the server.

[0257] Step 3:

[0258] The server analyzes the voice data and converts it into text.

[0259] The server converts the received voice data into text data using the Google Cloud Speech-to-Text API. The input is encrypted voice data, and the output is text data. This text data is the target for analysis.

[0260] Step 4:

[0261] Analyze text data using natural language processing.

[0262] The server analyzes the received text data using Hugging Face's transformers library. This analysis uses the Japanese BERT model to identify keywords and context and evaluate the risk of fraud. The input is text data, and the output is the analysis results, including keywords and context information.

[0263] Step 5:

[0264] A risk score is calculated based on the analysis results.

[0265] The server uses a machine learning model to calculate a risk score based on specific keywords and contextual information from the analysis results. For example, if a keyword such as "transfer" is detected, the risk score will be higher. The input is the analysis result, and the output is the risk score.

[0266] Step 6:

[0267] Generate warning messages based on risk scores.

[0268] If the risk score exceeds the set threshold, the server judges it to be a high risk and generates a warning message. This warning message is notified to the user. The input is the risk score, and the output is the warning message.

[0269] Step 7:

[0270] Send a warning message to the user's smartphone.

[0271] The generated warning message is sent from the server to the user's device. The receiving device immediately displays the warning message on its screen. The input is the warning message, and the output is the warning display on the user's smartphone.

[0272] Step 8:

[0273] Displays a warning message on the screen.

[0274] The user's smartphone immediately displays the received warning message on the screen to notify the user of the risk of fraud. The user can visually confirm this warning and take appropriate action against the fraud. The input is the warning message, and the output is the warning displayed on the screen.

[0275] Step 9:

[0276] Recording and transmitting video data (in the case of ATMs).

[0277] When a user operates an ATM, the security camera records the user's actions and the video data is encrypted and sent to the server. The input is the user's ATM operations, and the output is the encrypted video data sent to the server.

[0278] Step 10:

[0279] Analyze video data and identify behavioral patterns.

[0280] The server analyzes the received video data using a behavior recognition algorithm to identify the user's behavioral patterns. Unnatural actions such as frequent large cash withdrawals are detected. The input is encrypted video data, and the output is the results of behavioral pattern analysis.

[0281] Step 11:

[0282] A risk score is calculated based on the results of video data analysis.

[0283] The server uses a machine learning model to calculate a risk score based on the results of the behavioral pattern analysis. If certain unnatural behavior is detected, the risk score is set high. The input is the behavioral pattern analysis result, and the output is the risk score.

[0284] Step 12:

[0285] A warning message is generated and displayed on the ATM screen.

[0286] If the risk score is high, the server generates a warning message and displays it on the ATM screen. The user sees the warning: "You may be a victim of fraud, please be careful." The input is the risk score, and the output is the warning on the ATM screen.

[0287] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0288] This invention is a system designed to prevent financial frauds such as "I'm your son" frauds and "transfer frauds." The system utilizes audio data, video data, and an emotion engine to assess fraud risk in real time and issue warnings. The system is primarily composed of a server, terminals (smartphones and ATMs), and users.

[0289] Server-side processing

[0290] Data reception and initial analysis

[0291] The server receives voice and video data sent from the user's smartphone or ATM. For voice data, a speech recognition engine converts the speech into text, and a natural language processing (NLP) module analyzes keywords and context. An emotion engine analyzes the tone and speed of the voice to detect the user's emotions. For video data, a behavior recognition algorithm is applied to analyze the user's movements and behavior patterns. The emotion engine analyzes facial expressions and posture to recognize the user's emotions.

[0292] Risk Assessment

[0293] The server integrates the results of the audio data analysis, the video data analysis, and information from the emotion engine, and calculates a risk score using a machine learning model. For example, keywords that indicate fraud, such as "transfer" and "secret," as well as the user's emotions, such as impatience and tension, affect the risk score. The emotion engine adjusts the risk score based on these emotion indicators.

[0294] Alerting and Notifications

[0295] If the risk score exceeds a certain threshold, the server determines it to be a high risk and generates a warning message. This warning message is sent to the user's smartphone or ATM terminal. Security companies are also notified at the same time, enabling a prompt response.

[0296] Smartphone device processing

[0297] Audio recording and transmission

[0298] When a user makes a call through their smartphone, the device records the audio in the background. The recorded audio data is encrypted and sent to a server using the HTTPS protocol. The emotion engine also analyzes the audio and evaluates the user's emotions in real time.

[0299] Receiving and displaying alerts

[0300] When an alert is sent from the server, the smartphone receives a real-time warning message and displays it on the screen. This message may say something like, "This call may be fraudulent." By seeing this warning, the user can recognize the risk of fraud and take appropriate measures.

[0301] ATM terminal processing

[0302] Video recording and transmission

[0303] When a user operates an ATM, the operation is recorded by a security camera. This video data is encrypted and sent to a server. The server analyzes the video data to detect unusual behavior by the user or withdrawal of large amounts of cash. The emotion engine analyzes the user's facial expressions and movements to evaluate their emotional state.

[0304] Receiving and displaying alerts

[0305] When an alert from the server is sent to the ATM, a warning message will appear on the ATM screen, specifically stating, "Please be aware that you may be a victim of fraud." In some cases, the ATM may also include the option to pause the transaction and offer direct assistance to the user.

[0306] Specific examples

[0307] Specific examples when using a smartphone

[0308] 1. The user initiates a call with the scammer, and an app on their smartphone automatically records the audio in the background.

[0309] 2. The recorded voice data is encrypted and sent to the server using the HTTPS protocol. The emotion engine also analyzes the voice and evaluates the user's emotions.

[0310] 3. The server analyzes the voice data and detects keywords that indicate fraud, such as "transfer."

[0311] 4. The server determines the risk is high and sends a warning message to the user's smartphone. If the emotion engine detects that the user is anxious or nervous, the risk score will be further increased.

[0312] 5. The user's smartphone displays a warning that the call may be fraudulent.

[0313] Examples of ATM usage

[0314] 1. A user attempts to withdraw a large amount of cash from an ATM. A security camera records the footage, encrypts the data, and sends it to a server. The emotion engine analyzes the user's facial expressions and movements to evaluate their emotional state.

[0315] 2. The server analyzes the video data to detect unnatural trading behavior patterns. If the emotion engine detects tension in the user, it further increases the risk score.

[0316] 3. The server determines the risk to be high and sends a warning message to the ATM.

[0317] 4. A warning message appears on the ATM screen saying, "Please be aware that you may be a victim of fraud."

[0318] 5. If necessary, the ATM will temporarily suspend the transaction and provide the user with the option to contact the support center.

[0319] In this way, the system analyzes signs of fraud based on three elements: audio, video, and emotion analysis, assesses risk, and provides real-time warnings to users and enables them to take prompt action, thereby preventing fraud damage before it occurs.

[0320] The processing flow will be explained below.

[0321] Server-side processing

[0322] Step 1:

[0323] The server receives audio and video data sent from the user's smartphone or ATM via HTTP requests or WebSockets.

[0324] Step 2:

[0325] The server passes the received voice data to a voice recognition engine, which converts it into text data. At the same time, the voice data is sent to an emotion engine, which analyzes the user's emotions based on the tone and speed of the voice.

[0326] Step 3:

[0327] The text data converted by the speech recognition engine is sent to a natural language processing (NLP) module, where it is analyzed for keywords and context.

[0328] Step 4:

[0329] The server passes the received video data to a behavior recognition algorithm, which analyzes the user's movements and behavioral patterns. In parallel, the video data is sent to an emotion engine, which recognizes emotions from the user's facial expressions and movements.

[0330] Step 5:

[0331] The results of audio data analysis, video data analysis, and information from the emotion engine are integrated and input into a machine learning model to calculate a risk score. Emotional states (e.g., anxiety, tension, etc.) affect the risk score.

[0332] Step 6:

[0333] If the risk score exceeds a certain threshold, the server generates a warning message.

[0334] Step 7:

[0335] The server sends the generated warning message to the user's smartphone or ATM terminal, and simultaneously notifies the security company.

[0336] Smartphone device processing

[0337] Step 1:

[0338] A user initiates a call on their smartphone, and an app on the smartphone automatically records the audio in the background.

[0339] Step 2:

[0340] The emotion engine analyzes the tone and rate of speech to assess the user's emotional state (e.g., impatience or tension) in real time.

[0341] Step 3:

[0342] The recorded audio data is encrypted and sent to the server using the HTTPS protocol.

[0343] Step 4:

[0344] When an alert is sent from the server, the smartphone receives a warning message in real time.

[0345] Step 5:

[0346] The app displays a warning message on the user's screen, providing specific warning information and recommended actions.

[0347] ATM terminal processing

[0348] Step 1:

[0349] When a user begins to operate the ATM, the security camera records the activity.

[0350] Step 2:

[0351] The recorded video data is encrypted and sent to a server, where it is simultaneously sent to an emotion engine that analyzes emotions from the user's facial expressions and movements.

[0352] Step 3:

[0353] The server analyzes the video data to detect patterns of behavior, such as unusual user behavior or large cash withdrawals. If the emotion engine recognizes the user's emotions, such as nervousness or impatience, it will affect the risk score.

[0354] Step 4:

[0355] When an alert is sent from the server, the ATM receives a warning message in real time.

[0356] Step 5:

[0357] A warning message appears on the ATM screen, stating, "You may be a victim of fraud."

[0358] Step 6:

[0359] If a transaction is deemed high risk, the ATM terminal will temporarily suspend the transaction and prompt the user to contact the support center.

[0360] Specific examples

[0361] Specific examples when using a smartphone

[0362] Step 1:

[0363] The user initiates a call with the scammer, and an app on their phone automatically records the audio in the background.

[0364] Step 2:

[0365] The emotion engine analyzes the tone and speed of speech in real time to assess the user's emotional state.

[0366] Step 3:

[0367] The recorded audio data is encrypted and sent to the server using the HTTPS protocol.

[0368] Step 4:

[0369] The server analyzes the voice data and emotional state to detect keywords that indicate fraud, such as "transfer."

[0370] Step 5:

[0371] The server determines a high risk based on voice and emotional data and sends a warning message to the user's smartphone. If the emotion engine detects anxiety or tension, the risk score will be further increased.

[0372] Step 6:

[0373] The user's smartphone will display a warning that "This call may be fraudulent."

[0374] Examples of ATM usage

[0375] Step 1:

[0376] A user attempts to withdraw a large amount of cash from an ATM. The security camera records the footage, and the data is encrypted and sent to the server along with the emotion engine.

[0377] Step 2:

[0378] The server analyzes the video data to detect unnatural trading patterns, and the emotion engine analyzes the user's facial expressions and movements to detect tension or impatience.

[0379] Step 3:

[0380] The server calculates a risk score based on video data, emotional data, and behavioral analysis results. If the emotional engine detects tension, the risk score is further increased.

[0381] Step 4:

[0382] The server determines the risk is high and sends a warning message to the ATM.

[0383] Step 5:

[0384] A warning message appears on the ATM screen saying, "Please be careful, you may be a victim of fraud."

[0385] Step 6:

[0386] If necessary, the ATM will temporarily halt the transaction and give the user the option to contact a support center.

[0387] Example 2

[0388] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0389] Conventional financial fraud prevention systems simply analyze audio and video data, resulting in insufficient risk assessment. Furthermore, because real-time risk assessments that take into account the user's emotional state are not performed, potential fraud cannot be detected early, resulting in many cases of user harm. The objective of the present invention is to solve these problems and provide a more accurate and rapid fraud prevention system.

[0390] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0391] In this invention, the server includes means for receiving audio data and video data, means for analyzing the received audio data and video data, and means for calculating a risk score based on the results of the analysis of the audio data and video data and the results of the emotion analysis, thereby enabling a comprehensive risk assessment that takes into account not only audio but also video and the user's emotional state.

[0392] The present invention also includes a means for generating a warning message based on the risk score, a means for transmitting the generated warning message to a user terminal or an automated transaction machine, and a means for suspending a transaction or notifying a security company based on the transmitted warning message, thereby enabling users to recognize fraud risks in real time and take appropriate measures promptly.

[0393] Furthermore, the present invention includes a means for a user terminal to record voice data and transmit it to a server, a means for the server to analyze the voice data and detect specific keywords and contexts, a means for analyzing the user's emotional state in addition to analyzing the voice data, a means for calculating a risk score based on the detected keywords and emotional state, and a means for transmitting and displaying a generated warning message to the user terminal, thereby enabling early detection and prevention of fraud risks through multifaceted data analysis.

[0394] The system also includes a means for the automated teller machine to record video data and transmit it to a server, a means for the server to analyze the video data and detect unnatural behavioral patterns or specific behaviors, a means for analyzing the user's emotional state in addition to analyzing the video data, a means for calculating a risk score based on the detected behavioral patterns and emotional state, and a means for transmitting the generated warning message to the automated teller machine and displaying it, thereby enabling advanced risk assessment that takes physical behavior and emotional state into consideration.

[0395] "Voice data" refers to data in which the voice uttered by the user is recorded in digital format.

[0396] "Video data" refers to data that captures the movements and conditions of a user and their surroundings with a camera and records them in digital format.

[0397] A "server" is a computer system that communicates with multiple terminals over a network and receives, analyzes, stores, and transmits data.

[0398] "Analysis" is the process of breaking down data, deciphering its components and meaning, and extracting specific information.

[0399] "Emotion analysis" is a technology that detects a user's emotional state from audio and video data and analyzes the results.

[0400] A "risk score" is a numerical representation of the likelihood of fraud based on the analysis results, and is an evaluation index in which a higher value indicates a higher risk.

[0401] A "warning message" is a notification message that warns users and other related parties when it is determined that there is a high risk of fraud.

[0402] A "user terminal" is a device that is directly operated by a user, and includes smartphones, personal computers, and the like.

[0403] An "automated transaction machine" is a device that allows users to conduct financial transactions, and includes ATMs (automated teller machines).

[0404] "Notification" is the act of the system informing users and related parties of specific information.

[0405] This invention is a system that utilizes audio data, video data, and emotion analysis to assess fraud risk in real time and issue warnings. The system is primarily composed of a server, a user terminal (smartphone), and an automated teller machine (ATM). The specific operation of each device is described below.

[0406] Server Processing

[0407] Receiving and analyzing audio and video data

[0408] The server receives voice and video data sent from the user's smartphone or ATM. For the voice data, it converts the speech into text using a speech recognition engine (e.g., Google Cloud Speech-to-Text API or IBM Watson® Speech to Text) and analyzes keywords and context using a natural language processing (NLP) module (e.g., Python's NLTK library or spaCy). It also uses an emotion engine (e.g., Microsoft® Azure® Face API or Affectiva) to analyze the tone and speed of the speech and detect the user's emotions.

[0409] For video data, behavioral recognition algorithms (e.g., OpenCV or Google Vision API) are applied to analyze user movements and behavioral patterns. The emotion engine analyzes facial expressions and posture to recognize user emotions.

[0410] Risk Assessment

[0411] The server integrates the results of audio data analysis, video data analysis, and information from the emotion engine, and calculates a risk score using a machine learning model (e.g., Scikit-learn or TensorFlow). For example, keywords such as "transfer" and "secret" and emotions such as the user's impatience or tension affect the risk score.

[0412] Alerting and Notifications

[0413] If the risk score exceeds a certain threshold, the server generates a warning message, which is sent to the user's smartphone and ATM, and also notifies the security company, enabling a prompt response.

[0414] User device (smartphone) processing

[0415] Audio recording and transmission

[0416] When a user makes a call through their smartphone, the device records the voice in the background, encrypts the recorded voice data, and sends it to the server using the HTTPS protocol. The emotion engine also analyzes the voice and evaluates the user's emotions in real time.

[0417] Receiving and displaying alerts

[0418] When an alert is sent from the server, the smartphone receives a real-time warning message and displays it on the screen, such as "This call may be fraudulent."

[0419] Automatic transaction machine (ATM) processing

[0420] Video recording and transmission

[0421] When a user operates an ATM, the security camera records the operation, encrypts the video data, and sends it to a server. The server analyzes the video data to detect unusual behavior or large cash withdrawals. The emotion engine analyzes the user's facial expressions and movements to evaluate their emotional state.

[0422] Receiving and displaying alerts

[0423] When an alert from the server is sent to the ATM, a warning message is displayed on the ATM screen, such as "Please be aware that you may be a victim of fraud," and the ATM also includes the option to pause the transaction and offer direct assistance to the user if necessary.

[0424] Specific examples and prompts

[0425] Specific examples when using a smartphone

[0426] 1. The user initiates a call with the scammer, and an app on their smartphone automatically records the audio in the background.

[0427] 2. The recorded voice data is encrypted and sent to the server using the HTTPS protocol. The emotion engine also analyzes the voice and evaluates the user's emotions.

[0428] 3. The server analyzes the voice data and detects keywords that indicate fraud, such as "transfer."

[0429] 4. The server determines the risk is high and sends a warning message to the user's smartphone. If the emotion engine detects that the user is anxious or nervous, the risk score will be further increased.

[0430] 5. The user's smartphone displays a warning that the call may be fraudulent.

[0431] Examples of ATM usage

[0432] 1. A user attempts to withdraw a large amount of cash from an ATM. A security camera records the footage, encrypts the data, and sends it to a server. The emotion engine analyzes the user's facial expressions and movements to evaluate their emotional state.

[0433] 2. The server analyzes the video data to detect unnatural trading behavior patterns. If the emotion engine detects tension in the user, it further increases the risk score.

[0434] 3. The server determines the risk to be high and sends a warning message to the ATM.

[0435] 4. A warning message appears on the ATM screen saying, "Please be aware that you may be a victim of fraud."

[0436] 5. If necessary, the ATM will temporarily suspend the transaction and provide the user with the option to contact the support center.

[0437] Example prompts to input to the generative AI model

[0438] 1. Call Analysis Prompt:

[0439] A user is making a call on their smartphone. If the speech recognition engine detects keywords such as "transfer" and "secret," explain how a risk score is calculated and a warning message is generated.

[0440] 2. ATM Analysis Prompt:

[0441] A user is attempting to withdraw a large amount of cash from an ATM. If a security camera records the user's video and sends it to a server, explain how a risk score is calculated and a warning message is generated.

[0442] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0443] Step 1:

[0444] Data collection and transmission

[0445] When a user initiates a call, the smartphone automatically records audio in the background. This recorded audio data is encrypted and sent to a server using the HTTPS protocol. The audio data then becomes input data for analysis based on the content of the user's call.

[0446] As a specific operation, for example, while the user is talking to a fraudster, voice data containing the word "secret" is recorded and sent to the server.

[0447] Step 2:

[0448] Analysis of audio data

[0449] The server uses a speech recognition engine (e.g., Google Cloud Speech-to-Text) to convert the received voice data into text. This text data is then analyzed for keywords and context using a natural language processing (NLP) module (e.g., Python's NLTK library). The analysis results are also sent to an emotion engine, which detects the user's emotion from the tone and speed of the voice.

[0450] The input data is an audio file of audio data, and the output data is converted text data and emotional information. For example, the converted text data contains the keyword "transfer," and emotional analysis detects a tense tone.

[0451] Step 3:

[0452] Video data collection and transmission

[0453] When a user operates an ATM, the security camera records the footage, and the data is encrypted and sent to the server.

[0454] Specifically, when a user attempts to withdraw a large amount of cash from an ATM, this operation is recorded as video data and sent to the server.

[0455] Step 4:

[0456] Video data analysis

[0457] The server uses an action recognition algorithm (e.g., OpenCV) to analyze the video data. The user's movements and behavioral patterns are analyzed, and the emotion engine detects the user's emotions from their facial expressions and posture. The input data is recorded video data, and the output data is the user's behavioral patterns and emotional information. For example, video analysis may detect that the user is looking around unnaturally, and the emotion engine may identify that the user is in a state of tension from their facial expression.

[0458] Step 5:

[0459] Risk score calculation

[0460] The server integrates the analyzed audio data, video data, and emotional data and calculates a risk score using a machine learning model (e.g., Scikit-learn or TensorFlow). The risk score varies depending on specific keywords, unnatural behavior, and emotional state. The input data includes the results of audio analysis, video analysis, and emotional analysis, and the output data is a risk score. For example, if the keyword "transfer money" is found together with a tense tone and unnatural behavior, a high risk score is set.

[0461] Step 6:

[0462] Generate and send warning messages

[0463] If the risk score exceeds a certain threshold, the server generates a warning message. This warning message is sent to the user's smartphone or ATM. The security company is also notified. The input data is the risk score, and the output data is a warning message that is generated and sent to a specified terminal. For example, if the risk score is high, a message such as "This call may be fraudulent" or "You may be a victim of fraud, so please be careful" is displayed on the smartphone or ATM screen.

[0464] (Application example 2)

[0465] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0466] In recent years, financial frauds such as "I'm your son" frauds and "transfer frauds" have been on the rise, creating a need for effective prevention measures. However, conventional security systems lack the ability to detect signs of fraud in real time and issue prompt warnings. In particular, they lack the ability to integrate analysis of audio, video, and the user's emotional state, making it difficult to accurately assess fraud risk. Therefore, there is a need to develop a system that integrates audio, video, and emotional analysis to assess fraud risk in real time and issue warnings to users.

[0467] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving voice data, means for analyzing the received voice data, means for evaluating an emotional state, means for calculating a risk score based on the analysis results of the voice data and the emotional state, means for generating a warning message based on the risk score, and means for transmitting the generated warning message to a user terminal. This makes it possible to integrate the three data analyses of voice, video, and emotion analysis, accurately evaluate the risk of fraud in real time, and quickly issue a warning.

[0468] The "means for receiving voice data" is a device or module that receives voice data sent from a user terminal or a voice input device and converts it into a format that can be processed within the system.

[0469] The "means for analyzing received voice data" refers to a device or module that converts the acquired voice data into text using voice recognition technology and further analyzes it using natural language processing technology.

[0470] The "means for assessing emotional state" is a device or module for detecting the user's emotions from the tone and speed of the voice and the facial expressions of the video, and evaluating the results.

[0471] A "means for calculating a risk score" is a device or module that includes an algorithm or machine learning model for scoring the risk of fraud based on the analysis of voice data and the assessment of emotional state.

[0472] The "means for generating a warning message" is a device or module that creates a message to warn the user when the risk score exceeds a certain threshold.

[0473] The "means for transmitting the generated warning message to the user terminal" is a communication means for notifying the user's smartphone or other terminal of the generated warning message in real time.

[0474] The "means for receiving video data" is a device or module that receives video data sent from a user terminal or camera and converts it into a format that can be processed within the system.

[0475] The "means for analyzing received video data" refers to a device or module that analyzes the acquired video data using behavior recognition technology and detects unnatural behavior or actions.

[0476] "Means for a user terminal to record voice data and transmit it to a server" refers to a device or module that allows a terminal such as a user's smartphone to record voice and transmit that data to a server via a network.

[0477] "Means for the server to analyze the voice data and detect specific keywords and emotional states" refers to a device or module that allows the server to analyze the received voice data and identify keywords that indicate fraud and the user's emotional state.

[0478] To implement the present invention, the following system configuration and method are used.

[0479] The system is mainly composed of a server, a terminal (smartphone), and a user. The following explains in detail the configuration and the specific role of each means.

[0480] Server-side processing

[0481] Data reception and initial analysis

[0482] The server receives audio and video data sent from the user's smartphone. The audio data is converted into text using a speech recognition engine, and a natural language processing (NLP) module analyzes keywords and context. An emotion engine then analyzes the tone and speed of the voice to assess the user's emotional state. The video data is analyzed using a behavior recognition algorithm to analyze the user's movements and behavior patterns. Similarly, the emotion engine analyzes facial expressions and posture to assess the user's emotional state.

[0483] Risk Assessment

[0484] The server integrates the results of the audio data analysis, the video data analysis, and information from the emotion engine, and calculates a risk score using a machine learning model. Specifically, keywords that indicate fraud, such as "transfer" and "secret," as well as the user's impatience and tension, affect the risk score. The emotion engine adjusts the risk score based on these emotion indicators.

[0485] Alerting and Notifications

[0486] If the risk score exceeds a certain threshold, the server determines that the risk is high and generates a warning message. The generated warning message is sent to the user's smartphone. If necessary, a security company is also notified at the same time.

[0487] Smartphone device processing

[0488] Audio recording and transmission

[0489] When a user makes a call through their smartphone, the device records the voice in the background. The recorded voice data is encrypted and sent to the server using the HTTPS protocol. The emotion engine also analyzes the voice and evaluates the user's emotions in real time.

[0490] Receiving and displaying alerts

[0491] When an alert is sent from the server, the smartphone receives a real-time warning message and displays it on the screen. This message includes a warning such as "This call may be fraudulent." By seeing this warning, users can recognize the risk of fraud and take appropriate measures.

[0492] Specific examples

[0493] Specific examples when using a smartphone

[0494] 1. The user initiates a call with the scammer, and an app on their smartphone automatically records the audio in the background.

[0495] 2. The recorded voice data is encrypted and sent to the server using the HTTPS protocol. The emotion engine also analyzes the voice and evaluates the user's emotions.

[0496] 3. The server analyzes the voice data and detects keywords that indicate fraud, such as "transfer."

[0497] 4. The server determines the risk is high and sends a warning message to the user's smartphone. If the emotion engine detects that the user is anxious or nervous, the risk score will be further increased.

[0498] 5. The user's smartphone displays a warning that the call may be fraudulent.

[0499] As a concrete example, the prompt sentence to be input to the generative AI model is as follows:

[0500] "Generate proposals for a smartphone app that detects high-risk bank transfer fraud in real time and displays a warning message to the user. Analyze audio and video data, evaluate the user's emotional state with an emotion engine, and calculate a risk score with a machine learning model. If a risk exceeding a threshold is detected, display a warning message to the user."

[0501] Thus, an embodiment of the present invention provides a system that integrates three types of data: audio, video, and emotion analysis, to assess fraud risk in real time and quickly warn users.

[0502] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0503] Step 1:

[0504] A user initiates a call through a smartphone. The smartphone runs in the background and records the call. The input is the user's voice, and the output is the recorded voice data.

[0505] Step 2:

[0506] The device encrypts the recorded audio data and sends it to the server using the HTTPS protocol. The input here is the recorded audio data, and the output is the encrypted audio data sent to the server.

[0507] Step 3:

[0508] The server converts the received voice data into text using a voice recognition engine. The input here is encrypted voice data, and the output is analyzed text data.

[0509] Step 4:

[0510] The server analyzes the text of the voice data using a natural language processing (NLP) module and extracts specific keywords that indicate fraud. The input here is the analyzed text data, and the output is the extracted keywords.

[0511] Step 5:

[0512] The server uses an emotion engine to process the user's emotional state (e.g., tension or impatience) from the voice data. The input here is the voice data, and the output is the user's emotional state.

[0513] Step 6:

[0514] The server integrates the analysis results of the voice data and emotional state and processes them to calculate a risk score using a machine learning model, where the input is the extracted keywords and emotional state, and the output is the calculated risk score.

[0515] Step 7:

[0516] The server processes the risk score and generates a warning message based on the calculated risk score, where the input is the risk score and the output is the warning message.

[0517] Step 8:

[0518] The server processes the generated warning message and sends it to the user's smartphone. The input here is the warning message, and the output is the warning message displayed on the user's smartphone.

[0519] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0520] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0521] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0522] [Second embodiment]

[0523] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0524] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0525] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0526] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0527] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0528] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0529] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0530] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0531] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0532] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0533] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0534] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0535] This invention is a system designed to prevent financial frauds such as "I'm your son" frauds and "transfer frauds." The system utilizes audio and video data to assess fraud risks in real time and issue warnings. The system is primarily composed of a server, terminals (smartphones and ATMs), and users.

[0536] Server-side processing

[0537] Data reception and initial analysis

[0538] The server receives voice and video data sent from the user's smartphone or ATM. For voice data, a speech recognition engine converts the speech into text, and a natural language processing (NLP) module analyzes keywords and context. For video data, a behavior recognition algorithm is applied to analyze the user's movements and behavioral patterns.

[0539] Risk Assessment

[0540] The server uses a machine learning model to calculate a risk score based on the results of voice and video data analysis. For example, keywords that indicate fraud, such as "transfer" and "secret," as well as behavioral patterns such as frequent large cash withdrawals, affect the risk score.

[0541] Alerting and Notifications

[0542] If the risk score exceeds a certain threshold, the server determines it to be a high risk and generates a warning message. This warning message is sent to the user's smartphone or ATM terminal. Security companies are also notified at the same time, enabling a prompt response.

[0543] Smartphone device processing

[0544] Audio recording and transmission

[0545] When a user makes a call through their smartphone, the device records the audio in the background, and the recorded audio data is encrypted and sent to a server using the HTTPS protocol.

[0546] Receiving and displaying alerts

[0547] When an alert is sent from the server, the smartphone receives a real-time warning message and displays it on the screen. This message may say something like, "This call may be fraudulent." By seeing this warning, the user can recognize the risk of fraud and take appropriate measures.

[0548] ATM terminal processing

[0549] Video recording and transmission

[0550] When a user operates an ATM, the security camera records the operation. This video data is encrypted and sent to a server. The server analyzes the video data to detect unusual behavior by the user or large cash withdrawals.

[0551] Receiving and displaying alerts

[0552] When an alert from the server is sent to the ATM, a warning message will appear on the ATM screen, specifically stating, "Please be aware that you may be a victim of fraud." In some cases, the ATM may also include the option to pause the transaction and offer direct assistance to the user.

[0553] Specific examples

[0554] Specific examples when using a smartphone

[0555] 1. When a user initiates a call with a scammer, the device records the audio and sends it to a server.

[0556] 2. The server analyzes the voice data and detects keywords such as "transfer."

[0557] 3. The server determines the risk to be high and sends a warning message to the smartphone.

[0558] 4. The user's smartphone displays a warning that the call may be fraudulent.

[0559] Examples of ATM usage

[0560] 1. When a user attempts to withdraw a large amount of cash from an ATM, the security camera records the footage and sends it from the ATM to a server.

[0561] 2. The server analyzes behavioral patterns and detects unusual transactions.

[0562] 3. The server determines the risk to be high and sends a warning message to the ATM.

[0563] 4. The ATM screen displays the message, "Please be aware that you may be a victim of fraud."

[0564] 5. If necessary, trading will be temporarily suspended and the user will be advised to contact the support center.

[0565] In this way, the system analyzes both audio and video for signs of fraud and assesses the risk, enabling users to receive real-time warnings and take prompt action to prevent fraud before it occurs.

[0566] The processing flow will be explained below.

[0567] Server-side processing

[0568] Step 1:

[0569] The server receives audio and video data sent from the user's smartphone or ATM via HTTP requests or WebSockets.

[0570] Step 2:

[0571] The server passes the received voice data to a voice recognition engine and converts it into text data.

[0572] Step 3:

[0573] The text data converted by the speech recognition engine is sent to a natural language processing (NLP) module, where it is analyzed for keywords and context.

[0574] Step 4:

[0575] The server passes the received video data to a behavior recognition algorithm to analyze the user's movements and behavior patterns.

[0576] Step 5:

[0577] The results of audio data analysis and video data analysis are integrated and input into a machine learning model to calculate a risk score.

[0578] Step 6:

[0579] If the risk score exceeds a certain threshold, the server generates a warning message.

[0580] Step 7:

[0581] The server sends the generated warning message to the user's smartphone or ATM terminal, and simultaneously notifies the security company.

[0582] Smartphone device processing

[0583] Step 1:

[0584] A user initiates a call on their smartphone, and an app on the smartphone automatically records the audio in the background.

[0585] Step 2:

[0586] The recorded audio data is encrypted and sent to the server using the HTTPS protocol.

[0587] Step 3:

[0588] When an alert is sent from the server, the smartphone receives a warning message in real time.

[0589] Step 4:

[0590] The app displays a warning message on the user's screen, providing specific warning information and recommended actions.

[0591] ATM terminal processing

[0592] Step 1:

[0593] When a user begins to operate the ATM, the security camera records the activity.

[0594] Step 2:

[0595] The recorded video data is encrypted and sent to the server.

[0596] Step 3:

[0597] When an alert is sent from the server, the ATM receives a warning message in real time.

[0598] Step 4:

[0599] A warning message appears on the ATM screen, stating, "You may be a victim of fraud."

[0600] Step 5:

[0601] If a transaction is deemed high risk, the ATM terminal will temporarily suspend the transaction and prompt the user to contact the support center.

[0602] Specific examples

[0603] Specific examples when using a smartphone

[0604] Step 1:

[0605] The user initiates a call with the scammer, and an app on their phone automatically records the audio in the background.

[0606] Step 2:

[0607] The recorded audio data is encrypted and sent to the server using the HTTPS protocol.

[0608] Step 3:

[0609] The server analyzes the audio data and detects keywords that indicate fraud, such as "transfer."

[0610] Step 4:

[0611] The server determines the risk to be high and sends a warning message to the user's smartphone.

[0612] Step 5:

[0613] The user's smartphone will display a warning that "This call may be fraudulent."

[0614] Examples of ATM usage

[0615] Step 1:

[0616] A user attempts to withdraw a large amount of cash from an ATM. The security camera records the footage, encrypts it, and sends it to a server.

[0617] Step 2:

[0618] The server analyzes the video data and detects unnatural trading behavior patterns.

[0619] Step 3:

[0620] The server determines the risk is high and sends a warning message to the ATM.

[0621] Step 4:

[0622] A warning message appears on the ATM screen saying, "Please be careful, you may be a victim of fraud."

[0623] Step 5:

[0624] If necessary, the ATM will temporarily halt the transaction and give the user the option to contact a support center.

[0625] Example 1

[0626] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0627] In recent years, financial frauds such as "I'm your son" scams and bank transfer scams have been increasing, and many people have fallen victim to them. These scams are becoming more sophisticated and cannot be prevented by simple warnings or alerts. Elderly people and users who are not tech-savvy are particularly at high risk of falling victim to fraud, so there is a need for a system that can efficiently and quickly detect fraud and issue warnings to users. Conventional prevention methods rely on analyzing audio and video separately, which has limitations in detecting complex fraudulent behavior.

[0628] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0629] In this invention, the server includes means for receiving voice data, means for converting the received voice data into text, means for analyzing the converted text and identifying specific keywords and contexts, means for receiving video data, means for analyzing the received video data and identifying behavioral patterns, means for calculating a risk score based on the analysis results of the voice data and video data, means for generating a warning message based on the risk score, and means for transmitting the generated warning message to a user terminal and a monitoring system. This enables integrated analysis of the voice and video data to assess and warn of fraud risks in real time.

[0630] "Voice data" refers to data in which voice information uttered by a user is recorded in digital format.

[0631] "Means for converting to text" refers to technology that analyzes voice data and converts it into digital data of the corresponding characters.

[0632] "Means for identifying specific keywords and contexts" refers to technology that analyzes and extracts the meaning of specific words, phrases, or sentences that indicate signs of fraud from the converted text.

[0633] "Video data" refers to visual information that is recorded in digital format and that captures the user's actions and behavior.

[0634] "Means for identifying behavioral patterns" refers to technology that analyzes specific user actions and behaviors from video data and detects unusual or unnatural behaviors or specific patterns.

[0635] The "means for calculating risk scores" is a technology that quantifies the risk of fraud based on the results of analyzing audio and video data.

[0636] The "means for generating a warning message" is a technology that automatically creates a warning message indicating suspected fraud based on the calculated risk score.

[0637] A "user terminal" is a terminal device that is directly operated or used by a user, such as a smartphone or ATM terminal.

[0638] A "monitoring system" is a system used by security companies, supervisory agencies, etc. to continuously monitor and manage signs of fraud.

[0639] The "server" is a central processing unit that receives and analyzes various data, and performs risk assessments and generates warnings.

[0640] This invention is a system designed to prevent financial frauds such as "I'm your son" frauds and "bank transfer" frauds. The system is mainly composed of a server, terminals (smartphones and ATMs), and users. Detailed embodiments of this system are described below.

[0641] Server-side processing

[0642] The server receives data sent from the user's smartphone or ATM terminal. Specifically, it receives audio and video data. The received audio data is converted into text using a speech recognition engine (e.g., Google Cloud Speech-to-Text). The converted text is then analyzed by a natural language processing (NLP) module (e.g., Textrazor) to extract keywords and context.

[0643] The server also receives video data, which is then analyzed using behavior recognition algorithms (e.g., OpenPose) to detect user movements and patterns. For example, it can detect user behavior in front of an ATM.

[0644] Based on the analysis of this audio and video data, the server calculates a risk score using a machine learning model (e.g., Scikit-learn, TensorFlow). If the risk score exceeds a certain threshold, it is deemed to be high risk and a warning message is generated. This warning message is sent to the user's smartphone or ATM terminal. Security agencies are also notified at the same time.

[0645] Smartphone device processing

[0646] When a user makes a call through their smartphone, the device automatically records the audio in the background. This recorded audio data is encrypted and sent to a server using the HTTPS protocol. When a warning message is received from the server, a warning message is displayed in real time on the smartphone screen. For example, it may say, "This call may be fraudulent." This allows users to recognize the risk of fraud and take appropriate measures.

[0647] ATM terminal processing

[0648] When a user operates an ATM, the operation is recorded by a security camera. This video data is encoded in the H.264 standard and sent to a server using AES encryption. The server analyzes the video data and can detect abnormalities such as the user's unusual behavior or large cash withdrawals. When a warning message is sent from the server, the ATM screen displays the message, "Please be careful. You may be a victim of fraud." If necessary, the transaction is temporarily stopped and the user is prompted to contact the support center.

[0649] Specific examples

[0650] 1. Example when using a smartphone

[0651] When a user initiates a call with a scammer, the device records the audio and sends it to a server.

[0652] The server analyzes the voice data and detects keywords such as "transfer."

[0653] The server determines the risk is high and sends a warning message to the smartphone.

[0654] The user's smartphone will display a warning that "This call may be fraudulent."

[0655] 2. Example of using an ATM

[0656] When a user withdraws a large amount of cash from an ATM, the security camera records the footage and sends it from the ATM to a server.

[0657] The server analyzes behavioral patterns and detects unusual transactions.

[0658] The server determines the risk is high and sends a warning message to the ATM.

[0659] The ATM screen displays the message, "Please be aware that you may be a victim of fraud."

[0660] If necessary, trading will be temporarily suspended and users will be advised to contact support.

[0661] In this way, the system analyzes both audio and video for signs of fraud and assesses the risk, enabling users to receive real-time warnings and take prompt action, thereby preventing fraud before it occurs.

[0662] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0663] Step 1: Record and send audio data

[0664] When a user starts a call on their smartphone, the device records the audio in the background. This recorded audio data is encrypted with AES and sent to a server using the HTTPS protocol. Specifically, the smartphone's microphone captures the audio, temporarily stores it in local storage, and then uploads it to the server via an HTTP request. The server then receives the sent audio data.

[0665] Input: User's voice data

[0666] Output: Encrypted audio data sent to the server

[0667] Step 2: Convert audio data to text

[0668] The server passes the received voice data to a speech recognition engine (e.g., Google Cloud Speech-to-Text) to convert the voice into text, which is then sent to the next analysis step.

[0669] Input: Encrypted audio data

[0670] Output: Converted text data

[0671] Step 3: Analyzing the text data

[0672] The server analyzes the converted text using a natural language processing (NLP) module (e.g., Textrazor). This analysis extracts keywords and context. For example, keywords that indicate fraud, such as "transfer" or "secret," are detected. The analysis results are stored in an internal data store and used as input for calculating a risk score.

[0673] Input: Text data

[0674] Output: Keywords and context analysis results

[0675] Step 4: Record and transmit video data

[0676] When a user operates an ATM, the security camera records the video. The recorded video data is encoded in H.264, encrypted with AES, and then sent to a server using the HTTPS protocol. Specifically, the ATM camera captures the video and streams it to the server in real time.

[0677] Input: Video data including user actions

[0678] Output: Encrypted video data sent to the server

[0679] Step 5: Analyze the video data

[0680] The server passes the received video data to a behavioral recognition algorithm (e.g., OpenPose) to analyze the user's movements and behavioral patterns. As a result of the analysis, unnatural behaviors and specific behavioral patterns are extracted. These results are also stored in an internal data store and used to calculate the risk score.

[0681] Input: Encrypted video data

[0682] Output: Analysis results of movements and behavior patterns

[0683] Step 6: Calculate the risk score

[0684] The server calculates a risk score using a machine learning model (e.g., Scikit-learn, TensorFlow) based on the results of analyzing the audio and video data. The machine learning model uses pre-trained fraud behavior data and quantifies the risk of fraud from the analysis results. If this risk score exceeds a threshold, it is deemed to be high risk.

[0685] Input: Analysis results of audio and video data

[0686] Output: Risk score

[0687] Step 7: Generate and send a warning message

[0688] The server generates a warning message based on the calculated risk score if the call is deemed high risk. This warning message is sent to the user's smartphone, ATM terminal, and monitoring agency. Specific messages include "This call may be fraudulent" and "You may be a victim of fraud."

[0689] Input: Risk Score

[0690] Output: Generate and send warning messages

[0691] Step 8: Receive and view alerts

[0692] The user's smartphone or ATM terminal receives the warning message from the server and displays it to the user in real time. On smartphones, the screen displays "This call may be fraudulent." On ATMs, the screen displays "You may be a victim of fraud, please be careful."

[0693] Input: warning message

[0694] Output: A warning message that is displayed on the user's terminal.

[0695] (Application example 1)

[0696] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0697] Financial frauds such as "I'm your son" and "transfer fraud" are on the rise, causing serious economic damage to many people, especially the elderly. To address this issue, a system that can detect fraud risks in real time and issue warnings is needed. However, existing systems lack the ability to analyze audio and video data, making it difficult to detect fraud quickly and accurately. Therefore, there is a need for a system that can use both audio and video to provide users with warnings in real time.

[0698] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0699] In this invention, the server includes a means for receiving voice data and recording it in the background, a voice recognition means for analyzing the received voice data, a means for analyzing the received voice data and calculating a risk score by natural language processing, a means for generating a warning message in real time based on the risk score, and a means for displaying the generated warning message on a screen, thereby enabling fraud risks to be detected quickly and accurately and enabling users to take appropriate action immediately.

[0700] "Voice data" refers to data used to digitally record, store, and analyze words and sounds made by users.

[0701] "Background recording" is the process of automatically recording and saving audio during a call without the user being aware of it.

[0702] "Speech recognition means" is a technology that has the function of converting voice data into text data.

[0703] "Natural language processing" is a technology for analyzing text data and understanding the structure and meaning of language.

[0704] A "risk score" is a number that indicates the risk of fraud assessed based on the analysis results.

[0705] A "warning message" is a message consisting of text or audio that notifies the user of a risk.

[0706] "Generating in real time" refers to the process of generating an alert message immediately after receiving the analysis results.

[0707] "Display on screen" is the process of visually displaying the generated warning message on the screen of the user's terminal.

[0708] A "security camera" is an image capturing device installed for monitoring and recording purposes.

[0709] An "action recognition algorithm" is a technology that analyzes video data and identifies specific actions and behavior patterns.

[0710] "Encrypted data" is data that has been converted using a special code to prevent unauthorized access.

[0711] A "voice recognition engine" is software or hardware that analyzes voice data and identifies distinctive keywords and phrases.

[0712] The system based on this invention aims to analyze audio and video data, evaluate fraud risk in real time, and issue a warning. A specific implementation method of the invention will be described below.

[0713] Hardware and software used

[0714] Server: Use a server with high-performance processing power. For example, a cloud server from AWS or Google Cloud.

[0715] Speech recognition engine: To convert voice data into text data, a speech recognition engine such as the Google Cloud Speech-to-Text API is used.

[0716] Natural Language Processing (NLP) module: NLP analysis could include the Japanese BERT model using the Hugging Face transformers library.

[0717] Action recognition algorithm: OpenCV and deep learning frameworks (e.g., TensorFlow, PyTorch) are used for video data analysis.

[0718] User devices: These include smartphones and ATM terminals. We assume that these devices are general devices equipped with recording and camera functions.

[0719] Specific examples of voice data processing

[0720] 1. Audio recording and transmission:

[0721] When a user initiates a call, the smartphone records the audio data in the background, encrypts it, and sends it to a server.

[0722] 2. Audio data analysis:

[0723] The server converts the received voice data into text using a speech recognition engine and analyzes it using an NLP module. Keywords such as "transfer" and "secret" are detected as a result of the analysis.

[0724] 3. Calculating the risk score:

[0725] A machine learning model calculates a risk score based on the detected keywords and other analysis results.

[0726] 4. Generate and display warning messages:

[0727] If the risk score exceeds a certain threshold, the server immediately generates a warning message and sends it to the user's smartphone, which displays a warning such as "This call may be fraudulent."

[0728] Specific examples of video data processing

[0729] 1. Video recording and transmission:

[0730] When a user operates an ATM, the security camera records the footage and encrypts and transmits the data to a server.

[0731] 2. Video data analysis:

[0732] The server analyzes the transmitted video data using behavioral recognition algorithms to detect behavioral patterns such as unusual operations or frequent large cash withdrawals.

[0733] 3. Calculating the risk score:

[0734] A risk score is calculated from the analysis of the video data, also using a machine learning model.

[0735] 4. Generate and display warning messages:

[0736] If the risk score exceeds a certain threshold, the server generates a warning message that appears on the ATM screen saying, "Please be careful, you may be a victim of fraud."

[0737] Examples of concrete examples and prompts

[0738] Specific scenarios

[0739] If a user makes a call using the word "transfer," a warning message will appear on their smartphone.

[0740] When a user attempts to withdraw a large amount of cash from an ATM, security camera footage is sent to a server and a warning message appears stating, "You may be a victim of fraud."

[0741] Prompt Sentence Examples

[0742] "Please assess the risk of fraud based on the following passage: 'To avoid falling victim to bank transfer fraud...'"

[0743] "Analyze the following conversation and assess the risk of fraud: 'I received a call asking me to transfer money, but it seems suspicious...'"

[0744] This allows for analysis of both audio and video to quickly and accurately detect the risk of users being scammed and provide necessary warnings in real time.

[0745] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0746] Step 1:

[0747] The smartphone records audio data in the background.

[0748] When a user starts a call, the device automatically records audio data in the background. This audio data is saved as a file containing the waveform information of the recorded sound. The input is the user's voice, and the output is the recorded audio data file.

[0749] Step 2:

[0750] Send the audio data to the server.

[0751] The audio data recorded on the device is encrypted and sent to the server using a secure protocol (e.g., HTTPS). The input is the recorded audio file, and the output is the encrypted audio data sent to the server.

[0752] Step 3:

[0753] The server analyzes the voice data and converts it into text.

[0754] The server converts the received voice data into text data using the Google Cloud Speech-to-Text API. The input is encrypted voice data, and the output is text data. This text data is the target for analysis.

[0755] Step 4:

[0756] Analyze text data using natural language processing.

[0757] The server analyzes the received text data using Hugging Face's transformers library. This analysis uses the Japanese BERT model to identify keywords and context and evaluate the risk of fraud. The input is text data, and the output is the analysis results, including keywords and context information.

[0758] Step 5:

[0759] A risk score is calculated based on the analysis results.

[0760] The server uses a machine learning model to calculate a risk score based on specific keywords and contextual information from the analysis results. For example, if a keyword such as "transfer" is detected, the risk score will be higher. The input is the analysis result, and the output is the risk score.

[0761] Step 6:

[0762] Generate warning messages based on risk scores.

[0763] If the risk score exceeds the set threshold, the server judges it to be a high risk and generates a warning message. This warning message is notified to the user. The input is the risk score, and the output is the warning message.

[0764] Step 7:

[0765] Send a warning message to the user's smartphone.

[0766] The generated warning message is sent from the server to the user's device. The receiving device immediately displays the warning message on its screen. The input is the warning message, and the output is the warning display on the user's smartphone.

[0767] Step 8:

[0768] Displays a warning message on the screen.

[0769] The user's smartphone immediately displays the received warning message on the screen to notify the user of the risk of fraud. The user can visually confirm this warning and take appropriate action against the fraud. The input is the warning message, and the output is the warning displayed on the screen.

[0770] Step 9:

[0771] Recording and transmitting video data (in the case of ATMs).

[0772] When a user operates an ATM, the security camera records the user's actions and the video data is encrypted and sent to the server. The input is the user's ATM operations, and the output is the encrypted video data sent to the server.

[0773] Step 10:

[0774] Analyze video data and identify behavioral patterns.

[0775] The server analyzes the received video data using a behavior recognition algorithm to identify the user's behavioral patterns. Unnatural actions such as frequent large cash withdrawals are detected. The input is encrypted video data, and the output is the results of behavioral pattern analysis.

[0776] Step 11:

[0777] A risk score is calculated based on the results of video data analysis.

[0778] The server uses a machine learning model to calculate a risk score based on the results of the behavioral pattern analysis. If certain unnatural behavior is detected, the risk score is set high. The input is the behavioral pattern analysis result, and the output is the risk score.

[0779] Step 12:

[0780] A warning message is generated and displayed on the ATM screen.

[0781] If the risk score is high, the server generates a warning message and displays it on the ATM screen. The user sees the warning: "You may be a victim of fraud, please be careful." The input is the risk score, and the output is the warning on the ATM screen.

[0782] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0783] This invention is a system designed to prevent financial frauds such as "I'm your son" frauds and "transfer frauds." The system utilizes audio data, video data, and an emotion engine to assess fraud risk in real time and issue warnings. The system is primarily composed of a server, terminals (smartphones and ATMs), and users.

[0784] Server-side processing

[0785] Data reception and initial analysis

[0786] The server receives voice and video data sent from the user's smartphone or ATM. For voice data, a speech recognition engine converts the speech into text, and a natural language processing (NLP) module analyzes keywords and context. An emotion engine analyzes the tone and speed of the voice to detect the user's emotions. For video data, a behavior recognition algorithm is applied to analyze the user's movements and behavior patterns. The emotion engine analyzes facial expressions and posture to recognize the user's emotions.

[0787] Risk Assessment

[0788] The server integrates the results of the audio data analysis, the video data analysis, and information from the emotion engine, and calculates a risk score using a machine learning model. For example, keywords that indicate fraud, such as "transfer" and "secret," as well as the user's emotions, such as impatience and tension, affect the risk score. The emotion engine adjusts the risk score based on these emotion indicators.

[0789] Alerting and Notifications

[0790] If the risk score exceeds a certain threshold, the server determines it to be a high risk and generates a warning message. This warning message is sent to the user's smartphone or ATM terminal. Security companies are also notified at the same time, enabling a prompt response.

[0791] Smartphone device processing

[0792] Audio recording and transmission

[0793] When a user makes a call through their smartphone, the device records the audio in the background. The recorded audio data is encrypted and sent to a server using the HTTPS protocol. The emotion engine also analyzes the audio and evaluates the user's emotions in real time.

[0794] Receiving and displaying alerts

[0795] When an alert is sent from the server, the smartphone receives a real-time warning message and displays it on the screen. This message may say something like, "This call may be fraudulent." By seeing this warning, the user can recognize the risk of fraud and take appropriate measures.

[0796] ATM terminal processing

[0797] Video recording and transmission

[0798] When a user operates an ATM, the operation is recorded by a security camera. This video data is encrypted and sent to a server. The server analyzes the video data to detect unusual behavior by the user or withdrawal of large amounts of cash. The emotion engine analyzes the user's facial expressions and movements to evaluate their emotional state.

[0799] Receiving and displaying alerts

[0800] When an alert from the server is sent to the ATM, a warning message will appear on the ATM screen, specifically stating, "Please be aware that you may be a victim of fraud." In some cases, the ATM may also include the option to pause the transaction and offer direct assistance to the user.

[0801] Specific examples

[0802] Specific examples when using a smartphone

[0803] 1. The user initiates a call with the scammer, and an app on their smartphone automatically records the audio in the background.

[0804] 2. The recorded voice data is encrypted and sent to the server using the HTTPS protocol. The emotion engine also analyzes the voice and evaluates the user's emotions.

[0805] 3. The server analyzes the voice data and detects keywords that indicate fraud, such as "transfer."

[0806] 4. The server determines the risk is high and sends a warning message to the user's smartphone. If the emotion engine detects that the user is anxious or nervous, the risk score will be further increased.

[0807] 5. The user's smartphone displays a warning that the call may be fraudulent.

[0808] Examples of ATM usage

[0809] 1. A user attempts to withdraw a large amount of cash from an ATM. A security camera records the footage, encrypts the data, and sends it to a server. The emotion engine analyzes the user's facial expressions and movements to evaluate their emotional state.

[0810] 2. The server analyzes the video data to detect unnatural trading behavior patterns. If the emotion engine detects tension in the user, it further increases the risk score.

[0811] 3. The server determines the risk to be high and sends a warning message to the ATM.

[0812] 4. A warning message appears on the ATM screen saying, "Please be aware that you may be a victim of fraud."

[0813] 5. If necessary, the ATM will temporarily suspend the transaction and provide the user with the option to contact the support center.

[0814] In this way, the system analyzes signs of fraud based on three elements: audio, video, and emotion analysis, assesses risk, and provides real-time warnings to users and enables them to take prompt action, thereby preventing fraud damage before it occurs.

[0815] The processing flow will be explained below.

[0816] Server-side processing

[0817] Step 1:

[0818] The server receives audio and video data sent from the user's smartphone or ATM via HTTP requests or WebSockets.

[0819] Step 2:

[0820] The server passes the received voice data to a voice recognition engine, which converts it into text data. At the same time, the voice data is sent to an emotion engine, which analyzes the user's emotions based on the tone and speed of the voice.

[0821] Step 3:

[0822] The text data converted by the speech recognition engine is sent to a natural language processing (NLP) module, where it is analyzed for keywords and context.

[0823] Step 4:

[0824] The server passes the received video data to a behavior recognition algorithm, which analyzes the user's movements and behavioral patterns. In parallel, the video data is sent to an emotion engine, which recognizes emotions from the user's facial expressions and movements.

[0825] Step 5:

[0826] The results of audio data analysis, video data analysis, and information from the emotion engine are integrated and input into a machine learning model to calculate a risk score. Emotional states (e.g., anxiety, tension, etc.) affect the risk score.

[0827] Step 6:

[0828] If the risk score exceeds a certain threshold, the server generates a warning message.

[0829] Step 7:

[0830] The server sends the generated warning message to the user's smartphone or ATM terminal, and simultaneously notifies the security company.

[0831] Smartphone device processing

[0832] Step 1:

[0833] A user initiates a call on their smartphone, and an app on the smartphone automatically records the audio in the background.

[0834] Step 2:

[0835] The emotion engine analyzes the tone and rate of speech to assess the user's emotional state (e.g., impatience or tension) in real time.

[0836] Step 3:

[0837] The recorded audio data is encrypted and sent to the server using the HTTPS protocol.

[0838] Step 4:

[0839] When an alert is sent from the server, the smartphone receives a warning message in real time.

[0840] Step 5:

[0841] The app displays a warning message on the user's screen, providing specific warning information and recommended actions.

[0842] ATM terminal processing

[0843] Step 1:

[0844] When a user begins to operate the ATM, the security camera records the activity.

[0845] Step 2:

[0846] The recorded video data is encrypted and sent to a server, where it is simultaneously sent to an emotion engine that analyzes emotions from the user's facial expressions and movements.

[0847] Step 3:

[0848] The server analyzes the video data to detect patterns of behavior, such as unusual user behavior or large cash withdrawals. If the emotion engine recognizes the user's emotions, such as nervousness or impatience, it will affect the risk score.

[0849] Step 4:

[0850] When an alert is sent from the server, the ATM receives a warning message in real time.

[0851] Step 5:

[0852] A warning message appears on the ATM screen, stating, "You may be a victim of fraud."

[0853] Step 6:

[0854] If a transaction is deemed high risk, the ATM terminal will temporarily suspend the transaction and prompt the user to contact the support center.

[0855] Specific examples

[0856] Specific examples when using a smartphone

[0857] Step 1:

[0858] The user initiates a call with the scammer, and an app on their phone automatically records the audio in the background.

[0859] Step 2:

[0860] The emotion engine analyzes the tone and speed of speech in real time to assess the user's emotional state.

[0861] Step 3:

[0862] The recorded audio data is encrypted and sent to the server using the HTTPS protocol.

[0863] Step 4:

[0864] The server analyzes the voice data and emotional state to detect keywords that indicate fraud, such as "transfer."

[0865] Step 5:

[0866] The server determines a high risk based on voice and emotional data and sends a warning message to the user's smartphone. If the emotion engine detects anxiety or tension, the risk score will be further increased.

[0867] Step 6:

[0868] The user's smartphone will display a warning that "This call may be fraudulent."

[0869] Examples of ATM usage

[0870] Step 1:

[0871] A user attempts to withdraw a large amount of cash from an ATM. The security camera records the footage, and the data is encrypted and sent to the server along with the emotion engine.

[0872] Step 2:

[0873] The server analyzes the video data to detect unnatural trading patterns, and the emotion engine analyzes the user's facial expressions and movements to detect tension or impatience.

[0874] Step 3:

[0875] The server calculates a risk score based on video data, emotional data, and behavioral analysis results. If the emotional engine detects tension, the risk score is further increased.

[0876] Step 4:

[0877] The server determines the risk is high and sends a warning message to the ATM.

[0878] Step 5:

[0879] A warning message appears on the ATM screen saying, "Please be careful, you may be a victim of fraud."

[0880] Step 6:

[0881] If necessary, the ATM will temporarily halt the transaction and give the user the option to contact a support center.

[0882] Example 2

[0883] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0884] Conventional financial fraud prevention systems simply analyze audio and video data, resulting in insufficient risk assessment. Furthermore, because real-time risk assessments that take into account the user's emotional state are not performed, potential fraud cannot be detected early, resulting in many cases of user harm. The objective of the present invention is to solve these problems and provide a more accurate and rapid fraud prevention system.

[0885] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0886] In this invention, the server includes means for receiving audio data and video data, means for analyzing the received audio data and video data, and means for calculating a risk score based on the results of the analysis of the audio data and video data and the results of the emotion analysis, thereby enabling a comprehensive risk assessment that takes into account not only audio but also video and the user's emotional state.

[0887] The present invention also includes a means for generating a warning message based on the risk score, a means for transmitting the generated warning message to a user terminal or an automated transaction machine, and a means for suspending a transaction or notifying a security company based on the transmitted warning message, thereby enabling users to recognize fraud risks in real time and take appropriate measures promptly.

[0888] Furthermore, the present invention includes a means for a user terminal to record voice data and transmit it to a server, a means for the server to analyze the voice data and detect specific keywords and contexts, a means for analyzing the user's emotional state in addition to analyzing the voice data, a means for calculating a risk score based on the detected keywords and emotional state, and a means for transmitting and displaying a generated warning message to the user terminal, thereby enabling early detection and prevention of fraud risks through multifaceted data analysis.

[0889] The system also includes a means for the automated teller machine to record video data and transmit it to a server, a means for the server to analyze the video data and detect unnatural behavioral patterns or specific behaviors, a means for analyzing the user's emotional state in addition to analyzing the video data, a means for calculating a risk score based on the detected behavioral patterns and emotional state, and a means for transmitting the generated warning message to the automated teller machine and displaying it, thereby enabling advanced risk assessment that takes physical behavior and emotional state into consideration.

[0890] "Voice data" refers to data in which the voice uttered by the user is recorded in digital format.

[0891] "Video data" refers to data that captures the movements and conditions of a user and their surroundings with a camera and records them in digital format.

[0892] A "server" is a computer system that communicates with multiple terminals over a network and receives, analyzes, stores, and transmits data.

[0893] "Analysis" is the process of breaking down data, deciphering its components and meaning, and extracting specific information.

[0894] "Emotion analysis" is a technology that detects a user's emotional state from audio and video data and analyzes the results.

[0895] A "risk score" is a numerical representation of the likelihood of fraud based on the analysis results, and is an evaluation index in which a higher value indicates a higher risk.

[0896] A "warning message" is a notification message that warns users and other related parties when it is determined that there is a high risk of fraud.

[0897] A "user terminal" is a device that is directly operated by a user, and includes smartphones, personal computers, and the like.

[0898] An "automated transaction machine" is a device that allows users to conduct financial transactions, and includes ATMs (automated teller machines).

[0899] "Notification" is the act of the system informing users and related parties of specific information.

[0900] This invention is a system that utilizes audio data, video data, and emotion analysis to assess fraud risk in real time and issue warnings. The system is primarily composed of a server, a user terminal (smartphone), and an automated teller machine (ATM). The specific operation of each device is described below.

[0901] Server Processing

[0902] Receiving and analyzing audio and video data

[0903] The server receives voice and video data sent from the user's smartphone or ATM. For the voice data, it converts the speech into text using a speech recognition engine (e.g., Google Cloud Speech-to-Text API or IBM Watson Speech to Text), analyzes keywords and context using a natural language processing (NLP) module (e.g., Python's NLTK library or spaCy), and detects the user's emotions by analyzing the tone and speed of the speech using an emotion engine (e.g., Microsoft Azure Face API or Affectiva).

[0904] For video data, behavioral recognition algorithms (e.g., OpenCV or Google Vision API) are applied to analyze user movements and behavioral patterns. The emotion engine analyzes facial expressions and posture to recognize user emotions.

[0905] Risk Assessment

[0906] The server integrates the results of audio data analysis, video data analysis, and information from the emotion engine, and calculates a risk score using a machine learning model (e.g., Scikit-learn or TensorFlow). For example, keywords such as "transfer" and "secret" and emotions such as the user's impatience or tension affect the risk score.

[0907] Alerting and Notifications

[0908] If the risk score exceeds a certain threshold, the server generates a warning message, which is sent to the user's smartphone and ATM, and also notifies the security company, enabling a prompt response.

[0909] User device (smartphone) processing

[0910] Audio recording and transmission

[0911] When a user makes a call through their smartphone, the device records the voice in the background, encrypts the recorded voice data, and sends it to the server using the HTTPS protocol. The emotion engine also analyzes the voice and evaluates the user's emotions in real time.

[0912] Receiving and displaying alerts

[0913] When an alert is sent from the server, the smartphone receives a real-time warning message and displays it on the screen, such as "This call may be fraudulent."

[0914] Automatic transaction machine (ATM) processing

[0915] Video recording and transmission

[0916] When a user operates an ATM, the security camera records the operation, encrypts the video data, and sends it to a server. The server analyzes the video data to detect unusual behavior or large cash withdrawals. The emotion engine analyzes the user's facial expressions and movements to evaluate their emotional state.

[0917] Receiving and displaying alerts

[0918] When an alert from the server is sent to the ATM, a warning message is displayed on the ATM screen, such as "Please be aware that you may be a victim of fraud," and the ATM also includes the option to pause the transaction and offer direct assistance to the user if necessary.

[0919] Specific examples and prompts

[0920] Specific examples when using a smartphone

[0921] 1. The user initiates a call with the scammer, and an app on their smartphone automatically records the audio in the background.

[0922] 2. The recorded voice data is encrypted and sent to the server using the HTTPS protocol. The emotion engine also analyzes the voice and evaluates the user's emotions.

[0923] 3. The server analyzes the voice data and detects keywords that indicate fraud, such as "transfer."

[0924] 4. The server determines the risk is high and sends a warning message to the user's smartphone. If the emotion engine detects that the user is anxious or nervous, the risk score will be further increased.

[0925] 5. The user's smartphone displays a warning that the call may be fraudulent.

[0926] Examples of ATM usage

[0927] 1. A user attempts to withdraw a large amount of cash from an ATM. A security camera records the footage, encrypts the data, and sends it to a server. The emotion engine analyzes the user's facial expressions and movements to evaluate their emotional state.

[0928] 2. The server analyzes the video data to detect unnatural trading behavior patterns. If the emotion engine detects tension in the user, it further increases the risk score.

[0929] 3. The server determines the risk to be high and sends a warning message to the ATM.

[0930] 4. A warning message appears on the ATM screen saying, "Please be aware that you may be a victim of fraud."

[0931] 5. If necessary, the ATM will temporarily suspend the transaction and provide the user with the option to contact the support center.

[0932] Example prompts to input to the generative AI model

[0933] 1. Call Analysis Prompt:

[0934] A user is making a call on their smartphone. If the speech recognition engine detects keywords such as "transfer" and "secret," explain how a risk score is calculated and a warning message is generated.

[0935] 2. ATM Analysis Prompt:

[0936] A user is attempting to withdraw a large amount of cash from an ATM. If a security camera records the user's video and sends it to a server, explain how a risk score is calculated and a warning message is generated.

[0937] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0938] Step 1:

[0939] Data collection and transmission

[0940] When a user initiates a call, the smartphone automatically records audio in the background. This recorded audio data is encrypted and sent to a server using the HTTPS protocol. The audio data then becomes input data for analysis based on the content of the user's call.

[0941] As a specific operation, for example, while the user is talking to a fraudster, voice data containing the word "secret" is recorded and sent to the server.

[0942] Step 2:

[0943] Analysis of audio data

[0944] The server uses a speech recognition engine (e.g., Google Cloud Speech-to-Text) to convert the received voice data into text. This text data is then analyzed for keywords and context using a natural language processing (NLP) module (e.g., Python's NLTK library). The analysis results are also sent to an emotion engine, which detects the user's emotion from the tone and speed of the voice.

[0945] The input data is an audio file of audio data, and the output data is converted text data and emotional information. For example, the converted text data contains the keyword "transfer," and emotional analysis detects a tense tone.

[0946] Step 3:

[0947] Video data collection and transmission

[0948] When a user operates an ATM, the security camera records the footage, and the data is encrypted and sent to the server.

[0949] Specifically, when a user attempts to withdraw a large amount of cash from an ATM, this operation is recorded as video data and sent to the server.

[0950] Step 4:

[0951] Video data analysis

[0952] The server uses an action recognition algorithm (e.g., OpenCV) to analyze the video data. The user's movements and behavioral patterns are analyzed, and the emotion engine detects the user's emotions from their facial expressions and posture. The input data is recorded video data, and the output data is the user's behavioral patterns and emotional information. For example, video analysis may detect that the user is looking around unnaturally, and the emotion engine may identify that the user is in a state of tension from their facial expression.

[0953] Step 5:

[0954] Risk score calculation

[0955] The server integrates the analyzed audio data, video data, and emotional data and calculates a risk score using a machine learning model (e.g., Scikit-learn or TensorFlow). The risk score varies depending on specific keywords, unnatural behavior, and emotional state. The input data includes the results of audio analysis, video analysis, and emotional analysis, and the output data is a risk score. For example, if the keyword "transfer money" is found together with a tense tone and unnatural behavior, a high risk score is set.

[0956] Step 6:

[0957] Generate and send warning messages

[0958] If the risk score exceeds a certain threshold, the server generates a warning message. This warning message is sent to the user's smartphone or ATM. The security company is also notified. The input data is the risk score, and the output data is a warning message that is generated and sent to a specified terminal. For example, if the risk score is high, a message such as "This call may be fraudulent" or "You may be a victim of fraud, so please be careful" is displayed on the smartphone or ATM screen.

[0959] (Application example 2)

[0960] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0961] In recent years, financial frauds such as "I'm your son" frauds and "transfer frauds" have been on the rise, creating a need for effective prevention measures. However, conventional security systems lack the ability to detect signs of fraud in real time and issue prompt warnings. In particular, they lack the ability to integrate analysis of audio, video, and the user's emotional state, making it difficult to accurately assess fraud risk. Therefore, there is a need to develop a system that integrates audio, video, and emotional analysis to assess fraud risk in real time and issue warnings to users.

[0962] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving voice data, means for analyzing the received voice data, means for evaluating an emotional state, means for calculating a risk score based on the analysis results of the voice data and the emotional state, means for generating a warning message based on the risk score, and means for transmitting the generated warning message to a user terminal. This makes it possible to integrate the three data analyses of voice, video, and emotion analysis, accurately evaluate the risk of fraud in real time, and quickly issue a warning.

[0963] The "means for receiving voice data" is a device or module that receives voice data sent from a user terminal or a voice input device and converts it into a format that can be processed within the system.

[0964] The "means for analyzing received voice data" refers to a device or module that converts the acquired voice data into text using voice recognition technology and further analyzes it using natural language processing technology.

[0965] The "means for assessing emotional state" is a device or module for detecting the user's emotions from the tone and speed of the voice and the facial expressions of the video, and evaluating the results.

[0966] A "means for calculating a risk score" is a device or module that includes an algorithm or machine learning model for scoring the risk of fraud based on the analysis of voice data and the assessment of emotional state.

[0967] The "means for generating a warning message" is a device or module that creates a message to warn the user when the risk score exceeds a certain threshold.

[0968] The "means for transmitting the generated warning message to the user terminal" is a communication means for notifying the user's smartphone or other terminal of the generated warning message in real time.

[0969] The "means for receiving video data" is a device or module that receives video data sent from a user terminal or camera and converts it into a format that can be processed within the system.

[0970] The "means for analyzing received video data" refers to a device or module that analyzes the acquired video data using behavior recognition technology and detects unnatural behavior or actions.

[0971] "Means for a user terminal to record voice data and transmit it to a server" refers to a device or module that allows a terminal such as a user's smartphone to record voice and transmit that data to a server via a network.

[0972] "Means for the server to analyze the voice data and detect specific keywords and emotional states" refers to a device or module that allows the server to analyze the received voice data and identify keywords that indicate fraud and the user's emotional state.

[0973] To implement the present invention, the following system configuration and method are used.

[0974] The system is mainly composed of a server, a terminal (smartphone), and a user. The following explains in detail the configuration and the specific role of each means.

[0975] Server-side processing

[0976] Data reception and initial analysis

[0977] The server receives audio and video data sent from the user's smartphone. The audio data is converted into text using a speech recognition engine, and a natural language processing (NLP) module analyzes keywords and context. An emotion engine then analyzes the tone and speed of the voice to assess the user's emotional state. The video data is analyzed using a behavior recognition algorithm to analyze the user's movements and behavior patterns. Similarly, the emotion engine analyzes facial expressions and posture to assess the user's emotional state.

[0978] Risk Assessment

[0979] The server integrates the results of the audio data analysis, the video data analysis, and information from the emotion engine, and calculates a risk score using a machine learning model. Specifically, keywords that indicate fraud, such as "transfer" and "secret," as well as the user's impatience and tension, affect the risk score. The emotion engine adjusts the risk score based on these emotion indicators.

[0980] Alerting and Notifications

[0981] If the risk score exceeds a certain threshold, the server determines that the risk is high and generates a warning message. The generated warning message is sent to the user's smartphone. If necessary, a security company is also notified at the same time.

[0982] Smartphone device processing

[0983] Audio recording and transmission

[0984] When a user makes a call through their smartphone, the device records the voice in the background. The recorded voice data is encrypted and sent to the server using the HTTPS protocol. The emotion engine also analyzes the voice and evaluates the user's emotions in real time.

[0985] Receiving and displaying alerts

[0986] When an alert is sent from the server, the smartphone receives a real-time warning message and displays it on the screen. This message includes a warning such as "This call may be fraudulent." By seeing this warning, users can recognize the risk of fraud and take appropriate measures.

[0987] Specific examples

[0988] Specific examples when using a smartphone

[0989] 1. The user initiates a call with the scammer, and an app on their smartphone automatically records the audio in the background.

[0990] 2. The recorded voice data is encrypted and sent to the server using the HTTPS protocol. The emotion engine also analyzes the voice and evaluates the user's emotions.

[0991] 3. The server analyzes the voice data and detects keywords that indicate fraud, such as "transfer."

[0992] 4. The server determines the risk is high and sends a warning message to the user's smartphone. If the emotion engine detects that the user is anxious or nervous, the risk score will be further increased.

[0993] 5. The user's smartphone displays a warning that the call may be fraudulent.

[0994] As a concrete example, the prompt sentence to be input to the generative AI model is as follows:

[0995] "Generate proposals for a smartphone app that detects high-risk bank transfer fraud in real time and displays a warning message to the user. Analyze audio and video data, evaluate the user's emotional state with an emotion engine, and calculate a risk score with a machine learning model. If a risk exceeding a threshold is detected, display a warning message to the user."

[0996] Thus, an embodiment of the present invention provides a system that integrates three types of data: audio, video, and emotion analysis, to assess fraud risk in real time and quickly warn users.

[0997] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0998] Step 1:

[0999] A user initiates a call through a smartphone. The smartphone runs in the background and records the call. The input is the user's voice, and the output is the recorded voice data.

[1000] Step 2:

[1001] The device encrypts the recorded audio data and sends it to the server using the HTTPS protocol. The input here is the recorded audio data, and the output is the encrypted audio data sent to the server.

[1002] Step 3:

[1003] The server converts the received voice data into text using a voice recognition engine. The input here is encrypted voice data, and the output is analyzed text data.

[1004] Step 4:

[1005] The server analyzes the text of the voice data using a natural language processing (NLP) module and extracts specific keywords that indicate fraud. The input here is the analyzed text data, and the output is the extracted keywords.

[1006] Step 5:

[1007] The server uses an emotion engine to process the user's emotional state (e.g., tension or impatience) from the voice data. The input here is the voice data, and the output is the user's emotional state.

[1008] Step 6:

[1009] The server integrates the analysis results of the voice data and emotional state and processes them to calculate a risk score using a machine learning model, where the input is the extracted keywords and emotional state, and the output is the calculated risk score.

[1010] Step 7:

[1011] The server processes the risk score and generates a warning message based on the calculated risk score, where the input is the risk score and the output is the warning message.

[1012] Step 8:

[1013] The server processes the generated warning message and sends it to the user's smartphone. The input here is the warning message, and the output is the warning message displayed on the user's smartphone.

[1014] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1015] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1016] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1017] [Third embodiment]

[1018] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1019] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1020] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1021] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1022] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1023] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1024] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1025] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1026] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1027] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1028] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1029] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1030] This invention is a system designed to prevent financial frauds such as "I'm your son" frauds and "transfer frauds." The system utilizes audio and video data to assess fraud risks in real time and issue warnings. The system is primarily composed of a server, terminals (smartphones and ATMs), and users.

[1031] Server-side processing

[1032] Data reception and initial analysis

[1033] The server receives voice and video data sent from the user's smartphone or ATM. For voice data, a speech recognition engine converts the speech into text, and a natural language processing (NLP) module analyzes keywords and context. For video data, a behavior recognition algorithm is applied to analyze the user's movements and behavioral patterns.

[1034] Risk Assessment

[1035] The server uses a machine learning model to calculate a risk score based on the results of voice and video data analysis. For example, keywords that indicate fraud, such as "transfer" and "secret," as well as behavioral patterns such as frequent large cash withdrawals, affect the risk score.

[1036] Alerting and Notifications

[1037] If the risk score exceeds a certain threshold, the server determines it to be a high risk and generates a warning message. This warning message is sent to the user's smartphone or ATM terminal. Security companies are also notified at the same time, enabling a prompt response.

[1038] Smartphone device processing

[1039] Audio recording and transmission

[1040] When a user makes a call through their smartphone, the device records the audio in the background, and the recorded audio data is encrypted and sent to a server using the HTTPS protocol.

[1041] Receiving and displaying alerts

[1042] When an alert is sent from the server, the smartphone receives a real-time warning message and displays it on the screen. This message may say something like, "This call may be fraudulent." By seeing this warning, the user can recognize the risk of fraud and take appropriate measures.

[1043] ATM terminal processing

[1044] Video recording and transmission

[1045] When a user operates an ATM, the security camera records the operation. This video data is encrypted and sent to a server. The server analyzes the video data to detect unusual behavior by the user or large cash withdrawals.

[1046] Receiving and displaying alerts

[1047] When an alert from the server is sent to the ATM, a warning message will appear on the ATM screen, specifically stating, "Please be aware that you may be a victim of fraud." In some cases, the ATM may also include the option to pause the transaction and offer direct assistance to the user.

[1048] Specific examples

[1049] Specific examples when using a smartphone

[1050] 1. When a user initiates a call with a scammer, the device records the audio and sends it to a server.

[1051] 2. The server analyzes the voice data and detects keywords such as "transfer."

[1052] 3. The server determines the risk to be high and sends a warning message to the smartphone.

[1053] 4. The user's smartphone displays a warning that the call may be fraudulent.

[1054] Examples of ATM usage

[1055] 1. When a user attempts to withdraw a large amount of cash from an ATM, the security camera records the footage and sends it from the ATM to a server.

[1056] 2. The server analyzes behavioral patterns and detects unusual transactions.

[1057] 3. The server determines the risk to be high and sends a warning message to the ATM.

[1058] 4. The ATM screen displays the message, "Please be aware that you may be a victim of fraud."

[1059] 5. If necessary, trading will be temporarily suspended and the user will be advised to contact the support center.

[1060] In this way, the system analyzes both audio and video for signs of fraud and assesses the risk, enabling users to receive real-time warnings and take prompt action to prevent fraud before it occurs.

[1061] The processing flow will be explained below.

[1062] Server-side processing

[1063] Step 1:

[1064] The server receives audio and video data sent from the user's smartphone or ATM via HTTP requests or WebSockets.

[1065] Step 2:

[1066] The server passes the received voice data to a voice recognition engine and converts it into text data.

[1067] Step 3:

[1068] The text data converted by the speech recognition engine is sent to a natural language processing (NLP) module, where it is analyzed for keywords and context.

[1069] Step 4:

[1070] The server passes the received video data to a behavior recognition algorithm to analyze the user's movements and behavior patterns.

[1071] Step 5:

[1072] The results of audio data analysis and video data analysis are integrated and input into a machine learning model to calculate a risk score.

[1073] Step 6:

[1074] If the risk score exceeds a certain threshold, the server generates a warning message.

[1075] Step 7:

[1076] The server sends the generated warning message to the user's smartphone or ATM terminal, and simultaneously notifies the security company.

[1077] Smartphone device processing

[1078] Step 1:

[1079] A user initiates a call on their smartphone, and an app on the smartphone automatically records the audio in the background.

[1080] Step 2:

[1081] The recorded audio data is encrypted and sent to the server using the HTTPS protocol.

[1082] Step 3:

[1083] When an alert is sent from the server, the smartphone receives a warning message in real time.

[1084] Step 4:

[1085] The app displays a warning message on the user's screen, providing specific warning information and recommended actions.

[1086] ATM terminal processing

[1087] Step 1:

[1088] When a user begins to operate the ATM, the security camera records the activity.

[1089] Step 2:

[1090] The recorded video data is encrypted and sent to the server.

[1091] Step 3:

[1092] When an alert is sent from the server, the ATM receives a warning message in real time.

[1093] Step 4:

[1094] A warning message appears on the ATM screen, stating, "You may be a victim of fraud."

[1095] Step 5:

[1096] If a transaction is deemed high risk, the ATM terminal will temporarily suspend the transaction and prompt the user to contact the support center.

[1097] Specific examples

[1098] Specific examples when using a smartphone

[1099] Step 1:

[1100] The user initiates a call with the scammer, and an app on their phone automatically records the audio in the background.

[1101] Step 2:

[1102] The recorded audio data is encrypted and sent to the server using the HTTPS protocol.

[1103] Step 3:

[1104] The server analyzes the audio data and detects keywords that indicate fraud, such as "transfer."

[1105] Step 4:

[1106] The server determines the risk to be high and sends a warning message to the user's smartphone.

[1107] Step 5:

[1108] The user's smartphone will display a warning that "This call may be fraudulent."

[1109] Examples of ATM usage

[1110] Step 1:

[1111] A user attempts to withdraw a large amount of cash from an ATM. The security camera records the footage, encrypts it, and sends it to a server.

[1112] Step 2:

[1113] The server analyzes the video data and detects unnatural trading behavior patterns.

[1114] Step 3:

[1115] The server determines the risk is high and sends a warning message to the ATM.

[1116] Step 4:

[1117] A warning message appears on the ATM screen saying, "Please be careful, you may be a victim of fraud."

[1118] Step 5:

[1119] If necessary, the ATM will temporarily halt the transaction and give the user the option to contact a support center.

[1120] Example 1

[1121] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1122] In recent years, financial frauds such as "I'm your son" scams and bank transfer scams have been increasing, and many people have fallen victim to them. These scams are becoming more sophisticated and cannot be prevented by simple warnings or alerts. Elderly people and users who are not tech-savvy are particularly at high risk of falling victim to fraud, so there is a need for a system that can efficiently and quickly detect fraud and issue warnings to users. Conventional prevention methods rely on analyzing audio and video separately, which has limitations in detecting complex fraudulent behavior.

[1123] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1124] In this invention, the server includes means for receiving voice data, means for converting the received voice data into text, means for analyzing the converted text and identifying specific keywords and contexts, means for receiving video data, means for analyzing the received video data and identifying behavioral patterns, means for calculating a risk score based on the analysis results of the voice data and video data, means for generating a warning message based on the risk score, and means for transmitting the generated warning message to a user terminal and a monitoring system. This enables integrated analysis of the voice and video data to assess and warn of fraud risks in real time.

[1125] "Voice data" refers to data in which voice information uttered by a user is recorded in digital format.

[1126] "Means for converting to text" refers to technology that analyzes voice data and converts it into digital data of the corresponding characters.

[1127] "Means for identifying specific keywords and contexts" refers to technology that analyzes and extracts the meaning of specific words, phrases, or sentences that indicate signs of fraud from the converted text.

[1128] "Video data" refers to visual information that is recorded in digital format and that captures the user's actions and behavior.

[1129] "Means for identifying behavioral patterns" refers to technology that analyzes specific user actions and behaviors from video data and detects unusual or unnatural behaviors or specific patterns.

[1130] The "means for calculating risk scores" is a technology that quantifies the risk of fraud based on the results of analyzing audio and video data.

[1131] The "means for generating a warning message" is a technology that automatically creates a warning message indicating suspected fraud based on the calculated risk score.

[1132] A "user terminal" is a terminal device that is directly operated or used by a user, such as a smartphone or ATM terminal.

[1133] A "monitoring system" is a system used by security companies, supervisory agencies, etc. to continuously monitor and manage signs of fraud.

[1134] The "server" is a central processing unit that receives and analyzes various data, and performs risk assessments and generates warnings.

[1135] This invention is a system designed to prevent financial frauds such as "I'm your son" frauds and "bank transfer" frauds. The system is mainly composed of a server, terminals (smartphones and ATMs), and users. Detailed embodiments of this system are described below.

[1136] Server-side processing

[1137] The server receives data sent from the user's smartphone or ATM terminal. Specifically, it receives audio and video data. The received audio data is converted into text using a speech recognition engine (e.g., Google Cloud Speech-to-Text). The converted text is then analyzed by a natural language processing (NLP) module (e.g., Textrazor) to extract keywords and context.

[1138] The server also receives video data, which is then analyzed using behavior recognition algorithms (e.g., OpenPose) to detect user movements and patterns. For example, it can detect user behavior in front of an ATM.

[1139] Based on the analysis of this audio and video data, the server calculates a risk score using a machine learning model (e.g., Scikit-learn, TensorFlow). If the risk score exceeds a certain threshold, it is deemed to be high risk and a warning message is generated. This warning message is sent to the user's smartphone or ATM terminal. Security agencies are also notified at the same time.

[1140] Smartphone device processing

[1141] When a user makes a call through their smartphone, the device automatically records the audio in the background. This recorded audio data is encrypted and sent to a server using the HTTPS protocol. When a warning message is received from the server, a warning message is displayed in real time on the smartphone screen. For example, it may say, "This call may be fraudulent." This allows users to recognize the risk of fraud and take appropriate measures.

[1142] ATM terminal processing

[1143] When a user operates an ATM, the operation is recorded by a security camera. This video data is encoded in the H.264 standard and sent to a server using AES encryption. The server analyzes the video data and can detect abnormalities such as the user's unusual behavior or large cash withdrawals. When a warning message is sent from the server, the ATM screen displays the message, "Please be careful. You may be a victim of fraud." If necessary, the transaction is temporarily stopped and the user is prompted to contact the support center.

[1144] Specific examples

[1145] 1. Example when using a smartphone

[1146] When a user initiates a call with a scammer, the device records the audio and sends it to a server.

[1147] The server analyzes the voice data and detects keywords such as "transfer."

[1148] The server determines the risk is high and sends a warning message to the smartphone.

[1149] The user's smartphone will display a warning that "This call may be fraudulent."

[1150] 2. Example of using an ATM

[1151] When a user withdraws a large amount of cash from an ATM, the security camera records the footage and sends it from the ATM to a server.

[1152] The server analyzes behavioral patterns and detects unusual transactions.

[1153] The server determines the risk is high and sends a warning message to the ATM.

[1154] The ATM screen displays the message, "Please be aware that you may be a victim of fraud."

[1155] If necessary, trading will be temporarily suspended and users will be advised to contact support.

[1156] In this way, the system analyzes both audio and video for signs of fraud and assesses the risk, enabling users to receive real-time warnings and take prompt action, thereby preventing fraud before it occurs.

[1157] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1158] Step 1: Record and send audio data

[1159] When a user starts a call on their smartphone, the device records the audio in the background. This recorded audio data is encrypted with AES and sent to a server using the HTTPS protocol. Specifically, the smartphone's microphone captures the audio, temporarily stores it in local storage, and then uploads it to the server via an HTTP request. The server then receives the sent audio data.

[1160] Input: User's voice data

[1161] Output: Encrypted audio data sent to the server

[1162] Step 2: Convert audio data to text

[1163] The server passes the received voice data to a speech recognition engine (e.g., Google Cloud Speech-to-Text) to convert the voice into text, which is then sent to the next analysis step.

[1164] Input: Encrypted audio data

[1165] Output: Converted text data

[1166] Step 3: Analyzing the text data

[1167] The server analyzes the converted text using a natural language processing (NLP) module (e.g., Textrazor). This analysis extracts keywords and context. For example, keywords that indicate fraud, such as "transfer" or "secret," are detected. The analysis results are stored in an internal data store and used as input for calculating a risk score.

[1168] Input: Text data

[1169] Output: Keywords and context analysis results

[1170] Step 4: Record and transmit video data

[1171] When a user operates an ATM, the security camera records the video. The recorded video data is encoded in H.264, encrypted with AES, and then sent to a server using the HTTPS protocol. Specifically, the ATM camera captures the video and streams it to the server in real time.

[1172] Input: Video data including user actions

[1173] Output: Encrypted video data sent to the server

[1174] Step 5: Analyze the video data

[1175] The server passes the received video data to a behavioral recognition algorithm (e.g., OpenPose) to analyze the user's movements and behavioral patterns. As a result of the analysis, unnatural behaviors and specific behavioral patterns are extracted. These results are also stored in an internal data store and used to calculate the risk score.

[1176] Input: Encrypted video data

[1177] Output: Analysis results of movements and behavior patterns

[1178] Step 6: Calculate the risk score

[1179] The server calculates a risk score using a machine learning model (e.g., Scikit-learn, TensorFlow) based on the results of analyzing the audio and video data. The machine learning model uses pre-trained fraud behavior data and quantifies the risk of fraud from the analysis results. If this risk score exceeds a threshold, it is deemed to be high risk.

[1180] Input: Analysis results of audio and video data

[1181] Output: Risk score

[1182] Step 7: Generate and send a warning message

[1183] The server generates a warning message based on the calculated risk score if the call is deemed high risk. This warning message is sent to the user's smartphone, ATM terminal, and monitoring agency. Specific messages include "This call may be fraudulent" and "You may be a victim of fraud."

[1184] Input: Risk Score

[1185] Output: Generate and send warning messages

[1186] Step 8: Receive and view alerts

[1187] The user's smartphone or ATM terminal receives the warning message from the server and displays it to the user in real time. On smartphones, the screen displays "This call may be fraudulent." On ATMs, the screen displays "You may be a victim of fraud, please be careful."

[1188] Input: warning message

[1189] Output: A warning message that is displayed on the user's terminal.

[1190] (Application example 1)

[1191] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1192] Financial frauds such as "I'm your son" and "transfer fraud" are on the rise, causing serious economic damage to many people, especially the elderly. To address this issue, a system that can detect fraud risks in real time and issue warnings is needed. However, existing systems lack the ability to analyze audio and video data, making it difficult to detect fraud quickly and accurately. Therefore, there is a need for a system that can use both audio and video to provide users with warnings in real time.

[1193] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1194] In this invention, the server includes a means for receiving voice data and recording it in the background, a voice recognition means for analyzing the received voice data, a means for analyzing the received voice data and calculating a risk score by natural language processing, a means for generating a warning message in real time based on the risk score, and a means for displaying the generated warning message on a screen, thereby enabling fraud risks to be detected quickly and accurately and enabling users to take appropriate action immediately.

[1195] "Voice data" refers to data used to digitally record, store, and analyze words and sounds made by users.

[1196] "Background recording" is the process of automatically recording and saving audio during a call without the user being aware of it.

[1197] "Speech recognition means" is a technology that has the function of converting voice data into text data.

[1198] "Natural language processing" is a technology for analyzing text data and understanding the structure and meaning of language.

[1199] A "risk score" is a number that indicates the risk of fraud assessed based on the analysis results.

[1200] A "warning message" is a message consisting of text or audio that notifies the user of a risk.

[1201] "Generating in real time" refers to the process of generating an alert message immediately after receiving the analysis results.

[1202] "Display on screen" is the process of visually displaying the generated warning message on the screen of the user's terminal.

[1203] A "security camera" is an image capturing device installed for monitoring and recording purposes.

[1204] An "action recognition algorithm" is a technology that analyzes video data and identifies specific actions and behavior patterns.

[1205] "Encrypted data" is data that has been converted using a special code to prevent unauthorized access.

[1206] A "voice recognition engine" is software or hardware that analyzes voice data and identifies distinctive keywords and phrases.

[1207] The system based on this invention aims to analyze audio and video data, evaluate fraud risk in real time, and issue a warning. A specific implementation method of the invention will be described below.

[1208] Hardware and software used

[1209] Server: Use a server with high-performance processing power. For example, a cloud server from AWS or Google Cloud.

[1210] Speech recognition engine: To convert voice data into text data, a speech recognition engine such as the Google Cloud Speech-to-Text API is used.

[1211] Natural Language Processing (NLP) module: NLP analysis could include the Japanese BERT model using the Hugging Face transformers library.

[1212] Action recognition algorithm: OpenCV and deep learning frameworks (e.g., TensorFlow, PyTorch) are used for video data analysis.

[1213] User devices: These include smartphones and ATM terminals. We assume that these devices are general devices equipped with recording and camera functions.

[1214] Specific examples of voice data processing

[1215] 1. Audio recording and transmission:

[1216] When a user initiates a call, the smartphone records the audio data in the background, encrypts it, and sends it to a server.

[1217] 2. Audio data analysis:

[1218] The server converts the received voice data into text using a speech recognition engine and analyzes it using an NLP module. Keywords such as "transfer" and "secret" are detected as a result of the analysis.

[1219] 3. Calculating the risk score:

[1220] A machine learning model calculates a risk score based on the detected keywords and other analysis results.

[1221] 4. Generate and display warning messages:

[1222] If the risk score exceeds a certain threshold, the server immediately generates a warning message and sends it to the user's smartphone, which displays a warning such as "This call may be fraudulent."

[1223] Specific examples of video data processing

[1224] 1. Video recording and transmission:

[1225] When a user operates an ATM, the security camera records the footage and encrypts and transmits the data to a server.

[1226] 2. Video data analysis:

[1227] The server analyzes the transmitted video data using behavioral recognition algorithms to detect behavioral patterns such as unusual operations or frequent large cash withdrawals.

[1228] 3. Calculating the risk score:

[1229] A risk score is calculated from the analysis of the video data, also using a machine learning model.

[1230] 4. Generate and display warning messages:

[1231] If the risk score exceeds a certain threshold, the server generates a warning message that appears on the ATM screen saying, "Please be careful, you may be a victim of fraud."

[1232] Examples of concrete examples and prompts

[1233] Specific scenarios

[1234] If a user makes a call using the word "transfer," a warning message will appear on their smartphone.

[1235] When a user attempts to withdraw a large amount of cash from an ATM, security camera footage is sent to a server and a warning message appears stating, "You may be a victim of fraud."

[1236] Prompt Sentence Examples

[1237] "Please assess the risk of fraud based on the following passage: 'To avoid falling victim to bank transfer fraud...'"

[1238] "Analyze the following conversation and assess the risk of fraud: 'I received a call asking me to transfer money, but it seems suspicious...'"

[1239] This allows for analysis of both audio and video to quickly and accurately detect the risk of users being scammed and provide necessary warnings in real time.

[1240] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1241] Step 1:

[1242] The smartphone records audio data in the background.

[1243] When a user starts a call, the device automatically records audio data in the background. This audio data is saved as a file containing the waveform information of the recorded sound. The input is the user's voice, and the output is the recorded audio data file.

[1244] Step 2:

[1245] Send the audio data to the server.

[1246] The audio data recorded on the device is encrypted and sent to the server using a secure protocol (e.g., HTTPS). The input is the recorded audio file, and the output is the encrypted audio data sent to the server.

[1247] Step 3:

[1248] The server analyzes the voice data and converts it into text.

[1249] The server converts the received voice data into text data using the Google Cloud Speech-to-Text API. The input is encrypted voice data, and the output is text data. This text data is the target for analysis.

[1250] Step 4:

[1251] Analyze text data using natural language processing.

[1252] The server analyzes the received text data using Hugging Face's transformers library. This analysis uses the Japanese BERT model to identify keywords and context and evaluate the risk of fraud. The input is text data, and the output is the analysis results, including keywords and context information.

[1253] Step 5:

[1254] A risk score is calculated based on the analysis results.

[1255] The server uses a machine learning model to calculate a risk score based on specific keywords and contextual information from the analysis results. For example, if a keyword such as "transfer" is detected, the risk score will be higher. The input is the analysis result, and the output is the risk score.

[1256] Step 6:

[1257] Generate warning messages based on risk scores.

[1258] If the risk score exceeds the set threshold, the server judges it to be a high risk and generates a warning message. This warning message is notified to the user. The input is the risk score, and the output is the warning message.

[1259] Step 7:

[1260] Send a warning message to the user's smartphone.

[1261] The generated warning message is sent from the server to the user's device. The receiving device immediately displays the warning message on its screen. The input is the warning message, and the output is the warning display on the user's smartphone.

[1262] Step 8:

[1263] Displays a warning message on the screen.

[1264] The user's smartphone immediately displays the received warning message on the screen to notify the user of the risk of fraud. The user can visually confirm this warning and take appropriate action against the fraud. The input is the warning message, and the output is the warning displayed on the screen.

[1265] Step 9:

[1266] Recording and transmitting video data (in the case of ATMs).

[1267] When a user operates an ATM, the security camera records the user's actions and the video data is encrypted and sent to the server. The input is the user's ATM operations, and the output is the encrypted video data sent to the server.

[1268] Step 10:

[1269] Analyze video data and identify behavioral patterns.

[1270] The server analyzes the received video data using a behavior recognition algorithm to identify the user's behavioral patterns. Unnatural actions such as frequent large cash withdrawals are detected. The input is encrypted video data, and the output is the results of behavioral pattern analysis.

[1271] Step 11:

[1272] A risk score is calculated based on the results of video data analysis.

[1273] The server uses a machine learning model to calculate a risk score based on the results of the behavioral pattern analysis. If certain unnatural behavior is detected, the risk score is set high. The input is the behavioral pattern analysis result, and the output is the risk score.

[1274] Step 12:

[1275] A warning message is generated and displayed on the ATM screen.

[1276] If the risk score is high, the server generates a warning message and displays it on the ATM screen. The user sees the warning: "You may be a victim of fraud, please be careful." The input is the risk score, and the output is the warning on the ATM screen.

[1277] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1278] This invention is a system designed to prevent financial frauds such as "I'm your son" frauds and "transfer frauds." The system utilizes audio data, video data, and an emotion engine to assess fraud risk in real time and issue warnings. The system is primarily composed of a server, terminals (smartphones and ATMs), and users.

[1279] Server-side processing

[1280] Data reception and initial analysis

[1281] The server receives voice and video data sent from the user's smartphone or ATM. For voice data, a speech recognition engine converts the speech into text, and a natural language processing (NLP) module analyzes keywords and context. An emotion engine analyzes the tone and speed of the voice to detect the user's emotions. For video data, a behavior recognition algorithm is applied to analyze the user's movements and behavior patterns. The emotion engine analyzes facial expressions and posture to recognize the user's emotions.

[1282] Risk Assessment

[1283] The server integrates the results of the audio data analysis, the video data analysis, and information from the emotion engine, and calculates a risk score using a machine learning model. For example, keywords that indicate fraud, such as "transfer" and "secret," as well as the user's emotions, such as impatience and tension, affect the risk score. The emotion engine adjusts the risk score based on these emotion indicators.

[1284] Alerting and Notifications

[1285] If the risk score exceeds a certain threshold, the server determines it to be a high risk and generates a warning message. This warning message is sent to the user's smartphone or ATM terminal. Security companies are also notified at the same time, enabling a prompt response.

[1286] Smartphone device processing

[1287] Audio recording and transmission

[1288] When a user makes a call through their smartphone, the device records the audio in the background. The recorded audio data is encrypted and sent to a server using the HTTPS protocol. The emotion engine also analyzes the audio and evaluates the user's emotions in real time.

[1289] Receiving and displaying alerts

[1290] When an alert is sent from the server, the smartphone receives a real-time warning message and displays it on the screen. This message may say something like, "This call may be fraudulent." By seeing this warning, the user can recognize the risk of fraud and take appropriate measures.

[1291] ATM terminal processing

[1292] Video recording and transmission

[1293] When a user operates an ATM, the operation is recorded by a security camera. This video data is encrypted and sent to a server. The server analyzes the video data to detect unusual behavior by the user or withdrawal of large amounts of cash. The emotion engine analyzes the user's facial expressions and movements to evaluate their emotional state.

[1294] Receiving and displaying alerts

[1295] When an alert from the server is sent to the ATM, a warning message will appear on the ATM screen, specifically stating, "Please be aware that you may be a victim of fraud." In some cases, the ATM may also include the option to pause the transaction and offer direct assistance to the user.

[1296] Specific examples

[1297] Specific examples when using a smartphone

[1298] 1. The user initiates a call with the scammer, and an app on their smartphone automatically records the audio in the background.

[1299] 2. The recorded voice data is encrypted and sent to the server using the HTTPS protocol. The emotion engine also analyzes the voice and evaluates the user's emotions.

[1300] 3. The server analyzes the voice data and detects keywords that indicate fraud, such as "transfer."

[1301] 4. The server determines the risk is high and sends a warning message to the user's smartphone. If the emotion engine detects that the user is anxious or nervous, the risk score will be further increased.

[1302] 5. The user's smartphone displays a warning that the call may be fraudulent.

[1303] Examples of ATM usage

[1304] 1. A user attempts to withdraw a large amount of cash from an ATM. A security camera records the footage, encrypts the data, and sends it to a server. The emotion engine analyzes the user's facial expressions and movements to evaluate their emotional state.

[1305] 2. The server analyzes the video data to detect unnatural trading behavior patterns. If the emotion engine detects tension in the user, it further increases the risk score.

[1306] 3. The server determines the risk to be high and sends a warning message to the ATM.

[1307] 4. A warning message appears on the ATM screen saying, "Please be aware that you may be a victim of fraud."

[1308] 5. If necessary, the ATM will temporarily suspend the transaction and provide the user with the option to contact the support center.

[1309] In this way, the system analyzes signs of fraud based on three elements: audio, video, and emotion analysis, assesses risk, and provides real-time warnings to users and enables them to take prompt action, thereby preventing fraud damage before it occurs.

[1310] The processing flow will be explained below.

[1311] Server-side processing

[1312] Step 1:

[1313] The server receives audio and video data sent from the user's smartphone or ATM via HTTP requests or WebSockets.

[1314] Step 2:

[1315] The server passes the received voice data to a voice recognition engine, which converts it into text data. At the same time, the voice data is sent to an emotion engine, which analyzes the user's emotions based on the tone and speed of the voice.

[1316] Step 3:

[1317] The text data converted by the speech recognition engine is sent to a natural language processing (NLP) module, where it is analyzed for keywords and context.

[1318] Step 4:

[1319] The server passes the received video data to a behavior recognition algorithm, which analyzes the user's movements and behavioral patterns. In parallel, the video data is sent to an emotion engine, which recognizes emotions from the user's facial expressions and movements.

[1320] Step 5:

[1321] The results of audio data analysis, video data analysis, and information from the emotion engine are integrated and input into a machine learning model to calculate a risk score. Emotional states (e.g., anxiety, tension, etc.) affect the risk score.

[1322] Step 6:

[1323] If the risk score exceeds a certain threshold, the server generates a warning message.

[1324] Step 7:

[1325] The server sends the generated warning message to the user's smartphone or ATM terminal, and simultaneously notifies the security company.

[1326] Smartphone device processing

[1327] Step 1:

[1328] A user initiates a call on their smartphone, and an app on the smartphone automatically records the audio in the background.

[1329] Step 2:

[1330] The emotion engine analyzes the tone and rate of speech to assess the user's emotional state (e.g., impatience or tension) in real time.

[1331] Step 3:

[1332] The recorded audio data is encrypted and sent to the server using the HTTPS protocol.

[1333] Step 4:

[1334] When an alert is sent from the server, the smartphone receives a warning message in real time.

[1335] Step 5:

[1336] The app displays a warning message on the user's screen, providing specific warning information and recommended actions.

[1337] ATM terminal processing

[1338] Step 1:

[1339] When a user begins to operate the ATM, the security camera records the activity.

[1340] Step 2:

[1341] The recorded video data is encrypted and sent to a server, where it is simultaneously sent to an emotion engine that analyzes emotions from the user's facial expressions and movements.

[1342] Step 3:

[1343] The server analyzes the video data to detect patterns of behavior, such as unusual user behavior or large cash withdrawals. If the emotion engine recognizes the user's emotions, such as nervousness or impatience, it will affect the risk score.

[1344] Step 4:

[1345] When an alert is sent from the server, the ATM receives a warning message in real time.

[1346] Step 5:

[1347] A warning message appears on the ATM screen, stating, "You may be a victim of fraud."

[1348] Step 6:

[1349] If a transaction is deemed high risk, the ATM terminal will temporarily suspend the transaction and prompt the user to contact the support center.

[1350] Specific examples

[1351] Specific examples when using a smartphone

[1352] Step 1:

[1353] The user initiates a call with the scammer, and an app on their phone automatically records the audio in the background.

[1354] Step 2:

[1355] The emotion engine analyzes the tone and speed of speech in real time to assess the user's emotional state.

[1356] Step 3:

[1357] The recorded audio data is encrypted and sent to the server using the HTTPS protocol.

[1358] Step 4:

[1359] The server analyzes the voice data and emotional state to detect keywords that indicate fraud, such as "transfer."

[1360] Step 5:

[1361] The server determines a high risk based on voice and emotional data and sends a warning message to the user's smartphone. If the emotion engine detects anxiety or tension, the risk score will be further increased.

[1362] Step 6:

[1363] The user's smartphone will display a warning that "This call may be fraudulent."

[1364] Examples of ATM usage

[1365] Step 1:

[1366] A user attempts to withdraw a large amount of cash from an ATM. The security camera records the footage, and the data is encrypted and sent to the server along with the emotion engine.

[1367] Step 2:

[1368] The server analyzes the video data to detect unnatural trading patterns, and the emotion engine analyzes the user's facial expressions and movements to detect tension or impatience.

[1369] Step 3:

[1370] The server calculates a risk score based on video data, emotional data, and behavioral analysis results. If the emotional engine detects tension, the risk score is further increased.

[1371] Step 4:

[1372] The server determines the risk is high and sends a warning message to the ATM.

[1373] Step 5:

[1374] A warning message appears on the ATM screen saying, "Please be careful, you may be a victim of fraud."

[1375] Step 6:

[1376] If necessary, the ATM will temporarily halt the transaction and give the user the option to contact a support center.

[1377] Example 2

[1378] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1379] Conventional financial fraud prevention systems simply analyze audio and video data, resulting in insufficient risk assessment. Furthermore, because real-time risk assessments that take into account the user's emotional state are not performed, potential fraud cannot be detected early, resulting in many cases of user harm. The objective of the present invention is to solve these problems and provide a more accurate and rapid fraud prevention system.

[1380] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1381] In this invention, the server includes means for receiving audio data and video data, means for analyzing the received audio data and video data, and means for calculating a risk score based on the results of the analysis of the audio data and video data and the results of the emotion analysis, thereby enabling a comprehensive risk assessment that takes into account not only audio but also video and the user's emotional state.

[1382] The present invention also includes a means for generating a warning message based on the risk score, a means for transmitting the generated warning message to a user terminal or an automated transaction machine, and a means for suspending a transaction or notifying a security company based on the transmitted warning message, thereby enabling users to recognize fraud risks in real time and take appropriate measures promptly.

[1383] Furthermore, the present invention includes a means for a user terminal to record voice data and transmit it to a server, a means for the server to analyze the voice data and detect specific keywords and contexts, a means for analyzing the user's emotional state in addition to analyzing the voice data, a means for calculating a risk score based on the detected keywords and emotional state, and a means for transmitting and displaying a generated warning message to the user terminal, thereby enabling early detection and prevention of fraud risks through multifaceted data analysis.

[1384] The system also includes a means for the automated teller machine to record video data and transmit it to a server, a means for the server to analyze the video data and detect unnatural behavioral patterns or specific behaviors, a means for analyzing the user's emotional state in addition to analyzing the video data, a means for calculating a risk score based on the detected behavioral patterns and emotional state, and a means for transmitting the generated warning message to the automated teller machine and displaying it, thereby enabling advanced risk assessment that takes physical behavior and emotional state into consideration.

[1385] "Voice data" refers to data in which the voice uttered by the user is recorded in digital format.

[1386] "Video data" refers to data that captures the movements and conditions of a user and their surroundings with a camera and records them in digital format.

[1387] A "server" is a computer system that communicates with multiple terminals over a network and receives, analyzes, stores, and transmits data.

[1388] "Analysis" is the process of breaking down data, deciphering its components and meaning, and extracting specific information.

[1389] "Emotion analysis" is a technology that detects a user's emotional state from audio and video data and analyzes the results.

[1390] A "risk score" is a numerical representation of the likelihood of fraud based on the analysis results, and is an evaluation index in which a higher value indicates a higher risk.

[1391] A "warning message" is a notification message that warns users and other related parties when it is determined that there is a high risk of fraud.

[1392] A "user terminal" is a device that is directly operated by a user, and includes smartphones, personal computers, and the like.

[1393] An "automated transaction machine" is a device that allows users to conduct financial transactions, and includes ATMs (automated teller machines).

[1394] "Notification" is the act of the system informing users and related parties of specific information.

[1395] This invention is a system that utilizes audio data, video data, and emotion analysis to assess fraud risk in real time and issue warnings. The system is primarily composed of a server, a user terminal (smartphone), and an automated teller machine (ATM). The specific operation of each device is described below.

[1396] Server Processing

[1397] Receiving and analyzing audio and video data

[1398] The server receives voice and video data sent from the user's smartphone or ATM. For the voice data, it converts the speech into text using a speech recognition engine (e.g., Google Cloud Speech-to-Text API or IBM Watson Speech to Text), analyzes keywords and context using a natural language processing (NLP) module (e.g., Python's NLTK library or spaCy), and detects the user's emotions by analyzing the tone and speed of the speech using an emotion engine (e.g., Microsoft Azure Face API or Affectiva).

[1399] For video data, behavioral recognition algorithms (e.g., OpenCV or Google Vision API) are applied to analyze user movements and behavioral patterns. The emotion engine analyzes facial expressions and posture to recognize user emotions.

[1400] Risk Assessment

[1401] The server integrates the results of audio data analysis, video data analysis, and information from the emotion engine, and calculates a risk score using a machine learning model (e.g., Scikit-learn or TensorFlow). For example, keywords such as "transfer" and "secret" and emotions such as the user's impatience or tension affect the risk score.

[1402] Alerting and Notifications

[1403] If the risk score exceeds a certain threshold, the server generates a warning message, which is sent to the user's smartphone and ATM, and also notifies the security company, enabling a prompt response.

[1404] User device (smartphone) processing

[1405] Audio recording and transmission

[1406] When a user makes a call through their smartphone, the device records the voice in the background, encrypts the recorded voice data, and sends it to the server using the HTTPS protocol. The emotion engine also analyzes the voice and evaluates the user's emotions in real time.

[1407] Receiving and displaying alerts

[1408] When an alert is sent from the server, the smartphone receives a real-time warning message and displays it on the screen, such as "This call may be fraudulent."

[1409] Automatic transaction machine (ATM) processing

[1410] Video recording and transmission

[1411] When a user operates an ATM, the security camera records the operation, encrypts the video data, and sends it to a server. The server analyzes the video data to detect unusual behavior or large cash withdrawals. The emotion engine analyzes the user's facial expressions and movements to evaluate their emotional state.

[1412] Receiving and displaying alerts

[1413] When an alert from the server is sent to the ATM, a warning message is displayed on the ATM screen, such as "Please be aware that you may be a victim of fraud," and the ATM also includes the option to pause the transaction and offer direct assistance to the user if necessary.

[1414] Specific examples and prompts

[1415] Specific examples when using a smartphone

[1416] 1. The user initiates a call with the scammer, and an app on their smartphone automatically records the audio in the background.

[1417] 2. The recorded voice data is encrypted and sent to the server using the HTTPS protocol. The emotion engine also analyzes the voice and evaluates the user's emotions.

[1418] 3. The server analyzes the voice data and detects keywords that indicate fraud, such as "transfer."

[1419] 4. The server determines the risk is high and sends a warning message to the user's smartphone. If the emotion engine detects that the user is anxious or nervous, the risk score will be further increased.

[1420] 5. The user's smartphone displays a warning that the call may be fraudulent.

[1421] Examples of ATM usage

[1422] 1. A user attempts to withdraw a large amount of cash from an ATM. A security camera records the footage, encrypts the data, and sends it to a server. The emotion engine analyzes the user's facial expressions and movements to evaluate their emotional state.

[1423] 2. The server analyzes the video data to detect unnatural trading behavior patterns. If the emotion engine detects tension in the user, it further increases the risk score.

[1424] 3. The server determines the risk to be high and sends a warning message to the ATM.

[1425] 4. A warning message appears on the ATM screen saying, "Please be aware that you may be a victim of fraud."

[1426] 5. If necessary, the ATM will temporarily suspend the transaction and provide the user with the option to contact the support center.

[1427] Example prompts to input to the generative AI model

[1428] 1. Call Analysis Prompt:

[1429] A user is making a call on their smartphone. If the speech recognition engine detects keywords such as "transfer" and "secret," explain how a risk score is calculated and a warning message is generated.

[1430] 2. ATM Analysis Prompt:

[1431] A user is attempting to withdraw a large amount of cash from an ATM. If a security camera records the user's video and sends it to a server, explain how a risk score is calculated and a warning message is generated.

[1432] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1433] Step 1:

[1434] Data collection and transmission

[1435] When a user initiates a call, the smartphone automatically records audio in the background. This recorded audio data is encrypted and sent to a server using the HTTPS protocol. The audio data then becomes input data for analysis based on the content of the user's call.

[1436] As a specific operation, for example, while the user is talking to a fraudster, voice data containing the word "secret" is recorded and sent to the server.

[1437] Step 2:

[1438] Analysis of audio data

[1439] The server uses a speech recognition engine (e.g., Google Cloud Speech-to-Text) to convert the received voice data into text. This text data is then analyzed for keywords and context using a natural language processing (NLP) module (e.g., Python's NLTK library). The analysis results are also sent to an emotion engine, which detects the user's emotion from the tone and speed of the voice.

[1440] The input data is an audio file of audio data, and the output data is converted text data and emotional information. For example, the converted text data contains the keyword "transfer," and emotional analysis detects a tense tone.

[1441] Step 3:

[1442] Video data collection and transmission

[1443] When a user operates an ATM, the security camera records the footage, and the data is encrypted and sent to the server.

[1444] Specifically, when a user attempts to withdraw a large amount of cash from an ATM, this operation is recorded as video data and sent to the server.

[1445] Step 4:

[1446] Video data analysis

[1447] The server uses an action recognition algorithm (e.g., OpenCV) to analyze the video data. The user's movements and behavioral patterns are analyzed, and the emotion engine detects the user's emotions from their facial expressions and posture. The input data is recorded video data, and the output data is the user's behavioral patterns and emotional information. For example, video analysis may detect that the user is looking around unnaturally, and the emotion engine may identify that the user is in a state of tension from their facial expression.

[1448] Step 5:

[1449] Risk score calculation

[1450] The server integrates the analyzed audio data, video data, and emotional data and calculates a risk score using a machine learning model (e.g., Scikit-learn or TensorFlow). The risk score varies depending on specific keywords, unnatural behavior, and emotional state. The input data includes the results of audio analysis, video analysis, and emotional analysis, and the output data is a risk score. For example, if the keyword "transfer money" is found together with a tense tone and unnatural behavior, a high risk score is set.

[1451] Step 6:

[1452] Generate and send warning messages

[1453] If the risk score exceeds a certain threshold, the server generates a warning message. This warning message is sent to the user's smartphone or ATM. The security company is also notified. The input data is the risk score, and the output data is a warning message that is generated and sent to a specified terminal. For example, if the risk score is high, a message such as "This call may be fraudulent" or "You may be a victim of fraud, so please be careful" is displayed on the smartphone or ATM screen.

[1454] (Application example 2)

[1455] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1456] In recent years, financial frauds such as "I'm your son" frauds and "transfer frauds" have been on the rise, creating a need for effective prevention measures. However, conventional security systems lack the ability to detect signs of fraud in real time and issue prompt warnings. In particular, they lack the ability to integrate analysis of audio, video, and the user's emotional state, making it difficult to accurately assess fraud risk. Therefore, there is a need to develop a system that integrates audio, video, and emotional analysis to assess fraud risk in real time and issue warnings to users.

[1457] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving voice data, means for analyzing the received voice data, means for evaluating an emotional state, means for calculating a risk score based on the analysis results of the voice data and the emotional state, means for generating a warning message based on the risk score, and means for transmitting the generated warning message to a user terminal. This makes it possible to integrate the three data analyses of voice, video, and emotion analysis, accurately evaluate the risk of fraud in real time, and quickly issue a warning.

[1458] The "means for receiving voice data" is a device or module that receives voice data sent from a user terminal or a voice input device and converts it into a format that can be processed within the system.

[1459] The "means for analyzing received voice data" refers to a device or module that converts the acquired voice data into text using voice recognition technology and further analyzes it using natural language processing technology.

[1460] The "means for assessing emotional state" is a device or module for detecting the user's emotions from the tone and speed of the voice and the facial expressions of the video, and evaluating the results.

[1461] A "means for calculating a risk score" is a device or module that includes an algorithm or machine learning model for scoring the risk of fraud based on the analysis of voice data and the assessment of emotional state.

[1462] The "means for generating a warning message" is a device or module that creates a message to warn the user when the risk score exceeds a certain threshold.

[1463] The "means for transmitting the generated warning message to the user terminal" is a communication means for notifying the user's smartphone or other terminal of the generated warning message in real time.

[1464] The "means for receiving video data" is a device or module that receives video data sent from a user terminal or camera and converts it into a format that can be processed within the system.

[1465] The "means for analyzing received video data" refers to a device or module that analyzes the acquired video data using behavior recognition technology and detects unnatural behavior or actions.

[1466] "Means for a user terminal to record voice data and transmit it to a server" refers to a device or module that allows a terminal such as a user's smartphone to record voice and transmit that data to a server via a network.

[1467] "Means for the server to analyze the voice data and detect specific keywords and emotional states" refers to a device or module that allows the server to analyze the received voice data and identify keywords that indicate fraud and the user's emotional state.

[1468] To implement the present invention, the following system configuration and method are used.

[1469] The system is mainly composed of a server, a terminal (smartphone), and a user. The following explains in detail the configuration and the specific role of each means.

[1470] Server-side processing

[1471] Data reception and initial analysis

[1472] The server receives audio and video data sent from the user's smartphone. The audio data is converted into text using a speech recognition engine, and a natural language processing (NLP) module analyzes keywords and context. An emotion engine then analyzes the tone and speed of the voice to assess the user's emotional state. The video data is analyzed using a behavior recognition algorithm to analyze the user's movements and behavior patterns. Similarly, the emotion engine analyzes facial expressions and posture to assess the user's emotional state.

[1473] Risk Assessment

[1474] The server integrates the results of the audio data analysis, the video data analysis, and information from the emotion engine, and calculates a risk score using a machine learning model. Specifically, keywords that indicate fraud, such as "transfer" and "secret," as well as the user's impatience and tension, affect the risk score. The emotion engine adjusts the risk score based on these emotion indicators.

[1475] Alerting and Notifications

[1476] If the risk score exceeds a certain threshold, the server determines that the risk is high and generates a warning message. The generated warning message is sent to the user's smartphone. If necessary, a security company is also notified at the same time.

[1477] Smartphone device processing

[1478] Audio recording and transmission

[1479] When a user makes a call through their smartphone, the device records the voice in the background. The recorded voice data is encrypted and sent to the server using the HTTPS protocol. The emotion engine also analyzes the voice and evaluates the user's emotions in real time.

[1480] Receiving and displaying alerts

[1481] When an alert is sent from the server, the smartphone receives a real-time warning message and displays it on the screen. This message includes a warning such as "This call may be fraudulent." By seeing this warning, users can recognize the risk of fraud and take appropriate measures.

[1482] Specific examples

[1483] Specific examples when using a smartphone

[1484] 1. The user initiates a call with the scammer, and an app on their smartphone automatically records the audio in the background.

[1485] 2. The recorded voice data is encrypted and sent to the server using the HTTPS protocol. The emotion engine also analyzes the voice and evaluates the user's emotions.

[1486] 3. The server analyzes the voice data and detects keywords that indicate fraud, such as "transfer."

[1487] 4. The server determines the risk is high and sends a warning message to the user's smartphone. If the emotion engine detects that the user is anxious or nervous, the risk score will be further increased.

[1488] 5. The user's smartphone displays a warning that the call may be fraudulent.

[1489] As a concrete example, the prompt sentence to be input to the generative AI model is as follows:

[1490] "Generate proposals for a smartphone app that detects high-risk bank transfer fraud in real time and displays a warning message to the user. Analyze audio and video data, evaluate the user's emotional state with an emotion engine, and calculate a risk score with a machine learning model. If a risk exceeding a threshold is detected, display a warning message to the user."

[1491] Thus, an embodiment of the present invention provides a system that integrates three types of data: audio, video, and emotion analysis, to assess fraud risk in real time and quickly warn users.

[1492] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1493] Step 1:

[1494] A user initiates a call through a smartphone. The smartphone runs in the background and records the call. The input is the user's voice, and the output is the recorded voice data.

[1495] Step 2:

[1496] The device encrypts the recorded audio data and sends it to the server using the HTTPS protocol. The input here is the recorded audio data, and the output is the encrypted audio data sent to the server.

[1497] Step 3:

[1498] The server converts the received voice data into text using a voice recognition engine. The input here is encrypted voice data, and the output is analyzed text data.

[1499] Step 4:

[1500] The server analyzes the text of the voice data using a natural language processing (NLP) module and extracts specific keywords that indicate fraud. The input here is the analyzed text data, and the output is the extracted keywords.

[1501] Step 5:

[1502] The server uses an emotion engine to process the user's emotional state (e.g., tension or impatience) from the voice data. The input here is the voice data, and the output is the user's emotional state.

[1503] Step 6:

[1504] The server integrates the analysis results of the voice data and emotional state and processes them to calculate a risk score using a machine learning model, where the input is the extracted keywords and emotional state, and the output is the calculated risk score.

[1505] Step 7:

[1506] The server processes the risk score and generates a warning message based on the calculated risk score, where the input is the risk score and the output is the warning message.

[1507] Step 8:

[1508] The server processes the generated warning message and sends it to the user's smartphone. The input here is the warning message, and the output is the warning message displayed on the user's smartphone.

[1509] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1510] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1511] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1512] [Fourth embodiment]

[1513] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1514] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1515] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1516] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1517] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1518] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1519] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1520] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1521] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1522] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1523] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1524] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1525] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1526] This invention is a system designed to prevent financial frauds such as "I'm your son" frauds and "transfer frauds." The system utilizes audio and video data to assess fraud risks in real time and issue warnings. The system is primarily composed of a server, terminals (smartphones and ATMs), and users.

[1527] Server-side processing

[1528] Data reception and initial analysis

[1529] The server receives voice and video data sent from the user's smartphone or ATM. For voice data, a speech recognition engine converts the speech into text, and a natural language processing (NLP) module analyzes keywords and context. For video data, a behavior recognition algorithm is applied to analyze the user's movements and behavioral patterns.

[1530] Risk Assessment

[1531] The server uses a machine learning model to calculate a risk score based on the results of voice and video data analysis. For example, keywords that indicate fraud, such as "transfer" and "secret," as well as behavioral patterns such as frequent large cash withdrawals, affect the risk score.

[1532] Alerting and Notifications

[1533] If the risk score exceeds a certain threshold, the server determines it to be a high risk and generates a warning message. This warning message is sent to the user's smartphone or ATM terminal. Security companies are also notified at the same time, enabling a prompt response.

[1534] Smartphone device processing

[1535] Audio recording and transmission

[1536] When a user makes a call through their smartphone, the device records the audio in the background, and the recorded audio data is encrypted and sent to a server using the HTTPS protocol.

[1537] Receiving and displaying alerts

[1538] When an alert is sent from the server, the smartphone receives a real-time warning message and displays it on the screen. This message may say something like, "This call may be fraudulent." By seeing this warning, the user can recognize the risk of fraud and take appropriate measures.

[1539] ATM terminal processing

[1540] Video recording and transmission

[1541] When a user operates an ATM, the security camera records the operation. This video data is encrypted and sent to a server. The server analyzes the video data to detect unusual behavior by the user or large cash withdrawals.

[1542] Receiving and displaying alerts

[1543] When an alert from the server is sent to the ATM, a warning message will appear on the ATM screen, specifically stating, "Please be aware that you may be a victim of fraud." In some cases, the ATM may also include the option to pause the transaction and offer direct assistance to the user.

[1544] Specific examples

[1545] Specific examples when using a smartphone

[1546] 1. When a user initiates a call with a scammer, the device records the audio and sends it to a server.

[1547] 2. The server analyzes the voice data and detects keywords such as "transfer."

[1548] 3. The server determines the risk to be high and sends a warning message to the smartphone.

[1549] 4. The user's smartphone displays a warning that the call may be fraudulent.

[1550] Examples of ATM usage

[1551] 1. When a user attempts to withdraw a large amount of cash from an ATM, the security camera records the footage and sends it from the ATM to a server.

[1552] 2. The server analyzes behavioral patterns and detects unusual transactions.

[1553] 3. The server determines the risk to be high and sends a warning message to the ATM.

[1554] 4. The ATM screen displays the message, "Please be aware that you may be a victim of fraud."

[1555] 5. If necessary, trading will be temporarily suspended and the user will be advised to contact the support center.

[1556] In this way, the system analyzes both audio and video for signs of fraud and assesses the risk, enabling users to receive real-time warnings and take prompt action to prevent fraud before it occurs.

[1557] The processing flow will be explained below.

[1558] Server-side processing

[1559] Step 1:

[1560] The server receives audio and video data sent from the user's smartphone or ATM via HTTP requests or WebSockets.

[1561] Step 2:

[1562] The server passes the received voice data to a voice recognition engine and converts it into text data.

[1563] Step 3:

[1564] The text data converted by the speech recognition engine is sent to a natural language processing (NLP) module, where it is analyzed for keywords and context.

[1565] Step 4:

[1566] The server passes the received video data to a behavior recognition algorithm to analyze the user's movements and behavior patterns.

[1567] Step 5:

[1568] The results of audio data analysis and video data analysis are integrated and input into a machine learning model to calculate a risk score.

[1569] Step 6:

[1570] If the risk score exceeds a certain threshold, the server generates a warning message.

[1571] Step 7:

[1572] The server sends the generated warning message to the user's smartphone or ATM terminal, and simultaneously notifies the security company.

[1573] Smartphone device processing

[1574] Step 1:

[1575] A user initiates a call on their smartphone, and an app on the smartphone automatically records the audio in the background.

[1576] Step 2:

[1577] The recorded audio data is encrypted and sent to the server using the HTTPS protocol.

[1578] Step 3:

[1579] When an alert is sent from the server, the smartphone receives a warning message in real time.

[1580] Step 4:

[1581] The app displays a warning message on the user's screen, providing specific warning information and recommended actions.

[1582] ATM terminal processing

[1583] Step 1:

[1584] When a user begins to operate the ATM, the security camera records the activity.

[1585] Step 2:

[1586] The recorded video data is encrypted and sent to the server.

[1587] Step 3:

[1588] When an alert is sent from the server, the ATM receives a warning message in real time.

[1589] Step 4:

[1590] A warning message appears on the ATM screen, stating, "You may be a victim of fraud."

[1591] Step 5:

[1592] If a transaction is deemed high risk, the ATM terminal will temporarily suspend the transaction and prompt the user to contact the support center.

[1593] Specific examples

[1594] Specific examples when using a smartphone

[1595] Step 1:

[1596] The user initiates a call with the scammer, and an app on their phone automatically records the audio in the background.

[1597] Step 2:

[1598] The recorded audio data is encrypted and sent to the server using the HTTPS protocol.

[1599] Step 3:

[1600] The server analyzes the audio data and detects keywords that indicate fraud, such as "transfer."

[1601] Step 4:

[1602] The server determines the risk to be high and sends a warning message to the user's smartphone.

[1603] Step 5:

[1604] The user's smartphone will display a warning that "This call may be fraudulent."

[1605] Examples of ATM usage

[1606] Step 1:

[1607] A user attempts to withdraw a large amount of cash from an ATM. The security camera records the footage, encrypts it, and sends it to a server.

[1608] Step 2:

[1609] The server analyzes the video data and detects unnatural trading behavior patterns.

[1610] Step 3:

[1611] The server determines the risk is high and sends a warning message to the ATM.

[1612] Step 4:

[1613] A warning message appears on the ATM screen saying, "Please be careful, you may be a victim of fraud."

[1614] Step 5:

[1615] If necessary, the ATM will temporarily halt the transaction and give the user the option to contact a support center.

[1616] Example 1

[1617] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1618] In recent years, financial frauds such as "I'm your son" scams and bank transfer scams have been increasing, and many people have fallen victim to them. These scams are becoming more sophisticated and cannot be prevented by simple warnings or alerts. Elderly people and users who are not tech-savvy are particularly at high risk of falling victim to fraud, so there is a need for a system that can efficiently and quickly detect fraud and issue warnings to users. Conventional prevention methods rely on analyzing audio and video separately, which has limitations in detecting complex fraudulent behavior.

[1619] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1620] In this invention, the server includes means for receiving voice data, means for converting the received voice data into text, means for analyzing the converted text and identifying specific keywords and contexts, means for receiving video data, means for analyzing the received video data and identifying behavioral patterns, means for calculating a risk score based on the analysis results of the voice data and video data, means for generating a warning message based on the risk score, and means for transmitting the generated warning message to a user terminal and a monitoring system. This enables integrated analysis of the voice and video data to assess and warn of fraud risks in real time.

[1621] "Voice data" refers to data in which voice information uttered by a user is recorded in digital format.

[1622] "Means for converting to text" refers to technology that analyzes voice data and converts it into digital data of the corresponding characters.

[1623] "Means for identifying specific keywords and contexts" refers to technology that analyzes and extracts the meaning of specific words, phrases, or sentences that indicate signs of fraud from the converted text.

[1624] "Video data" refers to visual information that is recorded in digital format and that captures the user's actions and behavior.

[1625] "Means for identifying behavioral patterns" refers to technology that analyzes specific user actions and behaviors from video data and detects unusual or unnatural behaviors or specific patterns.

[1626] The "means for calculating risk scores" is a technology that quantifies the risk of fraud based on the results of analyzing audio and video data.

[1627] The "means for generating a warning message" is a technology that automatically creates a warning message indicating suspected fraud based on the calculated risk score.

[1628] A "user terminal" is a terminal device that is directly operated or used by a user, such as a smartphone or ATM terminal.

[1629] A "monitoring system" is a system used by security companies, supervisory agencies, etc. to continuously monitor and manage signs of fraud.

[1630] The "server" is a central processing unit that receives and analyzes various data, and performs risk assessments and generates warnings.

[1631] This invention is a system designed to prevent financial frauds such as "I'm your son" frauds and "bank transfer" frauds. The system is mainly composed of a server, terminals (smartphones and ATMs), and users. Detailed embodiments of this system are described below.

[1632] Server-side processing

[1633] The server receives data sent from the user's smartphone or ATM terminal. Specifically, it receives audio and video data. The received audio data is converted into text using a speech recognition engine (e.g., Google Cloud Speech-to-Text). The converted text is then analyzed by a natural language processing (NLP) module (e.g., Textrazor) to extract keywords and context.

[1634] The server also receives video data, which is then analyzed using behavior recognition algorithms (e.g., OpenPose) to detect user movements and patterns. For example, it can detect user behavior in front of an ATM.

[1635] Based on the analysis of this audio and video data, the server calculates a risk score using a machine learning model (e.g., Scikit-learn, TensorFlow). If the risk score exceeds a certain threshold, it is deemed to be high risk and a warning message is generated. This warning message is sent to the user's smartphone or ATM terminal. Security agencies are also notified at the same time.

[1636] Smartphone device processing

[1637] When a user makes a call through their smartphone, the device automatically records the audio in the background. This recorded audio data is encrypted and sent to a server using the HTTPS protocol. When a warning message is received from the server, a warning message is displayed in real time on the smartphone screen. For example, it may say, "This call may be fraudulent." This allows users to recognize the risk of fraud and take appropriate measures.

[1638] ATM terminal processing

[1639] When a user operates an ATM, the operation is recorded by a security camera. This video data is encoded in the H.264 standard and sent to a server using AES encryption. The server analyzes the video data and can detect abnormalities such as the user's unusual behavior or large cash withdrawals. When a warning message is sent from the server, the ATM screen displays the message, "Please be careful. You may be a victim of fraud." If necessary, the transaction is temporarily stopped and the user is prompted to contact the support center.

[1640] Specific examples

[1641] 1. Example when using a smartphone

[1642] When a user initiates a call with a scammer, the device records the audio and sends it to a server.

[1643] The server analyzes the voice data and detects keywords such as "transfer."

[1644] The server determines the risk is high and sends a warning message to the smartphone.

[1645] The user's smartphone will display a warning that "This call may be fraudulent."

[1646] 2. Example of using an ATM

[1647] When a user withdraws a large amount of cash from an ATM, the security camera records the footage and sends it from the ATM to a server.

[1648] The server analyzes behavioral patterns and detects unusual transactions.

[1649] The server determines the risk is high and sends a warning message to the ATM.

[1650] The ATM screen displays the message, "Please be aware that you may be a victim of fraud."

[1651] If necessary, trading will be temporarily suspended and users will be advised to contact support.

[1652] In this way, the system analyzes both audio and video for signs of fraud and assesses the risk, enabling users to receive real-time warnings and take prompt action, thereby preventing fraud before it occurs.

[1653] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1654] Step 1: Record and send audio data

[1655] When a user starts a call on their smartphone, the device records the audio in the background. This recorded audio data is encrypted with AES and sent to a server using the HTTPS protocol. Specifically, the smartphone's microphone captures the audio, temporarily stores it in local storage, and then uploads it to the server via an HTTP request. The server then receives the sent audio data.

[1656] Input: User's voice data

[1657] Output: Encrypted audio data sent to the server

[1658] Step 2: Convert audio data to text

[1659] The server passes the received voice data to a speech recognition engine (e.g., Google Cloud Speech-to-Text) to convert the voice into text, which is then sent to the next analysis step.

[1660] Input: Encrypted audio data

[1661] Output: Converted text data

[1662] Step 3: Analyzing the text data

[1663] The server analyzes the converted text using a natural language processing (NLP) module (e.g., Textrazor). This analysis extracts keywords and context. For example, keywords that indicate fraud, such as "transfer" or "secret," are detected. The analysis results are stored in an internal data store and used as input for calculating a risk score.

[1664] Input: Text data

[1665] Output: Keywords and context analysis results

[1666] Step 4: Record and transmit video data

[1667] When a user operates an ATM, the security camera records the video. The recorded video data is encoded in H.264, encrypted with AES, and then sent to a server using the HTTPS protocol. Specifically, the ATM camera captures the video and streams it to the server in real time.

[1668] Input: Video data including user actions

[1669] Output: Encrypted video data sent to the server

[1670] Step 5: Analyze the video data

[1671] The server passes the received video data to a behavioral recognition algorithm (e.g., OpenPose) to analyze the user's movements and behavioral patterns. As a result of the analysis, unnatural behaviors and specific behavioral patterns are extracted. These results are also stored in an internal data store and used to calculate the risk score.

[1672] Input: Encrypted video data

[1673] Output: Analysis results of movements and behavior patterns

[1674] Step 6: Calculate the risk score

[1675] The server calculates a risk score using a machine learning model (e.g., Scikit-learn, TensorFlow) based on the results of analyzing the audio and video data. The machine learning model uses pre-trained fraud behavior data and quantifies the risk of fraud from the analysis results. If this risk score exceeds a threshold, it is deemed to be high risk.

[1676] Input: Analysis results of audio and video data

[1677] Output: Risk score

[1678] Step 7: Generate and send a warning message

[1679] The server generates a warning message based on the calculated risk score if the call is deemed high risk. This warning message is sent to the user's smartphone, ATM terminal, and monitoring agency. Specific messages include "This call may be fraudulent" and "You may be a victim of fraud."

[1680] Input: Risk Score

[1681] Output: Generate and send warning messages

[1682] Step 8: Receive and view alerts

[1683] The user's smartphone or ATM terminal receives the warning message from the server and displays it to the user in real time. On smartphones, the screen displays "This call may be fraudulent." On ATMs, the screen displays "You may be a victim of fraud, please be careful."

[1684] Input: warning message

[1685] Output: A warning message that is displayed on the user's terminal.

[1686] (Application example 1)

[1687] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1688] Financial frauds such as "I'm your son" and "transfer fraud" are on the rise, causing serious economic damage to many people, especially the elderly. To address this issue, a system that can detect fraud risks in real time and issue warnings is needed. However, existing systems lack the ability to analyze audio and video data, making it difficult to detect fraud quickly and accurately. Therefore, there is a need for a system that can use both audio and video to provide users with warnings in real time.

[1689] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1690] In this invention, the server includes a means for receiving voice data and recording it in the background, a voice recognition means for analyzing the received voice data, a means for analyzing the received voice data and calculating a risk score by natural language processing, a means for generating a warning message in real time based on the risk score, and a means for displaying the generated warning message on a screen, thereby enabling fraud risks to be detected quickly and accurately and enabling users to take appropriate action immediately.

[1691] "Voice data" refers to data used to digitally record, store, and analyze words and sounds made by users.

[1692] "Background recording" is the process of automatically recording and saving audio during a call without the user being aware of it.

[1693] "Speech recognition means" is a technology that has the function of converting voice data into text data.

[1694] "Natural language processing" is a technology for analyzing text data and understanding the structure and meaning of language.

[1695] A "risk score" is a number that indicates the risk of fraud assessed based on the analysis results.

[1696] A "warning message" is a message consisting of text or audio that notifies the user of a risk.

[1697] "Generating in real time" refers to the process of generating an alert message immediately after receiving the analysis results.

[1698] "Display on screen" is the process of visually displaying the generated warning message on the screen of the user's terminal.

[1699] A "security camera" is an image capturing device installed for monitoring and recording purposes.

[1700] An "action recognition algorithm" is a technology that analyzes video data and identifies specific actions and behavior patterns.

[1701] "Encrypted data" is data that has been converted using a special code to prevent unauthorized access.

[1702] A "voice recognition engine" is software or hardware that analyzes voice data and identifies distinctive keywords and phrases.

[1703] The system based on this invention aims to analyze audio and video data, evaluate fraud risk in real time, and issue a warning. A specific implementation method of the invention will be described below.

[1704] Hardware and software used

[1705] Server: Use a server with high-performance processing power. For example, a cloud server from AWS or Google Cloud.

[1706] Speech recognition engine: To convert voice data into text data, a speech recognition engine such as the Google Cloud Speech-to-Text API is used.

[1707] Natural Language Processing (NLP) module: NLP analysis could include the Japanese BERT model using the Hugging Face transformers library.

[1708] Action recognition algorithm: OpenCV and deep learning frameworks (e.g., TensorFlow, PyTorch) are used for video data analysis.

[1709] User devices: These include smartphones and ATM terminals. We assume that these devices are general devices equipped with recording and camera functions.

[1710] Specific examples of voice data processing

[1711] 1. Audio recording and transmission:

[1712] When a user initiates a call, the smartphone records the audio data in the background, encrypts it, and sends it to a server.

[1713] 2. Audio data analysis:

[1714] The server converts the received voice data into text using a speech recognition engine and analyzes it using an NLP module. Keywords such as "transfer" and "secret" are detected as a result of the analysis.

[1715] 3. Calculating the risk score:

[1716] A machine learning model calculates a risk score based on the detected keywords and other analysis results.

[1717] 4. Generate and display warning messages:

[1718] If the risk score exceeds a certain threshold, the server immediately generates a warning message and sends it to the user's smartphone, which displays a warning such as "This call may be fraudulent."

[1719] Specific examples of video data processing

[1720] 1. Video recording and transmission:

[1721] When a user operates an ATM, the security camera records the footage and encrypts and transmits the data to a server.

[1722] 2. Video data analysis:

[1723] The server analyzes the transmitted video data using behavioral recognition algorithms to detect behavioral patterns such as unusual operations or frequent large cash withdrawals.

[1724] 3. Calculating the risk score:

[1725] A risk score is calculated from the analysis of the video data, also using a machine learning model.

[1726] 4. Generate and display warning messages:

[1727] If the risk score exceeds a certain threshold, the server generates a warning message that appears on the ATM screen saying, "Please be careful, you may be a victim of fraud."

[1728] Examples of concrete examples and prompts

[1729] Specific scenarios

[1730] If a user makes a call using the word "transfer," a warning message will appear on their smartphone.

[1731] When a user attempts to withdraw a large amount of cash from an ATM, security camera footage is sent to a server and a warning message appears stating, "You may be a victim of fraud."

[1732] Prompt Sentence Examples

[1733] "Please assess the risk of fraud based on the following passage: 'To avoid falling victim to bank transfer fraud...'"

[1734] "Analyze the following conversation and assess the risk of fraud: 'I received a call asking me to transfer money, but it seems suspicious...'"

[1735] This allows for analysis of both audio and video to quickly and accurately detect the risk of users being scammed and provide necessary warnings in real time.

[1736] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1737] Step 1:

[1738] The smartphone records audio data in the background.

[1739] When a user starts a call, the device automatically records audio data in the background. This audio data is saved as a file containing the waveform information of the recorded sound. The input is the user's voice, and the output is the recorded audio data file.

[1740] Step 2:

[1741] Send the audio data to the server.

[1742] The audio data recorded on the device is encrypted and sent to the server using a secure protocol (e.g., HTTPS). The input is the recorded audio file, and the output is the encrypted audio data sent to the server.

[1743] Step 3:

[1744] The server analyzes the voice data and converts it into text.

[1745] The server converts the received voice data into text data using the Google Cloud Speech-to-Text API. The input is encrypted voice data, and the output is text data. This text data is the target for analysis.

[1746] Step 4:

[1747] Analyze text data using natural language processing.

[1748] The server analyzes the received text data using Hugging Face's transformers library. This analysis uses the Japanese BERT model to identify keywords and context and evaluate the risk of fraud. The input is text data, and the output is the analysis results, including keywords and context information.

[1749] Step 5:

[1750] A risk score is calculated based on the analysis results.

[1751] The server uses a machine learning model to calculate a risk score based on specific keywords and contextual information from the analysis results. For example, if a keyword such as "transfer" is detected, the risk score will be higher. The input is the analysis result, and the output is the risk score.

[1752] Step 6:

[1753] Generate warning messages based on risk scores.

[1754] If the risk score exceeds the set threshold, the server judges it to be a high risk and generates a warning message. This warning message is notified to the user. The input is the risk score, and the output is the warning message.

[1755] Step 7:

[1756] Send a warning message to the user's smartphone.

[1757] The generated warning message is sent from the server to the user's device. The receiving device immediately displays the warning message on its screen. The input is the warning message, and the output is the warning display on the user's smartphone.

[1758] Step 8:

[1759] Displays a warning message on the screen.

[1760] The user's smartphone immediately displays the received warning message on the screen to notify the user of the risk of fraud. The user can visually confirm this warning and take appropriate action against the fraud. The input is the warning message, and the output is the warning displayed on the screen.

[1761] Step 9:

[1762] Recording and transmitting video data (in the case of ATMs).

[1763] When a user operates an ATM, the security camera records the user's actions and the video data is encrypted and sent to the server. The input is the user's ATM operations, and the output is the encrypted video data sent to the server.

[1764] Step 10:

[1765] Analyze video data and identify behavioral patterns.

[1766] The server analyzes the received video data using a behavior recognition algorithm to identify the user's behavioral patterns. Unnatural actions such as frequent large cash withdrawals are detected. The input is encrypted video data, and the output is the results of behavioral pattern analysis.

[1767] Step 11:

[1768] A risk score is calculated based on the results of video data analysis.

[1769] The server uses a machine learning model to calculate a risk score based on the results of the behavioral pattern analysis. If certain unnatural behavior is detected, the risk score is set high. The input is the behavioral pattern analysis result, and the output is the risk score.

[1770] Step 12:

[1771] A warning message is generated and displayed on the ATM screen.

[1772] If the risk score is high, the server generates a warning message and displays it on the ATM screen. The user sees the warning: "You may be a victim of fraud, please be careful." The input is the risk score, and the output is the warning on the ATM screen.

[1773] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1774] This invention is a system designed to prevent financial frauds such as "I'm your son" frauds and "transfer frauds." The system utilizes audio data, video data, and an emotion engine to assess fraud risk in real time and issue warnings. The system is primarily composed of a server, terminals (smartphones and ATMs), and users.

[1775] Server-side processing

[1776] Data reception and initial analysis

[1777] The server receives voice and video data sent from the user's smartphone or ATM. For voice data, a speech recognition engine converts the speech into text, and a natural language processing (NLP) module analyzes keywords and context. An emotion engine analyzes the tone and speed of the voice to detect the user's emotions. For video data, a behavior recognition algorithm is applied to analyze the user's movements and behavior patterns. The emotion engine analyzes facial expressions and posture to recognize the user's emotions.

[1778] Risk Assessment

[1779] The server integrates the results of the audio data analysis, the video data analysis, and information from the emotion engine, and calculates a risk score using a machine learning model. For example, keywords that indicate fraud, such as "transfer" and "secret," as well as the user's emotions, such as impatience and tension, affect the risk score. The emotion engine adjusts the risk score based on these emotion indicators.

[1780] Alerting and Notifications

[1781] If the risk score exceeds a certain threshold, the server determines it to be a high risk and generates a warning message. This warning message is sent to the user's smartphone or ATM terminal. Security companies are also notified at the same time, enabling a prompt response.

[1782] Smartphone device processing

[1783] Audio recording and transmission

[1784] When a user makes a call through their smartphone, the device records the audio in the background. The recorded audio data is encrypted and sent to a server using the HTTPS protocol. The emotion engine also analyzes the audio and evaluates the user's emotions in real time.

[1785] Receiving and displaying alerts

[1786] When an alert is sent from the server, the smartphone receives a real-time warning message and displays it on the screen. This message may say something like, "This call may be fraudulent." By seeing this warning, the user can recognize the risk of fraud and take appropriate measures.

[1787] ATM terminal processing

[1788] Video recording and transmission

[1789] When a user operates an ATM, the operation is recorded by a security camera. This video data is encrypted and sent to a server. The server analyzes the video data to detect unusual behavior by the user or withdrawal of large amounts of cash. The emotion engine analyzes the user's facial expressions and movements to evaluate their emotional state.

[1790] Receiving and displaying alerts

[1791] When an alert from the server is sent to the ATM, a warning message will appear on the ATM screen, specifically stating, "Please be aware that you may be a victim of fraud." In some cases, the ATM may also include the option to pause the transaction and offer direct assistance to the user.

[1792] Specific examples

[1793] Specific examples when using a smartphone

[1794] 1. The user initiates a call with the scammer, and an app on their smartphone automatically records the audio in the background.

[1795] 2. The recorded voice data is encrypted and sent to the server using the HTTPS protocol. The emotion engine also analyzes the voice and evaluates the user's emotions.

[1796] 3. The server analyzes the voice data and detects keywords that indicate fraud, such as "transfer."

[1797] 4. The server determines the risk is high and sends a warning message to the user's smartphone. If the emotion engine detects that the user is anxious or nervous, the risk score will be further increased.

[1798] 5. The user's smartphone displays a warning that the call may be fraudulent.

[1799] Examples of ATM usage

[1800] 1. A user attempts to withdraw a large amount of cash from an ATM. A security camera records the footage, encrypts the data, and sends it to a server. The emotion engine analyzes the user's facial expressions and movements to evaluate their emotional state.

[1801] 2. The server analyzes the video data to detect unnatural trading behavior patterns. If the emotion engine detects tension in the user, it further increases the risk score.

[1802] 3. The server determines the risk to be high and sends a warning message to the ATM.

[1803] 4. A warning message appears on the ATM screen saying, "Please be aware that you may be a victim of fraud."

[1804] 5. If necessary, the ATM will temporarily suspend the transaction and provide the user with the option to contact the support center.

[1805] In this way, the system analyzes signs of fraud based on three elements: audio, video, and emotion analysis, assesses risk, and provides real-time warnings to users and enables them to take prompt action, thereby preventing fraud damage before it occurs.

[1806] The processing flow will be explained below.

[1807] Server-side processing

[1808] Step 1:

[1809] The server receives audio and video data sent from the user's smartphone or ATM via HTTP requests or WebSockets.

[1810] Step 2:

[1811] The server passes the received voice data to a voice recognition engine, which converts it into text data. At the same time, the voice data is sent to an emotion engine, which analyzes the user's emotions based on the tone and speed of the voice.

[1812] Step 3:

[1813] The text data converted by the speech recognition engine is sent to a natural language processing (NLP) module, where it is analyzed for keywords and context.

[1814] Step 4:

[1815] The server passes the received video data to a behavior recognition algorithm, which analyzes the user's movements and behavioral patterns. In parallel, the video data is sent to an emotion engine, which recognizes emotions from the user's facial expressions and movements.

[1816] Step 5:

[1817] The results of audio data analysis, video data analysis, and information from the emotion engine are integrated and input into a machine learning model to calculate a risk score. Emotional states (e.g., anxiety, tension, etc.) affect the risk score.

[1818] Step 6:

[1819] If the risk score exceeds a certain threshold, the server generates a warning message.

[1820] Step 7:

[1821] The server sends the generated warning message to the user's smartphone or ATM terminal, and simultaneously notifies the security company.

[1822] Smartphone device processing

[1823] Step 1:

[1824] A user initiates a call on their smartphone, and an app on the smartphone automatically records the audio in the background.

[1825] Step 2:

[1826] The emotion engine analyzes the tone and rate of speech to assess the user's emotional state (e.g., impatience or tension) in real time.

[1827] Step 3:

[1828] The recorded audio data is encrypted and sent to the server using the HTTPS protocol.

[1829] Step 4:

[1830] When an alert is sent from the server, the smartphone receives a warning message in real time.

[1831] Step 5:

[1832] The app displays a warning message on the user's screen, providing specific warning information and recommended actions.

[1833] ATM terminal processing

[1834] Step 1:

[1835] When a user begins to operate the ATM, the security camera records the activity.

[1836] Step 2:

[1837] The recorded video data is encrypted and sent to a server, where it is simultaneously sent to an emotion engine that analyzes emotions from the user's facial expressions and movements.

[1838] Step 3:

[1839] The server analyzes the video data to detect patterns of behavior, such as unusual user behavior or large cash withdrawals. If the emotion engine recognizes the user's emotions, such as nervousness or impatience, it will affect the risk score.

[1840] Step 4:

[1841] When an alert is sent from the server, the ATM receives a warning message in real time.

[1842] Step 5:

[1843] A warning message appears on the ATM screen, stating, "You may be a victim of fraud."

[1844] Step 6:

[1845] If a transaction is deemed high risk, the ATM terminal will temporarily suspend the transaction and prompt the user to contact the support center.

[1846] Specific examples

[1847] Specific examples when using a smartphone

[1848] Step 1:

[1849] The user initiates a call with the scammer, and an app on their phone automatically records the audio in the background.

[1850] Step 2:

[1851] The emotion engine analyzes the tone and speed of speech in real time to assess the user's emotional state.

[1852] Step 3:

[1853] The recorded audio data is encrypted and sent to the server using the HTTPS protocol.

[1854] Step 4:

[1855] The server analyzes the voice data and emotional state to detect keywords that indicate fraud, such as "transfer."

[1856] Step 5:

[1857] The server determines a high risk based on voice and emotional data and sends a warning message to the user's smartphone. If the emotion engine detects anxiety or tension, the risk score will be further increased.

[1858] Step 6:

[1859] The user's smartphone will display a warning that "This call may be fraudulent."

[1860] Examples of ATM usage

[1861] Step 1:

[1862] A user attempts to withdraw a large amount of cash from an ATM. The security camera records the footage, and the data is encrypted and sent to the server along with the emotion engine.

[1863] Step 2:

[1864] The server analyzes the video data to detect unnatural trading patterns, and the emotion engine analyzes the user's facial expressions and movements to detect tension or impatience.

[1865] Step 3:

[1866] The server calculates a risk score based on video data, emotional data, and behavioral analysis results. If the emotional engine detects tension, the risk score is further increased.

[1867] Step 4:

[1868] The server determines the risk is high and sends a warning message to the ATM.

[1869] Step 5:

[1870] A warning message appears on the ATM screen saying, "Please be careful, you may be a victim of fraud."

[1871] Step 6:

[1872] If necessary, the ATM will temporarily halt the transaction and give the user the option to contact a support center.

[1873] Example 2

[1874] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1875] Conventional financial fraud prevention systems simply analyze audio and video data, resulting in insufficient risk assessment. Furthermore, because real-time risk assessments that take into account the user's emotional state are not performed, potential fraud cannot be detected early, resulting in many cases of user harm. The objective of the present invention is to solve these problems and provide a more accurate and rapid fraud prevention system.

[1876] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1877] In this invention, the server includes means for receiving audio data and video data, means for analyzing the received audio data and video data, and means for calculating a risk score based on the results of the analysis of the audio data and video data and the results of the emotion analysis, thereby enabling a comprehensive risk assessment that takes into account not only audio but also video and the user's emotional state.

[1878] The present invention also includes a means for generating a warning message based on the risk score, a means for transmitting the generated warning message to a user terminal or an automated transaction machine, and a means for suspending a transaction or notifying a security company based on the transmitted warning message, thereby enabling users to recognize fraud risks in real time and take appropriate measures promptly.

[1879] Furthermore, the present invention includes a means for a user terminal to record voice data and transmit it to a server, a means for the server to analyze the voice data and detect specific keywords and contexts, a means for analyzing the user's emotional state in addition to analyzing the voice data, a means for calculating a risk score based on the detected keywords and emotional state, and a means for transmitting and displaying a generated warning message to the user terminal, thereby enabling early detection and prevention of fraud risks through multifaceted data analysis.

[1880] The system also includes a means for the automated teller machine to record video data and transmit it to a server, a means for the server to analyze the video data and detect unnatural behavioral patterns or specific behaviors, a means for analyzing the user's emotional state in addition to analyzing the video data, a means for calculating a risk score based on the detected behavioral patterns and emotional state, and a means for transmitting the generated warning message to the automated teller machine and displaying it, thereby enabling advanced risk assessment that takes physical behavior and emotional state into consideration.

[1881] "Voice data" refers to data in which the voice uttered by the user is recorded in digital format.

[1882] "Video data" refers to data that captures the movements and conditions of a user and their surroundings with a camera and records them in digital format.

[1883] A "server" is a computer system that communicates with multiple terminals over a network and receives, analyzes, stores, and transmits data.

[1884] "Analysis" is the process of breaking down data, deciphering its components and meaning, and extracting specific information.

[1885] "Emotion analysis" is a technology that detects a user's emotional state from audio and video data and analyzes the results.

[1886] A "risk score" is a numerical representation of the likelihood of fraud based on the analysis results, and is an evaluation index in which a higher value indicates a higher risk.

[1887] A "warning message" is a notification message that warns users and other related parties when it is determined that there is a high risk of fraud.

[1888] A "user terminal" is a device that is directly operated by a user, and includes smartphones, personal computers, and the like.

[1889] An "automated transaction machine" is a device that allows users to conduct financial transactions, and includes ATMs (automated teller machines).

[1890] "Notification" is the act of the system informing users and related parties of specific information.

[1891] This invention is a system that utilizes audio data, video data, and emotion analysis to assess fraud risk in real time and issue warnings. The system is primarily composed of a server, a user terminal (smartphone), and an automated teller machine (ATM). The specific operation of each device is described below.

[1892] Server Processing

[1893] Receiving and analyzing audio and video data

[1894] The server receives voice and video data sent from the user's smartphone or ATM. For the voice data, it converts the speech into text using a speech recognition engine (e.g., Google Cloud Speech-to-Text API or IBM Watson Speech to Text), analyzes keywords and context using a natural language processing (NLP) module (e.g., Python's NLTK library or spaCy), and detects the user's emotions by analyzing the tone and speed of the speech using an emotion engine (e.g., Microsoft Azure Face API or Affectiva).

[1895] For video data, behavioral recognition algorithms (e.g., OpenCV or Google Vision API) are applied to analyze user movements and behavioral patterns. The emotion engine analyzes facial expressions and posture to recognize user emotions.

[1896] Risk Assessment

[1897] The server integrates the results of audio data analysis, video data analysis, and information from the emotion engine, and calculates a risk score using a machine learning model (e.g., Scikit-learn or TensorFlow). For example, keywords such as "transfer" and "secret" and emotions such as the user's impatience or tension affect the risk score.

[1898] Alerting and Notifications

[1899] If the risk score exceeds a certain threshold, the server generates a warning message, which is sent to the user's smartphone and ATM, and also notifies the security company, enabling a prompt response.

[1900] User device (smartphone) processing

[1901] Audio recording and transmission

[1902] When a user makes a call through their smartphone, the device records the voice in the background, encrypts the recorded voice data, and sends it to the server using the HTTPS protocol. The emotion engine also analyzes the voice and evaluates the user's emotions in real time.

[1903] Receiving and displaying alerts

[1904] When an alert is sent from the server, the smartphone receives a real-time warning message and displays it on the screen, such as "This call may be fraudulent."

[1905] Automatic transaction machine (ATM) processing

[1906] Video recording and transmission

[1907] When a user operates an ATM, the security camera records the operation, encrypts the video data, and sends it to a server. The server analyzes the video data to detect unusual behavior or large cash withdrawals. The emotion engine analyzes the user's facial expressions and movements to evaluate their emotional state.

[1908] Receiving and displaying alerts

[1909] When an alert from the server is sent to the ATM, a warning message is displayed on the ATM screen, such as "Please be aware that you may be a victim of fraud," and the ATM also includes the option to pause the transaction and offer direct assistance to the user if necessary.

[1910] Specific examples and prompts

[1911] Specific examples when using a smartphone

[1912] 1. The user initiates a call with the scammer, and an app on their smartphone automatically records the audio in the background.

[1913] 2. The recorded voice data is encrypted and sent to the server using the HTTPS protocol. The emotion engine also analyzes the voice and evaluates the user's emotions.

[1914] 3. The server analyzes the voice data and detects keywords that indicate fraud, such as "transfer."

[1915] 4. The server determines the risk is high and sends a warning message to the user's smartphone. If the emotion engine detects that the user is anxious or nervous, the risk score will be further increased.

[1916] 5. The user's smartphone displays a warning that the call may be fraudulent.

[1917] Examples of ATM usage

[1918] 1. A user attempts to withdraw a large amount of cash from an ATM. A security camera records the footage, encrypts the data, and sends it to a server. The emotion engine analyzes the user's facial expressions and movements to evaluate their emotional state.

[1919] 2. The server analyzes the video data to detect unnatural trading behavior patterns. If the emotion engine detects tension in the user, it further increases the risk score.

[1920] 3. The server determines the risk to be high and sends a warning message to the ATM.

[1921] 4. A warning message appears on the ATM screen saying, "Please be aware that you may be a victim of fraud."

[1922] 5. If necessary, the ATM will temporarily suspend the transaction and provide the user with the option to contact the support center.

[1923] Example prompts to input to the generative AI model

[1924] 1. Call Analysis Prompt:

[1925] A user is making a call on their smartphone. If the speech recognition engine detects keywords such as "transfer" and "secret," explain how a risk score is calculated and a warning message is generated.

[1926] 2. ATM Analysis Prompt:

[1927] A user is attempting to withdraw a large amount of cash from an ATM. If a security camera records the user's video and sends it to a server, explain how a risk score is calculated and a warning message is generated.

[1928] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1929] Step 1:

[1930] Data collection and transmission

[1931] When a user initiates a call, the smartphone automatically records audio in the background. This recorded audio data is encrypted and sent to a server using the HTTPS protocol. The audio data then becomes input data for analysis based on the content of the user's call.

[1932] As a specific operation, for example, while the user is talking to a fraudster, voice data containing the word "secret" is recorded and sent to the server.

[1933] Step 2:

[1934] Analysis of audio data

[1935] The server uses a speech recognition engine (e.g., Google Cloud Speech-to-Text) to convert the received voice data into text. This text data is then analyzed for keywords and context using a natural language processing (NLP) module (e.g., Python's NLTK library). The analysis results are also sent to an emotion engine, which detects the user's emotion from the tone and speed of the voice.

[1936] The input data is an audio file of audio data, and the output data is converted text data and emotional information. For example, the converted text data contains the keyword "transfer," and emotional analysis detects a tense tone.

[1937] Step 3:

[1938] Video data collection and transmission

[1939] When a user operates an ATM, the security camera records the footage, and the data is encrypted and sent to the server.

[1940] Specifically, when a user attempts to withdraw a large amount of cash from an ATM, this operation is recorded as video data and sent to the server.

[1941] Step 4:

[1942] Video data analysis

[1943] The server uses an action recognition algorithm (e.g., OpenCV) to analyze the video data. The user's movements and behavioral patterns are analyzed, and the emotion engine detects the user's emotions from their facial expressions and posture. The input data is recorded video data, and the output data is the user's behavioral patterns and emotional information. For example, video analysis may detect that the user is looking around unnaturally, and the emotion engine may identify that the user is in a state of tension from their facial expression.

[1944] Step 5:

[1945] Risk score calculation

[1946] The server integrates the analyzed audio data, video data, and emotional data and calculates a risk score using a machine learning model (e.g., Scikit-learn or TensorFlow). The risk score varies depending on specific keywords, unnatural behavior, and emotional state. The input data includes the results of audio analysis, video analysis, and emotional analysis, and the output data is a risk score. For example, if the keyword "transfer money" is found together with a tense tone and unnatural behavior, a high risk score is set.

[1947] Step 6:

[1948] Generate and send warning messages

[1949] If the risk score exceeds a certain threshold, the server generates a warning message. This warning message is sent to the user's smartphone or ATM. The security company is also notified. The input data is the risk score, and the output data is a warning message that is generated and sent to a specified terminal. For example, if the risk score is high, a message such as "This call may be fraudulent" or "You may be a victim of fraud, so please be careful" is displayed on the smartphone or ATM screen.

[1950] (Application example 2)

[1951] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1952] In recent years, financial frauds such as "I'm your son" frauds and "transfer frauds" have been on the rise, creating a need for effective prevention measures. However, conventional security systems lack the ability to detect signs of fraud in real time and issue prompt warnings. In particular, they lack the ability to integrate analysis of audio, video, and the user's emotional state, making it difficult to accurately assess fraud risk. Therefore, there is a need to develop a system that integrates audio, video, and emotional analysis to assess fraud risk in real time and issue warnings to users.

[1953] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving voice data, means for analyzing the received voice data, means for evaluating an emotional state, means for calculating a risk score based on the analysis results of the voice data and the emotional state, means for generating a warning message based on the risk score, and means for transmitting the generated warning message to a user terminal. This makes it possible to integrate the three data analyses of voice, video, and emotion analysis, accurately evaluate the risk of fraud in real time, and quickly issue a warning.

[1954] The "means for receiving voice data" is a device or module that receives voice data sent from a user terminal or a voice input device and converts it into a format that can be processed within the system.

[1955] The "means for analyzing received voice data" refers to a device or module that converts the acquired voice data into text using voice recognition technology and further analyzes it using natural language processing technology.

[1956] The "means for assessing emotional state" is a device or module for detecting the user's emotions from the tone and speed of the voice and the facial expressions of the video, and evaluating the results.

[1957] A "means for calculating a risk score" is a device or module that includes an algorithm or machine learning model for scoring the risk of fraud based on the analysis of voice data and the assessment of emotional state.

[1958] The "means for generating a warning message" is a device or module that creates a message to warn the user when the risk score exceeds a certain threshold.

[1959] The "means for transmitting the generated warning message to the user terminal" is a communication means for notifying the user's smartphone or other terminal of the generated warning message in real time.

[1960] The "means for receiving video data" is a device or module that receives video data sent from a user terminal or camera and converts it into a format that can be processed within the system.

[1961] The "means for analyzing received video data" refers to a device or module that analyzes the acquired video data using behavior recognition technology and detects unnatural behavior or actions.

[1962] "Means for a user terminal to record voice data and transmit it to a server" refers to a device or module that allows a terminal such as a user's smartphone to record voice and transmit that data to a server via a network.

[1963] "Means for the server to analyze the voice data and detect specific keywords and emotional states" refers to a device or module that allows the server to analyze the received voice data and identify keywords that indicate fraud and the user's emotional state.

[1964] To implement the present invention, the following system configuration and method are used.

[1965] The system is mainly composed of a server, a terminal (smartphone), and a user. The following explains in detail the configuration and the specific role of each means.

[1966] Server-side processing

[1967] Data reception and initial analysis

[1968] The server receives audio and video data sent from the user's smartphone. The audio data is converted into text using a speech recognition engine, and a natural language processing (NLP) module analyzes keywords and context. An emotion engine then analyzes the tone and speed of the voice to assess the user's emotional state. The video data is analyzed using a behavior recognition algorithm to analyze the user's movements and behavior patterns. Similarly, the emotion engine analyzes facial expressions and posture to assess the user's emotional state.

[1969] Risk Assessment

[1970] The server integrates the results of the audio data analysis, the video data analysis, and information from the emotion engine, and calculates a risk score using a machine learning model. Specifically, keywords that indicate fraud, such as "transfer" and "secret," as well as the user's impatience and tension, affect the risk score. The emotion engine adjusts the risk score based on these emotion indicators.

[1971] Alerting and Notifications

[1972] If the risk score exceeds a certain threshold, the server determines that the risk is high and generates a warning message. The generated warning message is sent to the user's smartphone. If necessary, a security company is also notified at the same time.

[1973] Smartphone device processing

[1974] Audio recording and transmission

[1975] When a user makes a call through their smartphone, the device records the voice in the background. The recorded voice data is encrypted and sent to the server using the HTTPS protocol. The emotion engine also analyzes the voice and evaluates the user's emotions in real time.

[1976] Receiving and displaying alerts

[1977] When an alert is sent from the server, the smartphone receives a real-time warning message and displays it on the screen. This message includes a warning such as "This call may be fraudulent." By seeing this warning, users can recognize the risk of fraud and take appropriate measures.

[1978] Specific examples

[1979] Specific examples when using a smartphone

[1980] 1. The user initiates a call with the scammer, and an app on their smartphone automatically records the audio in the background.

[1981] 2. The recorded voice data is encrypted and sent to the server using the HTTPS protocol. The emotion engine also analyzes the voice and evaluates the user's emotions.

[1982] 3. The server analyzes the voice data and detects keywords that indicate fraud, such as "transfer."

[1983] 4. The server determines the risk is high and sends a warning message to the user's smartphone. If the emotion engine detects that the user is anxious or nervous, the risk score will be further increased.

[1984] 5. The user's smartphone displays a warning that the call may be fraudulent.

[1985] As a concrete example, the prompt sentence to be input to the generative AI model is as follows:

[1986] "Generate proposals for a smartphone app that detects high-risk bank transfer fraud in real time and displays a warning message to the user. Analyze audio and video data, evaluate the user's emotional state with an emotion engine, and calculate a risk score with a machine learning model. If a risk exceeding a threshold is detected, display a warning message to the user."

[1987] Thus, an embodiment of the present invention provides a system that integrates three types of data: audio, video, and emotion analysis, to assess fraud risk in real time and quickly warn users.

[1988] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1989] Step 1:

[1990] A user initiates a call through a smartphone. The smartphone runs in the background and records the call. The input is the user's voice, and the output is the recorded voice data.

[1991] Step 2:

[1992] The device encrypts the recorded audio data and sends it to the server using the HTTPS protocol. The input here is the recorded audio data, and the output is the encrypted audio data sent to the server.

[1993] Step 3:

[1994] The server converts the received voice data into text using a voice recognition engine. The input here is encrypted voice data, and the output is analyzed text data.

[1995] Step 4:

[1996] The server analyzes the text of the voice data using a natural language processing (NLP) module and extracts specific keywords that indicate fraud. The input here is the analyzed text data, and the output is the extracted keywords.

[1997] Step 5:

[1998] The server uses an emotion engine to process the user's emotional state (e.g., tension or impatience) from the voice data. The input here is the voice data, and the output is the user's emotional state.

[1999] Step 6:

[2000] The server integrates the analysis results of the voice data and emotional state and processes them to calculate a risk score using a machine learning model, where the input is the extracted keywords and emotional state, and the output is the calculated risk score.

[2001] Step 7:

[2002] The server processes the risk score and generates a warning message based on the calculated risk score, where the input is the risk score and the output is the warning message.

[2003] Step 8:

[2004] The server processes the generated warning message and sends it to the user's smartphone. The input here is the warning message, and the output is the warning message displayed on the user's smartphone.

[2005] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2006] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2007] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2008] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2009] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2010] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2011] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2012] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2013] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2014] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2015] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2016] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2017] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2018] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installe...

Claims

1. means for receiving audio data; means for analyzing received audio data; means for calculating a risk score based on the analysis of the voice data; means for generating a warning message based on the risk score; means for transmitting the generated warning message to a user terminal; A system including:

2. means for receiving video data; means for analyzing the received video data; means for calculating a risk score based on the analysis of the video data; means for generating a warning message based on the risk score; means for transmitting the generated warning message to a user terminal; The system of claim 1 , comprising:

3. A means for the user terminal to record voice data and transmit it to the server; A means for the server to analyze the voice data and detect specific keywords; means for calculating a risk score based on the detected keywords; means for generating a warning message based on the risk score; means for transmitting the generated warning message to a user terminal; The system of claim 1 , comprising:

Images