System

Abstract

A system is provided.SOLUTION: A system comprising terminal means for collecting voice data and transmitting the voice data to a server, voice recognition means for converting the voice data into text data, analysis means for analyzing the text data and detecting a specific keyword, call origination location identification means for analyzing call origination location information of a call, AI model means for evaluating a possibility of fraud using the voice data, the keyword, and the call origination location information, warning generation means for generating a warning message based on a result of the evaluation of the possibility of fraud and transmitting the warning message to the terminal, and terminal means for notifying a user of the generated warning message.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] "It's me" fraud has become increasingly sophisticated in recent years, with losses reaching tens of billions of yen annually. Elderly people are particularly vulnerable, and those who care for their families and have few people to turn to for advice are often targeted. Conventional countermeasures have been insufficient to adequately prevent fraud. In light of the current problems, there is a need to provide a system that can more quickly and accurately detect possible "it's me" fraud and prevent fraud damage before it occurs. [Means for solving the problem]

[0005] The present invention provides a system that includes a terminal means for collecting voice data and transmitting it to a server, a voice recognition means for converting the voice data into text data, an analysis means for analyzing the text data and detecting specific keywords, a call origination location identification means for analyzing the call origination location information, and an AI model means for evaluating the possibility of fraud using the voice data, keywords, and call origination location information. This system includes a warning generation means for generating a warning message based on the results of the fraud likelihood evaluation and transmitting it to the terminal, and a terminal means for notifying the user of the generated warning message, thereby enabling the system to quickly and accurately detect calls that may be "I'm your son" fraud and immediately warn the user, thereby making it possible to prevent fraud damage before it occurs.

[0006] "Voice data" refers to voice signals collected from a terminal during a call.

[0007] A "terminal" is a device for making calls and has the function of collecting voice data and transmitting it to a server.

[0008] "Speech recognition means" refers to a technology or device that analyzes collected voice data and converts it into text data.

[0009] "Text data" refers to character information of voice data converted by a voice recognition means.

[0010] "Analysis means" refers to a technique or device that analyzes text data and detects specific keywords.

[0011] "Call origin identification means" refers to a technology or device that analyzes the call origin information and determines whether the location is suspicious.

[0012] "Keywords" refer to specific words or phrases that are likely to indicate telephone fraud.

[0013] "AI model means" refers to artificial intelligence technology that uses voice data, text data, keywords, and origin information to assess the likelihood of fraud.

[0014] "Warning generation means" refers to a technology or device that generates a message to warn the user based on the result of assessing the likelihood of fraud.

[0015] "Warning message" refers to the content of the warning that is generated and notified to the user when a possible fraud is detected.

[0016] "User" refers to a person who makes calls using the system and receives alert messages. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] This invention provides a system that detects "I am your son" fraud calls in real time and issues a prompt warning, targeting elderly people and other users who are prone to fraud. This system is an integrated system that includes voice data collection, voice analysis, keyword detection, caller location identification, fraud probability assessment using an AI model, and warning generation and notification.

[0039] System Configuration

[0040] Server-side configuration

[0041] 1. Receiving audio data

[0042] The server receives real-time voice data during a call, which is sent from the terminal.

[0043] 2. Voice Recognition

[0044] The server analyzes the received voice data using a speech recognition module and converts it into text data, with noise reduction and echo cancellation also being applied during this process.

[0045] 3. Keyword Discovery

[0046] The server analyzes the text data generated by the speech recognition to detect specific keywords, such as "money," "transfer," "accident," and "help."

[0047] 4. Identifying the source

[0048] The server analyzes the call origin information and if it detects any suspicious locations (such as public phones or calls from abroad), it records that information.

[0049] 5. AI-powered fraud assessment

[0050] The server inputs the voice data, detected keywords, and location information into an AI model to assess the likelihood of fraud, with reference to past data and known fraud patterns.

[0051] 6. Generating Warning Messages

[0052] If the server assesses a high probability of fraud, it will automatically generate a warning message after the call ends, including information about the suspected fraud and what steps should be taken.

[0053] 7. Sending Alerts

[0054] The server sends the generated warning message to the user's terminal.

[0055] Terminal configuration

[0056] 1. Collection of audio data

[0057] The device collects voice data during a call in real time and sends it to the server.

[0058] 2. Receiving a warning

[0059] When a warning message is sent from the server, the device notifies the user via voice call or message notification.

[0060] User-side configuration

[0061] 1. Check the contents of the call

[0062] The user will carry out the call as if it were a normal call.

[0063] 2. Check the warning

[0064] After the call ends, the user checks the warning message displayed on the device, which includes information about the suspected fraud and specific countermeasures.

[0065] Specific examples

[0066] 1. The device starts a call between elderly person C and the person in charge.

[0067] 2. The device transmits the voice data to the server in real time.

[0068] 3. The server converts the speech into text and detects the keyword "money."

[0069] 4. The server analyzes the call's location information and determines that the call originated from a domestic public telephone.

[0070] 5. The server inputs this information into an AI model and rates the likelihood of fraud.

[0071] 6. After the call ends, the server automatically generates a warning message and sends it to the terminal.

[0072] 7. The device will notify Mr. C of the warning message, so that he can immediately pay attention and take the necessary action.

[0073] In this way, the present invention can automate a series of processes from collecting voice data to evaluating the possibility of fraud and issuing a warning, thereby minimizing the risk of users becoming victims of fraud.

[0074] The processing flow will be explained below.

[0075] Step 1:

[0076] When a call is initiated, the device collects voice data in real time and transmits the voice data during the call to the server as is.

[0077] Step 2:

[0078] The server receives the voice data sent from the terminal and inputs it into a voice recognition module for real-time processing.

[0079] Step 3:

[0080] The server uses a speech recognition module to convert the voice data into text data, and pre-processing such as noise reduction and echo cancellation is performed during this process.

[0081] Step 4:

[0082] The server analyzes the text data and detects certain keywords, such as "money," "transfer," "accident," and "help."

[0083] Step 5:

[0084] The server analyzes the call origin information, identifying the origin using phone number, IP address, GPS information, etc., and records any suspicious locations (such as public telephones or calls from overseas).

[0085] Step 6:

[0086] The server inputs the voice data, text data, detected keywords, and location information into the AI ​​model, which then evaluates the likelihood of fraud based on this information.

[0087] Step 7:

[0088] If the server determines that there is a high possibility of fraud based on the fraud probability score evaluated by the AI ​​model, it will generate a warning message after the call ends, notifying the caller of the suspected fraud and providing specific countermeasures.

[0089] Step 8:

[0090] The server sends the generated warning message to the user's terminal, and the message arrives at the terminal in real time.

[0091] Step 9:

[0092] When the device receives the warning message sent from the server, it notifies the user, which is displayed as a voice call or a message.

[0093] Step 10:

[0094] The user checks the warning message on the device, understands the contents of the warning message, and takes necessary measures, such as rechecking the contents of calls that are suspected to be fraudulent and consulting the police or financial institution.

[0095] Through these steps, the system can quickly and accurately detect possible telephone fraud and issue an immediate warning to the user.

[0096] Example 1

[0097] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0098] There is a challenge in detecting "I'm your son" fraud calls in real time and issuing prompt warnings, targeting elderly people and other users who are prone to fraud. Conventional systems have difficulty immediately detecting potential fraud and issuing warnings, putting users at risk of becoming victims. In particular, since calls from suspicious locations and specific keywords are often not detected, there is a need for a means to prevent fraud in advance.

[0099] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0100] In this invention, the server includes terminal means for collecting voice data and transmitting it to the server, voice recognition means for converting the voice data into text data, analysis means for analyzing the text data and detecting specific keywords, call origination location identification means for analyzing call origination location information, voice processing means for performing noise reduction and echo cancellation, AI model means for evaluating the possibility of fraud using the voice data, keywords, and call origination location information, warning generation means for generating a warning message based on the result of the evaluation of the possibility of fraud and transmitting it to the terminal, and terminal means for notifying the user of the generated warning message. This makes it possible to detect fraudulent calls in real time and quickly issue a warning to the user.

[0101] "Voice data" refers to data converted into digital form of voice in a conversation between a user and another person.

[0102] "Terminal means" refers to a device that has the function of collecting voice data and transmitting it to a server.

[0103] "Speech recognition means" refers to software or hardware that has the function of analyzing received voice data and converting it into text data.

[0104] "Text data" refers to character string information generated from voice data by a voice recognition means.

[0105] "Analysis means" refers to software or hardware that has the function of analyzing text data and detecting specific keywords.

[0106] "Call origin identification means" refers to software or hardware that has the function of analyzing call origin information and identifying suspicious locations.

[0107] "Audio processing means" refers to software or hardware that performs noise reduction and echo cancellation and optimizes audio data.

[0108] "AI model means" refers to an artificial intelligence model that uses voice data, keywords, and location information to assess the likelihood of fraud.

[0109] "Warning generation means" refers to software or hardware that has the function of generating a warning message based on the result of assessing the likelihood of fraud and sending it to a terminal.

[0110] "Warning Message" means notification information containing warnings and countermeasures regarding potentially fraudulent calls.

[0111] This invention provides a system that detects "I am your son" fraud calls in real time and issues a prompt warning, targeting elderly people and other users who are prone to fraud. This system is an integrated system that includes voice data collection, voice analysis, keyword detection, caller location identification, fraud probability assessment using an AI model, and warning generation and notification.

[0112] Server-side configuration

[0113] 1. Receiving audio data

[0114] The server receives real-time audio data during a call. This audio data is sent from the device. The device collects audio during the call with a microphone and encodes it digitally using an AudioCodec. The encoded audio data is then sent to the server using HTTP or WebSocket protocols.

[0115] 2. Voice Recognition

[0116] The server analyzes the received voice data and converts it into text data using a speech recognition module. This process uses voice recognition software such as Google Cloud Speech-to-Text or IBM Watson. The server pre-processes the voice data by performing noise reduction and echo cancellation.

[0117] 3. Keyword Discovery

[0118] The server analyzes the text data generated by the speech recognition to detect specific keywords, such as "money," "transfer," "accident," and "help," using regular expressions and keyword matching algorithms.

[0119] 4. Identifying the source

[0120] The server analyzes the call origination information, using location analysis tools such as GeoIP databases to identify suspicious locations (e.g., domestic or international pay phones, or calls from abroad).

[0121] 5. AI-powered fraud assessment

[0122] The server inputs the voice data, detected keywords, and location information into an AI model to assess the likelihood of fraud. The AI ​​model was built using TensorFlow and PyTorch and is trained on past data and known fraud patterns.

[0123] 6. Generating Warning Messages

[0124] If the server assesses a high probability of fraud, it will automatically generate a warning message detailing the suspected fraud and recommending actions to take.

[0125] 7. Sending Alerts

[0126] The server then sends the generated warning message to the user's device via SMS, in-app notifications, etc.

[0127] Terminal configuration

[0128] 1. Collection of audio data

[0129] The device collects voice data during a call in real time and sends it to a server, where it is digitally encoded and transferred to the server using a protocol.

[0130] 2. Receiving a warning

[0131] The device receives the warning message from the server and notifies the user in the form of a pop-up message, a ringtone, or vibration.

[0132] User-side configuration

[0133] 1. Check the contents of the call

[0134] The user will be able to talk as if they were in a normal conversation. There is no need for them to be particularly conscious of it, but they will be able to talk naturally.

[0135] 2. Check the warning

[0136] After the call ends, the user checks the warning message that is sent to the terminal. The warning message contains information about the suspected fraud and specific countermeasures.

[0137] Specific examples

[0138] The device initiates a call between elderly person C and the person in charge. The device sends the voice data to the server in real time. The server converts the voice to text and detects the keyword "money." The server analyzes the call's origin information and determines that the call originated from a domestic public phone. The server inputs this information into an AI model and rates the possibility of fraud highly. After the call ends, the server automatically generates a warning message and sends it to the device. The device notifies C of the warning message, allowing him to immediately pay attention and take the necessary action.

[0139] In this way, the present invention can automate a series of processes from collecting voice data to evaluating the possibility of fraud and issuing a warning, thereby minimizing the risk of users becoming victims of fraud.

[0140] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0141] Step 1: Real-time collection and transmission of audio data

[0142] The device collects voice data during a call in real time, converts it into a digital voice data format (e.g., PCM format), compresses it using an AudioCodec, and transmits the data to the server using HTTP or WebSocket protocols.

[0143] Input: Audio during a call

[0144] Data processing: audio data capture and compression

[0145] Output: Digital audio data

[0146] Specific operation: A user starts a call on their smartphone, and the audio is collected by the device's microphone, compressed, and then sent to the server.

[0147] Step 2: Receiving and Preprocessing Audio Data

[0148] The server analyzes the received voice data in real time. First, it performs noise reduction and echo cancellation to improve the sound quality. Then, it converts the voice data into text using voice recognition software such as Google Cloud Speech-to-Text or IBM Watson.

[0149] Input: Digital audio data (sent to the server)

[0150] Data processing: noise reduction, echo cancellation, voice recognition

[0151] Output: Text data

[0152] Specific operation: The server calls the Google Cloud Speech-to-Text API and converts the spoken voice data of "Help, please help me" into the text "Help, please help me."

[0153] Step 3: Keyword Discovery

[0154] The server analyzes the text data generated by speech recognition to detect specific keywords (e.g., "money," "transfer," "accident," "help," etc.) using regular expressions and keyword matching algorithms.

[0155] Input: Text data converted by speech recognition

[0156] Data processing: Keyword detection using regular expressions and keyword matching

[0157] Output: A list of detected keywords

[0158] Specific operation: The server detects the keywords "money" and "transfer" from the text data "Please transfer money."

[0159] Step 4: Identify the source

[0160] The server analyzes the call origination information, using location analysis tools such as GeoIP databases to identify suspicious locations (e.g., international locations, pay phones).

[0161] Input: Data about the call's origin (IP address and GPS data)

[0162] Data processing: location information analysis

[0163] Output: Origination information

[0164] What it does: The server analyzes the IP address and determines that the caller is calling from overseas.

[0165] Step 5: AI fraud assessment

[0166] The server inputs the voice data, detected keywords, and location information into an AI model to assess the likelihood of fraud. The AI ​​model was built using TensorFlow and PyTorch and is trained on past data and known fraud patterns.

[0167] Input: Voice data, keywords, location information

[0168] Data manipulation: Assessing the likelihood of fraud with AI models

[0169] Output: Fraud probability assessment result

[0170] Specific operation: The server inputs the information "transfer," "help," and the origin of the call into the AI ​​model, and assesses the possibility of fraud.

[0171] Step 6: Generate a warning message

[0172] If the server assesses a high probability of fraud, it will automatically generate a warning message detailing the suspected fraud and recommending actions to take.

[0173] Input: Fraud probability assessment result

[0174] Data processing: Generate warning messages

[0175] Output: Warning message

[0176] Specific behavior: The server generates a warning message saying, "This call may be fraudulent. Please hang up immediately and report it to the police."

[0177] Step 7: Sending and notifying warning messages

[0178] The server sends the generated warning message to the user's device via SMS, in-app notification, etc. The device then notifies the user of the warning message received from the server via a pop-up message, ringtone, or vibration.

[0179] Input: warning message

[0180] Data Processing: Sending a warning message

[0181] Output: User notification

[0182] Specific operation: The server sends a warning message to the device as an SMS, and the device rings and notifies the user, "This is a potentially fraudulent call. Please stop the call and exercise caution."

[0183] Step 8: Review user warnings

[0184] After the call ends, the user checks the warning message displayed on the device, which includes information about the suspected fraud and specific countermeasures.

[0185] Input: warning message

[0186] Data processing: Check the warning message and take corrective action

[0187] Output: Response to suspected fraudulent calls

[0188] Specific actions: The user checks the warning message displayed on the device and then reports the incident to the police.

[0189] (Application example 1)

[0190] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0191] In recent years, fraudsters targeting elderly people and those prone to fraud have become increasingly sophisticated, resulting in the increasing incidence of frauds attempting to defraud users of money over the phone. To prevent such frauds, a system is needed that can detect possible fraud in real time during a call and quickly issue a warning to the user. However, current systems lack the functionality to instantly analyze and evaluate the content of calls and notify the user in real time. This creates a time lag between fraud detection and notification, making it difficult to prevent frauds from occurring.

[0192] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0193] In this invention, the server includes a voice recognition means for converting voice data into text data, an analysis means for analyzing the text data and detecting specific keywords, a call origination identification means for analyzing the call origination information, an AI model means for evaluating the possibility of fraud using the voice data, keywords, and call origination information, a warning generation means for generating a warning message based on the result of the fraud possibility evaluation and sending it to the terminal, and a terminal means for notifying the user of the generated warning message and displaying the contents of the warning message to the user in real time. This makes it possible to analyze voice data during a call in real time and immediately notify the user if there is a high possibility of fraud. This is expected to have the effect of preventing fraud damage before it occurs.

[0194] "Voice data" refers to electrical signals of voice obtained during a call, and is converted into text data by a voice recognition means.

[0195] "Terminal means" refers to a smartphone or other communication terminal used by a user, which is a device that collects voice data and transmits it to a server.

[0196] A "server" is a computer system that receives, analyzes, and evaluates voice data transmitted from a terminal means via a network.

[0197] "Speech recognition means" refers to a process or device that has the function of converting received voice data into text data, and performs noise reduction and echo cancellation processing.

[0198] The "analysis means" is a process or device for analyzing the converted text data and detecting specific keywords.

[0199] A "keyword" is a specific linguistic unit that suggests the possibility of fraud, such as "money," "transfer," "accident," or "help."

[0200] A "call origin determination means" is a process or device that analyzes the call origin information and identifies suspicious locations (e.g., calls from public phones or overseas).

[0201] An "AI model means" is a process or device that implements an artificial intelligence model to assess the likelihood of fraud using voice data, keywords, and origin information.

[0202] An "alert generator" is a process or device that generates an alert message based on the result of assessing the likelihood of fraud and sends it to the terminal.

[0203] A "warning message" is information intended to notify the user of a possible fraud and includes recommended actions to be taken.

[0204] "Real-time notification" is the process or function of immediately communicating generated alert messages to the user.

[0205] The system of the present invention has the function of detecting possible fraud during a call in real time and issuing a prompt warning to the user. This system is mainly composed of server-side processing and terminal-side processing.

[0206] Server-side configuration

[0207] 1. Receiving audio data

[0208] The server receives the voice data sent from the terminal in real time. The voice data is collected during the call.

[0209] 2. Voice Recognition

[0210] The server analyzes the received voice data using speech recognition and converts it into text data, using services such as the Google Cloud Speech-to-Text API, and also applies noise reduction and echo cancellation.

[0211] 3. Keyword Discovery

[0212] The converted text data is analyzed using analytical tools to detect specific keywords such as "money," "transfer," "accident," and "help." Python and SpaCy can be used for the analysis.

[0213] 4. Identifying the source

[0214] The server analyzes the call origin information using GeoIP, and if it detects a suspicious location (such as a public phone or a call from abroad), it records that information.

[0215] 5. AI-powered fraud assessment

[0216] The server inputs the voice data, detected keywords, and location information into an AI model using TensorFlow and PyTorch to assess the likelihood of fraud, with reference to past data and known fraud patterns.

[0217] 6. Generating and Sending Warning Messages

[0218] If the server judges that there is a high possibility of fraud, it generates a warning message by a warning generation means and sends it to the terminal. The warning message includes information about the suspected fraud and measures to be taken.

[0219] Terminal configuration

[0220] 1. Collection of audio data

[0221] The terminal has the ability to collect voice data during a call in real time and send it to a server.

[0222] 2. Receiving and Displaying Alerts

[0223] The terminal receives the warning message sent from the server and immediately notifies the user of the message by voice, message notification, or real-time display on the terminal screen.

[0224] User-side configuration

[0225] 1. Check the contents of the call

[0226] The user does not need to perform any special operations during the call, and the call proceeds as normal. The call content is automatically analyzed.

[0227] 2. Checking and responding to warnings

[0228] When a warning message is sent to the terminal, the user can check it and take appropriate measures.

[0229] Specific examples

[0230] For example, if an elderly person named A receives a phone call that they suspect is a scam, the following steps would be taken:

[0231] 1. Mr. A's smartphone sends the audio during the call to the server.

[0232] 2. The server converts the speech into text and detects keywords such as "money" and "transfer."

[0233] 3. The call origin is identified as a pay phone.

[0234] 4. AI models will assess the likelihood of fraud.

[0235] 5. A warning message will be sent to Mr. A's smartphone immediately during or after the call ends.

[0236] 6. Person A is able to accept the warning, pay immediate attention, and take the necessary action.

[0237] Prompt Sentence Examples

[0238] text

[0239] Audio files:<audio_file>

[0240] Origin: <location>

[0241] Location information:<location_info>

[0242] Detection Keywords:<keywords_detected>

[0243] Scam rating score:<fraud_score>

[0244] Warning message:<alarm_message>

[0245] In this way, the system of the present invention automatically performs a series of processes from collecting voice data to assessing the possibility of fraud and issuing a warning, thereby minimizing the risk of users becoming victims of fraud.

[0246] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0247] Step 1:

[0248] Audio data collection

[0249] The device collects voice data in real time during a call. This voice data is picked up by the device's microphone from the start to the end of the call. The collected voice data is temporarily stored in the device's memory as is.

[0250] Input: Voice data during a call

[0251] Output: Collected audio data

[0252] Step 2:

[0253] Sending audio data

[0254] The terminal sends the collected voice data to the server. The communication protocol is secure HTTP (HTTPS). The data sent includes voice data and origin information (e.g., location information obtained from GPS).

[0255] Input: Collected voice data, location information

[0256] Output: Sending voice data and location information to the server

[0257] Step 3:

[0258] Receiving audio data

[0259] The server receives the voice data sent from the device in real time and temporarily stores the received data in the server's storage.

[0260] Input: Voice data sent from the device, location information

[0261] Output: Audio data and location information stored on the server

[0262] Step 4:

[0263] Voice Recognition

[0264] The server converts the stored audio data into text using the Google Cloud Speech-to-Text API, applying noise reduction and echo cancellation. The converted text data is then passed on to the next analysis step.

[0265] Input: Audio data

[0266] Output: Text data

[0267] Step 5:

[0268] Keyword detection

[0269] The server analyzes the text data using Python and SpaCy to detect specific keywords (e.g., money, transfer, accident, help). The detected keywords are used in the next evaluation step.

[0270] Input: Text data

[0271] Output: Detected keywords

[0272] Step 6:

[0273] Identifying the source

[0274] The server analyzes the origin information using GeoIP to identify suspicious locations (e.g., payphones, overseas). If a location is suspicious, the identified location information is used in the next evaluation step.

[0275] Input: Location information

[0276] Output: Parsed origin information

[0277] Step 7:

[0278] AI-powered fraud assessment

[0279] The server uses the voice data, detected keywords, and location information to evaluate the likelihood of fraud using an AI model (using TensorFlow and PyTorch).The AI ​​model outputs a score indicating the likelihood of fraud based on the preprocessed data.

[0280] Input: Voice data, detected keywords, location information

[0281] Output: Fraud rating score

[0282] Step 8:

[0283] Generate and send warning messages

[0284] The server generates a warning message based on the fraud assessment score, including a warning that the fraud is suspected and specific countermeasures, and sends the generated warning message to the terminal in real time.

[0285] Input: Fraud rating score

[0286] Output: Warning message

[0287] Step 9:

[0288] Receiving and Viewing Alerts

[0289] The terminal receives the warning message sent from the server and immediately notifies the user by voice, message notification, or real-time display on the terminal screen.

[0290] Input: warning message

[0291] Output: User notification

[0292] For example, here is a prompt:

[0293] For example, "Audio file:<audio_file> Origin: <location>Location information:<location_info> Detection Keywords:<keywords_detected> Scam rating score:<fraud_score> Warning message:<alarm_message> " is generated.

[0294] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0295] The present invention provides a system that detects "I'm your son" fraud calls in real time and issues a prompt warning, targeting elderly users and those prone to fraud. This system is an integrated system that includes voice data collection, voice analysis, keyword detection, caller location identification, fraud probability assessment using an AI model, and warning generation and notification. One embodiment of the present invention further includes an emotion engine for recognizing user emotions.

[0296] System Configuration

[0297] Server-side configuration

[0298] 1. Receiving audio data

[0299] The server receives the voice data transmitted from the terminal during the call in real time.

[0300] 2. Voice Recognition

[0301] The server uses a speech recognition module to convert the voice data into text data, which also includes noise reduction and echo cancellation.

[0302] 3. Keyword Discovery

[0303] The server analyzes the text data generated by the speech recognition to detect specific keywords, such as "money," "transfer," "accident," and "help."

[0304] 4. Identifying the source

[0305] The server analyzes the call origin information, using phone numbers, IP addresses, and GPS information to identify the origin, and if a suspicious location is detected, it records that information.

[0306] 5. Emotion Recognition by Emotion Engine

[0307] The server then inputs the voice data into an emotion engine to recognize the user's emotions. The emotion engine analyzes the tone, pitch, and tempo of the user's voice to determine emotions such as tension or anxiety.

[0308] 6. AI-powered fraud assessment

[0309] The server inputs the voice data, text data, detected keywords, location information, and emotional information recognized by the emotion engine into an AI model to assess the likelihood of fraud, with reference to past data and known fraud patterns.

[0310] 7. Generating Warning Messages

[0311] If the AI ​​model judges the possibility of fraud high, the server automatically generates a warning message after the call ends, which includes information about the suspected fraud and specific countermeasures.

[0312] 8. Sending Alerts

[0313] The server sends the generated warning message to the user's terminal.

[0314] Terminal configuration

[0315] 1. Collection of audio data

[0316] The device collects voice data during a call in real time and sends it to the server.

[0317] 2. Receiving a warning

[0318] When a warning message is sent from the server, the device notifies the user via voice call or message notification.

[0319] User-side configuration

[0320] 1. Check the contents of the call

[0321] The user will carry out the call as if it were a normal call.

[0322] 2. Check the warning

[0323] After the call ends, the user checks the warning message displayed on their device, which includes information about a suspected scam and specific countermeasures.

[0324] Specific examples

[0325] 1. The device initiates a call with elderly person D and his daughter.

[0326] 2. The device transmits the voice data to the server in real time.

[0327] 3. The server converts the speech into text and detects the keyword "money."

[0328] 4. The server analyzes the call's location information and determines that the call came from a public phone.

[0329] 5. The server inputs the voice data into the emotion engine and recognizes that Mr. D is feeling nervous or anxious.

[0330] 6. The server inputs this information into an AI model and rates the likelihood of fraud.

[0331] 7. After the call ends, the server automatically generates a warning message and sends it to the terminal.

[0332] 8. The device will send a warning message to Mr. D, who can then take immediate action.

[0333] In this way, the present invention automates a series of processes from collecting voice data to recognizing emotional information, assessing the possibility of fraud, and issuing a warning, thereby minimizing the risk of users becoming victims of fraud.

[0334] The processing flow will be explained below.

[0335] Step 1:

[0336] When a call is initiated, the device collects voice data in real time, filters the data during the call, and sends it to the server.

[0337] Step 2:

[0338] The server receives the voice data sent from the terminal in real time and inputs it into a voice recognition module for analysis.

[0339] Step 3:

[0340] The server converts the voice data into text data using a speech recognition module, which performs pre-processing such as noise reduction and echo cancellation.

[0341] Step 4:

[0342] The server analyzes the text data generated by the voice recognition and detects predetermined keywords, such as "money," "transfer," "accident," and "help." The detected keywords are recorded along with the location information.

[0343] Step 5:

[0344] The server analyzes the call origin information, identifying the origin using phone number, IP address, GPS information, etc., and if it detects any suspicious locations (such as public telephones or calls from abroad), it records that information.

[0345] Step 6:

[0346] The server inputs the voice data into an emotion engine to recognize the user's emotions. The emotion engine analyzes the voice tone, pitch, and tempo to determine the user's emotional state. For example, if the user is feeling nervous or anxious, that information is also recorded.

[0347] Step 7:

[0348] The server inputs voice data, text data, detected keywords, location information, and emotional information into the AI ​​model, which uses this information to assess the likelihood of fraud. It then uses past data and known fraud patterns to generate a score.

[0349] Step 8:

[0350] If the AI ​​model evaluates the likelihood of fraud at a certain level, the server generates a warning message after the call ends, which includes information about the suspected fraud and specific countermeasures.

[0351] Step 9:

[0352] The server sends the generated warning message to the user's terminal, and the sent warning message arrives at the terminal in real time.

[0353] Step 10:

[0354] When the device receives the warning message sent from the server, it notifies the user, which is displayed as a voice call or message notification.

[0355] Step 11:

[0356] The user checks the warning message on the device, understands the contents of the warning message, and takes necessary measures, such as rechecking the contents of calls that are suspected to be fraudulent and consulting the police or financial institution.

[0357] By taking the above steps, the system can quickly and accurately detect the possibility of "It's me" fraud and issue an immediate warning to the user, thus preventing fraud damage before it occurs.

[0358] Example 2

[0359] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0360] In modern society, elderly people and certain user groups are increasingly becoming victims of fraud such as "it's me" scams. These frauds are becoming more sophisticated and are often difficult to detect using normal means. As a result, victims are likely to suffer significant financial losses. To solve this problem, a system is needed that can accurately detect potential fraud in real time and issue a prompt warning.

[0361] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0362] In this invention, the server includes: a voice recognition means for converting voice data into text data; an analysis means for analyzing the text data and detecting specific keywords; a call origin identification means for analyzing call origin information; an AI model means for evaluating the possibility of fraud using the voice data, keywords, and call origin information; a warning generation means for generating a warning message based on the result of the fraud possibility evaluation and sending it to the terminal; a terminal means for notifying the user of the generated warning message; an emotion recognition means for analyzing the voice data and recognizing the user's emotion; and an AI model means for inputting the emotion information into the AI ​​model and evaluating the possibility of fraud with high accuracy. This makes it possible to minimize the risk of users becoming victims of fraud.

[0363] "Audio data" means data in digital form obtained by recording or transmitting sound.

[0364] A "terminal means" is an electronic device for collecting and transmitting voice data.

[0365] A "server" is a central processing unit that receives, processes, and stores data from multiple terminals over a network.

[0366] "Speech recognition means" is a technology that analyzes voice data and converts the content into text data.

[0367] The "analysis means" is a device or software that has the function of analyzing text data and detecting specific keywords.

[0368] The "originating location identification means" is a technology that analyzes the originating location information of a call and identifies the area where the call was made.

[0369] An "AI modeling tool" is a device or software that uses artificial intelligence to assess the likelihood of fraud.

[0370] A "warning generator" is a device or software capable of generating a warning message based on the result of assessing the likelihood of fraud.

[0371] "Emotion recognition means" is a technology that analyzes voice data to determine the user's emotions.

[0372] A "warning message" is a message that notifies the user in case of possible fraud.

[0373] "User" means an individual or organization that uses the system.

[0374] This invention provides a system that detects "It's my son" fraud calls in real time and issues a prompt warning, targeting elderly people and other users who are prone to fraud. This system can automatically carry out a series of processes from collecting voice data to issuing a warning.

[0375] System Components

[0376] Server-side configuration

[0377] Receiving audio data:

[0378] The server receives the voice data sent from the device in real time during the call. Specifically, a WebSocket connection is used to exchange data bidirectionally between the device and the server.

[0379] Voice Recognition:

[0380] The server uses the Google Speech-to-Text API or a similar service to convert the received voice data into text data, and during this process, it also performs noise reduction and echo cancellation to generate highly accurate text data.

[0381] Keyword detection:

[0382] The server performs text analysis to detect specific keywords (e.g., "money," "transfer," "accident," "help," etc.) using regular expressions and pre-defined keyword lists.

[0383] Identifying the source:

[0384] The server analyzes the call origin information, using phone numbers, IP addresses, GPS information, etc. to detect calls from suspicious locations such as public phones.

[0385] Emotion recognition:

[0386] The server inputs the voice data into an emotion engine (such as Affectiva's emotion recognition API) that analyzes the tone, pitch, and tempo of the user's voice to determine their emotion. Based on this information, it can determine whether the user is feeling nervous or anxious.

[0387] AI Fraud Assessment:

[0388] The server inputs voice data, text data, keywords, source information, and emotional information into an AI model to assess the likelihood of fraud, which also refers to past data and known fraud patterns.

[0389] Generates a warning message:

[0390] If the AI ​​model judges the likelihood of fraud high, the server automatically generates a warning message containing the suspected fraud and specific countermeasures.

[0391] Sending alerts:

[0392] The server generates a warning message and sends it to the user's device. The message is promptly sent to the user via push notification or SMS.

[0393] Terminal configuration

[0394] Audio data collection:

[0395] The device collects voice data during a call in real time and immediately sends it to the server. It is set to start collecting voice data as soon as the call starts.

[0396] Receiving an alert:

[0397] The device receives a warning message from the server and notifies the user. Notification methods include voice calls and message notifications, and the user is alerted in a way that can be checked immediately.

[0398] User-side configuration

[0399] Check the call details:

[0400] Users make calls as usual, the system runs in the background and no special action is required.

[0401] Check for warnings:

[0402] After the call ends, the user can view the warning message on their device and take appropriate action. The message will include information about the suspected fraud and specific countermeasures.

[0403] Specific examples

[0404] If an elderly person named Mr. D is told by someone claiming to be his son that he needs money, the system works as follows:

[0405] 1. The device collects the conversation between Mr. D and the other party and sends the voice data to the server.

[0406] 2. The server uses a speech recognition module to convert this speech into text and detect the keyword "money."

[0407] 3. The server uses the call location identification means to identify that the call is coming from a public telephone.

[0408] 4. The server inputs the voice data into an emotion recognition engine and recognizes that Mr. D is feeling nervous or anxious.

[0409] 5. The server inputs this information into an AI model and scores the likelihood of fraud.

[0410] 6. After the call ends, the server generates a warning message and sends it to the terminal.

[0411] 7. The device will send a warning message to Mr. D, who can then take immediate action.

[0412] This system automates the entire process from collecting voice data to recognizing emotional information, assessing the likelihood of fraud, and issuing warning notifications, thereby minimizing the risk of users becoming victims of fraud.

[0413] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0414] Step 1:

[0415] Collection and transmission of voice data

[0416] The device collects audio data during the call in real time. Specifically, it uses the device's microphone to capture the audio of the call and sends the audio data to the server via WebSocket or HTTP. The input of this step is the audio data collected from the device's microphone, and the output is the audio data sent to the server.

[0417] Step 2:

[0418] Receiving audio data

[0419] The server receives the audio data sent from the device in real time. Specifically, it starts a WebSocket server to receive the audio data. The input of this step is the audio data sent from the device, and the output is the audio data stored in a temporary buffer on the server.

[0420] Step 3:

[0421] Voice Recognition

[0422] The server converts the received voice data into text data using a speech recognition module such as the Google Speech-to-Text API. During this process, noise reduction and echo cancellation are also performed to improve analysis accuracy. The input for this step is the voice data stored on the server, and the output is the result of converting the voice data into text data.

[0423] Step 4:

[0424] Keyword detection

[0425] The server analyzes the generated text data and detects specific keywords (e.g., "money," "transfer," "accident," "help," etc.). Specifically, the analysis is performed using regular expressions or a pre-defined keyword list. The input of this step is the text data, and the output is a list of detected keywords.

[0426] Step 5:

[0427] Identifying the source

[0428] The server analyzes the call's origin information. Specifically, it identifies the call's origin using the phone number, IP address, GPS information, etc. The input to this step is the metadata associated with the call, and the output is the origin information. If the call originates from a suspicious location, the location information is also included.

[0429] Step 6:

[0430] emotion recognition

[0431] The server inputs the voice data into an emotion recognition engine (e.g., Affectiva's emotion recognition API) and analyzes the tone, pitch, and tempo of the user's voice to determine their emotion. The input of this step is the voice data, and the output is the user's emotional information (e.g., tension, fear, relief, etc.).

[0432] Step 7:

[0433] AI-powered fraud assessment

[0434] The server inputs the voice data, text data, detected keywords, origin information, and emotion information into the AI ​​model to assess the likelihood of fraud. The input for this step is all of this data, and the output is a fraud risk assessment score. The AI ​​model also references known fraud patterns and past data to make the assessment.

[0435] Step 8:

[0436] Generate a warning message

[0437] If the AI ​​model evaluates the likelihood of fraud highly, the server automatically generates a warning message. This message includes the suspected fraud and specific countermeasures, and is provided to the user. The input of this step is the fraud risk assessment score, and the output is the generated warning message.

[0438] Step 9:

[0439] Sending an Alert

[0440] The server sends the generated warning message to the user's device. Specifically, it ensures that the warning message reaches the user promptly via push notification or SMS. The input of this step is the warning message, and the output is the warning message sent to the user's device.

[0441] Step 10:

[0442] Receiving and Acknowledging Alerts

[0443] The terminal receives the warning message from the server and notifies the user. Specifically, it notifies the user of the warning using a notification bar, a pop-up, or a sound notification. The input of this step is the warning message sent from the server, and the output is the warning message displayed in a format that can be confirmed by the user.

[0444] Step 11:

[0445] User response

[0446] The user checks the warning message sent to the device and takes appropriate action, such as reading the warning content, reporting it to the appropriate authorities, or ending the call. The input to this step is the warning message, and the output is the user's specific action.

[0447] Through these steps, the system can efficiently collect, analyze, evaluate, and warn users of voice data, minimizing the risk of fraud.

[0448] (Application example 2)

[0449] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0450] In recent years, there has been an increase in "I'm your son" frauds targeting elderly people and other users who are prone to fraud, creating a need for rapid and effective countermeasures. However, conventional systems have had difficulty detecting potential fraud in real time during voice calls and issuing prompt warnings to users. In addition, evaluations based solely on voice data have the problem of low detection accuracy and a high number of false positives.

[0451] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a voice recognition means for converting voice data into text data, an analysis means for analyzing the text data and detecting specific keywords, a call origin identification means for analyzing the call origin information, and an AI model means for evaluating the possibility of fraud using the voice data, keywords, call origin information, and user emotion data. This makes it possible to prevent elderly users and users who are prone to fraud from becoming victims of fraud.

[0452] The "terminal means" is a device that has the function of collecting voice data and transmitting it to a server.

[0453] A "voice recognition means" is a device or system that has the function of converting voice data into text data.

[0454] The "analysis means" is a system that has the function of analyzing text data and detecting specific keywords.

[0455] The "originating point identification means" is a system that has the function of analyzing the originating point information of a call and identifying the originating point.

[0456] The "AI model means" is a system that has the function of assessing the possibility of fraud using voice data, keywords, origin information, and user emotional data.

[0457] The "warning generation means" is a system that has the function of generating a warning message based on the result of assessing the possibility of fraud and sending it to the terminal.

[0458] "Emotion data" is information that represents the user's emotional state and is obtained by analyzing voice data.

[0459] This invention provides a system that detects fraudulent calls in real time and issues prompt warnings to elderly users and other users who are prone to fraud. This system is an integrated system that includes voice data collection, voice analysis, keyword detection, caller location identification, fraud probability assessment using an AI model, and warning generation and notification.

[0460] First, the terminal collects voice data and sends it to the server. The server converts the voice data into text data using a voice recognition means. The voice recognition means includes noise reduction and echo cancellation functions. Next, the server analyzes the text data to detect specific keywords. This analysis means detects keywords related to fraud (e.g., "money," "transfer," "accident," "help," etc.).

[0461] Furthermore, the server analyzes the call origination information and identifies suspicious locations using a call origination location identification means. The server also inputs the voice data into an emotion recognition engine to analyze the user's emotion data. This emotion data identifies emotions such as tension or anxiety based on the user's voice tone, pitch, tempo, etc.

[0462] This data (voice data, keywords, origin information, and emotion data) is input into the AI ​​model means to evaluate the likelihood of fraud. The AI ​​model means makes the evaluation by referring to past data and known fraud patterns. If it is evaluated that there is a high likelihood of fraud, the server generates a warning message and sends it to the terminal using the warning generation means. The terminal notifies the user of the generated warning message. This notification may take the form of a voice call or a message notification.

[0463] As a concrete example, consider a situation in which a device initiates a call between an elderly user and someone else. At this time, the device sends voice data to a server in real time. The server converts the voice to text and detects the keyword "money." The server then analyzes the call's origin information and determines that it is coming from a public phone. The server inputs the voice data into an emotion engine and recognizes that the user is feeling nervous or anxious. The server then inputs this information into an AI model and evaluates the likelihood of fraud. After the call ends, the server automatically generates a warning message and sends it to the device. The device notifies the user of the warning message, allowing them to take immediate action.

[0464] The server also handles notifications to notify the generated warning messages. The following is an example of a specific prompt to the AI ​​model:

[0465] "The following keywords were detected in this call: 'money', 'transfer'. The call originated from an unknown pay phone. The user's emotions are very tense. What is the likelihood of this being a scam?"

[0466] The present invention automates this series of processes, minimizing the risk of users falling victim to fraud.

[0467] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0468] Step 1:

[0469] The terminal collects voice data during a call. The collected voice data is sent to the server in real time. This allows the server to receive the user's call data in real time. The input is the call voice data, and the output is the voice data sent to the server.

[0470] Step 2:

[0471] The server converts the received voice data into text data using a voice recognition means. During this process, noise reduction and echo cancellation are performed to improve the accuracy of the voice data. The input is voice data, and the output is text data.

[0472] Step 3:

[0473] The server analyzes the text data generated by the speech recognition to detect specific keywords. Using the analysis means, it detects fraud-related keywords such as "money," "transfer," "accident," and "help." The input is the text data, and the output is a list of detected keywords.

[0474] Step 4:

[0475] The server analyzes the call origination information and uses the call origination identification means to identify suspicious locations by analyzing the phone number, IP address, and GPS information. The input is the call origination information, and the output is the analyzed call origination information.

[0476] Step 5:

[0477] The server inputs the voice data into an emotion recognition engine and analyzes the user's emotional data. The emotion recognition engine analyzes the tone, pitch, and tempo of the user's voice to determine emotions such as tension or anxiety. The input is voice data, and the output is the user's emotional data.

[0478] Step 6:

[0479] The server inputs the voice data, keywords, origin information, and emotion data into the AI ​​model means to evaluate the likelihood of fraud. The AI ​​model means makes an evaluation by referring to past data and known fraud patterns. The inputs are the voice data, keywords, origin information, and emotion data, and the output is the result of the evaluation of the likelihood of fraud.

[0480] Step 7:

[0481] If the server judges the possibility of fraud to be high, it generates a warning message using the warning generation means and sends it to the terminal. The generated warning message includes information about the suspected fraud and specific countermeasures. The input is the result of the fraud possibility assessment, and the output is the warning message.

[0482] Step 8:

[0483] The terminal receives the warning message sent from the server and notifies the user. The notification is in the form of a voice call or a message notification. The input is the warning message, and the output is the notification to the user.

[0484] Through the above series of processing steps, users can detect fraudulent calls in real time and take prompt action.

[0485] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0486] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0487] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0488] [Second embodiment]

[0489] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0490] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0491] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0492] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0493] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0494] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0495] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0496] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0497] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0498] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0499] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0500] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0501] This invention provides a system that detects "I am your son" fraud calls in real time and issues a prompt warning, targeting elderly people and other users who are prone to fraud. This system is an integrated system that includes voice data collection, voice analysis, keyword detection, caller location identification, fraud probability assessment using an AI model, and warning generation and notification.

[0502] System Configuration

[0503] Server-side configuration

[0504] 1. Receiving audio data

[0505] The server receives real-time voice data during a call, which is sent from the terminal.

[0506] 2. Voice Recognition

[0507] The server analyzes the received voice data using a speech recognition module and converts it into text data, with noise reduction and echo cancellation also being applied during this process.

[0508] 3. Keyword Discovery

[0509] The server analyzes the text data generated by the speech recognition to detect specific keywords, such as "money," "transfer," "accident," and "help."

[0510] 4. Identifying the source

[0511] The server analyzes the call origin information and if it detects any suspicious locations (such as public phones or calls from abroad), it records that information.

[0512] 5. AI-powered fraud assessment

[0513] The server inputs the voice data, detected keywords, and location information into an AI model to assess the likelihood of fraud, with reference to past data and known fraud patterns.

[0514] 6. Generating Warning Messages

[0515] If the server assesses a high probability of fraud, it will automatically generate a warning message after the call ends, including information about the suspected fraud and what steps should be taken.

[0516] 7. Sending Alerts

[0517] The server sends the generated warning message to the user's terminal.

[0518] Terminal configuration

[0519] 1. Collection of audio data

[0520] The device collects voice data during a call in real time and sends it to the server.

[0521] 2. Receiving a warning

[0522] When a warning message is sent from the server, the device notifies the user via voice call or message notification.

[0523] User-side configuration

[0524] 1. Check the contents of the call

[0525] The user will carry out the call as if it were a normal call.

[0526] 2. Check the warning

[0527] After the call ends, the user checks the warning message displayed on the device, which includes information about the suspected fraud and specific countermeasures.

[0528] Specific examples

[0529] 1. The device starts a call between elderly person C and the person in charge.

[0530] 2. The device transmits the voice data to the server in real time.

[0531] 3. The server converts the speech into text and detects the keyword "money."

[0532] 4. The server analyzes the call's location information and determines that the call originated from a domestic public telephone.

[0533] 5. The server inputs this information into an AI model and rates the likelihood of fraud.

[0534] 6. After the call ends, the server automatically generates a warning message and sends it to the terminal.

[0535] 7. The device will notify Mr. C of the warning message, so that he can immediately pay attention and take the necessary action.

[0536] In this way, the present invention can automate a series of processes from collecting voice data to evaluating the possibility of fraud and issuing a warning, thereby minimizing the risk of users becoming victims of fraud.

[0537] The processing flow will be explained below.

[0538] Step 1:

[0539] When a call is initiated, the device collects voice data in real time and transmits the voice data during the call to the server as is.

[0540] Step 2:

[0541] The server receives the voice data sent from the terminal and inputs it into a voice recognition module for real-time processing.

[0542] Step 3:

[0543] The server uses a speech recognition module to convert the voice data into text data, and pre-processing such as noise reduction and echo cancellation is performed during this process.

[0544] Step 4:

[0545] The server analyzes the text data and detects certain keywords, such as "money," "transfer," "accident," and "help."

[0546] Step 5:

[0547] The server analyzes the call origin information, identifying the origin using phone number, IP address, GPS information, etc., and records any suspicious locations (such as public telephones or calls from overseas).

[0548] Step 6:

[0549] The server inputs the voice data, text data, detected keywords, and location information into the AI ​​model, which then evaluates the likelihood of fraud based on this information.

[0550] Step 7:

[0551] If the server determines that there is a high possibility of fraud based on the fraud probability score evaluated by the AI ​​model, it will generate a warning message after the call ends, notifying the caller of the suspected fraud and providing specific countermeasures.

[0552] Step 8:

[0553] The server sends the generated warning message to the user's terminal, and the message arrives at the terminal in real time.

[0554] Step 9:

[0555] When the device receives the warning message sent from the server, it notifies the user, which is displayed as a voice call or a message.

[0556] Step 10:

[0557] The user checks the warning message on the device, understands the contents of the warning message, and takes necessary measures, such as rechecking the contents of calls that are suspected to be fraudulent and consulting the police or financial institution.

[0558] Through these steps, the system can quickly and accurately detect possible telephone fraud and issue an immediate warning to the user.

[0559] Example 1

[0560] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0561] There is a challenge in detecting "I'm your son" fraud calls in real time and issuing prompt warnings, targeting elderly people and other users who are prone to fraud. Conventional systems have difficulty immediately detecting potential fraud and issuing warnings, putting users at risk of becoming victims. In particular, since calls from suspicious locations and specific keywords are often not detected, there is a need for a means to prevent fraud in advance.

[0562] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0563] In this invention, the server includes terminal means for collecting voice data and transmitting it to the server, voice recognition means for converting the voice data into text data, analysis means for analyzing the text data and detecting specific keywords, call origination location identification means for analyzing call origination location information, voice processing means for performing noise reduction and echo cancellation, AI model means for evaluating the possibility of fraud using the voice data, keywords, and call origination location information, warning generation means for generating a warning message based on the result of the evaluation of the possibility of fraud and transmitting it to the terminal, and terminal means for notifying the user of the generated warning message. This makes it possible to detect fraudulent calls in real time and quickly issue a warning to the user.

[0564] "Voice data" refers to data converted into digital form of voice in a conversation between a user and another person.

[0565] "Terminal means" refers to a device that has the function of collecting voice data and transmitting it to a server.

[0566] "Speech recognition means" refers to software or hardware that has the function of analyzing received voice data and converting it into text data.

[0567] "Text data" refers to character string information generated from voice data by a voice recognition means.

[0568] "Analysis means" refers to software or hardware that has the function of analyzing text data and detecting specific keywords.

[0569] "Call origin identification means" refers to software or hardware that has the function of analyzing call origin information and identifying suspicious locations.

[0570] "Audio processing means" refers to software or hardware that performs noise reduction and echo cancellation and optimizes audio data.

[0571] "AI model means" refers to an artificial intelligence model that uses voice data, keywords, and location information to assess the likelihood of fraud.

[0572] "Warning generation means" refers to software or hardware that has the function of generating a warning message based on the result of assessing the likelihood of fraud and sending it to a terminal.

[0573] "Warning Message" means notification information containing warnings and countermeasures regarding potentially fraudulent calls.

[0574] This invention provides a system that detects "I am your son" fraud calls in real time and issues a prompt warning, targeting elderly people and other users who are prone to fraud. This system is an integrated system that includes voice data collection, voice analysis, keyword detection, caller location identification, fraud probability assessment using an AI model, and warning generation and notification.

[0575] Server-side configuration

[0576] 1. Receiving audio data

[0577] The server receives real-time audio data during a call. This audio data is sent from the device. The device collects audio during the call with a microphone and encodes it digitally using an AudioCodec. The encoded audio data is then sent to the server using HTTP or WebSocket protocols.

[0578] 2. Voice Recognition

[0579] The server analyzes the received voice data and converts it into text data using a speech recognition module. This process uses voice recognition software such as Google Cloud Speech-to-Text or IBM Watson. The server pre-processes the voice data by performing noise reduction and echo cancellation.

[0580] 3. Keyword Discovery

[0581] The server analyzes the text data generated by the speech recognition to detect specific keywords, such as "money," "transfer," "accident," and "help," using regular expressions and keyword matching algorithms.

[0582] 4. Identifying the source

[0583] The server analyzes the call origination information, using location analysis tools such as GeoIP databases to identify suspicious locations (e.g., domestic or international pay phones, or calls from abroad).

[0584] 5. AI-powered fraud assessment

[0585] The server inputs the voice data, detected keywords, and location information into an AI model to assess the likelihood of fraud. The AI ​​model was built using TensorFlow and PyTorch and is trained on past data and known fraud patterns.

[0586] 6. Generating Warning Messages

[0587] If the server assesses a high probability of fraud, it will automatically generate a warning message detailing the suspected fraud and recommending actions to take.

[0588] 7. Sending Alerts

[0589] The server then sends the generated warning message to the user's device via SMS, in-app notifications, etc.

[0590] Terminal configuration

[0591] 1. Collection of audio data

[0592] The device collects voice data during a call in real time and sends it to a server, where it is digitally encoded and transferred to the server using a protocol.

[0593] 2. Receiving a warning

[0594] The device receives the warning message from the server and notifies the user in the form of a pop-up message, a ringtone, or vibration.

[0595] User-side configuration

[0596] 1. Check the contents of the call

[0597] The user will be able to talk as if they were in a normal conversation. There is no need for them to be particularly conscious of it, but they will be able to talk naturally.

[0598] 2. Check the warning

[0599] After the call ends, the user checks the warning message that is sent to the terminal. The warning message contains information about the suspected fraud and specific countermeasures.

[0600] Specific examples

[0601] The device initiates a call between elderly person C and the person in charge. The device sends the voice data to the server in real time. The server converts the voice to text and detects the keyword "money." The server analyzes the call's origin information and determines that the call originated from a domestic public phone. The server inputs this information into an AI model and rates the possibility of fraud highly. After the call ends, the server automatically generates a warning message and sends it to the device. The device notifies C of the warning message, allowing him to immediately pay attention and take the necessary action.

[0602] In this way, the present invention can automate a series of processes from collecting voice data to evaluating the possibility of fraud and issuing a warning, thereby minimizing the risk of users becoming victims of fraud.

[0603] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0604] Step 1: Real-time collection and transmission of audio data

[0605] The device collects voice data during a call in real time, converts it into a digital voice data format (e.g., PCM format), compresses it using an AudioCodec, and transmits the data to the server using HTTP or WebSocket protocols.

[0606] Input: Audio during a call

[0607] Data processing: audio data capture and compression

[0608] Output: Digital audio data

[0609] Specific operation: A user starts a call on their smartphone, and the audio is collected by the device's microphone, compressed, and then sent to the server.

[0610] Step 2: Receiving and Preprocessing Audio Data

[0611] The server analyzes the received voice data in real time. First, it performs noise reduction and echo cancellation to improve the sound quality. Then, it converts the voice data into text using voice recognition software such as Google Cloud Speech-to-Text or IBM Watson.

[0612] Input: Digital audio data (sent to the server)

[0613] Data processing: noise reduction, echo cancellation, voice recognition

[0614] Output: Text data

[0615] Specific operation: The server calls the Google Cloud Speech-to-Text API and converts the spoken voice data of "Help, please help me" into the text "Help, please help me."

[0616] Step 3: Keyword Discovery

[0617] The server analyzes the text data generated by speech recognition to detect specific keywords (e.g., "money," "transfer," "accident," "help," etc.) using regular expressions and keyword matching algorithms.

[0618] Input: Text data converted by speech recognition

[0619] Data processing: Keyword detection using regular expressions and keyword matching

[0620] Output: A list of detected keywords

[0621] Specific operation: The server detects the keywords "money" and "transfer" from the text data "Please transfer money."

[0622] Step 4: Identify the source

[0623] The server analyzes the call origination information, using location analysis tools such as GeoIP databases to identify suspicious locations (e.g., international locations, pay phones).

[0624] Input: Data about the call's origin (IP address and GPS data)

[0625] Data processing: location information analysis

[0626] Output: Origination information

[0627] What it does: The server analyzes the IP address and determines that the caller is calling from overseas.

[0628] Step 5: AI fraud assessment

[0629] The server inputs the voice data, detected keywords, and location information into an AI model to assess the likelihood of fraud. The AI ​​model was built using TensorFlow and PyTorch and is trained on past data and known fraud patterns.

[0630] Input: Voice data, keywords, location information

[0631] Data manipulation: Assessing the likelihood of fraud with AI models

[0632] Output: Fraud probability assessment result

[0633] Specific operation: The server inputs the information "transfer," "help," and the origin of the call into the AI ​​model, and assesses the possibility of fraud.

[0634] Step 6: Generate a warning message

[0635] If the server assesses a high probability of fraud, it will automatically generate a warning message detailing the suspected fraud and recommending actions to take.

[0636] Input: Fraud probability assessment result

[0637] Data processing: Generate warning messages

[0638] Output: Warning message

[0639] Specific behavior: The server generates a warning message saying, "This call may be fraudulent. Please hang up immediately and report it to the police."

[0640] Step 7: Sending and notifying warning messages

[0641] The server sends the generated warning message to the user's device via SMS, in-app notification, etc. The device then notifies the user of the warning message received from the server via a pop-up message, ringtone, or vibration.

[0642] Input: warning message

[0643] Data Processing: Sending a warning message

[0644] Output: User notification

[0645] Specific operation: The server sends a warning message to the device as an SMS, and the device rings and notifies the user, "This is a potentially fraudulent call. Please stop the call and exercise caution."

[0646] Step 8: Review user warnings

[0647] After the call ends, the user checks the warning message displayed on the device, which includes information about the suspected fraud and specific countermeasures.

[0648] Input: warning message

[0649] Data processing: Check the warning message and take corrective action

[0650] Output: Response to suspected fraudulent calls

[0651] Specific actions: The user checks the warning message displayed on the device and then reports the incident to the police.

[0652] (Application example 1)

[0653] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0654] In recent years, fraudsters targeting elderly people and those prone to fraud have become increasingly sophisticated, resulting in the increasing incidence of frauds attempting to defraud users of money over the phone. To prevent such frauds, a system is needed that can detect possible fraud in real time during a call and quickly issue a warning to the user. However, current systems lack the functionality to instantly analyze and evaluate the content of calls and notify the user in real time. This creates a time lag between fraud detection and notification, making it difficult to prevent frauds from occurring.

[0655] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0656] In this invention, the server includes a voice recognition means for converting voice data into text data, an analysis means for analyzing the text data and detecting specific keywords, a call origination identification means for analyzing the call origination information, an AI model means for evaluating the possibility of fraud using the voice data, keywords, and call origination information, a warning generation means for generating a warning message based on the result of the fraud possibility evaluation and sending it to the terminal, and a terminal means for notifying the user of the generated warning message and displaying the contents of the warning message to the user in real time. This makes it possible to analyze voice data during a call in real time and immediately notify the user if there is a high possibility of fraud. This is expected to have the effect of preventing fraud damage before it occurs.

[0657] "Voice data" refers to electrical signals of voice obtained during a call, and is converted into text data by a voice recognition means.

[0658] "Terminal means" refers to a smartphone or other communication terminal used by a user, which is a device that collects voice data and transmits it to a server.

[0659] A "server" is a computer system that receives, analyzes, and evaluates voice data transmitted from a terminal means via a network.

[0660] "Speech recognition means" refers to a process or device that has the function of converting received voice data into text data, and performs noise reduction and echo cancellation processing.

[0661] The "analysis means" is a process or device for analyzing the converted text data and detecting specific keywords.

[0662] A "keyword" is a specific linguistic unit that suggests the possibility of fraud, such as "money," "transfer," "accident," or "help."

[0663] A "call origin determination means" is a process or device that analyzes the call origin information and identifies suspicious locations (e.g., calls from public phones or overseas).

[0664] An "AI model means" is a process or device that implements an artificial intelligence model to assess the likelihood of fraud using voice data, keywords, and origin information.

[0665] An "alert generator" is a process or device that generates an alert message based on the result of assessing the likelihood of fraud and sends it to the terminal.

[0666] A "warning message" is information intended to notify the user of a possible fraud and includes recommended actions to be taken.

[0667] "Real-time notification" is the process or function of immediately communicating generated alert messages to the user.

[0668] The system of the present invention has the function of detecting possible fraud during a call in real time and issuing a prompt warning to the user. This system is mainly composed of server-side processing and terminal-side processing.

[0669] Server-side configuration

[0670] 1. Receiving audio data

[0671] The server receives the voice data sent from the terminal in real time. The voice data is collected during the call.

[0672] 2. Voice Recognition

[0673] The server analyzes the received voice data using speech recognition and converts it into text data, using services such as the Google Cloud Speech-to-Text API, and also applies noise reduction and echo cancellation.

[0674] 3. Keyword Discovery

[0675] The converted text data is analyzed using analytical tools to detect specific keywords such as "money," "transfer," "accident," and "help." Python and SpaCy can be used for the analysis.

[0676] 4. Identifying the source

[0677] The server analyzes the call origin information using GeoIP, and if it detects a suspicious location (such as a public phone or a call from abroad), it records that information.

[0678] 5. AI-powered fraud assessment

[0679] The server inputs the voice data, detected keywords, and location information into an AI model using TensorFlow and PyTorch to assess the likelihood of fraud, with reference to past data and known fraud patterns.

[0680] 6. Generating and Sending Warning Messages

[0681] If the server judges that there is a high possibility of fraud, it generates a warning message by a warning generation means and sends it to the terminal. The warning message includes information about the suspected fraud and measures to be taken.

[0682] Terminal configuration

[0683] 1. Collection of audio data

[0684] The terminal has the ability to collect voice data during a call in real time and send it to a server.

[0685] 2. Receiving and Displaying Alerts

[0686] The terminal receives the warning message sent from the server and immediately notifies the user of the message by voice, message notification, or real-time display on the terminal screen.

[0687] User-side configuration

[0688] 1. Check the contents of the call

[0689] The user does not need to perform any special operations during the call, and the call proceeds as normal. The call content is automatically analyzed.

[0690] 2. Checking and responding to warnings

[0691] When a warning message is sent to the terminal, the user can check it and take appropriate measures.

[0692] Specific examples

[0693] For example, if an elderly person named A receives a phone call that they suspect is a scam, the following steps would be taken:

[0694] 1. Mr. A's smartphone sends the audio during the call to the server.

[0695] 2. The server converts the speech into text and detects keywords such as "money" and "transfer."

[0696] 3. The call origin is identified as a pay phone.

[0697] 4. AI models will assess the likelihood of fraud.

[0698] 5. A warning message will be sent to Mr. A's smartphone immediately during or after the call ends.

[0699] 6. Person A is able to accept the warning, pay immediate attention, and take the necessary action.

[0700] Prompt Sentence Examples

[0701] text

[0702] Audio files:<audio_file>

[0703] Origin: <location>

[0704] Location information:<location_info>

[0705] Detection Keywords:<keywords_detected>

[0706] Scam rating score:<fraud_score>

[0707] Warning message:<alarm_message>

[0708] In this way, the system of the present invention automatically performs a series of processes from collecting voice data to assessing the possibility of fraud and issuing a warning, thereby minimizing the risk of users becoming victims of fraud.

[0709] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0710] Step 1:

[0711] Audio data collection

[0712] The device collects voice data in real time during a call. This voice data is picked up by the device's microphone from the start to the end of the call. The collected voice data is temporarily stored in the device's memory as is.

[0713] Input: Voice data during a call

[0714] Output: Collected audio data

[0715] Step 2:

[0716] Sending audio data

[0717] The terminal sends the collected voice data to the server. The communication protocol is secure HTTP (HTTPS). The data sent includes voice data and origin information (e.g., location information obtained from GPS).

[0718] Input: Collected voice data, location information

[0719] Output: Sending voice data and location information to the server

[0720] Step 3:

[0721] Receiving audio data

[0722] The server receives the voice data sent from the device in real time and temporarily stores the received data in the server's storage.

[0723] Input: Voice data sent from the device, location information

[0724] Output: Audio data and location information stored on the server

[0725] Step 4:

[0726] Voice Recognition

[0727] The server converts the stored audio data into text using the Google Cloud Speech-to-Text API, applying noise reduction and echo cancellation. The converted text data is then passed on to the next analysis step.

[0728] Input: Audio data

[0729] Output: Text data

[0730] Step 5:

[0731] Keyword detection

[0732] The server analyzes the text data using Python and SpaCy to detect specific keywords (e.g., money, transfer, accident, help). The detected keywords are used in the next evaluation step.

[0733] Input: Text data

[0734] Output: Detected keywords

[0735] Step 6:

[0736] Identifying the source

[0737] The server analyzes the origin information using GeoIP to identify suspicious locations (e.g., payphones, overseas). If a location is suspicious, the identified location information is used in the next evaluation step.

[0738] Input: Location information

[0739] Output: Parsed origin information

[0740] Step 7:

[0741] AI-powered fraud assessment

[0742] The server uses the voice data, detected keywords, and location information to evaluate the likelihood of fraud using an AI model (using TensorFlow and PyTorch).The AI ​​model outputs a score indicating the likelihood of fraud based on the preprocessed data.

[0743] Input: Voice data, detected keywords, location information

[0744] Output: Fraud rating score

[0745] Step 8:

[0746] Generate and send warning messages

[0747] The server generates a warning message based on the fraud assessment score, including a warning that the fraud is suspected and specific countermeasures, and sends the generated warning message to the terminal in real time.

[0748] Input: Fraud rating score

[0749] Output: Warning message

[0750] Step 9:

[0751] Receiving and Viewing Alerts

[0752] The terminal receives the warning message sent from the server and immediately notifies the user by voice, message notification, or real-time display on the terminal screen.

[0753] Input: warning message

[0754] Output: User notification

[0755] For example, here is a prompt:

[0756] For example, "Audio file:<audio_file> Origin: <location>Location information:<location_info> Detection Keywords:<keywords_detected> Scam rating score:<fraud_score> Warning message:<alarm_message> " is generated.

[0757] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0758] The present invention provides a system that detects "I'm your son" fraud calls in real time and issues a prompt warning, targeting elderly users and those prone to fraud. This system is an integrated system that includes voice data collection, voice analysis, keyword detection, caller location identification, fraud probability assessment using an AI model, and warning generation and notification. One embodiment of the present invention further includes an emotion engine for recognizing user emotions.

[0759] System Configuration

[0760] Server-side configuration

[0761] 1. Receiving audio data

[0762] The server receives the voice data transmitted from the terminal during the call in real time.

[0763] 2. Voice Recognition

[0764] The server uses a speech recognition module to convert the voice data into text data, which also includes noise reduction and echo cancellation.

[0765] 3. Keyword Discovery

[0766] The server analyzes the text data generated by the speech recognition to detect specific keywords, such as "money," "transfer," "accident," and "help."

[0767] 4. Identifying the source

[0768] The server analyzes the call origin information, using phone numbers, IP addresses, and GPS information to identify the origin, and if a suspicious location is detected, it records that information.

[0769] 5. Emotion Recognition by Emotion Engine

[0770] The server then inputs the voice data into an emotion engine to recognize the user's emotions. The emotion engine analyzes the tone, pitch, and tempo of the user's voice to determine emotions such as tension or anxiety.

[0771] 6. AI-powered fraud assessment

[0772] The server inputs the voice data, text data, detected keywords, location information, and emotional information recognized by the emotion engine into an AI model to assess the likelihood of fraud, with reference to past data and known fraud patterns.

[0773] 7. Generating Warning Messages

[0774] If the AI ​​model judges the possibility of fraud high, the server automatically generates a warning message after the call ends, which includes information about the suspected fraud and specific countermeasures.

[0775] 8. Sending Alerts

[0776] The server sends the generated warning message to the user's terminal.

[0777] Terminal configuration

[0778] 1. Collection of audio data

[0779] The device collects voice data during a call in real time and sends it to the server.

[0780] 2. Receiving a warning

[0781] When a warning message is sent from the server, the device notifies the user via voice call or message notification.

[0782] User-side configuration

[0783] 1. Check the contents of the call

[0784] The user will carry out the call as if it were a normal call.

[0785] 2. Check the warning

[0786] After the call ends, the user checks the warning message displayed on their device, which includes information about a suspected scam and specific countermeasures.

[0787] Specific examples

[0788] 1. The device initiates a call with elderly person D and his daughter.

[0789] 2. The device transmits the voice data to the server in real time.

[0790] 3. The server converts the speech into text and detects the keyword "money."

[0791] 4. The server analyzes the call's location information and determines that the call came from a public phone.

[0792] 5. The server inputs the voice data into the emotion engine and recognizes that Mr. D is feeling nervous or anxious.

[0793] 6. The server inputs this information into an AI model and rates the likelihood of fraud.

[0794] 7. After the call ends, the server automatically generates a warning message and sends it to the terminal.

[0795] 8. The device will send a warning message to Mr. D, who can then take immediate action.

[0796] In this way, the present invention automates a series of processes from collecting voice data to recognizing emotional information, assessing the possibility of fraud, and issuing a warning, thereby minimizing the risk of users becoming victims of fraud.

[0797] The processing flow will be explained below.

[0798] Step 1:

[0799] When a call is initiated, the device collects voice data in real time, filters the data during the call, and sends it to the server.

[0800] Step 2:

[0801] The server receives the voice data sent from the terminal in real time and inputs it into a voice recognition module for analysis.

[0802] Step 3:

[0803] The server converts the voice data into text data using a speech recognition module, which performs pre-processing such as noise reduction and echo cancellation.

[0804] Step 4:

[0805] The server analyzes the text data generated by the voice recognition and detects predetermined keywords, such as "money," "transfer," "accident," and "help." The detected keywords are recorded along with the location information.

[0806] Step 5:

[0807] The server analyzes the call origin information, identifying the origin using phone number, IP address, GPS information, etc., and if it detects any suspicious locations (such as public telephones or calls from abroad), it records that information.

[0808] Step 6:

[0809] The server inputs the voice data into an emotion engine to recognize the user's emotions. The emotion engine analyzes the voice tone, pitch, and tempo to determine the user's emotional state. For example, if the user is feeling nervous or anxious, that information is also recorded.

[0810] Step 7:

[0811] The server inputs voice data, text data, detected keywords, location information, and emotional information into the AI ​​model, which uses this information to assess the likelihood of fraud. It then uses past data and known fraud patterns to generate a score.

[0812] Step 8:

[0813] If the AI ​​model evaluates the likelihood of fraud at a certain level, the server generates a warning message after the call ends, which includes information about the suspected fraud and specific countermeasures.

[0814] Step 9:

[0815] The server sends the generated warning message to the user's terminal, and the sent warning message arrives at the terminal in real time.

[0816] Step 10:

[0817] When the device receives the warning message sent from the server, it notifies the user, which is displayed as a voice call or message notification.

[0818] Step 11:

[0819] The user checks the warning message on the device, understands the contents of the warning message, and takes necessary measures, such as rechecking the contents of calls that are suspected to be fraudulent and consulting the police or financial institution.

[0820] By taking the above steps, the system can quickly and accurately detect the possibility of "It's me" fraud and issue an immediate warning to the user, thus preventing fraud damage before it occurs.

[0821] Example 2

[0822] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0823] In modern society, elderly people and certain user groups are increasingly becoming victims of fraud such as "it's me" scams. These frauds are becoming more sophisticated and are often difficult to detect using normal means. As a result, victims are likely to suffer significant financial losses. To solve this problem, a system is needed that can accurately detect potential fraud in real time and issue a prompt warning.

[0824] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0825] In this invention, the server includes: a voice recognition means for converting voice data into text data; an analysis means for analyzing the text data and detecting specific keywords; a call origin identification means for analyzing call origin information; an AI model means for evaluating the possibility of fraud using the voice data, keywords, and call origin information; a warning generation means for generating a warning message based on the result of the fraud possibility evaluation and sending it to the terminal; a terminal means for notifying the user of the generated warning message; an emotion recognition means for analyzing the voice data and recognizing the user's emotion; and an AI model means for inputting the emotion information into the AI ​​model and evaluating the possibility of fraud with high accuracy. This makes it possible to minimize the risk of users becoming victims of fraud.

[0826] "Audio data" means data in digital form obtained by recording or transmitting sound.

[0827] A "terminal means" is an electronic device for collecting and transmitting voice data.

[0828] A "server" is a central processing unit that receives, processes, and stores data from multiple terminals over a network.

[0829] "Speech recognition means" is a technology that analyzes voice data and converts the content into text data.

[0830] The "analysis means" is a device or software that has the function of analyzing text data and detecting specific keywords.

[0831] The "originating location identification means" is a technology that analyzes the originating location information of a call and identifies the area where the call was made.

[0832] An "AI modeling tool" is a device or software that uses artificial intelligence to assess the likelihood of fraud.

[0833] A "warning generator" is a device or software capable of generating a warning message based on the result of assessing the likelihood of fraud.

[0834] "Emotion recognition means" is a technology that analyzes voice data to determine the user's emotions.

[0835] A "warning message" is a message that notifies the user in case of possible fraud.

[0836] "User" means an individual or organization that uses the system.

[0837] This invention provides a system that detects "It's my son" fraud calls in real time and issues a prompt warning, targeting elderly people and other users who are prone to fraud. This system can automatically carry out a series of processes from collecting voice data to issuing a warning.

[0838] System Components

[0839] Server-side configuration

[0840] Receiving audio data:

[0841] The server receives the voice data sent from the device in real time during the call. Specifically, a WebSocket connection is used to exchange data bidirectionally between the device and the server.

[0842] Voice Recognition:

[0843] The server uses the Google Speech-to-Text API or a similar service to convert the received voice data into text data, and during this process, it also performs noise reduction and echo cancellation to generate highly accurate text data.

[0844] Keyword detection:

[0845] The server performs text analysis to detect specific keywords (e.g., "money," "transfer," "accident," "help," etc.) using regular expressions and pre-defined keyword lists.

[0846] Identifying the source:

[0847] The server analyzes the call origin information, using phone numbers, IP addresses, GPS information, etc. to detect calls from suspicious locations such as public phones.

[0848] Emotion recognition:

[0849] The server inputs the voice data into an emotion engine (such as Affectiva's emotion recognition API) that analyzes the tone, pitch, and tempo of the user's voice to determine their emotion. Based on this information, it can determine whether the user is feeling nervous or anxious.

[0850] AI Fraud Assessment:

[0851] The server inputs voice data, text data, keywords, source information, and emotional information into an AI model to assess the likelihood of fraud, which also refers to past data and known fraud patterns.

[0852] Generates a warning message:

[0853] If the AI ​​model judges the likelihood of fraud high, the server automatically generates a warning message containing the suspected fraud and specific countermeasures.

[0854] Sending alerts:

[0855] The server generates a warning message and sends it to the user's device. The message is promptly sent to the user via push notification or SMS.

[0856] Terminal configuration

[0857] Audio data collection:

[0858] The device collects voice data during a call in real time and immediately sends it to the server. It is set to start collecting voice data as soon as the call starts.

[0859] Receiving an alert:

[0860] The device receives a warning message from the server and notifies the user. Notification methods include voice calls and message notifications, and the user is alerted in a way that can be checked immediately.

[0861] User-side configuration

[0862] Check the call details:

[0863] Users make calls as usual, the system runs in the background and no special action is required.

[0864] Check for warnings:

[0865] After the call ends, the user can view the warning message on their device and take appropriate action. The message will include information about the suspected fraud and specific countermeasures.

[0866] Specific examples

[0867] If an elderly person named Mr. D is told by someone claiming to be his son that he needs money, the system works as follows:

[0868] 1. The device collects the conversation between Mr. D and the other party and sends the voice data to the server.

[0869] 2. The server uses a speech recognition module to convert this speech into text and detect the keyword "money."

[0870] 3. The server uses the call location identification means to identify that the call is coming from a public telephone.

[0871] 4. The server inputs the voice data into an emotion recognition engine and recognizes that Mr. D is feeling nervous or anxious.

[0872] 5. The server inputs this information into an AI model and scores the likelihood of fraud.

[0873] 6. After the call ends, the server generates a warning message and sends it to the terminal.

[0874] 7. The device will send a warning message to Mr. D, who can then take immediate action.

[0875] This system automates the entire process from collecting voice data to recognizing emotional information, assessing the likelihood of fraud, and issuing warning notifications, thereby minimizing the risk of users becoming victims of fraud.

[0876] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0877] Step 1:

[0878] Collection and transmission of voice data

[0879] The device collects audio data during the call in real time. Specifically, it uses the device's microphone to capture the audio of the call and sends the audio data to the server via WebSocket or HTTP. The input of this step is the audio data collected from the device's microphone, and the output is the audio data sent to the server.

[0880] Step 2:

[0881] Receiving audio data

[0882] The server receives the audio data sent from the device in real time. Specifically, it starts a WebSocket server to receive the audio data. The input of this step is the audio data sent from the device, and the output is the audio data stored in a temporary buffer on the server.

[0883] Step 3:

[0884] Voice Recognition

[0885] The server converts the received voice data into text data using a speech recognition module such as the Google Speech-to-Text API. During this process, noise reduction and echo cancellation are also performed to improve analysis accuracy. The input for this step is the voice data stored on the server, and the output is the result of converting the voice data into text data.

[0886] Step 4:

[0887] Keyword detection

[0888] The server analyzes the generated text data and detects specific keywords (e.g., "money," "transfer," "accident," "help," etc.). Specifically, the analysis is performed using regular expressions or a pre-defined keyword list. The input of this step is the text data, and the output is a list of detected keywords.

[0889] Step 5:

[0890] Identifying the source

[0891] The server analyzes the call's origin information. Specifically, it identifies the call's origin using the phone number, IP address, GPS information, etc. The input to this step is the metadata associated with the call, and the output is the origin information. If the call originates from a suspicious location, the location information is also included.

[0892] Step 6:

[0893] emotion recognition

[0894] The server inputs the voice data into an emotion recognition engine (e.g., Affectiva's emotion recognition API) and analyzes the tone, pitch, and tempo of the user's voice to determine their emotion. The input of this step is the voice data, and the output is the user's emotional information (e.g., tension, fear, relief, etc.).

[0895] Step 7:

[0896] AI-powered fraud assessment

[0897] The server inputs the voice data, text data, detected keywords, origin information, and emotion information into the AI ​​model to assess the likelihood of fraud. The input for this step is all of this data, and the output is a fraud risk assessment score. The AI ​​model also references known fraud patterns and past data to make the assessment.

[0898] Step 8:

[0899] Generate a warning message

[0900] If the AI ​​model evaluates the likelihood of fraud highly, the server automatically generates a warning message. This message includes the suspected fraud and specific countermeasures, and is provided to the user. The input of this step is the fraud risk assessment score, and the output is the generated warning message.

[0901] Step 9:

[0902] Sending an Alert

[0903] The server sends the generated warning message to the user's device. Specifically, it ensures that the warning message reaches the user promptly via push notification or SMS. The input of this step is the warning message, and the output is the warning message sent to the user's device.

[0904] Step 10:

[0905] Receiving and Acknowledging Alerts

[0906] The terminal receives the warning message from the server and notifies the user. Specifically, it notifies the user of the warning using a notification bar, a pop-up, or a sound notification. The input of this step is the warning message sent from the server, and the output is the warning message displayed in a format that can be confirmed by the user.

[0907] Step 11:

[0908] User response

[0909] The user checks the warning message sent to the device and takes appropriate action, such as reading the warning content, reporting it to the appropriate authorities, or ending the call. The input to this step is the warning message, and the output is the user's specific action.

[0910] Through these steps, the system can efficiently collect, analyze, evaluate, and warn users of voice data, minimizing the risk of fraud.

[0911] (Application example 2)

[0912] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0913] In recent years, there has been an increase in "I'm your son" frauds targeting elderly people and other users who are prone to fraud, creating a need for rapid and effective countermeasures. However, conventional systems have had difficulty detecting potential fraud in real time during voice calls and issuing prompt warnings to users. In addition, evaluations based solely on voice data have the problem of low detection accuracy and a high number of false positives.

[0914] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a voice recognition means for converting voice data into text data, an analysis means for analyzing the text data and detecting specific keywords, a call origin identification means for analyzing the call origin information, and an AI model means for evaluating the possibility of fraud using the voice data, keywords, call origin information, and user emotion data. This makes it possible to prevent elderly users and users who are prone to fraud from becoming victims of fraud.

[0915] The "terminal means" is a device that has the function of collecting voice data and transmitting it to a server.

[0916] A "voice recognition means" is a device or system that has the function of converting voice data into text data.

[0917] The "analysis means" is a system that has the function of analyzing text data and detecting specific keywords.

[0918] The "originating point identification means" is a system that has the function of analyzing the originating point information of a call and identifying the originating point.

[0919] The "AI model means" is a system that has the function of assessing the possibility of fraud using voice data, keywords, origin information, and user emotional data.

[0920] The "warning generation means" is a system that has the function of generating a warning message based on the result of assessing the possibility of fraud and sending it to the terminal.

[0921] "Emotion data" is information that represents the user's emotional state and is obtained by analyzing voice data.

[0922] This invention provides a system that detects fraudulent calls in real time and issues prompt warnings to elderly users and other users who are prone to fraud. This system is an integrated system that includes voice data collection, voice analysis, keyword detection, caller location identification, fraud probability assessment using an AI model, and warning generation and notification.

[0923] First, the terminal collects voice data and sends it to the server. The server converts the voice data into text data using a voice recognition means. The voice recognition means includes noise reduction and echo cancellation functions. Next, the server analyzes the text data to detect specific keywords. This analysis means detects keywords related to fraud (e.g., "money," "transfer," "accident," "help," etc.).

[0924] Furthermore, the server analyzes the call origination information and identifies suspicious locations using a call origination location identification means. The server also inputs the voice data into an emotion recognition engine to analyze the user's emotion data. This emotion data identifies emotions such as tension or anxiety based on the user's voice tone, pitch, tempo, etc.

[0925] This data (voice data, keywords, origin information, and emotion data) is input into the AI ​​model means to evaluate the likelihood of fraud. The AI ​​model means makes the evaluation by referring to past data and known fraud patterns. If it is evaluated that there is a high likelihood of fraud, the server generates a warning message and sends it to the terminal using the warning generation means. The terminal notifies the user of the generated warning message. This notification may take the form of a voice call or a message notification.

[0926] As a concrete example, consider a situation in which a device initiates a call between an elderly user and someone else. At this time, the device sends voice data to a server in real time. The server converts the voice to text and detects the keyword "money." The server then analyzes the call's origin information and determines that it is coming from a public phone. The server inputs the voice data into an emotion engine and recognizes that the user is feeling nervous or anxious. The server then inputs this information into an AI model and evaluates the likelihood of fraud. After the call ends, the server automatically generates a warning message and sends it to the device. The device notifies the user of the warning message, allowing them to take immediate action.

[0927] The server also handles notifications to notify the generated warning messages. The following is an example of a specific prompt to the AI ​​model:

[0928] "The following keywords were detected in this call: 'money', 'transfer'. The call originated from an unknown pay phone. The user's emotions are very tense. What is the likelihood of this being a scam?"

[0929] The present invention automates this series of processes, minimizing the risk of users falling victim to fraud.

[0930] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0931] Step 1:

[0932] The terminal collects voice data during a call. The collected voice data is sent to the server in real time. This allows the server to receive the user's call data in real time. The input is the call voice data, and the output is the voice data sent to the server.

[0933] Step 2:

[0934] The server converts the received voice data into text data using a voice recognition means. During this process, noise reduction and echo cancellation are performed to improve the accuracy of the voice data. The input is voice data, and the output is text data.

[0935] Step 3:

[0936] The server analyzes the text data generated by the speech recognition to detect specific keywords. Using the analysis means, it detects fraud-related keywords such as "money," "transfer," "accident," and "help." The input is the text data, and the output is a list of detected keywords.

[0937] Step 4:

[0938] The server analyzes the call origination information and uses the call origination identification means to identify suspicious locations by analyzing the phone number, IP address, and GPS information. The input is the call origination information, and the output is the analyzed call origination information.

[0939] Step 5:

[0940] The server inputs the voice data into an emotion recognition engine and analyzes the user's emotional data. The emotion recognition engine analyzes the tone, pitch, and tempo of the user's voice to determine emotions such as tension or anxiety. The input is voice data, and the output is the user's emotional data.

[0941] Step 6:

[0942] The server inputs the voice data, keywords, origin information, and emotion data into the AI ​​model means to evaluate the likelihood of fraud. The AI ​​model means makes an evaluation by referring to past data and known fraud patterns. The inputs are the voice data, keywords, origin information, and emotion data, and the output is the result of the evaluation of the likelihood of fraud.

[0943] Step 7:

[0944] If the server judges the possibility of fraud to be high, it generates a warning message using the warning generation means and sends it to the terminal. The generated warning message includes information about the suspected fraud and specific countermeasures. The input is the result of the fraud possibility assessment, and the output is the warning message.

[0945] Step 8:

[0946] The terminal receives the warning message sent from the server and notifies the user. The notification is in the form of a voice call or a message notification. The input is the warning message, and the output is the notification to the user.

[0947] Through the above series of processing steps, users can detect fraudulent calls in real time and take prompt action.

[0948] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0949] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0950] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0951] [Third embodiment]

[0952] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0953] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0954] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0955] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0956] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0957] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0958] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0959] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0960] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0961] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0962] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0963] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0964] This invention provides a system that detects "I am your son" fraud calls in real time and issues a prompt warning, targeting elderly people and other users who are prone to fraud. This system is an integrated system that includes voice data collection, voice analysis, keyword detection, caller location identification, fraud probability assessment using an AI model, and warning generation and notification.

[0965] System Configuration

[0966] Server-side configuration

[0967] 1. Receiving audio data

[0968] The server receives real-time voice data during a call, which is sent from the terminal.

[0969] 2. Voice Recognition

[0970] The server analyzes the received voice data using a speech recognition module and converts it into text data, with noise reduction and echo cancellation also being applied during this process.

[0971] 3. Keyword Discovery

[0972] The server analyzes the text data generated by the speech recognition to detect specific keywords, such as "money," "transfer," "accident," and "help."

[0973] 4. Identifying the source

[0974] The server analyzes the call origin information and if it detects any suspicious locations (such as public phones or calls from abroad), it records that information.

[0975] 5. AI-powered fraud assessment

[0976] The server inputs the voice data, detected keywords, and location information into an AI model to assess the likelihood of fraud, with reference to past data and known fraud patterns.

[0977] 6. Generating Warning Messages

[0978] If the server assesses a high probability of fraud, it will automatically generate a warning message after the call ends, including information about the suspected fraud and what steps should be taken.

[0979] 7. Sending Alerts

[0980] The server sends the generated warning message to the user's terminal.

[0981] Terminal configuration

[0982] 1. Collection of audio data

[0983] The device collects voice data during a call in real time and sends it to the server.

[0984] 2. Receiving a warning

[0985] When a warning message is sent from the server, the device notifies the user via voice call or message notification.

[0986] User-side configuration

[0987] 1. Check the contents of the call

[0988] The user will carry out the call as if it were a normal call.

[0989] 2. Check the warning

[0990] After the call ends, the user checks the warning message displayed on the device, which includes information about the suspected fraud and specific countermeasures.

[0991] Specific examples

[0992] 1. The device starts a call between elderly person C and the person in charge.

[0993] 2. The device transmits the voice data to the server in real time.

[0994] 3. The server converts the speech into text and detects the keyword "money."

[0995] 4. The server analyzes the call's location information and determines that the call originated from a domestic public telephone.

[0996] 5. The server inputs this information into an AI model and rates the likelihood of fraud.

[0997] 6. After the call ends, the server automatically generates a warning message and sends it to the terminal.

[0998] 7. The device will notify Mr. C of the warning message, so that he can immediately pay attention and take the necessary action.

[0999] In this way, the present invention can automate a series of processes from collecting voice data to evaluating the possibility of fraud and issuing a warning, thereby minimizing the risk of users becoming victims of fraud.

[1000] The processing flow will be explained below.

[1001] Step 1:

[1002] When a call is initiated, the device collects voice data in real time and transmits the voice data during the call to the server as is.

[1003] Step 2:

[1004] The server receives the voice data sent from the terminal and inputs it into a voice recognition module for real-time processing.

[1005] Step 3:

[1006] The server uses a speech recognition module to convert the voice data into text data, and pre-processing such as noise reduction and echo cancellation is performed during this process.

[1007] Step 4:

[1008] The server analyzes the text data and detects certain keywords, such as "money," "transfer," "accident," and "help."

[1009] Step 5:

[1010] The server analyzes the call origin information, identifying the origin using phone number, IP address, GPS information, etc., and records any suspicious locations (such as public telephones or calls from overseas).

[1011] Step 6:

[1012] The server inputs the voice data, text data, detected keywords, and location information into the AI ​​model, which then evaluates the likelihood of fraud based on this information.

[1013] Step 7:

[1014] If the server determines that there is a high possibility of fraud based on the fraud probability score evaluated by the AI ​​model, it will generate a warning message after the call ends, notifying the caller of the suspected fraud and providing specific countermeasures.

[1015] Step 8:

[1016] The server sends the generated warning message to the user's terminal, and the message arrives at the terminal in real time.

[1017] Step 9:

[1018] When the device receives the warning message sent from the server, it notifies the user, which is displayed as a voice call or a message.

[1019] Step 10:

[1020] The user checks the warning message on the device, understands the contents of the warning message, and takes necessary measures, such as rechecking the contents of calls that are suspected to be fraudulent and consulting the police or financial institution.

[1021] Through these steps, the system can quickly and accurately detect possible telephone fraud and issue an immediate warning to the user.

[1022] Example 1

[1023] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1024] There is a challenge in detecting "I'm your son" fraud calls in real time and issuing prompt warnings, targeting elderly people and other users who are prone to fraud. Conventional systems have difficulty immediately detecting potential fraud and issuing warnings, putting users at risk of becoming victims. In particular, since calls from suspicious locations and specific keywords are often not detected, there is a need for a means to prevent fraud in advance.

[1025] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1026] In this invention, the server includes terminal means for collecting voice data and transmitting it to the server, voice recognition means for converting the voice data into text data, analysis means for analyzing the text data and detecting specific keywords, call origination location identification means for analyzing call origination location information, voice processing means for performing noise reduction and echo cancellation, AI model means for evaluating the possibility of fraud using the voice data, keywords, and call origination location information, warning generation means for generating a warning message based on the result of the evaluation of the possibility of fraud and transmitting it to the terminal, and terminal means for notifying the user of the generated warning message. This makes it possible to detect fraudulent calls in real time and quickly issue a warning to the user.

[1027] "Voice data" refers to data converted into digital form of voice in a conversation between a user and another person.

[1028] "Terminal means" refers to a device that has the function of collecting voice data and transmitting it to a server.

[1029] "Speech recognition means" refers to software or hardware that has the function of analyzing received voice data and converting it into text data.

[1030] "Text data" refers to character string information generated from voice data by a voice recognition means.

[1031] "Analysis means" refers to software or hardware that has the function of analyzing text data and detecting specific keywords.

[1032] "Call origin identification means" refers to software or hardware that has the function of analyzing call origin information and identifying suspicious locations.

[1033] "Audio processing means" refers to software or hardware that performs noise reduction and echo cancellation and optimizes audio data.

[1034] "AI model means" refers to an artificial intelligence model that uses voice data, keywords, and location information to assess the likelihood of fraud.

[1035] "Warning generation means" refers to software or hardware that has the function of generating a warning message based on the result of assessing the likelihood of fraud and sending it to a terminal.

[1036] "Warning Message" means notification information containing warnings and countermeasures regarding potentially fraudulent calls.

[1037] This invention provides a system that detects "I am your son" fraud calls in real time and issues a prompt warning, targeting elderly people and other users who are prone to fraud. This system is an integrated system that includes voice data collection, voice analysis, keyword detection, caller location identification, fraud probability assessment using an AI model, and warning generation and notification.

[1038] Server-side configuration

[1039] 1. Receiving audio data

[1040] The server receives real-time audio data during a call. This audio data is sent from the device. The device collects audio during the call with a microphone and encodes it digitally using an AudioCodec. The encoded audio data is then sent to the server using HTTP or WebSocket protocols.

[1041] 2. Voice Recognition

[1042] The server analyzes the received voice data and converts it into text data using a speech recognition module. This process uses voice recognition software such as Google Cloud Speech-to-Text or IBM Watson. The server pre-processes the voice data by performing noise reduction and echo cancellation.

[1043] 3. Keyword Discovery

[1044] The server analyzes the text data generated by the speech recognition to detect specific keywords, such as "money," "transfer," "accident," and "help," using regular expressions and keyword matching algorithms.

[1045] 4. Identifying the source

[1046] The server analyzes the call origination information, using location analysis tools such as GeoIP databases to identify suspicious locations (e.g., domestic or international pay phones, or calls from abroad).

[1047] 5. AI-powered fraud assessment

[1048] The server inputs the voice data, detected keywords, and location information into an AI model to assess the likelihood of fraud. The AI ​​model was built using TensorFlow and PyTorch and is trained on past data and known fraud patterns.

[1049] 6. Generating Warning Messages

[1050] If the server assesses a high probability of fraud, it will automatically generate a warning message detailing the suspected fraud and recommending actions to take.

[1051] 7. Sending Alerts

[1052] The server then sends the generated warning message to the user's device via SMS, in-app notifications, etc.

[1053] Terminal configuration

[1054] 1. Collection of audio data

[1055] The device collects voice data during a call in real time and sends it to a server, where it is digitally encoded and transferred to the server using a protocol.

[1056] 2. Receiving a warning

[1057] The device receives the warning message from the server and notifies the user in the form of a pop-up message, a ringtone, or vibration.

[1058] User-side configuration

[1059] 1. Check the contents of the call

[1060] The user will be able to talk as if they were in a normal conversation. There is no need for them to be particularly conscious of it, but they will be able to talk naturally.

[1061] 2. Check the warning

[1062] After the call ends, the user checks the warning message that is sent to the terminal. The warning message contains information about the suspected fraud and specific countermeasures.

[1063] Specific examples

[1064] The device initiates a call between elderly person C and the person in charge. The device sends the voice data to the server in real time. The server converts the voice to text and detects the keyword "money." The server analyzes the call's origin information and determines that the call originated from a domestic public phone. The server inputs this information into an AI model and rates the possibility of fraud highly. After the call ends, the server automatically generates a warning message and sends it to the device. The device notifies C of the warning message, allowing him to immediately pay attention and take the necessary action.

[1065] In this way, the present invention can automate a series of processes from collecting voice data to evaluating the possibility of fraud and issuing a warning, thereby minimizing the risk of users becoming victims of fraud.

[1066] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1067] Step 1: Real-time collection and transmission of audio data

[1068] The device collects voice data during a call in real time, converts it into a digital voice data format (e.g., PCM format), compresses it using an AudioCodec, and transmits the data to the server using HTTP or WebSocket protocols.

[1069] Input: Audio during a call

[1070] Data processing: audio data capture and compression

[1071] Output: Digital audio data

[1072] Specific operation: A user starts a call on their smartphone, and the audio is collected by the device's microphone, compressed, and then sent to the server.

[1073] Step 2: Receiving and Preprocessing Audio Data

[1074] The server analyzes the received voice data in real time. First, it performs noise reduction and echo cancellation to improve the sound quality. Then, it converts the voice data into text using voice recognition software such as Google Cloud Speech-to-Text or IBM Watson.

[1075] Input: Digital audio data (sent to the server)

[1076] Data processing: noise reduction, echo cancellation, voice recognition

[1077] Output: Text data

[1078] Specific operation: The server calls the Google Cloud Speech-to-Text API and converts the spoken voice data of "Help, please help me" into the text "Help, please help me."

[1079] Step 3: Keyword Discovery

[1080] The server analyzes the text data generated by speech recognition to detect specific keywords (e.g., "money," "transfer," "accident," "help," etc.) using regular expressions and keyword matching algorithms.

[1081] Input: Text data converted by speech recognition

[1082] Data processing: Keyword detection using regular expressions and keyword matching

[1083] Output: A list of detected keywords

[1084] Specific operation: The server detects the keywords "money" and "transfer" from the text data "Please transfer money."

[1085] Step 4: Identify the source

[1086] The server analyzes the call origination information, using location analysis tools such as GeoIP databases to identify suspicious locations (e.g., international locations, pay phones).

[1087] Input: Data about the call's origin (IP address and GPS data)

[1088] Data processing: location information analysis

[1089] Output: Origination information

[1090] What it does: The server analyzes the IP address and determines that the caller is calling from overseas.

[1091] Step 5: AI fraud assessment

[1092] The server inputs the voice data, detected keywords, and location information into an AI model to assess the likelihood of fraud. The AI ​​model was built using TensorFlow and PyTorch and is trained on past data and known fraud patterns.

[1093] Input: Voice data, keywords, location information

[1094] Data manipulation: Assessing the likelihood of fraud with AI models

[1095] Output: Fraud probability assessment result

[1096] Specific operation: The server inputs the information "transfer," "help," and the origin of the call into the AI ​​model, and assesses the possibility of fraud.

[1097] Step 6: Generate a warning message

[1098] If the server assesses a high probability of fraud, it will automatically generate a warning message detailing the suspected fraud and recommending actions to take.

[1099] Input: Fraud probability assessment result

[1100] Data processing: Generate warning messages

[1101] Output: Warning message

[1102] Specific behavior: The server generates a warning message saying, "This call may be fraudulent. Please hang up immediately and report it to the police."

[1103] Step 7: Sending and notifying warning messages

[1104] The server sends the generated warning message to the user's device via SMS, in-app notification, etc. The device then notifies the user of the warning message received from the server via a pop-up message, ringtone, or vibration.

[1105] Input: warning message

[1106] Data Processing: Sending a warning message

[1107] Output: User notification

[1108] Specific operation: The server sends a warning message to the device as an SMS, and the device rings and notifies the user, "This is a potentially fraudulent call. Please stop the call and exercise caution."

[1109] Step 8: Review user warnings

[1110] After the call ends, the user checks the warning message displayed on the device, which includes information about the suspected fraud and specific countermeasures.

[1111] Input: warning message

[1112] Data processing: Check the warning message and take corrective action

[1113] Output: Response to suspected fraudulent calls

[1114] Specific actions: The user checks the warning message displayed on the device and then reports the incident to the police.

[1115] (Application example 1)

[1116] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1117] In recent years, fraudsters targeting elderly people and those prone to fraud have become increasingly sophisticated, resulting in the increasing incidence of frauds attempting to defraud users of money over the phone. To prevent such frauds, a system is needed that can detect possible fraud in real time during a call and quickly issue a warning to the user. However, current systems lack the functionality to instantly analyze and evaluate the content of calls and notify the user in real time. This creates a time lag between fraud detection and notification, making it difficult to prevent frauds from occurring.

[1118] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1119] In this invention, the server includes a voice recognition means for converting voice data into text data, an analysis means for analyzing the text data and detecting specific keywords, a call origination identification means for analyzing the call origination information, an AI model means for evaluating the possibility of fraud using the voice data, keywords, and call origination information, a warning generation means for generating a warning message based on the result of the fraud possibility evaluation and sending it to the terminal, and a terminal means for notifying the user of the generated warning message and displaying the contents of the warning message to the user in real time. This makes it possible to analyze voice data during a call in real time and immediately notify the user if there is a high possibility of fraud. This is expected to have the effect of preventing fraud damage before it occurs.

[1120] "Voice data" refers to electrical signals of voice obtained during a call, and is converted into text data by a voice recognition means.

[1121] "Terminal means" refers to a smartphone or other communication terminal used by a user, which is a device that collects voice data and transmits it to a server.

[1122] A "server" is a computer system that receives, analyzes, and evaluates voice data transmitted from a terminal means via a network.

[1123] "Speech recognition means" refers to a process or device that has the function of converting received voice data into text data, and performs noise reduction and echo cancellation processing.

[1124] The "analysis means" is a process or device for analyzing the converted text data and detecting specific keywords.

[1125] A "keyword" is a specific linguistic unit that suggests the possibility of fraud, such as "money," "transfer," "accident," or "help."

[1126] A "call origin determination means" is a process or device that analyzes the call origin information and identifies suspicious locations (e.g., calls from public phones or overseas).

[1127] An "AI model means" is a process or device that implements an artificial intelligence model to assess the likelihood of fraud using voice data, keywords, and origin information.

[1128] An "alert generator" is a process or device that generates an alert message based on the result of assessing the likelihood of fraud and sends it to the terminal.

[1129] A "warning message" is information intended to notify the user of a possible fraud and includes recommended actions to be taken.

[1130] "Real-time notification" is the process or function of immediately communicating generated alert messages to the user.

[1131] The system of the present invention has the function of detecting possible fraud during a call in real time and issuing a prompt warning to the user. This system is mainly composed of server-side processing and terminal-side processing.

[1132] Server-side configuration

[1133] 1. Receiving audio data

[1134] The server receives the voice data sent from the terminal in real time. The voice data is collected during the call.

[1135] 2. Voice Recognition

[1136] The server analyzes the received voice data using speech recognition and converts it into text data, using services such as the Google Cloud Speech-to-Text API, and also applies noise reduction and echo cancellation.

[1137] 3. Keyword Discovery

[1138] The converted text data is analyzed using analytical tools to detect specific keywords such as "money," "transfer," "accident," and "help." Python and SpaCy can be used for the analysis.

[1139] 4. Identifying the source

[1140] The server analyzes the call origin information using GeoIP, and if it detects a suspicious location (such as a public phone or a call from abroad), it records that information.

[1141] 5. AI-powered fraud assessment

[1142] The server inputs the voice data, detected keywords, and location information into an AI model using TensorFlow and PyTorch to assess the likelihood of fraud, with reference to past data and known fraud patterns.

[1143] 6. Generating and Sending Warning Messages

[1144] If the server judges that there is a high possibility of fraud, it generates a warning message by a warning generation means and sends it to the terminal. The warning message includes information about the suspected fraud and measures to be taken.

[1145] Terminal configuration

[1146] 1. Collection of audio data

[1147] The terminal has the ability to collect voice data during a call in real time and send it to a server.

[1148] 2. Receiving and Displaying Alerts

[1149] The terminal receives the warning message sent from the server and immediately notifies the user of the message by voice, message notification, or real-time display on the terminal screen.

[1150] User-side configuration

[1151] 1. Check the contents of the call

[1152] The user does not need to perform any special operations during the call, and the call proceeds as normal. The call content is automatically analyzed.

[1153] 2. Checking and responding to warnings

[1154] When a warning message is sent to the terminal, the user can check it and take appropriate measures.

[1155] Specific examples

[1156] For example, if an elderly person named A receives a phone call that they suspect is a scam, the following steps would be taken:

[1157] 1. Mr. A's smartphone sends the audio during the call to the server.

[1158] 2. The server converts the speech into text and detects keywords such as "money" and "transfer."

[1159] 3. The call origin is identified as a pay phone.

[1160] 4. AI models will assess the likelihood of fraud.

[1161] 5. A warning message will be sent to Mr. A's smartphone immediately during or after the call ends.

[1162] 6. Person A is able to accept the warning, pay immediate attention, and take the necessary action.

[1163] Prompt Sentence Examples

[1164] text

[1165] Audio files:<audio_file>

[1166] Origin: <location>

[1167] Location information:<location_info>

[1168] Detection Keywords:<keywords_detected>

[1169] Scam rating score:<fraud_score>

[1170] Warning message:<alarm_message>

[1171] In this way, the system of the present invention automatically performs a series of processes from collecting voice data to assessing the possibility of fraud and issuing a warning, thereby minimizing the risk of users becoming victims of fraud.

[1172] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1173] Step 1:

[1174] Audio data collection

[1175] The device collects voice data in real time during a call. This voice data is picked up by the device's microphone from the start to the end of the call. The collected voice data is temporarily stored in the device's memory as is.

[1176] Input: Voice data during a call

[1177] Output: Collected audio data

[1178] Step 2:

[1179] Sending audio data

[1180] The terminal sends the collected voice data to the server. The communication protocol is secure HTTP (HTTPS). The data sent includes voice data and origin information (e.g., location information obtained from GPS).

[1181] Input: Collected voice data, location information

[1182] Output: Sending voice data and location information to the server

[1183] Step 3:

[1184] Receiving audio data

[1185] The server receives the voice data sent from the device in real time and temporarily stores the received data in the server's storage.

[1186] Input: Voice data sent from the device, location information

[1187] Output: Audio data and location information stored on the server

[1188] Step 4:

[1189] Voice Recognition

[1190] The server converts the stored audio data into text using the Google Cloud Speech-to-Text API, applying noise reduction and echo cancellation. The converted text data is then passed on to the next analysis step.

[1191] Input: Audio data

[1192] Output: Text data

[1193] Step 5:

[1194] Keyword detection

[1195] The server analyzes the text data using Python and SpaCy to detect specific keywords (e.g., money, transfer, accident, help). The detected keywords are used in the next evaluation step.

[1196] Input: Text data

[1197] Output: Detected keywords

[1198] Step 6:

[1199] Identifying the source

[1200] The server analyzes the origin information using GeoIP to identify suspicious locations (e.g., payphones, overseas). If a location is suspicious, the identified location information is used in the next evaluation step.

[1201] Input: Location information

[1202] Output: Parsed origin information

[1203] Step 7:

[1204] AI-powered fraud assessment

[1205] The server uses the voice data, detected keywords, and location information to evaluate the likelihood of fraud using an AI model (using TensorFlow and PyTorch).The AI ​​model outputs a score indicating the likelihood of fraud based on the preprocessed data.

[1206] Input: Voice data, detected keywords, location information

[1207] Output: Fraud rating score

[1208] Step 8:

[1209] Generate and send warning messages

[1210] The server generates a warning message based on the fraud assessment score, including a warning that the fraud is suspected and specific countermeasures, and sends the generated warning message to the terminal in real time.

[1211] Input: Fraud rating score

[1212] Output: Warning message

[1213] Step 9:

[1214] Receiving and Viewing Alerts

[1215] The terminal receives the warning message sent from the server and immediately notifies the user by voice, message notification, or real-time display on the terminal screen.

[1216] Input: warning message

[1217] Output: User notification

[1218] For example, here is a prompt:

[1219] For example, "Audio file:<audio_file> Origin: <location>Location information:<location_info> Detection Keywords:<keywords_detected> Scam rating score:<fraud_score> Warning message:<alarm_message> " is generated.

[1220] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1221] The present invention provides a system that detects "I'm your son" fraud calls in real time and issues a prompt warning, targeting elderly users and those prone to fraud. This system is an integrated system that includes voice data collection, voice analysis, keyword detection, caller location identification, fraud probability assessment using an AI model, and warning generation and notification. One embodiment of the present invention further includes an emotion engine for recognizing user emotions.

[1222] System Configuration

[1223] Server-side configuration

[1224] 1. Receiving audio data

[1225] The server receives the voice data transmitted from the terminal during the call in real time.

[1226] 2. Voice Recognition

[1227] The server uses a speech recognition module to convert the voice data into text data, which also includes noise reduction and echo cancellation.

[1228] 3. Keyword Discovery

[1229] The server analyzes the text data generated by the speech recognition to detect specific keywords, such as "money," "transfer," "accident," and "help."

[1230] 4. Identifying the source

[1231] The server analyzes the call origin information, using phone numbers, IP addresses, and GPS information to identify the origin, and if a suspicious location is detected, it records that information.

[1232] 5. Emotion Recognition by Emotion Engine

[1233] The server then inputs the voice data into an emotion engine to recognize the user's emotions. The emotion engine analyzes the tone, pitch, and tempo of the user's voice to determine emotions such as tension or anxiety.

[1234] 6. AI-powered fraud assessment

[1235] The server inputs the voice data, text data, detected keywords, location information, and emotional information recognized by the emotion engine into an AI model to assess the likelihood of fraud, with reference to past data and known fraud patterns.

[1236] 7. Generating Warning Messages

[1237] If the AI ​​model judges the possibility of fraud high, the server automatically generates a warning message after the call ends, which includes information about the suspected fraud and specific countermeasures.

[1238] 8. Sending Alerts

[1239] The server sends the generated warning message to the user's terminal.

[1240] Terminal configuration

[1241] 1. Collection of audio data

[1242] The device collects voice data during a call in real time and sends it to the server.

[1243] 2. Receiving a warning

[1244] When a warning message is sent from the server, the device notifies the user via voice call or message notification.

[1245] User-side configuration

[1246] 1. Check the contents of the call

[1247] The user will carry out the call as if it were a normal call.

[1248] 2. Check the warning

[1249] After the call ends, the user checks the warning message displayed on their device, which includes information about a suspected scam and specific countermeasures.

[1250] Specific examples

[1251] 1. The device initiates a call with elderly person D and his daughter.

[1252] 2. The device transmits the voice data to the server in real time.

[1253] 3. The server converts the speech into text and detects the keyword "money."

[1254] 4. The server analyzes the call's location information and determines that the call came from a public phone.

[1255] 5. The server inputs the voice data into the emotion engine and recognizes that Mr. D is feeling nervous or anxious.

[1256] 6. The server inputs this information into an AI model and rates the likelihood of fraud.

[1257] 7. After the call ends, the server automatically generates a warning message and sends it to the terminal.

[1258] 8. The device will send a warning message to Mr. D, who can then take immediate action.

[1259] In this way, the present invention automates a series of processes from collecting voice data to recognizing emotional information, assessing the possibility of fraud, and issuing a warning, thereby minimizing the risk of users becoming victims of fraud.

[1260] The processing flow will be explained below.

[1261] Step 1:

[1262] When a call is initiated, the device collects voice data in real time, filters the data during the call, and sends it to the server.

[1263] Step 2:

[1264] The server receives the voice data sent from the terminal in real time and inputs it into a voice recognition module for analysis.

[1265] Step 3:

[1266] The server converts the voice data into text data using a speech recognition module, which performs pre-processing such as noise reduction and echo cancellation.

[1267] Step 4:

[1268] The server analyzes the text data generated by the voice recognition and detects predetermined keywords, such as "money," "transfer," "accident," and "help." The detected keywords are recorded along with the location information.

[1269] Step 5:

[1270] The server analyzes the call origin information, identifying the origin using phone number, IP address, GPS information, etc., and if it detects any suspicious locations (such as public telephones or calls from abroad), it records that information.

[1271] Step 6:

[1272] The server inputs the voice data into an emotion engine to recognize the user's emotions. The emotion engine analyzes the voice tone, pitch, and tempo to determine the user's emotional state. For example, if the user is feeling nervous or anxious, that information is also recorded.

[1273] Step 7:

[1274] The server inputs voice data, text data, detected keywords, location information, and emotional information into the AI ​​model, which uses this information to assess the likelihood of fraud. It then uses past data and known fraud patterns to generate a score.

[1275] Step 8:

[1276] If the AI ​​model evaluates the likelihood of fraud at a certain level, the server generates a warning message after the call ends, which includes information about the suspected fraud and specific countermeasures.

[1277] Step 9:

[1278] The server sends the generated warning message to the user's terminal, and the sent warning message arrives at the terminal in real time.

[1279] Step 10:

[1280] When the device receives the warning message sent from the server, it notifies the user, which is displayed as a voice call or message notification.

[1281] Step 11:

[1282] The user checks the warning message on the device, understands the contents of the warning message, and takes necessary measures, such as rechecking the contents of calls that are suspected to be fraudulent and consulting the police or financial institution.

[1283] By taking the above steps, the system can quickly and accurately detect the possibility of "It's me" fraud and issue an immediate warning to the user, thus preventing fraud damage before it occurs.

[1284] Example 2

[1285] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1286] In modern society, elderly people and certain user groups are increasingly becoming victims of fraud such as "it's me" scams. These frauds are becoming more sophisticated and are often difficult to detect using normal means. As a result, victims are likely to suffer significant financial losses. To solve this problem, a system is needed that can accurately detect potential fraud in real time and issue a prompt warning.

[1287] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1288] In this invention, the server includes: a voice recognition means for converting voice data into text data; an analysis means for analyzing the text data and detecting specific keywords; a call origin identification means for analyzing call origin information; an AI model means for evaluating the possibility of fraud using the voice data, keywords, and call origin information; a warning generation means for generating a warning message based on the result of the fraud possibility evaluation and sending it to the terminal; a terminal means for notifying the user of the generated warning message; an emotion recognition means for analyzing the voice data and recognizing the user's emotion; and an AI model means for inputting the emotion information into the AI ​​model and evaluating the possibility of fraud with high accuracy. This makes it possible to minimize the risk of users becoming victims of fraud.

[1289] "Audio data" means data in digital form obtained by recording or transmitting sound.

[1290] A "terminal means" is an electronic device for collecting and transmitting voice data.

[1291] A "server" is a central processing unit that receives, processes, and stores data from multiple terminals over a network.

[1292] "Speech recognition means" is a technology that analyzes voice data and converts the content into text data.

[1293] The "analysis means" is a device or software that has the function of analyzing text data and detecting specific keywords.

[1294] The "originating location identification means" is a technology that analyzes the originating location information of a call and identifies the area where the call was made.

[1295] An "AI modeling tool" is a device or software that uses artificial intelligence to assess the likelihood of fraud.

[1296] A "warning generator" is a device or software capable of generating a warning message based on the result of assessing the likelihood of fraud.

[1297] "Emotion recognition means" is a technology that analyzes voice data to determine the user's emotions.

[1298] A "warning message" is a message that notifies the user in case of possible fraud.

[1299] "User" means an individual or organization that uses the system.

[1300] This invention provides a system that detects "It's my son" fraud calls in real time and issues a prompt warning, targeting elderly people and other users who are prone to fraud. This system can automatically carry out a series of processes from collecting voice data to issuing a warning.

[1301] System Components

[1302] Server-side configuration

[1303] Receiving audio data:

[1304] The server receives the voice data sent from the device in real time during the call. Specifically, a WebSocket connection is used to exchange data bidirectionally between the device and the server.

[1305] Voice Recognition:

[1306] The server uses the Google Speech-to-Text API or a similar service to convert the received voice data into text data, and during this process, it also performs noise reduction and echo cancellation to generate highly accurate text data.

[1307] Keyword detection:

[1308] The server performs text analysis to detect specific keywords (e.g., "money," "transfer," "accident," "help," etc.) using regular expressions and pre-defined keyword lists.

[1309] Identifying the source:

[1310] The server analyzes the call origin information, using phone numbers, IP addresses, GPS information, etc. to detect calls from suspicious locations such as public phones.

[1311] Emotion recognition:

[1312] The server inputs the voice data into an emotion engine (such as Affectiva's emotion recognition API) that analyzes the tone, pitch, and tempo of the user's voice to determine their emotion. Based on this information, it can determine whether the user is feeling nervous or anxious.

[1313] AI Fraud Assessment:

[1314] The server inputs voice data, text data, keywords, source information, and emotional information into an AI model to assess the likelihood of fraud, which also refers to past data and known fraud patterns.

[1315] Generates a warning message:

[1316] If the AI ​​model judges the likelihood of fraud high, the server automatically generates a warning message containing the suspected fraud and specific countermeasures.

[1317] Sending alerts:

[1318] The server generates a warning message and sends it to the user's device. The message is promptly sent to the user via push notification or SMS.

[1319] Terminal configuration

[1320] Audio data collection:

[1321] The device collects voice data during a call in real time and immediately sends it to the server. It is set to start collecting voice data as soon as the call starts.

[1322] Receiving an alert:

[1323] The device receives a warning message from the server and notifies the user. Notification methods include voice calls and message notifications, and the user is alerted in a way that can be checked immediately.

[1324] User-side configuration

[1325] Check the call details:

[1326] Users make calls as usual, the system runs in the background and no special action is required.

[1327] Check for warnings:

[1328] After the call ends, the user can view the warning message on their device and take appropriate action. The message will include information about the suspected fraud and specific countermeasures.

[1329] Specific examples

[1330] If an elderly person named Mr. D is told by someone claiming to be his son that he needs money, the system works as follows:

[1331] 1. The device collects the conversation between Mr. D and the other party and sends the voice data to the server.

[1332] 2. The server uses a speech recognition module to convert this speech into text and detect the keyword "money."

[1333] 3. The server uses the call location identification means to identify that the call is coming from a public telephone.

[1334] 4. The server inputs the voice data into an emotion recognition engine and recognizes that Mr. D is feeling nervous or anxious.

[1335] 5. The server inputs this information into an AI model and scores the likelihood of fraud.

[1336] 6. After the call ends, the server generates a warning message and sends it to the terminal.

[1337] 7. The device will send a warning message to Mr. D, who can then take immediate action.

[1338] This system automates the entire process from collecting voice data to recognizing emotional information, assessing the likelihood of fraud, and issuing warning notifications, thereby minimizing the risk of users becoming victims of fraud.

[1339] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1340] Step 1:

[1341] Collection and transmission of voice data

[1342] The device collects audio data during the call in real time. Specifically, it uses the device's microphone to capture the audio of the call and sends the audio data to the server via WebSocket or HTTP. The input of this step is the audio data collected from the device's microphone, and the output is the audio data sent to the server.

[1343] Step 2:

[1344] Receiving audio data

[1345] The server receives the audio data sent from the device in real time. Specifically, it starts a WebSocket server to receive the audio data. The input of this step is the audio data sent from the device, and the output is the audio data stored in a temporary buffer on the server.

[1346] Step 3:

[1347] Voice Recognition

[1348] The server converts the received voice data into text data using a speech recognition module such as the Google Speech-to-Text API. During this process, noise reduction and echo cancellation are also performed to improve analysis accuracy. The input for this step is the voice data stored on the server, and the output is the result of converting the voice data into text data.

[1349] Step 4:

[1350] Keyword detection

[1351] The server analyzes the generated text data and detects specific keywords (e.g., "money," "transfer," "accident," "help," etc.). Specifically, the analysis is performed using regular expressions or a pre-defined keyword list. The input of this step is the text data, and the output is a list of detected keywords.

[1352] Step 5:

[1353] Identifying the source

[1354] The server analyzes the call's origin information. Specifically, it identifies the call's origin using the phone number, IP address, GPS information, etc. The input to this step is the metadata associated with the call, and the output is the origin information. If the call originates from a suspicious location, the location information is also included.

[1355] Step 6:

[1356] emotion recognition

[1357] The server inputs the voice data into an emotion recognition engine (e.g., Affectiva's emotion recognition API) and analyzes the tone, pitch, and tempo of the user's voice to determine their emotion. The input of this step is the voice data, and the output is the user's emotional information (e.g., tension, fear, relief, etc.).

[1358] Step 7:

[1359] AI-powered fraud assessment

[1360] The server inputs the voice data, text data, detected keywords, origin information, and emotion information into the AI ​​model to assess the likelihood of fraud. The input for this step is all of this data, and the output is a fraud risk assessment score. The AI ​​model also references known fraud patterns and past data to make the assessment.

[1361] Step 8:

[1362] Generate a warning message

[1363] If the AI ​​model evaluates the likelihood of fraud highly, the server automatically generates a warning message. This message includes the suspected fraud and specific countermeasures, and is provided to the user. The input of this step is the fraud risk assessment score, and the output is the generated warning message.

[1364] Step 9:

[1365] Sending an Alert

[1366] The server sends the generated warning message to the user's device. Specifically, it ensures that the warning message reaches the user promptly via push notification or SMS. The input of this step is the warning message, and the output is the warning message sent to the user's device.

[1367] Step 10:

[1368] Receiving and Acknowledging Alerts

[1369] The terminal receives the warning message from the server and notifies the user. Specifically, it notifies the user of the warning using a notification bar, a pop-up, or a sound notification. The input of this step is the warning message sent from the server, and the output is the warning message displayed in a format that can be confirmed by the user.

[1370] Step 11:

[1371] User response

[1372] The user checks the warning message sent to the device and takes appropriate action, such as reading the warning content, reporting it to the appropriate authorities, or ending the call. The input to this step is the warning message, and the output is the user's specific action.

[1373] Through these steps, the system can efficiently collect, analyze, evaluate, and warn users of voice data, minimizing the risk of fraud.

[1374] (Application example 2)

[1375] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1376] In recent years, there has been an increase in "I'm your son" frauds targeting elderly people and other users who are prone to fraud, creating a need for rapid and effective countermeasures. However, conventional systems have had difficulty detecting potential fraud in real time during voice calls and issuing prompt warnings to users. In addition, evaluations based solely on voice data have the problem of low detection accuracy and a high number of false positives.

[1377] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a voice recognition means for converting voice data into text data, an analysis means for analyzing the text data and detecting specific keywords, a call origin identification means for analyzing the call origin information, and an AI model means for evaluating the possibility of fraud using the voice data, keywords, call origin information, and user emotion data. This makes it possible to prevent elderly users and users who are prone to fraud from becoming victims of fraud.

[1378] The "terminal means" is a device that has the function of collecting voice data and transmitting it to a server.

[1379] A "voice recognition means" is a device or system that has the function of converting voice data into text data.

[1380] The "analysis means" is a system that has the function of analyzing text data and detecting specific keywords.

[1381] The "originating point identification means" is a system that has the function of analyzing the originating point information of a call and identifying the originating point.

[1382] The "AI model means" is a system that has the function of assessing the possibility of fraud using voice data, keywords, origin information, and user emotional data.

[1383] The "warning generation means" is a system that has the function of generating a warning message based on the result of assessing the possibility of fraud and sending it to the terminal.

[1384] "Emotion data" is information that represents the user's emotional state and is obtained by analyzing voice data.

[1385] This invention provides a system that detects fraudulent calls in real time and issues prompt warnings to elderly users and other users who are prone to fraud. This system is an integrated system that includes voice data collection, voice analysis, keyword detection, caller location identification, fraud probability assessment using an AI model, and warning generation and notification.

[1386] First, the terminal collects voice data and sends it to the server. The server converts the voice data into text data using a voice recognition means. The voice recognition means includes noise reduction and echo cancellation functions. Next, the server analyzes the text data to detect specific keywords. This analysis means detects keywords related to fraud (e.g., "money," "transfer," "accident," "help," etc.).

[1387] Furthermore, the server analyzes the call origination information and identifies suspicious locations using a call origination location identification means. The server also inputs the voice data into an emotion recognition engine to analyze the user's emotion data. This emotion data identifies emotions such as tension or anxiety based on the user's voice tone, pitch, tempo, etc.

[1388] This data (voice data, keywords, origin information, and emotion data) is input into the AI ​​model means to evaluate the likelihood of fraud. The AI ​​model means makes the evaluation by referring to past data and known fraud patterns. If it is evaluated that there is a high likelihood of fraud, the server generates a warning message and sends it to the terminal using the warning generation means. The terminal notifies the user of the generated warning message. This notification may take the form of a voice call or a message notification.

[1389] As a concrete example, consider a situation in which a device initiates a call between an elderly user and someone else. At this time, the device sends voice data to a server in real time. The server converts the voice to text and detects the keyword "money." The server then analyzes the call's origin information and determines that it is coming from a public phone. The server inputs the voice data into an emotion engine and recognizes that the user is feeling nervous or anxious. The server then inputs this information into an AI model and evaluates the likelihood of fraud. After the call ends, the server automatically generates a warning message and sends it to the device. The device notifies the user of the warning message, allowing them to take immediate action.

[1390] The server also handles notifications to notify the generated warning messages. The following is an example of a specific prompt to the AI ​​model:

[1391] "The following keywords were detected in this call: 'money', 'transfer'. The call originated from an unknown pay phone. The user's emotions are very tense. What is the likelihood of this being a scam?"

[1392] The present invention automates this series of processes, minimizing the risk of users falling victim to fraud.

[1393] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1394] Step 1:

[1395] The terminal collects voice data during a call. The collected voice data is sent to the server in real time. This allows the server to receive the user's call data in real time. The input is the call voice data, and the output is the voice data sent to the server.

[1396] Step 2:

[1397] The server converts the received voice data into text data using a voice recognition means. During this process, noise reduction and echo cancellation are performed to improve the accuracy of the voice data. The input is voice data, and the output is text data.

[1398] Step 3:

[1399] The server analyzes the text data generated by the speech recognition to detect specific keywords. Using the analysis means, it detects fraud-related keywords such as "money," "transfer," "accident," and "help." The input is the text data, and the output is a list of detected keywords.

[1400] Step 4:

[1401] The server analyzes the call origination information and uses the call origination identification means to identify suspicious locations by analyzing the phone number, IP address, and GPS information. The input is the call origination information, and the output is the analyzed call origination information.

[1402] Step 5:

[1403] The server inputs the voice data into an emotion recognition engine and analyzes the user's emotional data. The emotion recognition engine analyzes the tone, pitch, and tempo of the user's voice to determine emotions such as tension or anxiety. The input is voice data, and the output is the user's emotional data.

[1404] Step 6:

[1405] The server inputs the voice data, keywords, origin information, and emotion data into the AI ​​model means to evaluate the likelihood of fraud. The AI ​​model means makes an evaluation by referring to past data and known fraud patterns. The inputs are the voice data, keywords, origin information, and emotion data, and the output is the result of the evaluation of the likelihood of fraud.

[1406] Step 7:

[1407] If the server judges the possibility of fraud to be high, it generates a warning message using the warning generation means and sends it to the terminal. The generated warning message includes information about the suspected fraud and specific countermeasures. The input is the result of the fraud possibility assessment, and the output is the warning message.

[1408] Step 8:

[1409] The terminal receives the warning message sent from the server and notifies the user. The notification is in the form of a voice call or a message notification. The input is the warning message, and the output is the notification to the user.

[1410] Through the above series of processing steps, users can detect fraudulent calls in real time and take prompt action.

[1411] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1412] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1413] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1414] [Fourth embodiment]

[1415] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1416] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1417] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1418] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1419] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1420] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1421] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1422] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1423] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1424] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1425] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1426] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1427] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1428] This invention provides a system that detects "I am your son" fraud calls in real time and issues a prompt warning, targeting elderly people and other users who are prone to fraud. This system is an integrated system that includes voice data collection, voice analysis, keyword detection, caller location identification, fraud probability assessment using an AI model, and warning generation and notification.

[1429] System Configuration

[1430] Server-side configuration

[1431] 1. Receiving audio data

[1432] The server receives real-time voice data during a call, which is sent from the terminal.

[1433] 2. Voice Recognition

[1434] The server analyzes the received voice data using a speech recognition module and converts it into text data, with noise reduction and echo cancellation also being applied during this process.

[1435] 3. Keyword Discovery

[1436] The server analyzes the text data generated by the speech recognition to detect specific keywords, such as "money," "transfer," "accident," and "help."

[1437] 4. Identifying the source

[1438] The server analyzes the call origin information and if it detects any suspicious locations (such as public phones or calls from abroad), it records that information.

[1439] 5. AI-powered fraud assessment

[1440] The server inputs the voice data, detected keywords, and location information into an AI model to assess the likelihood of fraud, with reference to past data and known fraud patterns.

[1441] 6. Generating Warning Messages

[1442] If the server assesses a high probability of fraud, it will automatically generate a warning message after the call ends, including information about the suspected fraud and what steps should be taken.

[1443] 7. Sending Alerts

[1444] The server sends the generated warning message to the user's terminal.

[1445] Terminal configuration

[1446] 1. Collection of audio data

[1447] The device collects voice data during a call in real time and sends it to the server.

[1448] 2. Receiving a warning

[1449] When a warning message is sent from the server, the device notifies the user via voice call or message notification.

[1450] User-side configuration

[1451] 1. Check the contents of the call

[1452] The user will carry out the call as if it were a normal call.

[1453] 2. Check the warning

[1454] After the call ends, the user checks the warning message displayed on the device, which includes information about the suspected fraud and specific countermeasures.

[1455] Specific examples

[1456] 1. The device starts a call between elderly person C and the person in charge.

[1457] 2. The device transmits the voice data to the server in real time.

[1458] 3. The server converts the speech into text and detects the keyword "money."

[1459] 4. The server analyzes the call's location information and determines that the call originated from a domestic public telephone.

[1460] 5. The server inputs this information into an AI model and rates the likelihood of fraud.

[1461] 6. After the call ends, the server automatically generates a warning message and sends it to the terminal.

[1462] 7. The device will notify Mr. C of the warning message, so that he can immediately pay attention and take the necessary action.

[1463] In this way, the present invention can automate a series of processes from collecting voice data to evaluating the possibility of fraud and issuing a warning, thereby minimizing the risk of users becoming victims of fraud.

[1464] The processing flow will be explained below.

[1465] Step 1:

[1466] When a call is initiated, the device collects voice data in real time and transmits the voice data during the call to the server as is.

[1467] Step 2:

[1468] The server receives the voice data sent from the terminal and inputs it into a voice recognition module for real-time processing.

[1469] Step 3:

[1470] The server uses a speech recognition module to convert the voice data into text data, and pre-processing such as noise reduction and echo cancellation is performed during this process.

[1471] Step 4:

[1472] The server analyzes the text data and detects certain keywords, such as "money," "transfer," "accident," and "help."

[1473] Step 5:

[1474] The server analyzes the call origin information, identifying the origin using phone number, IP address, GPS information, etc., and records any suspicious locations (such as public telephones or calls from overseas).

[1475] Step 6:

[1476] The server inputs the voice data, text data, detected keywords, and location information into the AI ​​model, which then evaluates the likelihood of fraud based on this information.

[1477] Step 7:

[1478] If the server determines that there is a high possibility of fraud based on the fraud probability score evaluated by the AI ​​model, it will generate a warning message after the call ends, notifying the caller of the suspected fraud and providing specific countermeasures.

[1479] Step 8:

[1480] The server sends the generated warning message to the user's terminal, and the message arrives at the terminal in real time.

[1481] Step 9:

[1482] When the device receives the warning message sent from the server, it notifies the user, which is displayed as a voice call or a message.

[1483] Step 10:

[1484] The user checks the warning message on the device, understands the contents of the warning message, and takes necessary measures, such as rechecking the contents of calls that are suspected to be fraudulent and consulting the police or financial institution.

[1485] Through these steps, the system can quickly and accurately detect possible telephone fraud and issue an immediate warning to the user.

[1486] Example 1

[1487] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1488] There is a challenge in detecting "I'm your son" fraud calls in real time and issuing prompt warnings, targeting elderly people and other users who are prone to fraud. Conventional systems have difficulty immediately detecting potential fraud and issuing warnings, putting users at risk of becoming victims. In particular, since calls from suspicious locations and specific keywords are often not detected, there is a need for a means to prevent fraud in advance.

[1489] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1490] In this invention, the server includes terminal means for collecting voice data and transmitting it to the server, voice recognition means for converting the voice data into text data, analysis means for analyzing the text data and detecting specific keywords, call origination location identification means for analyzing call origination location information, voice processing means for performing noise reduction and echo cancellation, AI model means for evaluating the possibility of fraud using the voice data, keywords, and call origination location information, warning generation means for generating a warning message based on the result of the evaluation of the possibility of fraud and transmitting it to the terminal, and terminal means for notifying the user of the generated warning message. This makes it possible to detect fraudulent calls in real time and quickly issue a warning to the user.

[1491] "Voice data" refers to data converted into digital form of voice in a conversation between a user and another person.

[1492] "Terminal means" refers to a device that has the function of collecting voice data and transmitting it to a server.

[1493] "Speech recognition means" refers to software or hardware that has the function of analyzing received voice data and converting it into text data.

[1494] "Text data" refers to character string information generated from voice data by a voice recognition means.

[1495] "Analysis means" refers to software or hardware that has the function of analyzing text data and detecting specific keywords.

[1496] "Call origin identification means" refers to software or hardware that has the function of analyzing call origin information and identifying suspicious locations.

[1497] "Audio processing means" refers to software or hardware that performs noise reduction and echo cancellation and optimizes audio data.

[1498] "AI model means" refers to an artificial intelligence model that uses voice data, keywords, and location information to assess the likelihood of fraud.

[1499] "Warning generation means" refers to software or hardware that has the function of generating a warning message based on the result of assessing the likelihood of fraud and sending it to a terminal.

[1500] "Warning Message" means notification information containing warnings and countermeasures regarding potentially fraudulent calls.

[1501] This invention provides a system that detects "I am your son" fraud calls in real time and issues a prompt warning, targeting elderly people and other users who are prone to fraud. This system is an integrated system that includes voice data collection, voice analysis, keyword detection, caller location identification, fraud probability assessment using an AI model, and warning generation and notification.

[1502] Server-side configuration

[1503] 1. Receiving audio data

[1504] The server receives real-time audio data during a call. This audio data is sent from the device. The device collects audio during the call with a microphone and encodes it digitally using an AudioCodec. The encoded audio data is then sent to the server using HTTP or WebSocket protocols.

[1505] 2. Voice Recognition

[1506] The server analyzes the received voice data and converts it into text data using a speech recognition module. This process uses voice recognition software such as Google Cloud Speech-to-Text or IBM Watson. The server pre-processes the voice data by performing noise reduction and echo cancellation.

[1507] 3. Keyword Discovery

[1508] The server analyzes the text data generated by the speech recognition to detect specific keywords, such as "money," "transfer," "accident," and "help," using regular expressions and keyword matching algorithms.

[1509] 4. Identifying the source

[1510] The server analyzes the call origination information, using location analysis tools such as GeoIP databases to identify suspicious locations (e.g., domestic or international pay phones, or calls from abroad).

[1511] 5. AI-powered fraud assessment

[1512] The server inputs the voice data, detected keywords, and location information into an AI model to assess the likelihood of fraud. The AI ​​model was built using TensorFlow and PyTorch and is trained on past data and known fraud patterns.

[1513] 6. Generating Warning Messages

[1514] If the server assesses a high probability of fraud, it will automatically generate a warning message detailing the suspected fraud and recommending actions to take.

[1515] 7. Sending Alerts

[1516] The server then sends the generated warning message to the user's device via SMS, in-app notifications, etc.

[1517] Terminal configuration

[1518] 1. Collection of audio data

[1519] The device collects voice data during a call in real time and sends it to a server, where it is digitally encoded and transferred to the server using a protocol.

[1520] 2. Receiving a warning

[1521] The device receives the warning message from the server and notifies the user in the form of a pop-up message, a ringtone, or vibration.

[1522] User-side configuration

[1523] 1. Check the contents of the call

[1524] The user will be able to talk as if they were in a normal conversation. There is no need for them to be particularly conscious of it, but they will be able to talk naturally.

[1525] 2. Check the warning

[1526] After the call ends, the user checks the warning message that is sent to the terminal. The warning message contains information about the suspected fraud and specific countermeasures.

[1527] Specific examples

[1528] The device initiates a call between elderly person C and the person in charge. The device sends the voice data to the server in real time. The server converts the voice to text and detects the keyword "money." The server analyzes the call's origin information and determines that the call originated from a domestic public phone. The server inputs this information into an AI model and rates the possibility of fraud highly. After the call ends, the server automatically generates a warning message and sends it to the device. The device notifies C of the warning message, allowing him to immediately pay attention and take the necessary action.

[1529] In this way, the present invention can automate a series of processes from collecting voice data to evaluating the possibility of fraud and issuing a warning, thereby minimizing the risk of users becoming victims of fraud.

[1530] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1531] Step 1: Real-time collection and transmission of audio data

[1532] The device collects voice data during a call in real time, converts it into a digital voice data format (e.g., PCM format), compresses it using an AudioCodec, and transmits the data to the server using HTTP or WebSocket protocols.

[1533] Input: Audio during a call

[1534] Data processing: audio data capture and compression

[1535] Output: Digital audio data

[1536] Specific operation: A user starts a call on their smartphone, and the audio is collected by the device's microphone, compressed, and then sent to the server.

[1537] Step 2: Receiving and Preprocessing Audio Data

[1538] The server analyzes the received voice data in real time. First, it performs noise reduction and echo cancellation to improve the sound quality. Then, it converts the voice data into text using voice recognition software such as Google Cloud Speech-to-Text or IBM Watson.

[1539] Input: Digital audio data (sent to the server)

[1540] Data processing: noise reduction, echo cancellation, voice recognition

[1541] Output: Text data

[1542] Specific operation: The server calls the Google Cloud Speech-to-Text API and converts the spoken voice data of "Help, please help me" into the text "Help, please help me."

[1543] Step 3: Keyword Discovery

[1544] The server analyzes the text data generated by speech recognition to detect specific keywords (e.g., "money," "transfer," "accident," "help," etc.) using regular expressions and keyword matching algorithms.

[1545] Input: Text data converted by speech recognition

[1546] Data processing: Keyword detection using regular expressions and keyword matching

[1547] Output: A list of detected keywords

[1548] Specific operation: The server detects the keywords "money" and "transfer" from the text data "Please transfer money."

[1549] Step 4: Identify the source

[1550] The server analyzes the call origination information, using location analysis tools such as GeoIP databases to identify suspicious locations (e.g., international locations, pay phones).

[1551] Input: Data about the call's origin (IP address and GPS data)

[1552] Data processing: location information analysis

[1553] Output: Origination information

[1554] What it does: The server analyzes the IP address and determines that the caller is calling from overseas.

[1555] Step 5: AI fraud assessment

[1556] The server inputs the voice data, detected keywords, and location information into an AI model to assess the likelihood of fraud. The AI ​​model was built using TensorFlow and PyTorch and is trained on past data and known fraud patterns.

[1557] Input: Voice data, keywords, location information

[1558] Data manipulation: Assessing the likelihood of fraud with AI models

[1559] Output: Fraud probability assessment result

[1560] Specific operation: The server inputs the information "transfer," "help," and the origin of the call into the AI ​​model, and assesses the possibility of fraud.

[1561] Step 6: Generate a warning message

[1562] If the server assesses a high probability of fraud, it will automatically generate a warning message detailing the suspected fraud and recommending actions to take.

[1563] Input: Fraud probability assessment result

[1564] Data processing: Generate warning messages

[1565] Output: Warning message

[1566] Specific behavior: The server generates a warning message saying, "This call may be fraudulent. Please hang up immediately and report it to the police."

[1567] Step 7: Sending and notifying warning messages

[1568] The server sends the generated warning message to the user's device via SMS, in-app notification, etc. The device then notifies the user of the warning message received from the server via a pop-up message, ringtone, or vibration.

[1569] Input: warning message

[1570] Data Processing: Sending a warning message

[1571] Output: User notification

[1572] Specific operation: The server sends a warning message to the device as an SMS, and the device rings and notifies the user, "This is a potentially fraudulent call. Please stop the call and exercise caution."

[1573] Step 8: Review user warnings

[1574] After the call ends, the user checks the warning message displayed on the device, which includes information about the suspected fraud and specific countermeasures.

[1575] Input: warning message

[1576] Data processing: Check the warning message and take corrective action

[1577] Output: Response to suspected fraudulent calls

[1578] Specific actions: The user checks the warning message displayed on the device and then reports the incident to the police.

[1579] (Application example 1)

[1580] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1581] In recent years, fraudsters targeting elderly people and those prone to fraud have become increasingly sophisticated, resulting in the increasing incidence of frauds attempting to defraud users of money over the phone. To prevent such frauds, a system is needed that can detect possible fraud in real time during a call and quickly issue a warning to the user. However, current systems lack the functionality to instantly analyze and evaluate the content of calls and notify the user in real time. This creates a time lag between fraud detection and notification, making it difficult to prevent frauds from occurring.

[1582] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1583] In this invention, the server includes a voice recognition means for converting voice data into text data, an analysis means for analyzing the text data and detecting specific keywords, a call origination identification means for analyzing the call origination information, an AI model means for evaluating the possibility of fraud using the voice data, keywords, and call origination information, a warning generation means for generating a warning message based on the result of the fraud possibility evaluation and sending it to the terminal, and a terminal means for notifying the user of the generated warning message and displaying the contents of the warning message to the user in real time. This makes it possible to analyze voice data during a call in real time and immediately notify the user if there is a high possibility of fraud. This is expected to have the effect of preventing fraud damage before it occurs.

[1584] "Voice data" refers to electrical signals of voice obtained during a call, and is converted into text data by a voice recognition means.

[1585] "Terminal means" refers to a smartphone or other communication terminal used by a user, which is a device that collects voice data and transmits it to a server.

[1586] A "server" is a computer system that receives, analyzes, and evaluates voice data transmitted from a terminal means via a network.

[1587] "Speech recognition means" refers to a process or device that has the function of converting received voice data into text data, and performs noise reduction and echo cancellation processing.

[1588] The "analysis means" is a process or device for analyzing the converted text data and detecting specific keywords.

[1589] A "keyword" is a specific linguistic unit that suggests the possibility of fraud, such as "money," "transfer," "accident," or "help."

[1590] A "call origin determination means" is a process or device that analyzes the call origin information and identifies suspicious locations (e.g., calls from public phones or overseas).

[1591] An "AI model means" is a process or device that implements an artificial intelligence model to assess the likelihood of fraud using voice data, keywords, and origin information.

[1592] An "alert generator" is a process or device that generates an alert message based on the result of assessing the likelihood of fraud and sends it to the terminal.

[1593] A "warning message" is information intended to notify the user of a possible fraud and includes recommended actions to be taken.

[1594] "Real-time notification" is the process or function of immediately communicating generated alert messages to the user.

[1595] The system of the present invention has the function of detecting possible fraud during a call in real time and issuing a prompt warning to the user. This system is mainly composed of server-side processing and terminal-side processing.

[1596] Server-side configuration

[1597] 1. Receiving audio data

[1598] The server receives the voice data sent from the terminal in real time. The voice data is collected during the call.

[1599] 2. Voice Recognition

[1600] The server analyzes the received voice data using speech recognition and converts it into text data, using services such as the Google Cloud Speech-to-Text API, and also applies noise reduction and echo cancellation.

[1601] 3. Keyword Discovery

[1602] The converted text data is analyzed using analytical tools to detect specific keywords such as "money," "transfer," "accident," and "help." Python and SpaCy can be used for the analysis.

[1603] 4. Identifying the source

[1604] The server analyzes the call origin information using GeoIP, and if it detects a suspicious location (such as a public phone or a call from abroad), it records that information.

[1605] 5. AI-powered fraud assessment

[1606] The server inputs the voice data, detected keywords, and location information into an AI model using TensorFlow and PyTorch to assess the likelihood of fraud, with reference to past data and known fraud patterns.

[1607] 6. Generating and Sending Warning Messages

[1608] If the server judges that there is a high possibility of fraud, it generates a warning message by a warning generation means and sends it to the terminal. The warning message includes information about the suspected fraud and measures to be taken.

[1609] Terminal configuration

[1610] 1. Collection of audio data

[1611] The terminal has the ability to collect voice data during a call in real time and send it to a server.

[1612] 2. Receiving and Displaying Alerts

[1613] The terminal receives the warning message sent from the server and immediately notifies the user of the message by voice, message notification, or real-time display on the terminal screen.

[1614] User-side configuration

[1615] 1. Check the contents of the call

[1616] The user does not need to perform any special operations during the call, and the call proceeds as normal. The call content is automatically analyzed.

[1617] 2. Checking and responding to warnings

[1618] When a warning message is sent to the terminal, the user can check it and take appropriate measures.

[1619] Specific examples

[1620] For example, if an elderly person named A receives a phone call that they suspect is a scam, the following steps would be taken:

[1621] 1. Mr. A's smartphone sends the audio during the call to the server.

[1622] 2. The server converts the speech into text and detects keywords such as "money" and "transfer."

[1623] 3. The call origin is identified as a pay phone.

[1624] 4. AI models will assess the likelihood of fraud.

[1625] 5. A warning message will be sent to Mr. A's smartphone immediately during or after the call ends.

[1626] 6. Person A is able to accept the warning, pay immediate attention, and take the necessary action.

[1627] Prompt Sentence Examples

[1628] text

[1629] Audio files:<audio_file>

[1630] Origin: <location>

[1631] Location information:<location_info>

[1632] Detection Keywords:<keywords_detected>

[1633] Scam rating score:<fraud_score>

[1634] Warning message:<alarm_message>

[1635] In this way, the system of the present invention automatically performs a series of processes from collecting voice data to assessing the possibility of fraud and issuing a warning, thereby minimizing the risk of users becoming victims of fraud.

[1636] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1637] Step 1:

[1638] Audio data collection

[1639] The device collects voice data in real time during a call. This voice data is picked up by the device's microphone from the start to the end of the call. The collected voice data is temporarily stored in the device's memory as is.

[1640] Input: Voice data during a call

[1641] Output: Collected audio data

[1642] Step 2:

[1643] Sending audio data

[1644] The terminal sends the collected voice data to the server. The communication protocol is secure HTTP (HTTPS). The data sent includes voice data and origin information (e.g., location information obtained from GPS).

[1645] Input: Collected voice data, location information

[1646] Output: Sending voice data and location information to the server

[1647] Step 3:

[1648] Receiving audio data

[1649] The server receives the voice data sent from the device in real time and temporarily stores the received data in the server's storage.

[1650] Input: Voice data sent from the device, location information

[1651] Output: Audio data and location information stored on the server

[1652] Step 4:

[1653] Voice Recognition

[1654] The server converts the stored audio data into text using the Google Cloud Speech-to-Text API, applying noise reduction and echo cancellation. The converted text data is then passed on to the next analysis step.

[1655] Input: Audio data

[1656] Output: Text data

[1657] Step 5:

[1658] Keyword detection

[1659] The server analyzes the text data using Python and SpaCy to detect specific keywords (e.g., money, transfer, accident, help). The detected keywords are used in the next evaluation step.

[1660] Input: Text data

[1661] Output: Detected keywords

[1662] Step 6:

[1663] Identifying the source

[1664] The server analyzes the origin information using GeoIP to identify suspicious locations (e.g., payphones, overseas). If a location is suspicious, the identified location information is used in the next evaluation step.

[1665] Input: Location information

[1666] Output: Parsed origin information

[1667] Step 7:

[1668] AI-powered fraud assessment

[1669] The server uses the voice data, detected keywords, and location information to evaluate the likelihood of fraud using an AI model (using TensorFlow and PyTorch).The AI ​​model outputs a score indicating the likelihood of fraud based on the preprocessed data.

[1670] Input: Voice data, detected keywords, location information

[1671] Output: Fraud rating score

[1672] Step 8:

[1673] Generate and send warning messages

[1674] The server generates a warning message based on the fraud assessment score, including a warning that the fraud is suspected and specific countermeasures, and sends the generated warning message to the terminal in real time.

[1675] Input: Fraud rating score

[1676] Output: Warning message

[1677] Step 9:

[1678] Receiving and Viewing Alerts

[1679] The terminal receives the warning message sent from the server and immediately notifies the user by voice, message notification, or real-time display on the terminal screen.

[1680] Input: warning message

[1681] Output: User notification

[1682] For example, here is a prompt:

[1683] For example, "Audio file:<audio_file> Origin: <location>Location information:<location_info> Detection Keywords:<keywords_detected> Scam rating score:<fraud_score> Warning message:<alarm_message> " is generated.

[1684] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1685] The present invention provides a system that detects "I'm your son" fraud calls in real time and issues a prompt warning, targeting elderly users and those prone to fraud. This system is an integrated system that includes voice data collection, voice analysis, keyword detection, caller location identification, fraud probability assessment using an AI model, and warning generation and notification. One embodiment of the present invention further includes an emotion engine for recognizing user emotions.

[1686] System Configuration

[1687] Server-side configuration

[1688] 1. Receiving audio data

[1689] The server receives the voice data transmitted from the terminal during the call in real time.

[1690] 2. Voice Recognition

[1691] The server uses a speech recognition module to convert the voice data into text data, which also includes noise reduction and echo cancellation.

[1692] 3. Keyword Discovery

[1693] The server analyzes the text data generated by the speech recognition to detect specific keywords, such as "money," "transfer," "accident," and "help."

[1694] 4. Identifying the source

[1695] The server analyzes the call origin information, using phone numbers, IP addresses, and GPS information to identify the origin, and if a suspicious location is detected, it records that information.

[1696] 5. Emotion Recognition by Emotion Engine

[1697] The server then inputs the voice data into an emotion engine to recognize the user's emotions. The emotion engine analyzes the tone, pitch, and tempo of the user's voice to determine emotions such as tension or anxiety.

[1698] 6. AI-powered fraud assessment

[1699] The server inputs the voice data, text data, detected keywords, location information, and emotional information recognized by the emotion engine into an AI model to assess the likelihood of fraud, with reference to past data and known fraud patterns.

[1700] 7. Generating Warning Messages

[1701] If the AI ​​model judges the possibility of fraud high, the server automatically generates a warning message after the call ends, which includes information about the suspected fraud and specific countermeasures.

[1702] 8. Sending Alerts

[1703] The server sends the generated warning message to the user's terminal.

[1704] Terminal configuration

[1705] 1. Collection of audio data

[1706] The device collects voice data during a call in real time and sends it to the server.

[1707] 2. Receiving a warning

[1708] When a warning message is sent from the server, the device notifies the user via voice call or message notification.

[1709] User-side configuration

[1710] 1. Check the contents of the call

[1711] The user will carry out the call as if it were a normal call.

[1712] 2. Check the warning

[1713] After the call ends, the user checks the warning message displayed on their device, which includes information about a suspected scam and specific countermeasures.

[1714] Specific examples

[1715] 1. The device initiates a call with elderly person D and his daughter.

[1716] 2. The device transmits the voice data to the server in real time.

[1717] 3. The server converts the speech into text and detects the keyword "money."

[1718] 4. The server analyzes the call's location information and determines that the call came from a public phone.

[1719] 5. The server inputs the voice data into the emotion engine and recognizes that Mr. D is feeling nervous or anxious.

[1720] 6. The server inputs this information into an AI model and rates the likelihood of fraud.

[1721] 7. After the call ends, the server automatically generates a warning message and sends it to the terminal.

[1722] 8. The device will send a warning message to Mr. D, who can then take immediate action.

[1723] In this way, the present invention automates a series of processes from collecting voice data to recognizing emotional information, assessing the possibility of fraud, and issuing a warning, thereby minimizing the risk of users becoming victims of fraud.

[1724] The processing flow will be explained below.

[1725] Step 1:

[1726] When a call is initiated, the device collects voice data in real time, filters the data during the call, and sends it to the server.

[1727] Step 2:

[1728] The server receives the voice data sent from the terminal in real time and inputs it into a voice recognition module for analysis.

[1729] Step 3:

[1730] The server converts the voice data into text data using a speech recognition module, which performs pre-processing such as noise reduction and echo cancellation.

[1731] Step 4:

[1732] The server analyzes the text data generated by the voice recognition and detects predetermined keywords, such as "money," "transfer," "accident," and "help." The detected keywords are recorded along with the location information.

[1733] Step 5:

[1734] The server analyzes the call origin information, identifying the origin using phone number, IP address, GPS information, etc., and if it detects any suspicious locations (such as public telephones or calls from abroad), it records that information.

[1735] Step 6:

[1736] The server inputs the voice data into an emotion engine to recognize the user's emotions. The emotion engine analyzes the voice tone, pitch, and tempo to determine the user's emotional state. For example, if the user is feeling nervous or anxious, that information is also recorded.

[1737] Step 7:

[1738] The server inputs voice data, text data, detected keywords, location information, and emotional information into the AI ​​model, which uses this information to assess the likelihood of fraud. It then uses past data and known fraud patterns to generate a score.

[1739] Step 8:

[1740] If the AI ​​model evaluates the likelihood of fraud at a certain level, the server generates a warning message after the call ends, which includes information about the suspected fraud and specific countermeasures.

[1741] Step 9:

[1742] The server sends the generated warning message to the user's terminal, and the sent warning message arrives at the terminal in real time.

[1743] Step 10:

[1744] When the device receives the warning message sent from the server, it notifies the user, which is displayed as a voice call or message notification.

[1745] Step 11:

[1746] The user checks the warning message on the device, understands the contents of the warning message, and takes necessary measures, such as rechecking the contents of calls that are suspected to be fraudulent and consulting the police or financial institution.

[1747] By taking the above steps, the system can quickly and accurately detect the possibility of "It's me" fraud and issue an immediate warning to the user, thus preventing fraud damage before it occurs.

[1748] Example 2

[1749] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1750] In modern society, elderly people and certain user groups are increasingly becoming victims of fraud such as "it's me" scams. These frauds are becoming more sophisticated and are often difficult to detect using normal means. As a result, victims are likely to suffer significant financial losses. To solve this problem, a system is needed that can accurately detect potential fraud in real time and issue a prompt warning.

[1751] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1752] In this invention, the server includes: a voice recognition means for converting voice data into text data; an analysis means for analyzing the text data and detecting specific keywords; a call origin identification means for analyzing call origin information; an AI model means for evaluating the possibility of fraud using the voice data, keywords, and call origin information; a warning generation means for generating a warning message based on the result of the fraud possibility evaluation and sending it to the terminal; a terminal means for notifying the user of the generated warning message; an emotion recognition means for analyzing the voice data and recognizing the user's emotion; and an AI model means for inputting the emotion information into the AI ​​model and evaluating the possibility of fraud with high accuracy. This makes it possible to minimize the risk of users becoming victims of fraud.

[1753] "Audio data" means data in digital form obtained by recording or transmitting sound.

[1754] A "terminal means" is an electronic device for collecting and transmitting voice data.

[1755] A "server" is a central processing unit that receives, processes, and stores data from multiple terminals over a network.

[1756] "Speech recognition means" is a technology that analyzes voice data and converts the content into text data.

[1757] The "analysis means" is a device or software that has the function of analyzing text data and detecting specific keywords.

[1758] The "originating location identification means" is a technology that analyzes the originating location information of a call and identifies the area where the call was made.

[1759] An "AI modeling tool" is a device or software that uses artificial intelligence to assess the likelihood of fraud.

[1760] A "warning generator" is a device or software capable of generating a warning message based on the result of assessing the likelihood of fraud.

[1761] "Emotion recognition means" is a technology that analyzes voice data to determine the user's emotions.

[1762] A "warning message" is a message that notifies the user in case of possible fraud.

[1763] "User" means an individual or organization that uses the system.

[1764] This invention provides a system that detects "It's my son" fraud calls in real time and issues a prompt warning, targeting elderly people and other users who are prone to fraud. This system can automatically carry out a series of processes from collecting voice data to issuing a warning.

[1765] System Components

[1766] Server-side configuration

[1767] Receiving audio data:

[1768] The server receives the voice data sent from the device in real time during the call. Specifically, a WebSocket connection is used to exchange data bidirectionally between the device and the server.

[1769] Voice Recognition:

[1770] The server uses the Google Speech-to-Text API or a similar service to convert the received voice data into text data, and during this process, it also performs noise reduction and echo cancellation to generate highly accurate text data.

[1771] Keyword detection:

[1772] The server performs text analysis to detect specific keywords (e.g., "money," "transfer," "accident," "help," etc.) using regular expressions and pre-defined keyword lists.

[1773] Identifying the source:

[1774] The server analyzes the call origin information, using phone numbers, IP addresses, GPS information, etc. to detect calls from suspicious locations such as public phones.

[1775] Emotion recognition:

[1776] The server inputs the voice data into an emotion engine (such as Affectiva's emotion recognition API) that analyzes the tone, pitch, and tempo of the user's voice to determine their emotion. Based on this information, it can determine whether the user is feeling nervous or anxious.

[1777] AI Fraud Assessment:

[1778] The server inputs voice data, text data, keywords, source information, and emotional information into an AI model to assess the likelihood of fraud, which also refers to past data and known fraud patterns.

[1779] Generates a warning message:

[1780] If the AI ​​model judges the likelihood of fraud high, the server automatically generates a warning message containing the suspected fraud and specific countermeasures.

[1781] Sending alerts:

[1782] The server generates a warning message and sends it to the user's device. The message is promptly sent to the user via push notification or SMS.

[1783] Terminal configuration

[1784] Audio data collection:

[1785] The device collects voice data during a call in real time and immediately sends it to the server. It is set to start collecting voice data as soon as the call starts.

[1786] Receiving an alert:

[1787] The device receives a warning message from the server and notifies the user. Notification methods include voice calls and message notifications, and the user is alerted in a way that can be checked immediately.

[1788] User-side configuration

[1789] Check the call details:

[1790] Users make calls as usual, the system runs in the background and no special action is required.

[1791] Check for warnings:

[1792] After the call ends, the user can view the warning message on their device and take appropriate action. The message will include information about the suspected fraud and specific countermeasures.

[1793] Specific examples

[1794] If an elderly person named Mr. D is told by someone claiming to be his son that he needs money, the system works as follows:

[1795] 1. The device collects the conversation between Mr. D and the other party and sends the voice data to the server.

[1796] 2. The server uses a speech recognition module to convert this speech into text and detect the keyword "money."

[1797] 3. The server uses the call location identification means to identify that the call is coming from a public telephone.

[1798] 4. The server inputs the voice data into an emotion recognition engine and recognizes that Mr. D is feeling nervous or anxious.

[1799] 5. The server inputs this information into an AI model and scores the likelihood of fraud.

[1800] 6. After the call ends, the server generates a warning message and sends it to the terminal.

[1801] 7. The device will send a warning message to Mr. D, who can then take immediate action.

[1802] This system automates the entire process from collecting voice data to recognizing emotional information, assessing the likelihood of fraud, and issuing warning notifications, thereby minimizing the risk of users becoming victims of fraud.

[1803] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1804] Step 1:

[1805] Collection and transmission of voice data

[1806] The device collects audio data during the call in real time. Specifically, it uses the device's microphone to capture the audio of the call and sends the audio data to the server via WebSocket or HTTP. The input of this step is the audio data collected from the device's microphone, and the output is the audio data sent to the server.

[1807] Step 2:

[1808] Receiving audio data

[1809] The server receives the audio data sent from the device in real time. Specifically, it starts a WebSocket server to receive the audio data. The input of this step is the audio data sent from the device, and the output is the audio data stored in a temporary buffer on the server.

[1810] Step 3:

[1811] Voice Recognition

[1812] The server converts the received voice data into text data using a speech recognition module such as the Google Speech-to-Text API. During this process, noise reduction and echo cancellation are also performed to improve analysis accuracy. The input for this step is the voice data stored on the server, and the output is the result of converting the voice data into text data.

[1813] Step 4:

[1814] Keyword detection

[1815] The server analyzes the generated text data and detects specific keywords (e.g., "money," "transfer," "accident," "help," etc.). Specifically, the analysis is performed using regular expressions or a pre-defined keyword list. The input of this step is the text data, and the output is a list of detected keywords.

[1816] Step 5:

[1817] Identifying the source

[1818] The server analyzes the call's origin information. Specifically, it identifies the call's origin using the phone number, IP address, GPS information, etc. The input to this step is the metadata associated with the call, and the output is the origin information. If the call originates from a suspicious location, the location information is also included.

[1819] Step 6:

[1820] emotion recognition

[1821] The server inputs the voice data into an emotion recognition engine (e.g., Affectiva's emotion recognition API) and analyzes the tone, pitch, and tempo of the user's voice to determine their emotion. The input of this step is the voice data, and the output is the user's emotional information (e.g., tension, fear, relief, etc.).

[1822] Step 7:

[1823] AI-powered fraud assessment

[1824] The server inputs the voice data, text data, detected keywords, origin information, and emotion information into the AI ​​model to assess the likelihood of fraud. The input for this step is all of this data, and the output is a fraud risk assessment score. The AI ​​model also references known fraud patterns and past data to make the assessment.

[1825] Step 8:

[1826] Generate a warning message

[1827] If the AI ​​model evaluates the likelihood of fraud highly, the server automatically generates a warning message. This message includes the suspected fraud and specific countermeasures, and is provided to the user. The input of this step is the fraud risk assessment score, and the output is the generated warning message.

[1828] Step 9:

[1829] Sending an Alert

[1830] The server sends the generated warning message to the user's device. Specifically, it ensures that the warning message reaches the user promptly via push notification or SMS. The input of this step is the warning message, and the output is the warning message sent to the user's device.

[1831] Step 10:

[1832] Receiving and Acknowledging Alerts

[1833] The terminal receives the warning message from the server and notifies the user. Specifically, it notifies the user of the warning using a notification bar, a pop-up, or a sound notification. The input of this step is the warning message sent from the server, and the output is the warning message displayed in a format that can be confirmed by the user.

[1834] Step 11:

[1835] User response

[1836] The user checks the warning message sent to the device and takes appropriate action, such as reading the warning content, reporting it to the appropriate authorities, or ending the call. The input to this step is the warning message, and the output is the user's specific action.

[1837] Through these steps, the system can efficiently collect, analyze, evaluate, and warn users of voice data, minimizing the risk of fraud.

[1838] (Application example 2)

[1839] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1840] In recent years, there has been an increase in "I'm your son" frauds targeting elderly people and other users who are prone to fraud, creating a need for rapid and effective countermeasures. However, conventional systems have had difficulty detecting potential fraud in real time during voice calls and issuing prompt warnings to users. In addition, evaluations based solely on voice data have the problem of low detection accuracy and a high number of false positives.

[1841] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a voice recognition means for converting voice data into text data, an analysis means for analyzing the text data and detecting specific keywords, a call origin identification means for analyzing the call origin information, and an AI model means for evaluating the possibility of fraud using the voice data, keywords, call origin information, and user emotion data. This makes it possible to prevent elderly users and users who are prone to fraud from becoming victims of fraud.

[1842] The "terminal means" is a device that has the function of collecting voice data and transmitting it to a server.

[1843] A "voice recognition means" is a device or system that has the function of converting voice data into text data.

[1844] The "analysis means" is a system that has the function of analyzing text data and detecting specific keywords.

[1845] The "originating point identification means" is a system that has the function of analyzing the originating point information of a call and identifying the originating point.

[1846] The "AI model means" is a system that has the function of assessing the possibility of fraud using voice data, keywords, origin information, and user emotional data.

[1847] The "warning generation means" is a system that has the function of generating a warning message based on the result of assessing the possibility of fraud and sending it to the terminal.

[1848] "Emotion data" is information that represents the user's emotional state and is obtained by analyzing voice data.

[1849] This invention provides a system that detects fraudulent calls in real time and issues prompt warnings to elderly users and other users who are prone to fraud. This system is an integrated system that includes voice data collection, voice analysis, keyword detection, caller location identification, fraud probability assessment using an AI model, and warning generation and notification.

[1850] First, the terminal collects voice data and sends it to the server. The server converts the voice data into text data using a voice recognition means. The voice recognition means includes noise reduction and echo cancellation functions. Next, the server analyzes the text data to detect specific keywords. This analysis means detects keywords related to fraud (e.g., "money," "transfer," "accident," "help," etc.).

[1851] Furthermore, the server analyzes the call origination information and identifies suspicious locations using a call origination location identification means. The server also inputs the voice data into an emotion recognition engine to analyze the user's emotion data. This emotion data identifies emotions such as tension or anxiety based on the user's voice tone, pitch, tempo, etc.

[1852] This data (voice data, keywords, origin information, and emotion data) is input into the AI ​​model means to evaluate the likelihood of fraud. The AI ​​model means makes the evaluation by referring to past data and known fraud patterns. If it is evaluated that there is a high likelihood of fraud, the server generates a warning message and sends it to the terminal using the warning generation means. The terminal notifies the user of the generated warning message. This notification may take the form of a voice call or a message notification.

[1853] As a concrete example, consider a situation in which a device initiates a call between an elderly user and someone else. At this time, the device sends voice data to a server in real time. The server converts the voice to text and detects the keyword "money." The server then analyzes the call's origin information and determines that it is coming from a public phone. The server inputs the voice data into an emotion engine and recognizes that the user is feeling nervous or anxious. The server then inputs this information into an AI model and evaluates the likelihood of fraud. After the call ends, the server automatically generates a warning message and sends it to the device. The device notifies the user of the warning message, allowing them to take immediate action.

[1854] The server also handles notifications to notify the generated warning messages. The following is an example of a specific prompt to the AI ​​model:

[1855] "The following keywords were detected in this call: 'money', 'transfer'. The call originated from an unknown pay phone. The user's emotions are very tense. What is the likelihood of this being a scam?"

[1856] The present invention automates this series of processes, minimizing the risk of users falling victim to fraud.

[1857] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1858] Step 1:

[1859] The terminal collects voice data during a call. The collected voice data is sent to the server in real time. This allows the server to receive the user's call data in real time. The input is the call voice data, and the output is the voice data sent to the server.

[1860] Step 2:

[1861] The server converts the received voice data into text data using a voice recognition means. During this process, noise reduction and echo cancellation are performed to improve the accuracy of the voice data. The input is voice data, and the output is text data.

[1862] Step 3:

[1863] The server analyzes the text data generated by the speech recognition to detect specific keywords. Using the analysis means, it detects fraud-related keywords such as "money," "transfer," "accident," and "help." The input is the text data, and the output is a list of detected keywords.

[1864] Step 4:

[1865] The server analyzes the call origination information and uses the call origination identification means to identify suspicious locations by analyzing the phone number, IP address, and GPS information. The input is the call origination information, and the output is the analyzed call origination information.

[1866] Step 5:

[1867] The server inputs the voice data into an emotion recognition engine and analyzes the user's emotional data. The emotion recognition engine analyzes the tone, pitch, and tempo of the user's voice to determine emotions such as tension or anxiety. The input is voice data, and the output is the user's emotional data.

[1868] Step 6:

[1869] The server inputs the voice data, keywords, origin information, and emotion data into the AI ​​model means to evaluate the likelihood of fraud. The AI ​​model means makes an evaluation by referring to past data and known fraud patterns. The inputs are the voice data, keywords, origin information, and emotion data, and the output is the result of the evaluation of the likelihood of fraud.

[1870] Step 7:

[1871] If the server judges the possibility of fraud to be high, it generates a warning message using the warning generation means and sends it to the terminal. The generated warning message includes information about the suspected fraud and specific countermeasures. The input is the result of the fraud possibility assessment, and the output is the warning message.

[1872] Step 8:

[1873] The terminal receives the warning message sent from the server and notifies the user. The notification is in the form of a voice call or a message notification. The input is the warning message, and the output is the notification to the user.

[1874] Through the above series of processing steps, users can detect fraudulent calls in real time and take prompt action.

[1875] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1876] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1877] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1878] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1879] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1880] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1881] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1882] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1883] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1884] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1885] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1886] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1887] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1888] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1889] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1890] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1891] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1892] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1893] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1894] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1895] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1896] The following is further disclosed regarding the above embodiment.

[1897] (Claim 1)

[1898] a terminal means for collecting voice data and transmitting the voice data to a server;

[1899] a speech recognition means for converting speech data into text data;

[1900] an analysis means for analyzing the text data and detecting specific keywords;

[1901] a call origination location determination means for analyzing call origination location information;

[1902] An AI model that uses voice data, keywords, and location information to assess the likelihood of fraud;

[1903] a warning generating means for generating a warning message based on the result of evaluating the possibility of fraud and transmitting the warning message to the terminal;

[1904] The system includes terminal means for notifying a user of the generated warning message.

[1905] (Claim 2)

[1906] The system of claim 1, further comprising an AI model means for evaluating the likelihood of fraud if the call origin information indicates a suspicious location.

[1907] (Claim 3)

[1908] 2. The system according to claim 1, further comprising a speech recognition means for performing noise reduction and echo cancellation when a keyword is detected.

[1909] "Example 1"

[1910] (Claim 1)

[1911] a terminal means for collecting voice data and transmitting the voice data to a server;

[1912] a speech recognition means for converting speech data into text data;

[1913] an analysis means for analyzing the text data and detecting specific keywords;

[1914] a call origination location determination means for analyzing call origination location information;

[1915] an audio processing means for performing noise reduction and echo cancellation;

[1916] An AI model that uses voice data, keywords, and location information to assess the likelihood of fraud;

[1917] a warning generating means for generating a warning message based on the result of evaluating the possibility of fraud and transmitting the warning message to the terminal;

[1918] The system includes terminal means for notifying a user of the generated warning message.

[1919] (Claim 2)

[1920] The system of claim 1, further comprising an AI model means for evaluating the likelihood of fraud if the call origin information indicates a suspicious location.

[1921] (Claim 3)

[1922] 2. The system of claim 1, further comprising means for transmitting and receiving audio data using a protocol.

[1923] "Application Example 1"

[1924] (Claim 1)

[1925] a terminal means for collecting voice data and transmitting the voice data to a server;

[1926] a speech recognition means for converting speech data into text data;

[1927] an analysis means for analyzing the text data and detecting specific keywords;

[1928] a call origination location determination means for analyzing call origination location information;

[1929] An AI model that uses voice data, keywords, and location information to assess the likelihood of fraud;

[1930] a warning generating means for generating a warning message based on the result of evaluating the possibility of fraud and transmitting the warning message to the terminal;

[1931] The system includes a terminal means for notifying a user of the generated warning message and displaying the contents of the warning message to the user in real time.

[1932] (Claim 2)

[1933] The system of claim 1, further comprising an AI model means for evaluating the likelihood of fraud if the call origin information indicates a suspicious location.

[1934] (Claim 3)

[1935] 2. The system according to claim 1, further comprising a speech recognition means for performing noise reduction and echo cancellation when detecting a keyword, and a function for immediately notifying the result of keyword detection to the user's terminal at the same time.

[1936] "Example 2: Combining Emotion Engines"

[1937] (Claim 1)

[1938] a terminal means for collecting voice data and transmitting the voice data to a server;

[1939] a speech recognition means for converting speech data into text data;

[1940] an analysis means for analyzing the text data and detecting specific keywords;

[1941] a call origination location determination means for analyzing call origination location information;

[1942] An AI model that uses voice data, keywords, and location information to assess the likelihood of fraud;

[1943] a warning generating means for generating a warning message based on the result of evaluating the possibility of fraud and transmitting the warning message to the terminal;

[1944] a terminal means for notifying a user of the generated warning message;

[1945] emotion recognition means for analyzing voice data and recognizing the emotion of a user;

[1946] A system including an AI model means for inputting emotional information into an AI model and evaluating the likelihood of fraud with high accuracy.

[1947] (Claim 2)

[1948] The system of claim 1, further comprising an AI model means for evaluating the likelihood of fraud if the call origin information indicates a suspicious location.

[1949] (Claim 3)

[1950] 2. The system according to claim 1, further comprising a speech recognition means for performing noise reduction and echo cancellation when a keyword is detected.

[1951] "Application example 2 when combining emotion engines"

[1952] (Claim 1)

[1953] a terminal means for collecting voice data and transmitting the voice data to a server;

[1954] a speech recognition means for converting speech data into text data;

[1955] an analysis means for analyzing the text data and detecting specific keywords;

[1956] a call origination location determination means for analyzing call origination location information;

[1957] An AI model means for assessing the likelihood of fraud using voice data, keywords, origin information, and user emotion data;

[1958] a warning generating means for generating a warning message based on the result of evaluating the possibility of fraud and transmitting the warning message to the terminal;

[1959] The system includes terminal means for notifying a user of the generated warning message.

[1960] (Claim 2)

[1961] The system of claim 1, further comprising an AI model means for evaluating the likelihood of fraud if the call origin information indicates a suspicious location.

[1962] (Claim 3)

[1963] 2. The system according to claim 1, further comprising a speech recognition means for performing noise reduction and echo cancellation when a keyword is detected. [Explanation of symbols]

[1964] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / location> < / location> < / url:> < / location> < / location> < / url:> < / location> < / location> < / url:> < / location> < / location>

Claims

1. a terminal means for collecting voice data and transmitting the voice data to a server; a speech recognition means for converting speech data into text data; an analysis means for analyzing the text data and detecting specific keywords; a call origination location determination means for analyzing call origination location information; An AI model that uses voice data, keywords, and location information to assess the likelihood of fraud; a warning generating means for generating a warning message based on the result of evaluating the possibility of fraud and transmitting the warning message to the terminal; The system includes terminal means for notifying a user of the generated warning message.

2. The system of claim 1, further comprising an AI model means for evaluating the possibility of fraud highly when the call origin information indicates a suspicious location.

3. 2. The system according to claim 1, further comprising a speech recognition means for performing noise reduction and echo cancellation when a keyword is detected.

Images