system

Abstract

システムを提供する。【解決手段】音響データを受信し、これを文字列データに変換する音響認識手段と、変換された文字列データに対して自然言語処理を適用し、危険な単語や表現を識別する言語解析手段と、識別された単語や表現をもとに、詐欺的要素を持つかどうかを判定する詐欺成分検出手段と、詐欺的要素が検出された際にユーザーに警告を生成する注意生成手段と、生成された警告をユーザーに視覚的または聴覚的に通知する注意通知手段と、ユーザーが不審な会話の真偽を確認するための具体的な質問を自動的に生成する問い合わせ案作成手段と、ユーザーに対して視覚的に情報を表示する通知視覚化手段と、を含むシステム。

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In recent years, sophisticated sales talks and fraudulent conversations have been increasing, and especially the elderly, general consumers, and employees of companies are vulnerable to such damages. This problem has brought about an economic and psychological burden of fraud damage and has become a social problem. Therefore, there is a need for technical means to give a warning before a user is involved in these fraudulent conversations and to confirm the authenticity of the other party with appropriate questions.

Means for Solving the Problems

[0005] This invention provides a system for detecting fraudulent elements by converting user conversations into text in real time using speech recognition means and analyzing the text data with natural language processing means. If fraudulent elements are detected, the system immediately issues a warning to the user using a warning generation means and communicates the warning to the user visually or audibly through a warning notification means and a notification display means. Furthermore, by using a question generation means to present the user with specific questions to verify the authenticity of the other party, the risk of fraud can be reduced.

[0006] "Speech recognition means" refers to a device or system equipped with technology for converting speech data into text data.

[0007] "Natural language processing means" refers to a device or system equipped with technology for analyzing text data to understand its meaning and context, and for extracting or processing specific information.

[0008] A "fraudulent element detection means" is a device or system equipped with technology for identifying dangerous words or phrases that could lead to fraud within text data.

[0009] A "warning generation means" is a device or system that generates a warning message to inform the user of danger based on detected fraudulent elements.

[0010] "Warning notification means" refers to a device or system equipped with technology for sending or communicating a generated warning message to a user.

[0011] A "question draft generation means" is a device or system equipped with technology to generate specific questions for the user to verify the authenticity of the other party.

[0012] "Notification display means" refers to a device or system equipped with technology for presenting warnings or suggested questions to the user visually or audibly. [Brief explanation of the drawing]

[0013] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2, which combines an emotion engine. [Modes for carrying out the invention]

[0014] An example of an embodiment of the system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0017] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0018] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0019] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0021] [First Embodiment]

[0022] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0023] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0026] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0029] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0033] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0034] This invention is a system that reduces the risk of fraud by detecting fraudulent elements in a user's conversation in real time, immediately issuing a warning to the user, and suggesting questions. This system is implemented using speech recognition technology, natural language processing technology, and warning / question generation technology.

[0035] Users use the device for everyday phone calls and face-to-face conversations. The device uses its built-in microphone to capture high-precision audio data of the conversation. The captured audio data is sent to a server, where it is converted into text data by speech recognition technology.

[0036] The server applies natural language processing to the converted text data to detect dangerous words and phrases that contain fraudulent elements. For example, phrases such as "guaranteed profit" or "special investment opportunity" fall into this category. When fraudulent elements are detected, the server generates a warning message using a warning generation system and transmits this message to the user through a warning notification system.

[0037] The user's device displays warnings sent from the server in a highly visible format. Visual or audible notifications allow the user to immediately understand the situation. The server also uses a question generation system to create and display on the user's device questions to help them verify the authenticity of the other party. For example, a question such as "Please send me the detailed contract information by email later" might be suggested.

[0038] As a concrete example, when a user receives an unknown investment offer over the phone, the device transmits the conversation to a server in real time. The server detects dangerous phrases from this conversation and immediately generates a warning, alerting the user. At the same time, it provides specific questions the user can ask the caller, allowing the user to recognize the possibility of fraud in advance and take appropriate action. This process contributes to safe and effective communication.

[0039] The following describes the processing flow.

[0040] Step 1:

[0041] The user initiates a conversation using the device. The device uses its built-in microphone to capture this conversation as audio data in real time.

[0042] Step 2:

[0043] The terminal converts the captured audio data into a digital signal and sends it to the server. This allows the server to begin processing.

[0044] Step 3:

[0045] The server uses speech recognition to convert the received audio data into text data.

[0046] Step 4:

[0047] The server applies natural language processing techniques to the converted text data to analyze whether it contains any fraudulent words or phrases.

[0048] Step 5:

[0049] If fraudulent elements are detected, the server will use a warning generation mechanism to create a warning message. This message will inform the user of the potential for fraud.

[0050] Step 6:

[0051] The server uses a warning notification mechanism to send the generated warning message to the user's terminal.

[0052] Step 7:

[0053] The device receives a warning message and displays it to the user visually or audibly. This allows the user to recognize the possibility of fraud.

[0054] Step 8:

[0055] The server then uses a question generation mechanism to generate specific questions that the user should ask the other party.

[0056] Step 9:

[0057] The server sends the generated question draft to the user's terminal, which then displays it to the user. The user can then use this as a reference to decide how to respond to the other party.

[0058] (Example 1)

[0059] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0060] In modern society, advancements in communication technology allow people to utilize diverse means of communication, but at the same time, fraudulent activities have become more sophisticated, and the risk exists even in everyday conversations. With conventional technology, it has been difficult to detect these fraudulent elements in real time and deal with them appropriately. Furthermore, there has been a lack of concrete means for users to quickly detect danger and protect themselves. Therefore, there is a need to establish technology that can instantly capture fraud risks lurking in conversations and provide users with appropriate warnings and countermeasures.

[0061] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0062] In this invention, the server includes speech recognition means, natural language processing means, and fraud element detection means. This makes it possible to instantly detect fraud elements hidden in everyday conversation, warn the user in real time, and provide specific question suggestions.

[0063] A "voice acquisition means" is a device that has the function of collecting a user's everyday conversations as digital voice data.

[0064] "Speech recognition means" refers to technology that analyzes collected speech data and converts it into text data.

[0065] "Natural language processing methods" are techniques that analyze text data, understand the meaning and structure of the language, and identify parts that contain fraudulent elements.

[0066] A "fraudulent element detection method" is a method that uses natural language processing to identify and extract dangerous words and phrases within text data.

[0067] A "warning generation method" is a technology that creates a warning message for the user based on detected fraudulent elements.

[0068] "Warning notification means" refers to a function for quickly communicating generated warning messages to the user.

[0069] A "question generation method" is a technology that automatically generates appropriate questions for users to determine whether or not they are being scammed.

[0070] A "notification display method" is a function that displays warning messages and suggested questions in an easy-to-read format on the user's device.

[0071] This system detects potential fraudulent elements in users' everyday phone calls and face-to-face conversations in real time and issues warnings based on that detection.

[0072] The user engages in normal conversation using the device. The device is equipped with a high-precision microphone that can collect the conversation audio as digital audio data. This audio data is transmitted to a server in real time via the internet.

[0073] The server converts the received audio data into text data using speech recognition technology. This process utilizes APIs commonly used as speech recognition engines. For example, it is expected that Google's Speech-to-Text API or a general-purpose speech recognition engine will be applied.

[0074] Natural language processing is applied to the converted text data. Here, the server uses a generative AI model to identify dangerous words and phrases within the text. Specifically, it utilizes OpenAI's GPT model and others to detect phrases that indicate potential fraud, such as "guaranteed profits" or "special investment opportunity." The prompt used at this time is "Please extract fraudulent elements from this text."

[0075] If the server detects fraudulent elements, a warning generation mechanism creates a warning message. This alerts the user to the potential for fraud. The warning message is immediately sent to the user's terminal by a warning notification mechanism. The terminal notifies the user of this information visually or audibly.

[0076] Furthermore, the server uses a question generation mechanism to generate questions for the user to verify the authenticity of the scam. This involves an AI generation model that automatically creates appropriate phrases and presents the user with specific countermeasures. For example, a question such as "Please send me the detailed contract information by email later" might be suggested.

[0077] For example, when a user receives an unknown investment offer over the phone, this system immediately detects risk elements in the conversation, displays a warning, and suggests appropriate questions. In this way, the user has an effective means of reducing the risk of fraud.

[0078] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0079] Step 1:

[0080] The user engages in everyday conversation, and their voice is captured with high precision by the device's built-in microphone. The input is the user's spoken voice, which is recorded as digital data. The device performs audio processing, such as noise cancellation, to prepare clear audio data.

[0081] Step 2:

[0082] The terminal transmits the acquired audio data to the server in real time. The input here is the audio data generated in step 1. The output is the audio file sent to the server, which is securely transmitted over the internet after data compression and encryption.

[0083] Step 3:

[0084] The server uses speech recognition to convert received audio data into text data. The input is an audio file, and the output is text information. This process uses an acoustic model and a dictionary to convert speech to text and applies a language model to prevent misrecognition.

[0085] Step 4:

[0086] The server uses natural language processing to analyze the converted text data. In this step, text data is used as input, and text fragments that may contain fraudulent elements are generated as output. A generative AI model is used to perform the analysis using the prompt "Extract fraudulent elements from this text."

[0087] Step 5:

[0088] If the server detects fraudulent elements, it uses a warning generation mechanism to create a warning message. The input here is the text information flagged in step 4, and the output is the warning message conveyed to the user. The generated message includes specific information about the potential for fraud.

[0089] Step 6:

[0090] The server sends a warning message to the user's terminal through a warning notification system. The input is the warning message, and the output is a visual or audible alert in the terminal's notification system. The terminal notifies the user of this as a display or audible notification.

[0091] Step 7:

[0092] The server then uses a question generation mechanism to create a proposed dialogue for the user. In this step, the input is the text data from the previous stage and the detected fraud elements, and the output is a proposed specific dialogue question. For example, a question such as "Please send me the detailed contract information by email later" might be generated.

[0093] Step 8:

[0094] The user reviews the warnings and suggested questions notified by the device and responds appropriately to conversations that may involve fraud. The input is the warning message and suggested questions from the device, and the output is the action chosen by the user. The user can use the generated questions to ask the other party questions to verify their trustworthiness.

[0095] (Application Example 1)

[0096] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0097] In today's communication-driven society, individuals are at increasing risk of becoming involved in fraudulent conversations. These fraudulent conversations often take place via telephone, online calls, and face-to-face interactions, making them particularly difficult to counter. The challenge lies in providing users with means to detect fraud in real time and deal with it safely.

[0098] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0099] In this invention, the server includes acoustic recognition means, language analysis means, and fraudulent component detection means. This makes it possible for the user to instantly detect fraudulent elements even in everyday conversations and be notified with a warning and specific countermeasures.

[0100] "Acoustic recognition means" refers to technology for receiving acoustic data and converting it into string data.

[0101] "Language analysis means" refers to techniques that apply natural language processing to converted string data to identify dangerous words and expressions.

[0102] A "fraudulent element detection method" is a technology that determines whether or not something contains fraudulent elements based on identified words or expressions.

[0103] A "warning generation method" is a technology that generates a warning to the user when fraudulent elements are detected.

[0104] A "warning notification method" is a technology that notifies the user of a generated warning visually or audibly.

[0105] The "inquiry draft generation method" is a technology that automatically generates specific questions for users to verify the authenticity of suspicious conversations.

[0106] A "notification visualization method" is a technology that visually displays information to the user, allowing them to immediately confirm its content.

[0107] This invention is a system that detects and warns users in real time about fraudulent elements in conversations they receive. First, the user uses a device such as a smartphone to have a phone or face-to-face conversation. During this time, a microphone built into the device acquires acoustic data with high accuracy. The acquired acoustic data is transmitted to a server using a communication means.

[0108] The server uses the Google Cloud Speech-to-Text API to convert the received audio data into text data. This text data is then analyzed by a language analysis tool using the Python library "spaCy". During this analysis, the server checks for the presence of dangerous and fraudulent words or expressions. If fraudulent content is detected, the server generates a warning message using a warning generation tool and communicates it to the user through a warning notification tool.

[0109] The user's device visually or audibly notifies them of warnings sent from the server through a display device. In addition, the server uses a query generation mechanism to generate specific questions for the user to verify the truthfulness of what the other party is saying, and presents them on the device. This allows the user to continue safe communication while protecting their conversation from the risk of fraud.

[0110] As a concrete example, consider a scenario where a user receives a phone call offering a "special investment opportunity." When an acoustic recognition system transmits the conversation to a server in real time, a language analysis system detects dangerous phrases and immediately generates a warning. Simultaneously, a prompt such as "Please email me the contract details later" is presented, allowing the user to respond appropriately.

[0111] Examples of prompt statements are as follows:

[0112] Conversation text: "There is a special investment opportunity. You will regret it if you miss this opportunity."

[0113] Please generate a draft question:

[0114] "Please send me the details of this proposal via email."

[0115] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0116] Step 1:

[0117] The user's device uses its built-in microphone to acquire acoustic data during a conversation. In this process, the device converts the microphone input into a digital format and outputs the captured audio in real time. The acquired acoustic data is then sent directly to the server, ready for speech recognition.

[0118] Step 2:

[0119] The server receives audio data sent from the terminal and converts it into text data using the Google Cloud Speech-to-Text API. Here, the input is audio waveform data, which the API analyzes and outputs as a corresponding text string. This process converts speech to text.

[0120] Step 3:

[0121] The server processes the converted string data using the Python library "spaCy" for natural language processing. In this step, the server receives the text data as input, analyzes the words and expressions it contains, and checks for any fraudulent elements. The output is the result of the determination of whether or not fraudulent elements are present.

[0122] Step 4:

[0123] If fraudulent elements are detected, the server activates its alert generation mechanism and generates a warning message. This process uses the analysis results as input to construct the content of the warning. Once the warning message is generated, it is ready to alert the user.

[0124] Step 5:

[0125] The server sends a generated warning message to the terminal. The terminal receives this and displays the warning to the user visually or audibly using a notification visualization device. Here, the input is the warning message, and the output is a notification in a format recognizable to the user. This allows the user to recognize fraudulent elements in real time.

[0126] Step 6:

[0127] In addition, the server uses a query generation mechanism to generate specific questions for the user to ask the other party in the conversation for confirmation. The input for this step is the detected fraudulent elements, which are used to generate the questions. The output is a set of question suggestions that the user can use immediately, and these are displayed on the terminal.

[0128] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0129] This invention is a system that, while a user is having a conversation, uses speech recognition technology, natural language processing technology, and an emotion engine to analyze the user's emotional state in real time, detect fraudulent elements, and provide appropriate warnings and suggested questions tailored to the user's psychological state.

[0130] When a user engages in conversation using a device, the device collects audio through its microphone. The collected audio data is sent to a server, where it is converted into text data by speech recognition. This text data is then analyzed by natural language processing to check for any potentially fraudulent words or phrases. The server also uses an emotion engine to analyze the user's voice to determine their emotional state. This analysis helps determine whether the user is feeling anxious, excited, or relaxed.

[0131] If fraudulent elements are detected, the server uses a warning generation mechanism to create a warning message tailored to the user's emotional state and sends it to the user's device via a warning notification mechanism. By adjusting the content and tone of the message based on the user's emotional state, more effective warnings can be provided. The device presents the warning visually or audibly to help the user understand the situation.

[0132] Furthermore, the server uses a question generation mechanism to generate questions that allow the user to efficiently verify the authenticity of the other party. These question suggestions are also adjusted to take the user's emotional state into account. For example, if the user is feeling anxious, questions that provide greater reassurance will be suggested.

[0133] As a concrete example, when a user hears a special investment pitch over the phone, the device sends the conversation to a server where speech recognition and natural language processing are performed in real time. The server detects fraudulent elements and, simultaneously, uses an emotion engine to determine that the user is feeling uneasy. Based on this, the server creates reassuring questions for the user, such as "What are the risks of this investment?", and provides the user with related warning messages. This process allows the user to assess the risk of fraud and take appropriate action while being psychologically supported.

[0134] The following describes the processing flow.

[0135] Step 1:

[0136] The user initiates a conversation using the device. The device uses its built-in microphone to capture audio data of the conversation in real time.

[0137] Step 2:

[0138] The terminal converts the acquired audio data into a digital signal and sends it to the server. This is so that the server can process the audio data.

[0139] Step 3:

[0140] The server uses speech recognition to convert the received audio data into text data. This conversion is performed in real time.

[0141] Step 4:

[0142] The server applies natural language processing techniques to text data to analyze and detect fraudulent words and phrases.

[0143] Step 5:

[0144] Simultaneously, the server uses an emotion engine to analyze the user's emotional state from the voice data. This allows it to determine whether the user is feeling anxious or stressed.

[0145] Step 6:

[0146] If fraudulent elements are detected, the server generates a warning message using a warning generation mechanism based on the user's emotional state.

[0147] Step 7:

[0148] The server sends the generated warning message to the user's terminal via a warning notification system.

[0149] Step 8:

[0150] The device receives a warning message and presents it to the user visually or audibly. Through this, the user becomes aware of the potential for fraud.

[0151] Step 9:

[0152] The server uses a question generation mechanism to generate specific questions that allow the user to verify the truthfulness of the other party's statements.

[0153] Step 10:

[0154] The server sends the generated question drafts to the user's terminal, which then displays them to the user. The user can then use this information to make a better judgment about the other party's trustworthiness.

[0155] (Example 2)

[0156] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0157] In conversations involving fraud or dishonest activities, there is a challenge in preventing users from making poor decisions due to anxiety, and in allowing them to continue the conversation with peace of mind. Furthermore, it is necessary to understand the user's emotional state and provide appropriate warnings and countermeasures based on the situation. By addressing these challenges, users can communicate calmly and protect themselves from potential dangers.

[0158] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0159] In this invention, the server includes speech recognition means for converting audio signals into text information, natural language processing means for analyzing the text information to detect elements indicating potential misconduct, and sentiment analysis means for analyzing emotional states. This makes it possible to analyze the user's situation in real time and generate and display appropriate warnings and suggested questions.

[0160] "Audio signals" are information that represents sound as electrical signals, which are then converted into a format that can be processed by a speech recognition system.

[0161] "Speech recognition means" refers to a technology that converts speech signals into textual information, and has the function of mechanically analyzing speech and transcribing it into text.

[0162] "Textual information" refers to information in text format converted from audio signals, which can be visually displayed and analyzed.

[0163] "Natural language processing means" refers to technical means that analyze textual information, understand the context and meaning contained therein, and identify elements that indicate the possibility of fraudulent activity.

[0164] "Emotion analysis means" refers to a technology that estimates a user's emotional state from audio signals and text information, and analyzes the type and intensity of that emotion in real time.

[0165] A "warning generation means" is a technical means that creates appropriate warning content for the user based on the analyzed information.

[0166] A "warning notification means" is a technical means for communicating a generated warning message to the user, and presents the warning visually or audibly.

[0167] A "question generation method" is a technical means that allows a user to generate questions to verify the truthfulness of information provided by their conversation partner, and provides appropriate questions based on the flow of the conversation.

[0168] A "notification display means" is a technical means that presents warning messages or suggested questions to the user, and has the function of conveying information using a screen or audio output.

[0169] This invention is a system that analyzes the content of conversations between users and provides a safe and secure dialogue environment. It mainly uses the following hardware and software. The terminal includes a microphone for collecting voice and a display or speaker for providing information to the user. The server is a high-performance computing device that is responsible for the computational processing required to analyze the voice data.

[0170] The server utilizes a speech recognition engine to convert audio signals into text. Specifically, it uses a major cloud-based speech recognition service. The text information is analyzed by a natural language processing engine to detect specific patterns that may indicate fraudulent activity. This process employs natural language processing libraries and models. The server also uses an emotion analysis engine to estimate the user's emotional state and analyzes those emotions in real time.

[0171] Furthermore, the server uses a warning generation mechanism to create an appropriate warning message based on the analysis results. This warning is sent to the user's terminal via a warning notification mechanism, and the terminal presents the information to the user via display or audio. Depending on the user's emotional state, a question suggestion generation mechanism proposes questions to verify the authenticity of the person they are interacting with.

[0172] As a concrete example, when a user receives a special investment offer over the phone, the device sends the audio to a server, where speech recognition and natural language processing are performed. The server detects fraudulent elements, and at the same time, an emotion analysis engine determines that the user is feeling uneasy. Based on this information, the server generates a question such as, "What are the specific risks of this investment?" and provides a relevant warning message.

[0173] An example of a prompt message is, "Analyze the emotions detected during the conversation and provide appropriate warnings and countermeasures." In this way, the system can analyze the user's conversation and support safe dialogue in real time.

[0174] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0175] Step 1:

[0176] The terminal collects the user's speech as an audio signal through the microphone. It then converts the collected audio signal into data packets and prepares them for transmission to the server. The input is the user's voice, and the output is the audio data sent to the server.

[0177] Step 2:

[0178] The server receives audio data sent from the terminal. It uses a speech recognition engine to convert the audio signal into text information. The input is audio data, and the output is text data.

[0179] Step 3:

[0180] The server analyzes the generated text data using a natural language processing engine. This analysis identifies language patterns and specific keywords that indicate fraudulent activity. The input is text data, and the output is an assessment of fraud risk.

[0181] Step 4:

[0182] The server uses an emotion analysis engine to estimate and analyze the user's emotional state from voice data. This allows it to determine what emotions the user is experiencing. While the input is voice data, text information may also be included in the analysis. The output is data indicating the user's emotional state.

[0183] Step 5:

[0184] The server generates a warning message using a warning generation mechanism based on fraud risk and emotional state. The content and tone of the message are adjusted according to the user's emotions. The input is fraud risk assessment information and emotional state data, and the output is the warning message.

[0185] Step 6:

[0186] The terminal receives warning messages sent from the server. It then notifies the user visually or audibly, presenting the information in a way that is easy for the user to understand. The input is the warning message, and the output is the notification to the user.

[0187] Step 7:

[0188] The server uses a question generation mechanism to generate questions for the user to verify the truthfulness of information provided by the conversation partner. The content is adjusted according to the user's emotional state. The input is emotional state data, and the output is the question proposal.

[0189] Step 8:

[0190] The terminal presents the generated question proposals to the user, supporting the user in making decisions. The input is the question proposal, and the output is the presentation of the question proposals.

[0191] (Application Example 2)

[0192] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0193] There is a need to mitigate the risk of fraud that users experience through conversations, while simultaneously reducing the psychological burden. Traditional fraud detection systems do not take into account the user's emotional state, making it difficult to provide appropriate warnings and guidance, and thus insufficient to help users understand the situation. In particular, it is important for the system to respond flexibly when the user feels distressed or anxious in a fraudulent situation.

[0194] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0195] In this invention, the server includes speech recognition means, natural language processing means, and sentiment analysis means. This makes it possible to analyze the user's conversation in real time and detect fraudulent elements, as well as to understand the user's emotional state and provide appropriate warnings or questions based on that emotion.

[0196] "Speech recognition means" refers to a means for converting speech data into text data.

[0197] "Natural language processing means" are methods for analyzing obtained text data and extracting fraudulent elements and important information.

[0198] A "fraudulent element detection method" is a means of identifying potentially fraudulent words or phrases from conversation content.

[0199] A "warning generation method" is a means for creating a warning message to notify a user of the possibility of fraud.

[0200] A "warning notification method" is a means of delivering a generated warning message to the user.

[0201] A "question proposal generation method" is a means of suggesting questions that help the user better understand the content of a conversation.

[0202] A "notification display means" is a means of presenting warnings or questions to the user visually or audibly.

[0203] An "emotional analysis tool" is a means of analyzing a user's voice to understand their emotional state and psychological condition.

[0204] "Emotional state-based warning adjustment means" refers to a means of adjusting the content and tone of warning messages according to the user's emotional state.

[0205] The "station / train response information generation means" is a means for generating additional information according to the situation the user is dealing with.

[0206] The system for implementing this invention aims to detect fraud risks and provide appropriate warnings and questions to the user by analyzing the user's voice in real time during their conversation. The server uses speech recognition technology to convert the voice into text data, which is then processed using natural language processing. This process utilizes the Google Cloud Speech Recognition API and the Hugging Face Transformers library. This allows for the immediate detection of fraudulent elements hidden in the user's conversation.

[0207] The server also uses sentiment analysis technology to analyze the user's voice and evaluates emotions using the Sentiment Analysis library. This analysis allows the server to understand the user's emotional state during a conversation and adjust the content and tone of warnings accordingly.

[0208] The device notifies the user of warning messages and suggested questions via Firebase, presenting them visually or audibly. This allows the user to intuitively understand the risk of fraud and respond with confidence.

[0209] For example, if a user is listening to suspicious product information provided by a salesperson over the phone, the system will automatically generate a question such as, "What risks are associated with this product?" to support the user in making a decision with confidence.

[0210] Examples of prompts for a generative AI model:

[0211] "Assess the potential for fraud based on the following conversation: Conversation Text. Generate appropriate warnings and questions based on the user's emotional state."

[0212] This system allows users to stay safe during everyday conversations while quickly addressing non-face-to-face fraud.

[0213] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0214] Step 1:

[0215] The device captures the user's conversation in audio format. It collects audio data through the microphone and transmits it to the server in real time. At this stage, the input data is an analog audio signal.

[0216] Step 2:

[0217] The server digitizes the received audio data and converts it into text data using the Google Cloud Speech Recognition API. The input is audio data, and the output is text data. This conversion makes subsequent natural language processing easier.

[0218] Step 3:

[0219] The server uses the Hugging Face Transformers library to perform natural language processing on text data. This process identifies words and phrases that contain fraudulent elements. The input is text data, and the output is information about the fraudulent sections.

[0220] Step 4:

[0221] The server uses the Sentiment Analysis library to analyze the user's emotional state from their voice. The input is voice data, and the output is data indicating the user's emotional state. This analysis result is used to adjust warning messages.

[0222] Step 5:

[0223] The server generates appropriate warning messages and suggested questions using a warning generation mechanism, based on the analysis results of fraudulent elements and emotional state. The input for this step is information on fraudulent elements and emotional state data, and the output is the warning message and suggested questions to be presented to the user.

[0224] Step 6:

[0225] The device notifies the user of warning messages and suggested questions via Firebase, presenting them visually or audibly as needed. The input is the generated warning messages and suggested questions, and the output is what is presented to the user visually or audibly.

[0226] Based on these warnings and suggested questions, users can confidently evaluate the conversation and take appropriate action.

[0227] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0228] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0229] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0230] [Second Embodiment]

[0231] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0232] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0233] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0234] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0235] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0236] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0237] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0238] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0239] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0240] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0241] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0242] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0243] This invention is a system that reduces the risk of fraud by detecting fraudulent elements in a user's conversation in real time, immediately issuing a warning to the user, and suggesting questions. This system is implemented using speech recognition technology, natural language processing technology, and warning / question generation technology.

[0244] Users use the device for everyday phone calls and face-to-face conversations. The device uses its built-in microphone to capture high-precision audio data of the conversation. The captured audio data is sent to a server, where it is converted into text data by speech recognition technology.

[0245] The server applies natural language processing to the converted text data to detect dangerous words and phrases that contain fraudulent elements. For example, phrases such as "guaranteed profit" or "special investment opportunity" fall into this category. When fraudulent elements are detected, the server generates a warning message using a warning generation system and transmits this message to the user through a warning notification system.

[0246] The user's device displays warnings sent from the server in a highly visible format. Visual or audible notifications allow the user to immediately understand the situation. The server also uses a question generation system to create and display on the user's device questions to help them verify the authenticity of the other party. For example, a question such as "Please send me the detailed contract information by email later" might be suggested.

[0247] As a concrete example, when a user receives an unknown investment offer over the phone, the device transmits the conversation to a server in real time. The server detects dangerous phrases from this conversation and immediately generates a warning, alerting the user. At the same time, it provides specific questions the user can ask the caller, allowing the user to recognize the possibility of fraud in advance and take appropriate action. This process contributes to safe and effective communication.

[0248] The following describes the processing flow.

[0249] Step 1:

[0250] The user initiates a conversation using the device. The device uses its built-in microphone to capture this conversation as audio data in real time.

[0251] Step 2:

[0252] The terminal converts the captured audio data into a digital signal and sends it to the server. This allows the server to begin processing.

[0253] Step 3:

[0254] The server uses speech recognition to convert the received audio data into text data.

[0255] Step 4:

[0256] The server applies natural language processing techniques to the converted text data to analyze whether it contains any fraudulent words or phrases.

[0257] Step 5:

[0258] If fraudulent elements are detected, the server will use a warning generation mechanism to create a warning message. This message will inform the user of the potential for fraud.

[0259] Step 6:

[0260] The server uses a warning notification mechanism to send the generated warning message to the user's terminal.

[0261] Step 7:

[0262] The device receives a warning message and displays it to the user visually or audibly. This allows the user to recognize the possibility of fraud.

[0263] Step 8:

[0264] The server then uses a question generation mechanism to generate specific questions that the user should ask the other party.

[0265] Step 9:

[0266] The server sends the generated question draft to the user's terminal, which then displays it to the user. The user can then use this as a reference to decide how to respond to the other party.

[0267] (Example 1)

[0268] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0269] In modern society, advancements in communication technology allow people to utilize diverse means of communication, but at the same time, fraudulent activities have become more sophisticated, and the risk exists even in everyday conversations. With conventional technology, it has been difficult to detect these fraudulent elements in real time and deal with them appropriately. Furthermore, there has been a lack of concrete means for users to quickly detect danger and protect themselves. Therefore, there is a need to establish technology that can instantly capture fraud risks lurking in conversations and provide users with appropriate warnings and countermeasures.

[0270] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0271] In this invention, the server includes speech recognition means, natural language processing means, and fraud element detection means. This makes it possible to instantly detect fraud elements hidden in everyday conversation, warn the user in real time, and provide specific question suggestions.

[0272] A "voice acquisition means" is a device that has the function of collecting a user's everyday conversations as digital voice data.

[0273] "Speech recognition means" refers to technology that analyzes collected speech data and converts it into text data.

[0274] "Natural language processing methods" are techniques that analyze text data, understand the meaning and structure of the language, and identify parts that contain fraudulent elements.

[0275] A "fraudulent element detection method" is a method that uses natural language processing to identify and extract dangerous words and phrases within text data.

[0276] A "warning generation method" is a technology that creates a warning message for the user based on detected fraudulent elements.

[0277] "Warning notification means" refers to a function for quickly communicating generated warning messages to the user.

[0278] A "question generation method" is a technology that automatically generates appropriate questions for users to determine whether or not they are being scammed.

[0279] A "notification display method" is a function that displays warning messages and suggested questions in an easy-to-read format on the user's device.

[0280] This system detects potential fraudulent elements in users' everyday phone calls and face-to-face conversations in real time and issues warnings based on that detection.

[0281] The user engages in normal conversation using the device. The device is equipped with a high-precision microphone that can collect the conversation audio as digital audio data. This audio data is transmitted to a server in real time via the internet.

[0282] The server converts the received audio data into text data using speech recognition technology. This process utilizes commonly used APIs for speech recognition engines. For example, the Google Speech-to-Text API or a general-purpose speech recognition engine is expected to be used.

[0283] Natural language processing is applied to the converted text data. Here, the server uses a generative AI model to identify dangerous words and phrases within the text. Specifically, it utilizes OpenAI's GPT model to detect phrases that indicate potential fraud, such as "guaranteed profits" or "special investment opportunity." The prompt used during this process is "Please extract fraudulent elements from this text."

[0284] When the server detects a fraud element, a warning message is created by the warning generation means. This raises awareness about the possibility of fraud. The warning message is immediately sent to the user's terminal by the warning notification means. The terminal notifies the user of this information in a visual or audible form.

[0285] Furthermore, the server uses question generation means to generate questions for the user to confirm the truth of the fraud. For this, the generation AI model automatically creates appropriate phrases and presents specific countermeasures to the user. For example, a question such as "Please send the detailed contract content by email later" is proposed.

[0286] As a specific example, when the user receives an unknown investment proposal by phone, this system immediately detects the risk elements during the conversation, displays a warning, and presents an appropriate questionnaire. In this way, the user can obtain an effective means to reduce the risk of fraud.

[0287] The flow of the specific process in Example 1 will be described using FIG. 11.

[0288] Step 1:

[0289] The user conducts daily conversations, and the voice is accurately acquired by the built-in microphone of the terminal. The input is the user's spoken voice, and this voice is recorded as digital data. The terminal performs voice processing such as noise cancellation to prepare clear voice data.

[0290] Step 2:

[0291] The terminal sends the acquired voice data to the server in real time. The input here is the voice data generated in Step 1. The output is the voice file sent to the server, which is securely transmitted via the Internet after data compression and encryption.

[0292] Step 3:

[0293] The server uses speech recognition to convert received audio data into text data. The input is an audio file, and the output is text information. This process uses an acoustic model and a dictionary to convert speech to text and applies a language model to prevent misrecognition.

[0294] Step 4:

[0295] The server uses natural language processing to analyze the converted text data. In this step, text data is used as input, and text fragments that may contain fraudulent elements are generated as output. A generative AI model is used to perform the analysis using the prompt "Extract fraudulent elements from this text."

[0296] Step 5:

[0297] If the server detects fraudulent elements, it uses a warning generation mechanism to create a warning message. The input here is the text information flagged in step 4, and the output is the warning message conveyed to the user. The generated message includes specific information about the potential for fraud.

[0298] Step 6:

[0299] The server sends a warning message to the user's terminal through a warning notification system. The input is the warning message, and the output is a visual or audible alert in the terminal's notification system. The terminal notifies the user of this as a display or audible notification.

[0300] Step 7:

[0301] The server then uses a question generation mechanism to create a proposed dialogue for the user. In this step, the input is the text data from the previous stage and the detected fraud elements, and the output is a proposed specific dialogue question. For example, a question such as "Please send me the detailed contract information by email later" might be generated.

[0302] Step 8:

[0303] The user checks the warnings and questionnaires notified from the terminal and appropriately responds to conversations including the possibility of fraud. The input is the warning message and questionnaire from the terminal, and the output is the action selected by the user. The user can use the generated questions to ask questions to confirm the reliability of the other party.

[0304] (Application Example 1)

[0305] Next, Application Example 1 will be described. In the following description, the data processing device 12 is referred to as a "server", and the smart glasses 214 are referred to as a "terminal".

[0306] In modern communication society, the risk that an individual is involved in a fraudulent conversation is increasing. Such fraudulent conversations are often conducted through phone calls, online calls, and face-to-face conversations, and are particularly difficult to countermeasure. In contrast, it is an issue to provide a means for users to detect fraud in real time and handle it safely.

[0307] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0308] In this invention, the server includes an acoustic recognition means, a language analysis means, and a fraud component detection means. Thereby, it becomes possible for a user to immediately detect a fraudulent element even during daily conversations and notify a warning and specific countermeasures.

[0309] The "acoustic recognition means" is a technology for receiving acoustic data and converting it into character string data.

[0310] The "language analysis means" is a technology for applying natural language processing to the converted character string data to identify dangerous words and expressions.

[0311] A "fraudulent element detection method" is a technology that determines whether or not something contains fraudulent elements based on identified words or expressions.

[0312] A "warning generation method" is a technology that generates a warning to the user when fraudulent elements are detected.

[0313] A "warning notification method" is a technology that notifies the user of a generated warning visually or audibly.

[0314] The "inquiry draft generation method" is a technology that automatically generates specific questions for users to verify the authenticity of suspicious conversations.

[0315] A "notification visualization method" is a technology that visually displays information to the user, allowing them to immediately confirm its content.

[0316] This invention is a system that detects and warns users in real time about fraudulent elements in conversations they receive. First, the user uses a device such as a smartphone to have a phone or face-to-face conversation. During this time, a microphone built into the device acquires acoustic data with high accuracy. The acquired acoustic data is transmitted to a server using a communication means.

[0317] The server uses the Google Cloud Speech-to-Text API to convert the received audio data into text data. This text data is then analyzed by a language analysis tool using the Python library "spaCy". During this analysis, the server checks for the presence of dangerous and fraudulent words or expressions. If fraudulent content is detected, the server generates a warning message using a warning generation tool and communicates it to the user through a warning notification tool.

[0318] The user's device visually or audibly notifies them of warnings sent from the server through a display device. In addition, the server uses a query generation mechanism to generate specific questions for the user to verify the truthfulness of what the other party is saying, and presents them on the device. This allows the user to continue safe communication while protecting their conversation from the risk of fraud.

[0319] As a concrete example, consider a scenario where a user receives a phone call offering a "special investment opportunity." When an acoustic recognition system transmits the conversation to a server in real time, a language analysis system detects dangerous phrases and immediately generates a warning. Simultaneously, a prompt such as "Please email me the contract details later" is presented, allowing the user to respond appropriately.

[0320] Examples of prompt statements are as follows:

[0321] Conversation text: "There is a special investment opportunity. You will regret it if you miss this opportunity."

[0322] Please generate a draft question:

[0323] "Please send me the details of this proposal via email."

[0324] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0325] Step 1:

[0326] The user's device uses its built-in microphone to acquire acoustic data during a conversation. In this process, the device converts the microphone input into a digital format and outputs the captured audio in real time. The acquired acoustic data is then sent directly to the server, ready for speech recognition.

[0327] Step 2:

[0328] The server receives audio data sent from the terminal and converts it into text data using the Google Cloud Speech-to-Text API. Here, the input is audio waveform data, which the API analyzes and outputs as a corresponding text string. This process converts speech to text.

[0329] Step 3:

[0330] The server processes the converted string data using the Python library "spaCy" for natural language processing. In this step, the server receives the text data as input, analyzes the words and expressions it contains, and checks for any fraudulent elements. The output is the result of the determination of whether or not fraudulent elements are present.

[0331] Step 4:

[0332] If fraudulent elements are detected, the server activates its alert generation mechanism and generates a warning message. This process uses the analysis results as input to construct the content of the warning. Once the warning message is generated, it is ready to alert the user.

[0333] Step 5:

[0334] The server sends a generated warning message to the terminal. The terminal receives this and displays the warning to the user visually or audibly using a notification visualization device. Here, the input is the warning message, and the output is a notification in a format recognizable to the user. This allows the user to recognize fraudulent elements in real time.

[0335] Step 6:

[0336] In addition, the server uses a query generation mechanism to generate specific questions for the user to ask the other party in the conversation for confirmation. The input for this step is the detected fraudulent elements, which are used to generate the questions. The output is a set of question suggestions that the user can use immediately, and these are displayed on the terminal.

[0337] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0338] This invention is a system that, while a user is having a conversation, uses speech recognition technology, natural language processing technology, and an emotion engine to analyze the user's emotional state in real time, detect fraudulent elements, and provide appropriate warnings and suggested questions tailored to the user's psychological state.

[0339] When a user engages in conversation using a device, the device collects audio through its microphone. The collected audio data is sent to a server, where it is converted into text data by speech recognition. This text data is then analyzed by natural language processing to check for any potentially fraudulent words or phrases. The server also uses an emotion engine to analyze the user's voice to determine their emotional state. This analysis helps determine whether the user is feeling anxious, excited, or relaxed.

[0340] If fraudulent elements are detected, the server uses a warning generation mechanism to create a warning message tailored to the user's emotional state and sends it to the user's device via a warning notification mechanism. By adjusting the content and tone of the message based on the user's emotional state, more effective warnings can be provided. The device presents the warning visually or audibly to help the user understand the situation.

[0341] Furthermore, the server uses a question generation mechanism to generate questions that allow the user to efficiently verify the authenticity of the other party. These question suggestions are also adjusted to take the user's emotional state into account. For example, if the user is feeling anxious, questions that provide greater reassurance will be suggested.

[0342] As a concrete example, when a user hears a special investment pitch over the phone, the device sends the conversation to a server where speech recognition and natural language processing are performed in real time. The server detects fraudulent elements and, simultaneously, uses an emotion engine to determine that the user is feeling uneasy. Based on this, the server creates reassuring questions for the user, such as "What are the risks of this investment?", and provides the user with related warning messages. This process allows the user to assess the risk of fraud and take appropriate action while being psychologically supported.

[0343] The following describes the processing flow.

[0344] Step 1:

[0345] The user initiates a conversation using the device. The device uses its built-in microphone to capture audio data of the conversation in real time.

[0346] Step 2:

[0347] The terminal converts the acquired audio data into a digital signal and sends it to the server. This is so that the server can process the audio data.

[0348] Step 3:

[0349] The server uses speech recognition to convert the received audio data into text data. This conversion is performed in real time.

[0350] Step 4:

[0351] The server applies natural language processing techniques to text data to analyze and detect fraudulent words and phrases.

[0352] Step 5:

[0353] Simultaneously, the server uses an emotion engine to analyze the user's emotional state from the voice data. This allows it to determine whether the user is feeling anxious or stressed.

[0354] Step 6:

[0355] If fraudulent elements are detected, the server generates a warning message using a warning generation mechanism based on the user's emotional state.

[0356] Step 7:

[0357] The server sends the generated warning message to the user's terminal via a warning notification system.

[0358] Step 8:

[0359] The device receives a warning message and presents it to the user visually or audibly. Through this, the user becomes aware of the potential for fraud.

[0360] Step 9:

[0361] The server uses a question generation mechanism to generate specific questions that allow the user to verify the truthfulness of the other party's statements.

[0362] Step 10:

[0363] The server sends the generated question drafts to the user's terminal, which then displays them to the user. The user can then use this information to make a better judgment about the other party's trustworthiness.

[0364] (Example 2)

[0365] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0366] In conversations involving fraud or dishonest activities, there is a challenge in preventing users from making poor decisions due to anxiety, and in allowing them to continue the conversation with peace of mind. Furthermore, it is necessary to understand the user's emotional state and provide appropriate warnings and countermeasures based on the situation. By addressing these challenges, users can communicate calmly and protect themselves from potential dangers.

[0367] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0368] In this invention, the server includes speech recognition means for converting audio signals into text information, natural language processing means for analyzing the text information to detect elements indicating potential misconduct, and sentiment analysis means for analyzing emotional states. This makes it possible to analyze the user's situation in real time and generate and display appropriate warnings and suggested questions.

[0369] "Audio signals" are information that represents sound as electrical signals, which are then converted into a format that can be processed by a speech recognition system.

[0370] "Speech recognition means" refers to a technology that converts speech signals into textual information, and has the function of mechanically analyzing speech and transcribing it into text.

[0371] "Textual information" refers to information in text format converted from audio signals, which can be visually displayed and analyzed.

[0372] "Natural language processing means" refers to technical means that analyze textual information, understand the context and meaning contained therein, and identify elements that indicate the possibility of fraudulent activity.

[0373] "Emotion analysis means" refers to a technology that estimates a user's emotional state from audio signals and text information, and analyzes the type and intensity of that emotion in real time.

[0374] A "warning generation means" is a technical means that creates appropriate warning content for the user based on the analyzed information.

[0375] A "warning notification means" is a technical means for communicating a generated warning message to the user, and presents the warning visually or audibly.

[0376] A "question generation method" is a technical means that allows a user to generate questions to verify the truthfulness of information provided by their conversation partner, and provides appropriate questions based on the flow of the conversation.

[0377] A "notification display means" is a technical means that presents warning messages or suggested questions to the user, and has the function of conveying information using a screen or audio output.

[0378] This invention is a system that analyzes the content of conversations between users and provides a safe and secure dialogue environment. It mainly uses the following hardware and software. The terminal includes a microphone for collecting voice and a display or speaker for providing information to the user. The server is a high-performance computing device that is responsible for the computational processing required to analyze the voice data.

[0379] The server utilizes a speech recognition engine to convert audio signals into text. Specifically, it uses a major cloud-based speech recognition service. The text information is analyzed by a natural language processing engine to detect specific patterns that may indicate fraudulent activity. This process employs natural language processing libraries and models. The server also uses an emotion analysis engine to estimate the user's emotional state and analyzes those emotions in real time.

[0380] Furthermore, the server uses a warning generation mechanism to create an appropriate warning message based on the analysis results. This warning is sent to the user's terminal via a warning notification mechanism, and the terminal presents the information to the user via display or audio. Depending on the user's emotional state, a question suggestion generation mechanism proposes questions to verify the authenticity of the person they are interacting with.

[0381] As a concrete example, when a user receives a special investment offer over the phone, the device sends the audio to a server, where speech recognition and natural language processing are performed. The server detects fraudulent elements, and at the same time, an emotion analysis engine determines that the user is feeling uneasy. Based on this information, the server generates a question such as, "What are the specific risks of this investment?" and provides a relevant warning message.

[0382] An example of a prompt message is, "Analyze the emotions detected during the conversation and provide appropriate warnings and countermeasures." In this way, the system can analyze the user's conversation and support safe dialogue in real time.

[0383] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0384] Step 1:

[0385] The terminal collects the user's speech as an audio signal through the microphone. It then converts the collected audio signal into data packets and prepares them for transmission to the server. The input is the user's voice, and the output is the audio data sent to the server.

[0386] Step 2:

[0387] The server receives audio data sent from the terminal. It uses a speech recognition engine to convert the audio signal into text information. The input is audio data, and the output is text data.

[0388] Step 3:

[0389] The server analyzes the generated text data using a natural language processing engine. This analysis identifies language patterns and specific keywords that indicate fraudulent activity. The input is text data, and the output is an assessment of fraud risk.

[0390] Step 4:

[0391] The server uses an emotion analysis engine to estimate and analyze the user's emotional state from voice data. This allows it to determine what emotions the user is experiencing. While the input is voice data, text information may also be included in the analysis. The output is data indicating the user's emotional state.

[0392] Step 5:

[0393] The server generates a warning message using a warning generation mechanism based on fraud risk and emotional state. The content and tone of the message are adjusted according to the user's emotions. The input is fraud risk assessment information and emotional state data, and the output is the warning message.

[0394] Step 6:

[0395] The terminal receives warning messages sent from the server. It then notifies the user visually or audibly, presenting the information in a way that is easy for the user to understand. The input is the warning message, and the output is the notification to the user.

[0396] Step 7:

[0397] The server uses a question generation mechanism to generate questions for the user to verify the truthfulness of information provided by the conversation partner. The content is adjusted according to the user's emotional state. The input is emotional state data, and the output is the question proposal.

[0398] Step 8:

[0399] The terminal presents the generated question proposals to the user, supporting the user in making decisions. The input is the question proposal, and the output is the presentation of the question proposals.

[0400] (Application Example 2)

[0401] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0402] There is a need to mitigate the risk of fraud that users experience through conversations, while simultaneously reducing the psychological burden. Traditional fraud detection systems do not take into account the user's emotional state, making it difficult to provide appropriate warnings and guidance, and thus insufficient to help users understand the situation. In particular, it is important for the system to respond flexibly when the user feels distressed or anxious in a fraudulent situation.

[0403] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0404] In this invention, the server includes speech recognition means, natural language processing means, and sentiment analysis means. This makes it possible to analyze the user's conversation in real time and detect fraudulent elements, as well as to understand the user's emotional state and provide appropriate warnings or questions based on that emotion.

[0405] "Speech recognition means" refers to a means for converting speech data into text data.

[0406] "Natural language processing means" are methods for analyzing obtained text data and extracting fraudulent elements and important information.

[0407] A "fraudulent element detection method" is a means of identifying potentially fraudulent words or phrases from conversation content.

[0408] A "warning generation method" is a means for creating a warning message to notify a user of the possibility of fraud.

[0409] A "warning notification method" is a means of delivering a generated warning message to the user.

[0410] A "question proposal generation method" is a means of suggesting questions that help the user better understand the content of a conversation.

[0411] A "notification display means" is a means of presenting warnings or questions to the user visually or audibly.

[0412] An "emotional analysis tool" is a means of analyzing a user's voice to understand their emotional state and psychological condition.

[0413] "Emotional state-based warning adjustment means" refers to a means of adjusting the content and tone of warning messages according to the user's emotional state.

[0414] The "station / train response information generation means" is a means for generating additional information according to the situation the user is dealing with.

[0415] The system for implementing this invention aims to detect fraud risks and provide appropriate warnings and questions to the user by analyzing the user's voice in real time during their conversation. The server uses speech recognition technology to convert the voice into text data, which is then processed using natural language processing. This process utilizes the Google Cloud Speech Recognition API and the Hugging Face Transformers library. This allows for the immediate detection of fraudulent elements hidden in the user's conversation.

[0416] The server also uses sentiment analysis technology to analyze the user's voice and evaluates emotions using the Sentiment Analysis library. This analysis allows the server to understand the user's emotional state during a conversation and adjust the content and tone of warnings accordingly.

[0417] The device notifies the user of warning messages and suggested questions via Firebase, presenting them visually or audibly. This allows the user to intuitively understand the risk of fraud and respond with confidence.

[0418] For example, if a user is listening to suspicious product information provided by a salesperson over the phone, the system will automatically generate a question such as, "What risks are associated with this product?" to support the user in making a decision with confidence.

[0419] Examples of prompts for a generative AI model:

[0420] "Assess the potential for fraud based on the following conversation: Conversation Text. Generate appropriate warnings and questions based on the user's emotional state."

[0421] This system allows users to stay safe during everyday conversations while quickly addressing non-face-to-face fraud.

[0422] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0423] Step 1:

[0424] The device captures the user's conversation in audio format. It collects audio data through the microphone and transmits it to the server in real time. At this stage, the input data is an analog audio signal.

[0425] Step 2:

[0426] The server digitizes the received audio data and converts it into text data using the Google Cloud Speech Recognition API. The input is audio data, and the output is text data. This conversion makes subsequent natural language processing easier.

[0427] Step 3:

[0428] The server uses the Hugging Face Transformers library to perform natural language processing on text data. This process identifies words and phrases that contain fraudulent elements. The input is text data, and the output is information about the fraudulent sections.

[0429] Step 4:

[0430] The server uses the Sentiment Analysis library to analyze the user's emotional state from their voice. The input is voice data, and the output is data indicating the user's emotional state. This analysis result is used to adjust warning messages.

[0431] Step 5:

[0432] The server generates appropriate warning messages and suggested questions using a warning generation mechanism, based on the analysis results of fraudulent elements and emotional state. The input for this step is information on fraudulent elements and emotional state data, and the output is the warning message and suggested questions to be presented to the user.

[0433] Step 6:

[0434] The device notifies the user of warning messages and suggested questions via Firebase, presenting them visually or audibly as needed. The input is the generated warning messages and suggested questions, and the output is what is presented to the user visually or audibly.

[0435] Based on these warnings and suggested questions, users can confidently evaluate the conversation and take appropriate action.

[0436] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0437] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0438] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0439] [Third Embodiment]

[0440] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0441] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0442] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0443] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0444] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0445] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0446] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0447] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0448] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0449] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0450] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0451] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0452] This invention is a system that reduces the risk of fraud by detecting fraudulent elements in a user's conversation in real time, immediately issuing a warning to the user, and suggesting questions. This system is implemented using speech recognition technology, natural language processing technology, and warning / question generation technology.

[0453] Users use the device for everyday phone calls and face-to-face conversations. The device uses its built-in microphone to capture high-precision audio data of the conversation. The captured audio data is sent to a server, where it is converted into text data by speech recognition technology.

[0454] The server applies natural language processing to the converted text data to detect dangerous words and phrases that contain fraudulent elements. For example, phrases such as "guaranteed profit" or "special investment opportunity" fall into this category. When fraudulent elements are detected, the server generates a warning message using a warning generation system and transmits this message to the user through a warning notification system.

[0455] The user's device displays warnings sent from the server in a highly visible format. Visual or audible notifications allow the user to immediately understand the situation. The server also uses a question generation system to create and display on the user's device questions to help them verify the authenticity of the other party. For example, a question such as "Please send me the detailed contract information by email later" might be suggested.

[0456] As a concrete example, when a user receives an unknown investment offer over the phone, the device transmits the conversation to a server in real time. The server detects dangerous phrases from this conversation and immediately generates a warning, alerting the user. At the same time, it provides specific questions the user can ask the caller, allowing the user to recognize the possibility of fraud in advance and take appropriate action. This process contributes to safe and effective communication.

[0457] The following describes the processing flow.

[0458] Step 1:

[0459] The user initiates a conversation using the device. The device uses its built-in microphone to capture this conversation as audio data in real time.

[0460] Step 2:

[0461] The terminal converts the captured audio data into a digital signal and sends it to the server. This allows the server to begin processing.

[0462] Step 3:

[0463] The server uses speech recognition to convert the received audio data into text data.

[0464] Step 4:

[0465] The server applies natural language processing techniques to the converted text data to analyze whether it contains any fraudulent words or phrases.

[0466] Step 5:

[0467] If fraudulent elements are detected, the server will use a warning generation mechanism to create a warning message. This message will inform the user of the potential for fraud.

[0468] Step 6:

[0469] The server uses a warning notification mechanism to send the generated warning message to the user's terminal.

[0470] Step 7:

[0471] The device receives a warning message and displays it to the user visually or audibly. This allows the user to recognize the possibility of fraud.

[0472] Step 8:

[0473] The server then uses a question generation mechanism to generate specific questions that the user should ask the other party.

[0474] Step 9:

[0475] The server sends the generated question draft to the user's terminal, which then displays it to the user. The user can then use this as a reference to decide how to respond to the other party.

[0476] (Example 1)

[0477] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0478] In modern society, advancements in communication technology allow people to utilize diverse means of communication, but at the same time, fraudulent activities have become more sophisticated, and the risk exists even in everyday conversations. With conventional technology, it has been difficult to detect these fraudulent elements in real time and deal with them appropriately. Furthermore, there has been a lack of concrete means for users to quickly detect danger and protect themselves. Therefore, there is a need to establish technology that can instantly capture fraud risks lurking in conversations and provide users with appropriate warnings and countermeasures.

[0479] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0480] In this invention, the server includes speech recognition means, natural language processing means, and fraud element detection means. This makes it possible to instantly detect fraud elements hidden in everyday conversation, warn the user in real time, and provide specific question suggestions.

[0481] A "voice acquisition means" is a device that has the function of collecting a user's everyday conversations as digital voice data.

[0482] "Speech recognition means" refers to technology that analyzes collected speech data and converts it into text data.

[0483] "Natural language processing methods" are techniques that analyze text data, understand the meaning and structure of the language, and identify parts that contain fraudulent elements.

[0484] A "fraudulent element detection method" is a method that uses natural language processing to identify and extract dangerous words and phrases within text data.

[0485] A "warning generation method" is a technology that creates a warning message for the user based on detected fraudulent elements.

[0486] "Warning notification means" refers to a function for quickly communicating generated warning messages to the user.

[0487] A "question generation method" is a technology that automatically generates appropriate questions for users to determine whether or not they are being scammed.

[0488] A "notification display method" is a function that displays warning messages and suggested questions in an easy-to-read format on the user's device.

[0489] This system detects potential fraudulent elements in users' everyday phone calls and face-to-face conversations in real time and issues warnings based on that detection.

[0490] The user engages in normal conversation using the device. The device is equipped with a high-precision microphone that can collect the conversation audio as digital audio data. This audio data is transmitted to a server in real time via the internet.

[0491] The server converts the received audio data into text data using speech recognition technology. This process utilizes commonly used APIs for speech recognition engines. For example, the Google Speech-to-Text API or a general-purpose speech recognition engine is expected to be used.

[0492] Natural language processing is applied to the converted text data. Here, the server uses a generative AI model to identify dangerous words and phrases within the text. Specifically, it utilizes OpenAI's GPT model to detect phrases that indicate potential fraud, such as "guaranteed profits" or "special investment opportunity." The prompt used during this process is "Please extract fraudulent elements from this text."

[0493] If the server detects fraudulent elements, a warning generation mechanism creates a warning message. This alerts the user to the potential for fraud. The warning message is immediately sent to the user's terminal by a warning notification mechanism. The terminal notifies the user of this information visually or audibly.

[0494] Furthermore, the server uses a question generation mechanism to generate questions for the user to verify the authenticity of the scam. This involves an AI generation model that automatically creates appropriate phrases and presents the user with specific countermeasures. For example, a question such as "Please send me the detailed contract information by email later" might be suggested.

[0495] For example, when a user receives an unknown investment offer over the phone, this system immediately detects risk elements in the conversation, displays a warning, and suggests appropriate questions. In this way, the user has an effective means of reducing the risk of fraud.

[0496] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0497] Step 1:

[0498] The user engages in everyday conversation, and their voice is captured with high precision by the device's built-in microphone. The input is the user's spoken voice, which is recorded as digital data. The device performs audio processing, such as noise cancellation, to prepare clear audio data.

[0499] Step 2:

[0500] The terminal transmits the acquired audio data to the server in real time. The input here is the audio data generated in step 1. The output is the audio file sent to the server, which is securely transmitted over the internet after data compression and encryption.

[0501] Step 3:

[0502] The server uses speech recognition to convert received audio data into text data. The input is an audio file, and the output is text information. This process uses an acoustic model and a dictionary to convert speech to text and applies a language model to prevent misrecognition.

[0503] Step 4:

[0504] The server uses natural language processing to analyze the converted text data. In this step, text data is used as input, and text fragments that may contain fraudulent elements are generated as output. A generative AI model is used to perform the analysis using the prompt "Extract fraudulent elements from this text."

[0505] Step 5:

[0506] If the server detects fraudulent elements, it uses a warning generation mechanism to create a warning message. The input here is the text information flagged in step 4, and the output is the warning message conveyed to the user. The generated message includes specific information about the potential for fraud.

[0507] Step 6:

[0508] The server sends a warning message to the user's terminal through a warning notification system. The input is the warning message, and the output is a visual or audible alert in the terminal's notification system. The terminal notifies the user of this as a display or audible notification.

[0509] Step 7:

[0510] The server then uses a question generation mechanism to create a proposed dialogue for the user. In this step, the input is the text data from the previous stage and the detected fraud elements, and the output is a proposed specific dialogue question. For example, a question such as "Please send me the detailed contract information by email later" might be generated.

[0511] Step 8:

[0512] The user reviews the warnings and suggested questions notified by the device and responds appropriately to conversations that may involve fraud. The input is the warning message and suggested questions from the device, and the output is the action chosen by the user. The user can use the generated questions to ask the other party questions to verify their trustworthiness.

[0513] (Application Example 1)

[0514] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0515] In today's communication-driven society, individuals are at increasing risk of becoming involved in fraudulent conversations. These fraudulent conversations often take place via telephone, online calls, and face-to-face interactions, making them particularly difficult to counter. The challenge lies in providing users with means to detect fraud in real time and deal with it safely.

[0516] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0517] In this invention, the server includes acoustic recognition means, language analysis means, and fraudulent component detection means. This makes it possible for the user to instantly detect fraudulent elements even in everyday conversations and be notified with a warning and specific countermeasures.

[0518] "Acoustic recognition means" refers to technology for receiving acoustic data and converting it into string data.

[0519] "Language analysis means" refers to techniques that apply natural language processing to converted string data to identify dangerous words and expressions.

[0520] A "fraudulent element detection method" is a technology that determines whether or not something contains fraudulent elements based on identified words or expressions.

[0521] A "warning generation method" is a technology that generates a warning to the user when fraudulent elements are detected.

[0522] A "warning notification method" is a technology that notifies the user of a generated warning visually or audibly.

[0523] The "inquiry draft generation method" is a technology that automatically generates specific questions for users to verify the authenticity of suspicious conversations.

[0524] A "notification visualization method" is a technology that visually displays information to the user, allowing them to immediately confirm its content.

[0525] This invention is a system that detects and warns users in real time about fraudulent elements in conversations they receive. First, the user uses a device such as a smartphone to have a phone or face-to-face conversation. During this time, a microphone built into the device acquires acoustic data with high accuracy. The acquired acoustic data is transmitted to a server using a communication means.

[0526] The server uses the Google Cloud Speech-to-Text API to convert the received audio data into text data. This text data is then analyzed by a language analysis tool using the Python library "spaCy". During this analysis, the server checks for the presence of dangerous and fraudulent words or expressions. If fraudulent content is detected, the server generates a warning message using a warning generation tool and communicates it to the user through a warning notification tool.

[0527] The user's device visually or audibly notifies them of warnings sent from the server through a display device. In addition, the server uses a query generation mechanism to generate specific questions for the user to verify the truthfulness of what the other party is saying, and presents them on the device. This allows the user to continue safe communication while protecting their conversation from the risk of fraud.

[0528] As a concrete example, consider a scenario where a user receives a phone call offering a "special investment opportunity." When an acoustic recognition system transmits the conversation to a server in real time, a language analysis system detects dangerous phrases and immediately generates a warning. Simultaneously, a prompt such as "Please email me the contract details later" is presented, allowing the user to respond appropriately.

[0529] Examples of prompt statements are as follows:

[0530] Conversation text: "There is a special investment opportunity. You will regret it if you miss this opportunity."

[0531] Please generate a draft question:

[0532] "Please send me the details of this proposal via email."

[0533] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0534] Step 1:

[0535] The user's device uses its built-in microphone to acquire acoustic data during a conversation. In this process, the device converts the microphone input into a digital format and outputs the captured audio in real time. The acquired acoustic data is then sent directly to the server, ready for speech recognition.

[0536] Step 2:

[0537] The server receives audio data sent from the terminal and converts it into text data using the Google Cloud Speech-to-Text API. Here, the input is audio waveform data, which the API analyzes and outputs as a corresponding text string. This process converts speech to text.

[0538] Step 3:

[0539] The server processes the converted string data using the Python library "spaCy" for natural language processing. In this step, the server receives the text data as input, analyzes the words and expressions it contains, and checks for any fraudulent elements. The output is the result of the determination of whether or not fraudulent elements are present.

[0540] Step 4:

[0541] If fraudulent elements are detected, the server activates its alert generation mechanism and generates a warning message. This process uses the analysis results as input to construct the content of the warning. Once the warning message is generated, it is ready to alert the user.

[0542] Step 5:

[0543] The server sends a generated warning message to the terminal. The terminal receives this and displays the warning to the user visually or audibly using a notification visualization device. Here, the input is the warning message, and the output is a notification in a format recognizable to the user. This allows the user to recognize fraudulent elements in real time.

[0544] Step 6:

[0545] In addition, the server uses a query generation mechanism to generate specific questions for the user to ask the other party in the conversation for confirmation. The input for this step is the detected fraudulent elements, which are used to generate the questions. The output is a set of question suggestions that the user can use immediately, and these are displayed on the terminal.

[0546] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0547] This invention is a system that, while a user is having a conversation, uses speech recognition technology, natural language processing technology, and an emotion engine to analyze the user's emotional state in real time, detect fraudulent elements, and provide appropriate warnings and suggested questions tailored to the user's psychological state.

[0548] When a user engages in conversation using a device, the device collects audio through its microphone. The collected audio data is sent to a server, where it is converted into text data by speech recognition. This text data is then analyzed by natural language processing to check for any potentially fraudulent words or phrases. The server also uses an emotion engine to analyze the user's voice to determine their emotional state. This analysis helps determine whether the user is feeling anxious, excited, or relaxed.

[0549] If fraudulent elements are detected, the server uses a warning generation mechanism to create a warning message tailored to the user's emotional state and sends it to the user's device via a warning notification mechanism. By adjusting the content and tone of the message based on the user's emotional state, more effective warnings can be provided. The device presents the warning visually or audibly to help the user understand the situation.

[0550] Furthermore, the server uses a question generation mechanism to generate questions that allow the user to efficiently verify the authenticity of the other party. These question suggestions are also adjusted to take the user's emotional state into account. For example, if the user is feeling anxious, questions that provide greater reassurance will be suggested.

[0551] As a concrete example, when a user hears a special investment pitch over the phone, the device sends the conversation to a server where speech recognition and natural language processing are performed in real time. The server detects fraudulent elements and, simultaneously, uses an emotion engine to determine that the user is feeling uneasy. Based on this, the server creates reassuring questions for the user, such as "What are the risks of this investment?", and provides the user with related warning messages. This process allows the user to assess the risk of fraud and take appropriate action while being psychologically supported.

[0552] The following describes the processing flow.

[0553] Step 1:

[0554] The user initiates a conversation using the device. The device uses its built-in microphone to capture audio data of the conversation in real time.

[0555] Step 2:

[0556] The terminal converts the acquired audio data into a digital signal and sends it to the server. This is so that the server can process the audio data.

[0557] Step 3:

[0558] The server uses speech recognition to convert the received audio data into text data. This conversion is performed in real time.

[0559] Step 4:

[0560] The server applies natural language processing techniques to text data to analyze and detect fraudulent words and phrases.

[0561] Step 5:

[0562] Simultaneously, the server uses an emotion engine to analyze the user's emotional state from the voice data. This allows it to determine whether the user is feeling anxious or stressed.

[0563] Step 6:

[0564] If fraudulent elements are detected, the server generates a warning message using a warning generation mechanism based on the user's emotional state.

[0565] Step 7:

[0566] The server sends the generated warning message to the user's terminal via a warning notification system.

[0567] Step 8:

[0568] The device receives a warning message and presents it to the user visually or audibly. Through this, the user becomes aware of the potential for fraud.

[0569] Step 9:

[0570] The server uses a question generation mechanism to generate specific questions that allow the user to verify the truthfulness of the other party's statements.

[0571] Step 10:

[0572] The server sends the generated question drafts to the user's terminal, which then displays them to the user. The user can then use this information to make a better judgment about the other party's trustworthiness.

[0573] (Example 2)

[0574] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0575] In conversations involving fraud or dishonest activities, there is a challenge in preventing users from making poor decisions due to anxiety, and in allowing them to continue the conversation with peace of mind. Furthermore, it is necessary to understand the user's emotional state and provide appropriate warnings and countermeasures based on the situation. By addressing these challenges, users can communicate calmly and protect themselves from potential dangers.

[0576] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0577] In this invention, the server includes speech recognition means for converting audio signals into text information, natural language processing means for analyzing the text information to detect elements indicating potential misconduct, and sentiment analysis means for analyzing emotional states. This makes it possible to analyze the user's situation in real time and generate and display appropriate warnings and suggested questions.

[0578] "Audio signals" are information that represents sound as electrical signals, which are then converted into a format that can be processed by a speech recognition system.

[0579] "Speech recognition means" refers to a technology that converts speech signals into textual information, and has the function of mechanically analyzing speech and transcribing it into text.

[0580] "Textual information" refers to information in text format converted from audio signals, which can be visually displayed and analyzed.

[0581] "Natural language processing means" refers to technical means that analyze textual information, understand the context and meaning contained therein, and identify elements that indicate the possibility of fraudulent activity.

[0582] "Emotion analysis means" refers to a technology that estimates a user's emotional state from audio signals and text information, and analyzes the type and intensity of that emotion in real time.

[0583] A "warning generation means" is a technical means that creates appropriate warning content for the user based on the analyzed information.

[0584] A "warning notification means" is a technical means for communicating a generated warning message to the user, and presents the warning visually or audibly.

[0585] A "question generation method" is a technical means that allows a user to generate questions to verify the truthfulness of information provided by their conversation partner, and provides appropriate questions based on the flow of the conversation.

[0586] A "notification display means" is a technical means that presents warning messages or suggested questions to the user, and has the function of conveying information using a screen or audio output.

[0587] This invention is a system that analyzes the content of conversations between users and provides a safe and secure dialogue environment. It mainly uses the following hardware and software. The terminal includes a microphone for collecting voice and a display or speaker for providing information to the user. The server is a high-performance computing device that is responsible for the computational processing required to analyze the voice data.

[0588] The server utilizes a speech recognition engine to convert audio signals into text. Specifically, it uses a major cloud-based speech recognition service. The text information is analyzed by a natural language processing engine to detect specific patterns that may indicate fraudulent activity. This process employs natural language processing libraries and models. The server also uses an emotion analysis engine to estimate the user's emotional state and analyzes those emotions in real time.

[0589] Furthermore, the server uses a warning generation mechanism to create an appropriate warning message based on the analysis results. This warning is sent to the user's terminal via a warning notification mechanism, and the terminal presents the information to the user via display or audio. Depending on the user's emotional state, a question suggestion generation mechanism proposes questions to verify the authenticity of the person they are interacting with.

[0590] As a concrete example, when a user receives a special investment offer over the phone, the device sends the audio to a server, where speech recognition and natural language processing are performed. The server detects fraudulent elements, and at the same time, an emotion analysis engine determines that the user is feeling uneasy. Based on this information, the server generates a question such as, "What are the specific risks of this investment?" and provides a relevant warning message.

[0591] An example of a prompt message is, "Analyze the emotions detected during the conversation and provide appropriate warnings and countermeasures." In this way, the system can analyze the user's conversation and support safe dialogue in real time.

[0592] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0593] Step 1:

[0594] The terminal collects the user's speech as an audio signal through the microphone. It then converts the collected audio signal into data packets and prepares them for transmission to the server. The input is the user's voice, and the output is the audio data sent to the server.

[0595] Step 2:

[0596] The server receives audio data sent from the terminal. It uses a speech recognition engine to convert the audio signal into text information. The input is audio data, and the output is text data.

[0597] Step 3:

[0598] The server analyzes the generated text data using a natural language processing engine. This analysis identifies language patterns and specific keywords that indicate fraudulent activity. The input is text data, and the output is an assessment of fraud risk.

[0599] Step 4:

[0600] The server uses an emotion analysis engine to estimate and analyze the user's emotional state from voice data. This allows it to determine what emotions the user is experiencing. While the input is voice data, text information may also be included in the analysis. The output is data indicating the user's emotional state.

[0601] Step 5:

[0602] The server generates a warning message using a warning generation mechanism based on fraud risk and emotional state. The content and tone of the message are adjusted according to the user's emotions. The input is fraud risk assessment information and emotional state data, and the output is the warning message.

[0603] Step 6:

[0604] The terminal receives warning messages sent from the server. It then notifies the user visually or audibly, presenting the information in a way that is easy for the user to understand. The input is the warning message, and the output is the notification to the user.

[0605] Step 7:

[0606] The server uses a question generation mechanism to generate questions for the user to verify the truthfulness of information provided by the conversation partner. The content is adjusted according to the user's emotional state. The input is emotional state data, and the output is the question proposal.

[0607] Step 8:

[0608] The terminal presents the generated question proposals to the user, supporting the user in making decisions. The input is the question proposal, and the output is the presentation of the question proposals.

[0609] (Application Example 2)

[0610] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0611] There is a need to mitigate the risk of fraud that users experience through conversations, while simultaneously reducing the psychological burden. Traditional fraud detection systems do not take into account the user's emotional state, making it difficult to provide appropriate warnings and guidance, and thus insufficient to help users understand the situation. In particular, it is important for the system to respond flexibly when the user feels distressed or anxious in a fraudulent situation.

[0612] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0613] In this invention, the server includes speech recognition means, natural language processing means, and sentiment analysis means. This makes it possible to analyze the user's conversation in real time and detect fraudulent elements, as well as to understand the user's emotional state and provide appropriate warnings or questions based on that emotion.

[0614] "Speech recognition means" refers to a means for converting speech data into text data.

[0615] "Natural language processing means" are methods for analyzing obtained text data and extracting fraudulent elements and important information.

[0616] A "fraudulent element detection method" is a means of identifying potentially fraudulent words or phrases from conversation content.

[0617] A "warning generation method" is a means for creating a warning message to notify a user of the possibility of fraud.

[0618] A "warning notification method" is a means of delivering a generated warning message to the user.

[0619] A "question proposal generation method" is a means of suggesting questions that help the user better understand the content of a conversation.

[0620] A "notification display means" is a means of presenting warnings or questions to the user visually or audibly.

[0621] An "emotional analysis tool" is a means of analyzing a user's voice to understand their emotional state and psychological condition.

[0622] "Emotional state-based warning adjustment means" refers to a means of adjusting the content and tone of warning messages according to the user's emotional state.

[0623] The "station / train response information generation means" is a means for generating additional information according to the situation the user is dealing with.

[0624] The system for implementing this invention aims to detect fraud risks and provide appropriate warnings and questions to the user by analyzing the user's voice in real time during their conversation. The server uses speech recognition technology to convert the voice into text data, which is then processed using natural language processing. This process utilizes the Google Cloud Speech Recognition API and the Hugging Face Transformers library. This allows for the immediate detection of fraudulent elements hidden in the user's conversation.

[0625] The server also uses sentiment analysis technology to analyze the user's voice and evaluates emotions using the Sentiment Analysis library. This analysis allows the server to understand the user's emotional state during a conversation and adjust the content and tone of warnings accordingly.

[0626] The device notifies the user of warning messages and suggested questions via Firebase, presenting them visually or audibly. This allows the user to intuitively understand the risk of fraud and respond with confidence.

[0627] For example, if a user is listening to suspicious product information provided by a salesperson over the phone, the system will automatically generate a question such as, "What risks are associated with this product?" to support the user in making a decision with confidence.

[0628] Examples of prompts for a generative AI model:

[0629] "Assess the potential for fraud based on the following conversation: Conversation Text. Generate appropriate warnings and questions based on the user's emotional state."

[0630] This system allows users to stay safe during everyday conversations while quickly addressing non-face-to-face fraud.

[0631] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0632] Step 1:

[0633] The device captures the user's conversation in audio format. It collects audio data through the microphone and transmits it to the server in real time. At this stage, the input data is an analog audio signal.

[0634] Step 2:

[0635] The server digitizes the received audio data and converts it into text data using the Google Cloud Speech Recognition API. The input is audio data, and the output is text data. This conversion makes subsequent natural language processing easier.

[0636] Step 3:

[0637] The server uses the Hugging Face Transformers library to perform natural language processing on text data. This process identifies words and phrases that contain fraudulent elements. The input is text data, and the output is information about the fraudulent sections.

[0638] Step 4:

[0639] The server uses the Sentiment Analysis library to analyze the user's emotional state from their voice. The input is voice data, and the output is data indicating the user's emotional state. This analysis result is used to adjust warning messages.

[0640] Step 5:

[0641] The server generates appropriate warning messages and suggested questions using a warning generation mechanism, based on the analysis results of fraudulent elements and emotional state. The input for this step is information on fraudulent elements and emotional state data, and the output is the warning message and suggested questions to be presented to the user.

[0642] Step 6:

[0643] The device notifies the user of warning messages and suggested questions via Firebase, presenting them visually or audibly as needed. The input is the generated warning messages and suggested questions, and the output is what is presented to the user visually or audibly.

[0644] Based on these warnings and suggested questions, users can confidently evaluate the conversation and take appropriate action.

[0645] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0646] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0647] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0648] [Fourth Embodiment]

[0649] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0650] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0651] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0652] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0653] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0654] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0655] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0656] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0657] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0658] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0659] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0660] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0661] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0662] This invention is a system that reduces the risk of fraud by detecting fraudulent elements in a user's conversation in real time, immediately issuing a warning to the user, and suggesting questions. This system is implemented using speech recognition technology, natural language processing technology, and warning / question generation technology.

[0663] Users use the device for everyday phone calls and face-to-face conversations. The device uses its built-in microphone to capture high-precision audio data of the conversation. The captured audio data is sent to a server, where it is converted into text data by speech recognition technology.

[0664] The server applies natural language processing to the converted text data to detect dangerous words and phrases that contain fraudulent elements. For example, phrases such as "guaranteed profit" or "special investment opportunity" fall into this category. When fraudulent elements are detected, the server generates a warning message using a warning generation system and transmits this message to the user through a warning notification system.

[0665] The user's device displays warnings sent from the server in a highly visible format. Visual or audible notifications allow the user to immediately understand the situation. The server also uses a question generation system to create and display on the user's device questions to help them verify the authenticity of the other party. For example, a question such as "Please send me the detailed contract information by email later" might be suggested.

[0666] As a concrete example, when a user receives an unknown investment offer over the phone, the device transmits the conversation to a server in real time. The server detects dangerous phrases from this conversation and immediately generates a warning, alerting the user. At the same time, it provides specific questions the user can ask the caller, allowing the user to recognize the possibility of fraud in advance and take appropriate action. This process contributes to safe and effective communication.

[0667] The following describes the processing flow.

[0668] Step 1:

[0669] The user initiates a conversation using the device. The device uses its built-in microphone to capture this conversation as audio data in real time.

[0670] Step 2:

[0671] The terminal converts the captured audio data into a digital signal and sends it to the server. This allows the server to begin processing.

[0672] Step 3:

[0673] The server uses speech recognition to convert the received audio data into text data.

[0674] Step 4:

[0675] The server applies natural language processing techniques to the converted text data to analyze whether it contains any fraudulent words or phrases.

[0676] Step 5:

[0677] If fraudulent elements are detected, the server will use a warning generation mechanism to create a warning message. This message will inform the user of the potential for fraud.

[0678] Step 6:

[0679] The server uses a warning notification mechanism to send the generated warning message to the user's terminal.

[0680] Step 7:

[0681] The device receives a warning message and displays it to the user visually or audibly. This allows the user to recognize the possibility of fraud.

[0682] Step 8:

[0683] The server then uses a question generation mechanism to generate specific questions that the user should ask the other party.

[0684] Step 9:

[0685] The server sends the generated question draft to the user's terminal, which then displays it to the user. The user can then use this as a reference to decide how to respond to the other party.

[0686] (Example 1)

[0687] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0688] In modern society, advancements in communication technology allow people to utilize diverse means of communication, but at the same time, fraudulent activities have become more sophisticated, and the risk exists even in everyday conversations. With conventional technology, it has been difficult to detect these fraudulent elements in real time and deal with them appropriately. Furthermore, there has been a lack of concrete means for users to quickly detect danger and protect themselves. Therefore, there is a need to establish technology that can instantly capture fraud risks lurking in conversations and provide users with appropriate warnings and countermeasures.

[0689] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0690] In this invention, the server includes speech recognition means, natural language processing means, and fraud element detection means. This makes it possible to instantly detect fraud elements hidden in everyday conversation, warn the user in real time, and provide specific question suggestions.

[0691] A "voice acquisition means" is a device that has the function of collecting a user's everyday conversations as digital voice data.

[0692] "Speech recognition means" refers to technology that analyzes collected speech data and converts it into text data.

[0693] "Natural language processing methods" are techniques that analyze text data, understand the meaning and structure of the language, and identify parts that contain fraudulent elements.

[0694] A "fraudulent element detection method" is a method that uses natural language processing to identify and extract dangerous words and phrases within text data.

[0695] A "warning generation method" is a technology that creates a warning message for the user based on detected fraudulent elements.

[0696] "Warning notification means" refers to a function for quickly communicating generated warning messages to the user.

[0697] A "question generation method" is a technology that automatically generates appropriate questions for users to determine whether or not they are being scammed.

[0698] A "notification display method" is a function that displays warning messages and suggested questions in an easy-to-read format on the user's device.

[0699] This system detects potential fraudulent elements in users' everyday phone calls and face-to-face conversations in real time and issues warnings based on that detection.

[0700] The user engages in normal conversation using the device. The device is equipped with a high-precision microphone that can collect the conversation audio as digital audio data. This audio data is transmitted to a server in real time via the internet.

[0701] The server converts the received audio data into text data using speech recognition technology. This process utilizes commonly used APIs for speech recognition engines. For example, the Google Speech-to-Text API or a general-purpose speech recognition engine is expected to be used.

[0702] Natural language processing is applied to the converted text data. Here, the server uses a generative AI model to identify dangerous words and phrases within the text. Specifically, it utilizes OpenAI's GPT model to detect phrases that indicate potential fraud, such as "guaranteed profits" or "special investment opportunity." The prompt used during this process is "Please extract fraudulent elements from this text."

[0703] If the server detects fraudulent elements, a warning generation mechanism creates a warning message. This alerts the user to the potential for fraud. The warning message is immediately sent to the user's terminal by a warning notification mechanism. The terminal notifies the user of this information visually or audibly.

[0704] Furthermore, the server uses a question generation mechanism to generate questions for the user to verify the authenticity of the scam. This involves an AI generation model that automatically creates appropriate phrases and presents the user with specific countermeasures. For example, a question such as "Please send me the detailed contract information by email later" might be suggested.

[0705] For example, when a user receives an unknown investment offer over the phone, this system immediately detects risk elements in the conversation, displays a warning, and suggests appropriate questions. In this way, the user has an effective means of reducing the risk of fraud.

[0706] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0707] Step 1:

[0708] The user engages in everyday conversation, and their voice is captured with high precision by the device's built-in microphone. The input is the user's spoken voice, which is recorded as digital data. The device performs audio processing, such as noise cancellation, to prepare clear audio data.

[0709] Step 2:

[0710] The terminal transmits the acquired audio data to the server in real time. The input here is the audio data generated in step 1. The output is the audio file sent to the server, which is securely transmitted over the internet after data compression and encryption.

[0711] Step 3:

[0712] The server uses speech recognition to convert received audio data into text data. The input is an audio file, and the output is text information. This process uses an acoustic model and a dictionary to convert speech to text and applies a language model to prevent misrecognition.

[0713] Step 4:

[0714] The server uses natural language processing to analyze the converted text data. In this step, text data is used as input, and text fragments that may contain fraudulent elements are generated as output. A generative AI model is used to perform the analysis using the prompt "Extract fraudulent elements from this text."

[0715] Step 5:

[0716] If the server detects fraudulent elements, it uses a warning generation mechanism to create a warning message. The input here is the text information flagged in step 4, and the output is the warning message conveyed to the user. The generated message includes specific information about the potential for fraud.

[0717] Step 6:

[0718] The server sends a warning message to the user's terminal through a warning notification system. The input is the warning message, and the output is a visual or audible alert in the terminal's notification system. The terminal notifies the user of this as a display or audible notification.

[0719] Step 7:

[0720] The server then uses a question generation mechanism to create a proposed dialogue for the user. In this step, the input is the text data from the previous stage and the detected fraud elements, and the output is a proposed specific dialogue question. For example, a question such as "Please send me the detailed contract information by email later" might be generated.

[0721] Step 8:

[0722] The user reviews the warnings and suggested questions notified by the device and responds appropriately to conversations that may involve fraud. The input is the warning message and suggested questions from the device, and the output is the action chosen by the user. The user can use the generated questions to ask the other party questions to verify their trustworthiness.

[0723] (Application Example 1)

[0724] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0725] In today's communication-driven society, individuals are at increasing risk of becoming involved in fraudulent conversations. These fraudulent conversations often take place via telephone, online calls, and face-to-face interactions, making them particularly difficult to counter. The challenge lies in providing users with means to detect fraud in real time and deal with it safely.

[0726] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0727] In this invention, the server includes acoustic recognition means, language analysis means, and fraudulent component detection means. This makes it possible for the user to instantly detect fraudulent elements even in everyday conversations and be notified with a warning and specific countermeasures.

[0728] "Acoustic recognition means" refers to technology for receiving acoustic data and converting it into string data.

[0729] "Language analysis means" refers to techniques that apply natural language processing to converted string data to identify dangerous words and expressions.

[0730] A "fraudulent element detection method" is a technology that determines whether or not something contains fraudulent elements based on identified words or expressions.

[0731] A "warning generation method" is a technology that generates a warning to the user when fraudulent elements are detected.

[0732] A "warning notification method" is a technology that notifies the user of a generated warning visually or audibly.

[0733] The "inquiry draft generation method" is a technology that automatically generates specific questions for users to verify the authenticity of suspicious conversations.

[0734] A "notification visualization method" is a technology that visually displays information to the user, allowing them to immediately confirm its content.

[0735] This invention is a system that detects and warns users in real time about fraudulent elements in conversations they receive. First, the user uses a device such as a smartphone to have a phone or face-to-face conversation. During this time, a microphone built into the device acquires acoustic data with high accuracy. The acquired acoustic data is transmitted to a server using a communication means.

[0736] The server uses the Google Cloud Speech-to-Text API to convert the received audio data into text data. This text data is then analyzed by a language analysis tool using the Python library "spaCy". During this analysis, the server checks for the presence of dangerous and fraudulent words or expressions. If fraudulent content is detected, the server generates a warning message using a warning generation tool and communicates it to the user through a warning notification tool.

[0737] The user's device visually or audibly notifies them of warnings sent from the server through a display device. In addition, the server uses a query generation mechanism to generate specific questions for the user to verify the truthfulness of what the other party is saying, and presents them on the device. This allows the user to continue safe communication while protecting their conversation from the risk of fraud.

[0738] As a concrete example, consider a scenario where a user receives a phone call offering a "special investment opportunity." When an acoustic recognition system transmits the conversation to a server in real time, a language analysis system detects dangerous phrases and immediately generates a warning. Simultaneously, a prompt such as "Please email me the contract details later" is presented, allowing the user to respond appropriately.

[0739] Examples of prompt statements are as follows:

[0740] Conversation text: "There is a special investment opportunity. You will regret it if you miss this opportunity."

[0741] Please generate a draft question:

[0742] "Please send me the details of this proposal via email."

[0743] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0744] Step 1:

[0745] The user's device uses its built-in microphone to acquire acoustic data during a conversation. In this process, the device converts the microphone input into a digital format and outputs the captured audio in real time. The acquired acoustic data is then sent directly to the server, ready for speech recognition.

[0746] Step 2:

[0747] The server receives audio data sent from the terminal and converts it into text data using the Google Cloud Speech-to-Text API. Here, the input is audio waveform data, which the API analyzes and outputs as a corresponding text string. This process converts speech to text.

[0748] Step 3:

[0749] The server processes the converted string data using the Python library "spaCy" for natural language processing. In this step, the server receives the text data as input, analyzes the words and expressions it contains, and checks for any fraudulent elements. The output is the result of the determination of whether or not fraudulent elements are present.

[0750] Step 4:

[0751] If fraudulent elements are detected, the server activates its alert generation mechanism and generates a warning message. This process uses the analysis results as input to construct the content of the warning. Once the warning message is generated, it is ready to alert the user.

[0752] Step 5:

[0753] The server sends a generated warning message to the terminal. The terminal receives this and displays the warning to the user visually or audibly using a notification visualization device. Here, the input is the warning message, and the output is a notification in a format recognizable to the user. This allows the user to recognize fraudulent elements in real time.

[0754] Step 6:

[0755] In addition, the server uses a query generation mechanism to generate specific questions for the user to ask the other party in the conversation for confirmation. The input for this step is the detected fraudulent elements, which are used to generate the questions. The output is a set of question suggestions that the user can use immediately, and these are displayed on the terminal.

[0756] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0757] This invention is a system that, while a user is having a conversation, uses speech recognition technology, natural language processing technology, and an emotion engine to analyze the user's emotional state in real time, detect fraudulent elements, and provide appropriate warnings and suggested questions tailored to the user's psychological state.

[0758] When a user engages in conversation using a device, the device collects audio through its microphone. The collected audio data is sent to a server, where it is converted into text data by speech recognition. This text data is then analyzed by natural language processing to check for any potentially fraudulent words or phrases. The server also uses an emotion engine to analyze the user's voice to determine their emotional state. This analysis helps determine whether the user is feeling anxious, excited, or relaxed.

[0759] If fraudulent elements are detected, the server uses a warning generation mechanism to create a warning message tailored to the user's emotional state and sends it to the user's device via a warning notification mechanism. By adjusting the content and tone of the message based on the user's emotional state, more effective warnings can be provided. The device presents the warning visually or audibly to help the user understand the situation.

[0760] Furthermore, the server uses a question generation mechanism to generate questions that allow the user to efficiently verify the authenticity of the other party. These question suggestions are also adjusted to take the user's emotional state into account. For example, if the user is feeling anxious, questions that provide greater reassurance will be suggested.

[0761] As a concrete example, when a user hears a special investment pitch over the phone, the device sends the conversation to a server where speech recognition and natural language processing are performed in real time. The server detects fraudulent elements and, simultaneously, uses an emotion engine to determine that the user is feeling uneasy. Based on this, the server creates reassuring questions for the user, such as "What are the risks of this investment?", and provides the user with related warning messages. This process allows the user to assess the risk of fraud and take appropriate action while being psychologically supported.

[0762] The following describes the processing flow.

[0763] Step 1:

[0764] The user initiates a conversation using the device. The device uses its built-in microphone to capture audio data of the conversation in real time.

[0765] Step 2:

[0766] The terminal converts the acquired audio data into a digital signal and sends it to the server. This is so that the server can process the audio data.

[0767] Step 3:

[0768] The server uses speech recognition to convert the received audio data into text data. This conversion is performed in real time.

[0769] Step 4:

[0770] The server applies natural language processing techniques to text data to analyze and detect fraudulent words and phrases.

[0771] Step 5:

[0772] Simultaneously, the server uses an emotion engine to analyze the user's emotional state from the voice data. This allows it to determine whether the user is feeling anxious or stressed.

[0773] Step 6:

[0774] If fraudulent elements are detected, the server generates a warning message using a warning generation mechanism based on the user's emotional state.

[0775] Step 7:

[0776] The server sends the generated warning message to the user's terminal via a warning notification system.

[0777] Step 8:

[0778] The device receives a warning message and presents it to the user visually or audibly. Through this, the user becomes aware of the potential for fraud.

[0779] Step 9:

[0780] The server uses a question generation mechanism to generate specific questions that allow the user to verify the truthfulness of the other party's statements.

[0781] Step 10:

[0782] The server sends the generated question drafts to the user's terminal, which then displays them to the user. The user can then use this information to make a better judgment about the other party's trustworthiness.

[0783] (Example 2)

[0784] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0785] In conversations involving fraud or dishonest activities, there is a challenge in preventing users from making poor decisions due to anxiety, and in allowing them to continue the conversation with peace of mind. Furthermore, it is necessary to understand the user's emotional state and provide appropriate warnings and countermeasures based on the situation. By addressing these challenges, users can communicate calmly and protect themselves from potential dangers.

[0786] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0787] In this invention, the server includes speech recognition means for converting audio signals into text information, natural language processing means for analyzing the text information to detect elements indicating potential misconduct, and sentiment analysis means for analyzing emotional states. This makes it possible to analyze the user's situation in real time and generate and display appropriate warnings and suggested questions.

[0788] "Audio signals" are information that represents sound as electrical signals, which are then converted into a format that can be processed by a speech recognition system.

[0789] "Speech recognition means" refers to a technology that converts speech signals into textual information, and has the function of mechanically analyzing speech and transcribing it into text.

[0790] "Textual information" refers to information in text format converted from audio signals, which can be visually displayed and analyzed.

[0791] "Natural language processing means" refers to technical means that analyze textual information, understand the context and meaning contained therein, and identify elements that indicate the possibility of fraudulent activity.

[0792] "Emotion analysis means" refers to a technology that estimates a user's emotional state from audio signals and text information, and analyzes the type and intensity of that emotion in real time.

[0793] A "warning generation means" is a technical means that creates appropriate warning content for the user based on the analyzed information.

[0794] A "warning notification means" is a technical means for communicating a generated warning message to the user, and presents the warning visually or audibly.

[0795] A "question generation method" is a technical means that allows a user to generate questions to verify the truthfulness of information provided by their conversation partner, and provides appropriate questions based on the flow of the conversation.

[0796] A "notification display means" is a technical means that presents warning messages or suggested questions to the user, and has the function of conveying information using a screen or audio output.

[0797] This invention is a system that analyzes the content of conversations between users and provides a safe and secure dialogue environment. It mainly uses the following hardware and software. The terminal includes a microphone for collecting voice and a display or speaker for providing information to the user. The server is a high-performance computing device that is responsible for the computational processing required to analyze the voice data.

[0798] The server utilizes a speech recognition engine to convert audio signals into text. Specifically, it uses a major cloud-based speech recognition service. The text information is analyzed by a natural language processing engine to detect specific patterns that may indicate fraudulent activity. This process employs natural language processing libraries and models. The server also uses an emotion analysis engine to estimate the user's emotional state and analyzes those emotions in real time.

[0799] Furthermore, the server uses a warning generation mechanism to create an appropriate warning message based on the analysis results. This warning is sent to the user's terminal via a warning notification mechanism, and the terminal presents the information to the user via display or audio. Depending on the user's emotional state, a question suggestion generation mechanism proposes questions to verify the authenticity of the person they are interacting with.

[0800] As a concrete example, when a user receives a special investment offer over the phone, the device sends the audio to a server, where speech recognition and natural language processing are performed. The server detects fraudulent elements, and at the same time, an emotion analysis engine determines that the user is feeling uneasy. Based on this information, the server generates a question such as, "What are the specific risks of this investment?" and provides a relevant warning message.

[0801] An example of a prompt message is, "Analyze the emotions detected during the conversation and provide appropriate warnings and countermeasures." In this way, the system can analyze the user's conversation and support safe dialogue in real time.

[0802] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0803] Step 1:

[0804] The terminal collects the user's speech as an audio signal through the microphone. It then converts the collected audio signal into data packets and prepares them for transmission to the server. The input is the user's voice, and the output is the audio data sent to the server.

[0805] Step 2:

[0806] The server receives audio data sent from the terminal. It uses a speech recognition engine to convert the audio signal into text information. The input is audio data, and the output is text data.

[0807] Step 3:

[0808] The server analyzes the generated text data using a natural language processing engine. This analysis identifies language patterns and specific keywords that indicate fraudulent activity. The input is text data, and the output is an assessment of fraud risk.

[0809] Step 4:

[0810] The server uses an emotion analysis engine to estimate and analyze the user's emotional state from voice data. This allows it to determine what emotions the user is experiencing. While the input is voice data, text information may also be included in the analysis. The output is data indicating the user's emotional state.

[0811] Step 5:

[0812] The server generates a warning message using a warning generation mechanism based on fraud risk and emotional state. The content and tone of the message are adjusted according to the user's emotions. The input is fraud risk assessment information and emotional state data, and the output is the warning message.

[0813] Step 6:

[0814] The terminal receives warning messages sent from the server. It then notifies the user visually or audibly, presenting the information in a way that is easy for the user to understand. The input is the warning message, and the output is the notification to the user.

[0815] Step 7:

[0816] The server uses a question generation mechanism to generate questions for the user to verify the truthfulness of information provided by the conversation partner. The content is adjusted according to the user's emotional state. The input is emotional state data, and the output is the question proposal.

[0817] Step 8:

[0818] The terminal presents the generated question proposals to the user, supporting the user in making decisions. The input is the question proposal, and the output is the presentation of the question proposals.

[0819] (Application Example 2)

[0820] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0821] There is a need to mitigate the risk of fraud that users experience through conversations, while simultaneously reducing the psychological burden. Traditional fraud detection systems do not take into account the user's emotional state, making it difficult to provide appropriate warnings and guidance, and thus insufficient to help users understand the situation. In particular, it is important for the system to respond flexibly when the user feels distressed or anxious in a fraudulent situation.

[0822] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0823] In this invention, the server includes speech recognition means, natural language processing means, and sentiment analysis means. This makes it possible to analyze the user's conversation in real time and detect fraudulent elements, as well as to understand the user's emotional state and provide appropriate warnings or questions based on that emotion.

[0824] "Speech recognition means" refers to a means for converting speech data into text data.

[0825] "Natural language processing means" are methods for analyzing obtained text data and extracting fraudulent elements and important information.

[0826] A "fraudulent element detection method" is a means of identifying potentially fraudulent words or phrases from conversation content.

[0827] A "warning generation method" is a means for creating a warning message to notify a user of the possibility of fraud.

[0828] A "warning notification method" is a means of delivering a generated warning message to the user.

[0829] A "question proposal generation method" is a means of suggesting questions that help the user better understand the content of a conversation.

[0830] A "notification display means" is a means of presenting warnings or questions to the user visually or audibly.

[0831] An "emotional analysis tool" is a means of analyzing a user's voice to understand their emotional state and psychological condition.

[0832] "Emotional state-based warning adjustment means" refers to a means of adjusting the content and tone of warning messages according to the user's emotional state.

[0833] The "station / train response information generation means" is a means for generating additional information according to the situation the user is dealing with.

[0834] The system for implementing this invention aims to detect fraud risks and provide appropriate warnings and questions to the user by analyzing the user's voice in real time during their conversation. The server uses speech recognition technology to convert the voice into text data, which is then processed using natural language processing. This process utilizes the Google Cloud Speech Recognition API and the Hugging Face Transformers library. This allows for the immediate detection of fraudulent elements hidden in the user's conversation.

[0835] The server also uses sentiment analysis technology to analyze the user's voice and evaluates emotions using the Sentiment Analysis library. This analysis allows the server to understand the user's emotional state during a conversation and adjust the content and tone of warnings accordingly.

[0836] The device notifies the user of warning messages and suggested questions via Firebase, presenting them visually or audibly. This allows the user to intuitively understand the risk of fraud and respond with confidence.

[0837] For example, if a user is listening to suspicious product information provided by a salesperson over the phone, the system will automatically generate a question such as, "What risks are associated with this product?" to support the user in making a decision with confidence.

[0838] Examples of prompts for a generative AI model:

[0839] "Assess the potential for fraud based on the following conversation: Conversation Text. Generate appropriate warnings and questions based on the user's emotional state."

[0840] This system allows users to stay safe during everyday conversations while quickly addressing non-face-to-face fraud.

[0841] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0842] Step 1:

[0843] The device captures the user's conversation in audio format. It collects audio data through the microphone and transmits it to the server in real time. At this stage, the input data is an analog audio signal.

[0844] Step 2:

[0845] The server digitizes the received audio data and converts it into text data using the Google Cloud Speech Recognition API. The input is audio data, and the output is text data. This conversion makes subsequent natural language processing easier.

[0846] Step 3:

[0847] The server uses the Hugging Face Transformers library to perform natural language processing on text data. This process identifies words and phrases that contain fraudulent elements. The input is text data, and the output is information about the fraudulent sections.

[0848] Step 4:

[0849] The server uses the Sentiment Analysis library to analyze the user's emotional state from their voice. The input is voice data, and the output is data indicating the user's emotional state. This analysis result is used to adjust warning messages.

[0850] Step 5:

[0851] The server generates appropriate warning messages and suggested questions using a warning generation mechanism, based on the analysis results of fraudulent elements and emotional state. The input for this step is information on fraudulent elements and emotional state data, and the output is the warning message and suggested questions to be presented to the user.

[0852] Step 6:

[0853] The device notifies the user of warning messages and suggested questions via Firebase, presenting them visually or audibly as needed. The input is the generated warning messages and suggested questions, and the output is what is presented to the user visually or audibly.

[0854] Based on these warnings and suggested questions, users can confidently evaluate the conversation and take appropriate action.

[0855] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0856] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0857] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0858] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0859] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0860] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0861] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0862] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0863] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0864] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0865] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0866] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0867] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0868] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0869] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0870] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0871] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0872] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0873] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0874] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0875] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0876] The following is further disclosed regarding the embodiments described above.

[0877] (Claim 1)

[0878] Voice recognition means and

[0879] Natural language processing means,

[0880] Fraudulent element detection method,

[0881] Warning generation means and

[0882] Warning notification means,

[0883] A means for generating draft questions,

[0884] Notification display means,

[0885] A system that includes this.

[0886] (Claim 2)

[0887] The system according to claim 1, comprising speech recognition means for converting speech data into text data.

[0888] (Claim 3)

[0889] The system according to claim 1, comprising a natural language processing means for identifying dangerous words or phrases in text data.

[0890] "Example 1"

[0891] (Claim 1)

[0892] A means of acquiring sound,

[0893] Voice recognition means and

[0894] Natural language processing means,

[0895] Fraudulent element detection method,

[0896] Warning generation means and

[0897] Warning notification means,

[0898] A means for generating draft questions,

[0899] Notification display means,

[0900] A system that includes this.

[0901] (Claim 2)

[0902] The system according to claim 1, comprising a speech recognition means for acquiring voice data in real time, transmitting it via data communication, and converting the voice data into text data.

[0903] (Claim 3)

[0904] The system according to claim 1, comprising natural language processing means for identifying dangerous words and expressions in text data and detecting fraudulent elements using a generative AI model.

[0905] "Application Example 1"

[0906] (Claim 1)

[0907] Acoustic recognition means,

[0908] Language analysis tools,

[0909] Methods for detecting fraudulent ingredients,

[0910] Attention generation means,

[0911] Warning notification means,

[0912] Methods for creating inquiry proposals,

[0913] Notification visualization means,

[0914] A system that includes this.

[0915] (Claim 2)

[0916] The system according to claim 1, comprising an acoustic recognition means for converting acoustic data into string data.

[0917] (Claim 3)

[0918] The system according to claim 1, comprising a language analysis means for identifying dangerous words and expressions in string data.

[0919] "Example 2 of combining an emotion engine"

[0920] (Claim 1)

[0921] A device for collecting audio signals,

[0922] A speech recognition means that converts collected audio signals into text information,

[0923] A natural language processing means that analyzes textual information and detects elements that indicate the possibility of fraudulent activity,

[0924] A means of analyzing emotional states,

[0925] A warning generation means that creates a warning message that prompts appropriate attention based on emotional state,

[0926] A warning notification means for sending a warning message,

[0927] A question generation method for generating questions to verify the authenticity of the person being spoken to,

[0928] A notification display means that presents a warning statement and a draft question,

[0929] A system that includes this.

[0930] (Claim 2)

[0931] The system according to claim 1, comprising speech recognition means for converting audio signals into text information.

[0932] (Claim 3)

[0933] The system according to claim 1, comprising a natural language processing means for identifying words or phrases in textual information that indicate potential danger.

[0934] "Application example 2 when combining with an emotional engine"

[0935] (Claim 1)

[0936] Voice recognition means and

[0937] Natural language processing means,

[0938] Fraudulent element detection method,

[0939] Warning generation means and

[0940] Warning notification means,

[0941] A means for generating draft questions,

[0942] Notification display means,

[0943] Emotion analysis methods,

[0944] A warning adjustment mechanism based on emotional state,

[0945] Station train response information generation means,

[0946] A system that includes this.

[0947] (Claim 2)

[0948] The system according to claim 1, comprising speech recognition means for converting speech data into text data.

[0949] (Claim 3)

[0950] The system according to claim 1, comprising a natural language processing means for identifying dangerous words or phrases in text data. [Explanation of Symbols]

[0951] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A sound recognition means that receives sound data and converts it into string data, A language analysis means that applies natural language processing to the converted string data to identify dangerous words and expressions, A fraudulent component detection method that determines whether an identified word or expression has fraudulent elements, A warning generation method that generates a warning to the user when fraudulent elements are detected, A warning notification means that visually or audibly notifies the user of the generated warning, A means for generating inquiry proposals that automatically generates specific questions for users to verify the authenticity of suspicious conversations, A notification visualization means that visually displays information to the user, A system that includes this.

2. The system according to claim 1, comprising an acoustic recognition means for converting acoustic data into string data.

3. The system according to claim 1, comprising a language analysis means for identifying dangerous words and expressions in string data.

Images