System

Abstract

A system is provided.SOLUTION: A system comprising: an artificial intelligence apparatus that responds to an incoming call; means for converting information of a caller into text using a voice recognition technique; means for calculating reliability of a call based on attribute information of the caller; and means for notifying a recipient of the call based on the reliability.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In modern society, special frauds and nuisance sales calls using telephones have become a serious problem. In particular, elderly people and those with dementia are often caught up in these scams, and responding to nuisance sales calls is also a burden for companies. A method to solve these problems and ensure safe and smooth telephone responses is needed. [Means for solving the problem]

[0005] This invention solves the above problems by the following means. First, an artificial intelligence device is installed that answers incoming calls. Next, a means is provided to convert the caller's information into text using voice recognition technology and generate text data. Furthermore, a means is included to calculate the reliability of the call based on the caller's attribute information. Also, by providing a means to notify the recipient of the call based on the reliability, it becomes possible to make only calls that are truly necessary. The system also includes a means to analyze the content of the conversation during the call and check whether it matches the initial information, and a means to store the caller's attribute information in a database and adjust the reliability based on past call history.

[0006] An "artificial intelligence device" is a computer system that runs a program to automatically answer telephone calls and collect and analyze caller information.

[0007] "Speech recognition technology" is a technology that receives voice data and converts it into text data.

[0008] "Text conversion" refers to the process of expressing information input through speech as text.

[0009] "Caller demographic information" refers to detailed information about the caller, such as the caller's phone number, location, company name, and past call history.

[0010] "Confidence" refers to an index, such as a percentage, calculated to assess whether a caller is a legitimate caller.

[0011] "Means for notifying a call" refers to a mechanism for notifying the recipient's terminal of the existence of a call based on the trustworthiness of the caller.

[0012] "Conversation content analysis" refers to the process of analyzing conversations during a call in real time and converting the content into text data.

[0013] A "database" is a collection of information that stores caller attribute information and past call history, and can be searched and updated as needed.

[0014] "Past call history" refers to records of previous calls, such as the date and time, caller information, and call content. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] This invention is a system that uses artificial intelligence (AI) to handle primary telephone inquiries, eliminating special frauds and nuisance sales calls and realizing safe and efficient telephone response. Specific embodiments are described below.

[0037] System configuration

[0038] This system consists of three elements: a server, a terminal, and a user.

[0039] Server: The core part of the system where AI is executed. It analyzes caller information, calculates reliability, decides whether to notify the caller, and analyzes the content of the conversation during the call.

[0040] Terminal: A device used by a user. It detects incoming calls, plays an automatic answering message, and receives notifications from the server and notifies the user.

[0041] User: The person who uses this system, such as an elderly person, a home resident, or a company employee. The user receives notifications through the terminal and answers the call as needed.

[0042] Program processing

[0043] 1. Receiving calls and initial response

[0044] The server detects an incoming call via the device and instructs the device to play an automated response message: "This is an AI support desk. Please tell us your name and request," and records the caller's voice.

[0045] 2. Collection and analysis of caller information

[0046] The recorded voice data of the caller is sent to a server, which uses speech recognition technology to convert this voice data into text, extracting and summarizing who is calling, who the call is addressed to, and what the request is.

[0047] 3. Calculating reliability

[0048] The server retrieves the caller's attribute information (such as phone number, area, company name, and past call history) from a database and calculates the caller's trustworthiness based on this information. If the caller has withheld their number, the trustworthiness is set significantly lower.

[0049] 4. Notifications and call connections

[0050] If the reliability exceeds a certain threshold (e.g., 90%), the server converts the result from text to speech and notifies the user via the device. The user can then confirm the content of the notification and choose to accept or reject the call. The device will then connect or reject the call according to the user's instructions.

[0051] 5. Monitoring and updating attribute information during a call

[0052] During the call, the server analyzes the conversation in real time to check whether it matches the initial information. If there is a mismatch, the server issues a warning and updates the attribute information in the database.

[0053] Specific examples

[0054] When an elderly person answers a phone call

[0055] 1. The device detects an incoming call and notifies the server.

[0056] 2. The device plays an automated answering message and records the voice of the caller, who identifies himself as "Yamada Taro." The recording data is sent to the server.

[0057] 3. The server converts the voice data into text and summarizes it as "Caller: Yamada Taro, Recipient: Grandpa, Purpose: Check status."

[0058] 4. The server inspects the attribute information and calculates the reliability as 99%.

[0059] 5. The user (grandfather) is notified via the terminal that "Yamada Taro is calling to check on your status."

[0060] 6. The user presses the answer button to connect the call.

[0061] 7. During the call, the server monitors the conversation and verifies that it matches.

[0062] When a company receives nuisance calls

[0063] 1. The device detects an incoming call and notifies the server.

[0064] 2. The device plays an automated answering message and records the voice of the anonymous caller.

[0065] 3. The recorded data is sent to a server, where it is converted into text. The summary is "Caller: Anonymous, Destination: Unknown, Requirement: Sales."

[0066] 4. The server checks the attribute information and calculates the reliability as 10%.

[0067] 5. The server decides not to notify and the terminal does not connect the call.

[0068] In this way, the system can automatically answer calls under certain conditions, notifying and connecting the call recipient only if the call is trustworthy, effectively filtering out fraudulent and nuisance calls.

[0069] The processing flow will be explained below.

[0070] Step 1:

[0071] The terminal detects the incoming call and notifies the server of the information.

[0072] Specific operation: When the terminal receives an incoming call signal from the telephone line, it notifies the server of the incoming call.

[0073] Step 2:

[0074] The server instructs the terminal to play back the automatic response message.

[0075] Specific operation: The server sends a pre-prepared response message to the device and commands the device to play it. The message played is, "This is the AI ​​support desk. Please tell us your name and requirements."

[0076] Step 3:

[0077] The terminal records the caller's voice and transmits the voice data to the server.

[0078] Specific operation: When the caller answers and states their name and requirements, the terminal records the voice and transmits the data to the server in real time.

[0079] Step 4:

[0080] The server sends the voice data to a voice recognition engine and converts it into text data.

[0081] Specific operation: The server sends the received voice data to a cloud-based voice recognition service, converts the caller's speech into text, and receives it.

[0082] Step 5:

[0083] The server analyzes the text, extracts information such as "who is calling," "who is it addressed to," and "what the requirements are," and generates a summary.

[0084] Specific operation: Analyzes text using natural language processing (NLP) algorithms to obtain sender information, recipient information, and requirement information.

[0085] Step 6:

[0086] The server obtains the sender's attribute information from the database and calculates the reliability.

[0087] Specific operation: The server retrieves data such as the caller's phone number, area, company name, and past call history from a database, and calculates its reliability using a statistical model.

[0088] Step 7:

[0089] The server determines whether the confidence level exceeds a set threshold (e.g., 90%) and issues a result.

[0090] Specific operation: If the reliability is 90% or more, it is determined to be a legitimate call and proceeds to the next step. If it is less than 90%, the call is not notified and the call is terminated.

[0091] Step 8:

[0092] The server converts the results from text to speech and notifies the device.

[0093] Specific operation: The server sends the summarized text data to a speech synthesis engine to generate an audio file, which is then sent to the terminal for playback.

[0094] Step 9:

[0095] The terminal plays the notification message to the user.

[0096] Specific operation: The terminal plays the message "This is a call from Mr. / Ms. XX. The request is XX" to the user.

[0097] Step 10:

[0098] The user checks the notification and decides whether to respond.

[0099] Specific operation: After checking the notification message, the user can choose to press the answer button or the reject button.

[0100] Step 11:

[0101] The terminal connects or rejects the call according to the user's instructions.

[0102] Specific behavior: If the user presses the answer button, the call will be connected. If the user presses the reject button, the call will be ended.

[0103] Step 12:

[0104] The server analyzes the content of the call in real time and checks whether it matches the initial information.

[0105] Specific operation: The server analyzes the contents of the call using voice recognition and NLP, and if any content differs from the summary, it issues an alert.

[0106] In this way, the system reduces the risk of fraud and unwanted calls and efficiently directs only necessary calls.

[0107] Example 1

[0108] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0109] In modern times, telephone fraud and nuisance sales calls are on the rise. These calls are a major problem, especially for the elderly and businesses, and there is a need for methods to prevent these attacks. However, conventional telephone answering systems are unable to adequately address these issues. An effective system is needed to automatically filter out unreliable callers and notify recipients of only reliable calls.

[0110] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0111] In this invention, the server includes an artificial intelligence device that responds to incoming calls, a means for converting caller information into text using voice recognition technology, a means for calculating the reliability of the call based on the caller's attribute information, and a means for notifying the recipient of the call when the reliability exceeds a certain threshold. This makes it possible to automatically filter out calls from unreliable callers and efficiently deliver only reliable calls to the recipient. Furthermore, by providing a means for monitoring the content of conversations during calls in real time and issuing an alert if there is a discrepancy, further protection from fraud and nuisance calls can be achieved. The reliability calculation uses a machine learning algorithm, improving the accuracy and reliability of the system.

[0112] An "artificial intelligence device that answers incoming calls" is a device that detects an incoming call and automatically plays a response message.

[0113] "Means of converting caller information into text using voice recognition technology" refers to technology for analyzing recorded voice data and converting it into text data.

[0114] The "means for calculating the reliability of a call based on the caller's attribute information" is a means for evaluating attribute information such as the caller's telephone number, region, company name, and past call history, and calculating its reliability.

[0115] The "means for notifying the recipient of a call when the reliability exceeds a certain threshold" refers to a means for notifying the recipient of a call when the calculated reliability exceeds a set threshold.

[0116] "Means for monitoring the content of conversations during calls in real time and issuing an alert if there is a discrepancy" refers to a means for constantly monitoring the content of conversations during calls and issuing an alert if a discrepancy is detected between the information obtained the first time and the content.

[0117] A "machine learning algorithm" is a computer algorithm that learns patterns from data and uses them to make future predictions and classifications.

[0118] This invention is a system that uses artificial intelligence (AI) to handle the primary telephone contact point, eliminating special frauds and nuisance sales calls and realizing safe and efficient telephone response. This system consists of three elements: a server, a terminal, and a user. Specific embodiments are described below.

[0119] System configuration

[0120] 1. Server

[0121] This is the core part of the system where the AI ​​runs. The server is responsible for detecting incoming calls, voice recognition, calculating reliability, and monitoring call content in real time. It mainly uses the following software:

[0122] Google Speech-to-Text API or IBM Watson Speech to Text API: For speech recognition technology.

[0123] Amazon Polly or Google Cloud Text-to-Speech: For converting text to speech.

[0124] scikit-learn or XGBoost: For implementing machine learning algorithms.

[0125] 2. Terminal

[0126] A device used by a user. It detects incoming calls, plays an automatic answering message, and sends the audio data to a server. It also receives notifications from the server and conveys them to the user. Specifically, it is a smartphone or dedicated telephone terminal that the user owns.

[0127] 3. Users

[0128] The person using the system can be an elderly person, a home resident, a company employee, etc. The user receives a notification through the terminal and can answer the call if necessary.

[0129] Specific operation of the system

[0130] 1. Receiving calls and initial response

[0131] The device detects an incoming call and sends the detected call information to the server.

[0132] The server instructs the terminal to play an automatic response message.

[0133] The device plays a message saying, "This is an AI support desk. Please tell us your name and request," and records the caller's voice.

[0134] 2. Collection and analysis of caller information

[0135] The recorded audio data is sent from the terminal to the server.

[0136] The server uses voice recognition technology to convert the voice data into text, extracting information such as "who is calling," "who is it addressed to," and "what the requirement is."

[0137] 3. Calculating reliability

[0138] The server retrieves the caller's attribute information from a database, including phone number, area, company name, and past call history.

[0139] Calculate the trustworthiness of the sender using a machine learning algorithm, such as the Random Forest or XGBoost algorithms from scikit-learn.

[0140] 4. Notifications and call connections

[0141] If the reliability exceeds a set threshold, the server converts the text into speech and sends the speech data to the terminal.

[0142] The terminal plays the notification content to the user, who can then choose to respond or reject it.

[0143] If the user chooses to answer, the terminal connects the call.

[0144] 5. Monitoring and updating attribute information during a call

[0145] During the call, the server analyzes the conversation in real time to see if it matches the initial information.

[0146] If there is no match, the server issues a warning and notifies the user via the terminal.

[0147] The attribute information in the database is updated as needed.

[0148] Specific examples

[0149] When an elderly person answers a phone call

[0150] 1. The device detects an incoming call and notifies the server.

[0151] 2. The device plays an automated response message and records the caller's voice.

[0152] 3. The recorded data is sent to the server, where it is converted into text. The text is summarized as "Caller: XXX, Recipient: Grandpa, Message: Status check."

[0153] 4. The server checks the attribute information and calculates the reliability.

[0154] 5. Since the server has a high reliability, it sends the result to the terminal and notifies the user, "Mr. / Ms. XXX is calling to check on your status."

[0155] 6. The user presses the answer button to connect the call.

[0156] 7. During the call, the server monitors the conversation to ensure it matches.

[0157] When a company receives nuisance calls

[0158] 1. The device detects an incoming call and notifies the server.

[0159] 2. The device plays an automated response message and records the caller's voice.

[0160] 3. The recorded data is sent to a server, where it is converted into text. The text is summarized as "Caller: Anonymous, Destination: Unknown, Requirement: Sales."

[0161] 4. The server checks the attribute information and calculates the reliability.

[0162] 5. The server decides not to notify and the terminal does not connect the call.

[0163] Example prompt sentence:

[0164] "Please tell me the specific steps for a system that uses AI to prevent nuisance and fraudulent calls when elderly people receive phone calls."

[0165] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0166] Step 1:

[0167] The device detects an incoming call and sends the incoming call information to the server. The input includes the phone number and the time of the call. The device collects this data and sends it to the server to notify the call.

[0168] Specific operation: When the phone rings, the device's sensor reacts, captures the incoming call information (phone number and time) in the internal memory, and immediately transfers it to the server.

[0169] Step 2:

[0170] The server receives the incoming call information and sends an instruction to the terminal to play the automatic response message. The server receives the incoming call information from the terminal as input. Based on the received information, the server generates a response message and sends that instruction to the terminal.

[0171] Specific operation: The server analyzes the received telephone number and time information, and sends a command to the terminal to play the automatic response message.

[0172] Step 3:

[0173] The device plays a message saying, "This is an AI support desk. Please tell us your name and requirements," and records the caller's voice. As input, it receives a playback instruction from the server. The device receives the playback instruction and starts recording the voice while playing the specified message to the caller.

[0174] Specific operation: The device's speaker plays the message according to the instructions, the microphone records the caller's voice, and the recorded voice data is stored in temporary memory.

[0175] Step 4:

[0176] The recorded voice data is sent from the device to the server. As input, the recorded voice data is retrieved from the device's memory. The device uploads the voice data to the server.

[0177] Specific operation: To send the audio file stored in the temporary memory to the server, the terminal divides the data into packets and transmits them over the network.

[0178] Step 5:

[0179] The server uses voice recognition technology to convert the voice data into text and extracts information such as "Who is calling?", "Who is the caller?", and "What is the requirement?". The server receives the voice data sent from the device as input. The server uses voice recognition software such as the Google Speech-to-Text API to convert the voice data into text and extract the necessary information.

[0180] Specific operation: Voice data is sent to a voice recognition API, and the returned text is analyzed to extract and summarize important information (sender, recipient, requirements).

[0181] Step 6:

[0182] The server retrieves the caller's attribute information from the database and calculates the caller's reliability using a machine learning algorithm. The input is the text data obtained by speech recognition and the caller information in the database. The server uses, for example, a scikit-learn algorithm to calculate the reliability.

[0183] Specific operation: A database query is issued to obtain the caller's attribute information, which is then passed as input to a machine learning model to calculate a confidence score.

[0184] Step 7:

[0185] If the confidence score exceeds a set threshold, the server sends the result to the device and instructs it to play a notification message to the user.The server uses the calculated confidence score as input.The server converts the text to speech and sends it to the device.

[0186] Specific operation: After converting text to speech, the speech data is sent to the terminal, and the terminal plays the specified notification message to the user.

[0187] Step 8:

[0188] The user checks the notification message and chooses whether to respond or reject it. The notification is received from the device as input. The user selects a response by operating the device's buttons or touch screen.

[0189] Specific operation: The device detects the user's selection and sends that information to the server.

[0190] Step 9:

[0191] The terminal connects or rejects the call based on the user's instruction. The terminal receives the user's selection as input. The terminal performs the call connection operation based on the selection.

[0192] Specific operation: The terminal sends out a call connection signal or a call rejection signal to control the telephone line appropriately.

[0193] Step 10:

[0194] During the call, the server analyzes the conversation in real time to check whether it matches the initial information. The input includes the voice data from the call. The server again uses voice recognition technology to convert it into text data and verify the information.

[0195] What it does: Continuously samples the audio during a call, transcribes it, compares it with the initial data, and generates a warning if there is a mismatch.

[0196] Step 11:

[0197] If a mismatch occurs, the server issues a warning to the device and updates the attribute information in the database. The mismatch detection information is used as input. The server adds new information to the database and updates it.

[0198] Specific operation: Add a new entry to the database based on the detected discrepancy and send a warning signal to the terminal.

[0199] (Application example 1)

[0200] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0201] In today's world, with the increase in special frauds and nuisance sales calls, it is important for call recipients to be protected from these fraudulent calls. Senior citizens and businesses are particularly at risk of business disruption and financial loss from these fraudulent and nuisance calls. Traditional manual responses are insufficient, and more efficient and reliable countermeasures are needed.

[0202] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0203] In this invention, the server includes an artificial intelligence device that responds to incoming calls, a means for converting caller information into text using voice recognition technology, and a means for calculating the reliability of the call based on the caller's attribute information. This makes it possible to eliminate special frauds and nuisance sales calls and notify the recipient of only reliable calls.

[0204] An "artificial intelligence device that answers incoming calls" is a device that has the function of detecting an incoming call and automatically playing a response message.

[0205] "Means of converting to text using voice recognition technology" refers to technology that analyzes recorded voice data and extracts and converts the caller's voice into text.

[0206] The "means for calculating trustworthiness" is a technology that quantifies the trustworthiness of a caller based on the caller's attribute information (telephone number, region, company name, past call history, etc.).

[0207] The "means for notifying the recipient of a call" is a technology that has the function of notifying the recipient of a highly reliable call based on the calculated reliability.

[0208] The "means for transmitting notification content to the recipient by voice synthesis" is a technology for transmitting notification content generated by the server to the recipient as voice using voice synthesis technology.

[0209] "Means for analyzing the content of conversations during a call" refers to technology that monitors the conversation between the caller and recipient in real time during a call and checks whether it matches information obtained in advance.

[0210] "Means for storing caller attribute information in a database" refers to technology that has the function of recording and storing information such as caller characteristics and call history in a database.

[0211] "Means for adjusting reliability based on past call history" refers to a technology that analyzes accumulated call data and updates and adjusts the reliability of callers in real time.

[0212] "Means for providing response options" refers to a technique that presents a reliable call so that the recipient can choose whether to respond.

[0213] "Means for automatically disconnecting a call" refers to technology that has the function of automatically terminating a call when an unreliable call is detected.

[0214] MODE FOR CARRYING OUT THE INVENTION

[0215] System configuration

[0216] This invention is a system that uses AI to handle the primary telephone call counter and eliminate special frauds and nuisance sales calls. The system consists of three elements: a server, a terminal, and a user.

[0217] 1. Server: This is the core part of the system, and it analyzes caller information, calculates reliability, decides whether to notify, and analyzes the content of conversations during calls. The software used includes Python, speech_recognition, pyttsx3, and requests.

[0218] 2. Terminal: A device used by a user. It detects incoming calls, plays an automatic answering message, and receives notifications from the server and notifies the user.

[0219] 3. User: The person who uses this system, such as an elderly person, a home resident, or a company employee. The user receives notifications through the terminal and answers the call as needed.

[0220] System Operation

[0221] Receiving and initial phone calls

[0222] The server detects an incoming call via the device and instructs the device to play an automated answering message: "This is an AI support desk. Please tell us your name and request," and records the caller's voice.

[0223] Collection and analysis of caller information

[0224] The recorded voice data of the caller is sent to a server, which uses speech recognition technology to convert the data into text, extracting and summarizing who is calling, who the call is addressed to, and what the request is.

[0225] Calculating reliability

[0226] The server retrieves the caller's attribute information (such as phone number, area, company name, and past call history) from a database and calculates the caller's trustworthiness based on this information. If the caller has withheld their number, the trustworthiness is set significantly lower.

[0227] Notifications and call connections

[0228] If the reliability exceeds a certain threshold (e.g., 90%), the server converts the result from text to speech and notifies the user via the device. The user can then confirm the content of the notification and choose whether to answer or reject the call. The device will then connect or reject the call according to the user's instructions.

[0229] Monitoring and updating attribute information during a call

[0230] During the call, the server analyzes the conversation in real time to check whether it matches the initial information. If there is a mismatch, the server issues a warning and updates the attribute information in the database.

[0231] Specific examples

[0232] When an elderly person answers a phone call

[0233] 1. The device detects an incoming call and notifies the server.

[0234] 2. The device plays an automated response message and records the caller's voice. The recording is then sent to the server.

[0235] 3. The server converts the voice data into text and summarizes it as "Sender: Mr. X, Address: Recipient, Request: Status check."

[0236] 4. The server checks the attribute information and calculates the reliability.

[0237] 5. The user is notified via the terminal that "Mr. X is calling to check on your status."

[0238] 6. The user presses the answer button to connect the call.

[0239] 7. During the call, the server monitors the conversation and verifies that it matches.

[0240] When a company receives nuisance calls

[0241] 1. The device detects an incoming call and notifies the server.

[0242] 2. The device plays an automated answering message and records the voice of the anonymous caller.

[0243] 3. The recorded data is sent to the server, where it is converted into text. The text is summarized as "Caller: Anonymous, Address: Recipient, Requirement: Sales."

[0244] 4. The server checks the attribute information and calculates the reliability.

[0245] 5. Due to low reliability, the server decides not to notify and the device does not connect the call.

[0246] Prompt Sentence Examples

[0247] "We use an AI model that analyzes audio files, identifies the caller's name, destination, and requirements, and calculates the reliability. For example, the following is from: Caller: Mr. X, Destination: Recipient, Requirements: Status check."

[0248] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0249] Step 1:

[0250] The server detects an incoming call via the terminal. It receives the incoming call signal as input and sends an instruction to the terminal to play an automatic answering message as output. Specifically, the terminal's microphone is enabled and goes into recording mode.

[0251] Step 2:

[0252] The device plays an automated answering message saying, "This is an AI support desk. Please tell us your name and request," and records the caller's voice. It receives incoming call signals and playback instructions as input, generates voice data as output, and sends it to the server. Specifically, the message is played from the speaker, and the caller's voice is recorded by the microphone.

[0253] Step 3:

[0254] The server receives the recorded voice data and converts it into text using speech recognition technology. It receives voice data as input and generates text data as output. Specifically, it analyzes the voice data using the speech_recognition library and converts the caller's name and requirements into text.

[0255] Step 4:

[0256] The server analyzes the text data and extracts and summarizes information such as "who is calling," "who is the caller," and "what are the requirements?" It receives text data as input and generates summary data as output. Specifically, it uses a generative AI model to extract the necessary information from the text.

[0257] Step 5:

[0258] The server retrieves the caller's attribute information from a database and calculates the reliability. It takes the caller's number and past call history as input and generates a reliability score as output. Specifically, it executes a database query and calculates the reliability based on the obtained data.

[0259] Step 6:

[0260] The server determines whether the confidence score exceeds a certain threshold (for example, 90%) and notifies the device. It receives the confidence score as input and converts the notification content into audio as output, which is then sent to the device. Specifically, it uses the pyttsx3 library to convert text to audio.

[0261] Step 7:

[0262] The device receives the notification from the server and conveys the notification content to the recipient. The recipient checks the notification and chooses whether to answer or reject the call. The device receives the notification content as input and sends the recipient's choice to the server as output. Specific operations include playing the notification content from a speaker and displaying an interface for receiving user input.

[0263] Step 8:

[0264] If the user chooses to answer the call, the terminal connects the call by receiving the user's selection as input and generating a call connection signal as output, specifically by opening a line for sending and receiving the call's voice data in real time.

[0265] Step 9:

[0266] During the call, the server analyzes the conversation in real time to check whether it matches the initial information. It receives the call audio data as input and generates the analysis results as output. Specifically, it uses the speech_recognition library again to convert the acquired audio data into text and compare it with the initial information.

[0267] Step 10:

[0268] If a mismatch occurs in the call content, the server issues a warning and updates the attribute information in the database. It receives the analysis results as input and generates updated attribute information as output. Specifically, it displays a mismatch warning on the terminal and executes an update query on the database.

[0269] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0270] This invention is a system that uses artificial intelligence (AI) as the primary point of contact for telephone calls, and combines it with an emotion engine that recognizes the user's emotions to eliminate special frauds and nuisance sales calls, providing greater safety and efficiency. Specific embodiments are described in detail below.

[0271] System configuration

[0272] This system consists of three elements: a server, a terminal, and a user, and is further combined with an emotion engine to perform advanced analysis.

[0273] Server: The core of the system, where the AI ​​and emotion engine run. It analyzes caller information, calculates reliability, decides whether to notify the user, and analyzes the content and emotions of the conversation during the call.

[0274] Terminal: A device used by a user. It detects incoming calls, plays an automatic answering message, and receives notifications from the server and notifies the user.

[0275] User: The person who uses this system, such as an elderly person, a home resident, or a company employee. The user receives notifications through the terminal and answers the call as needed.

[0276] Program processing

[0277] 1. Receiving calls and initial response

[0278] The server detects the incoming call via the terminal and instructs the terminal to play an automated response message: "This is an AI support desk. Please tell us your name and request," and records the caller's voice.

[0279] 2. Collection and analysis of caller information

[0280] The recorded voice data of the caller is sent to a server, which uses speech recognition technology to convert this voice data into text, extracting and summarizing who is calling, who the call is addressed to, and what the request is.

[0281] 3. Calculating reliability

[0282] The server retrieves the caller's attribute information from the database and calculates the caller's trustworthiness based on this information. If the caller's number is withheld, the trustworthiness is set significantly lower.

[0283] 4. Analysis by Emotion Engine

[0284] The server uses an emotion engine to extract emotional data from the caller's recorded voice, analyzes the emotional data, and understands the caller's emotional state.

[0285] 5. Notifications and call connections

[0286] The server comprehensively evaluates the reliability and emotional data, and if the result exceeds a certain threshold, converts the text to speech and notifies the user via the device. The user can then review the notification and choose to accept or reject the call. The device will then connect or reject the call according to the user's instructions.

[0287] 6. Call monitoring and sentiment analysis

[0288] During a call, the server analyzes the conversation content and emotional data in real time. It issues alerts based on whether the initial information matches the conversation content and changes in emotion. This allows it to quickly detect suspicious activity and issue a warning alert if necessary.

[0289] Specific examples

[0290] Flow when a user receives a call

[0291] 1. The device detects an incoming call and notifies the server.

[0292] 2. The terminal plays an automated response message, and after the caller responds, the voice is recorded and sent to the server.

[0293] 3. The server converts the recorded voice into text and summarizes the caller information. At the same time, it analyzes the emotional data using an emotion engine.

[0294] 4. The database is referenced to obtain the sender's attribute information, and the trustworthiness and emotional data are evaluated comprehensively.

[0295] 5. If the confidence and emotion data exceed a certain threshold, the notification is converted from text to speech and notified to the user again via the device.

[0296] 6. The user checks the notification and chooses whether to connect or reject the call. The device follows this choice.

[0297] 7. During the call, the server monitors the conversation content and emotional changes in real time. If it detects any discrepancies with the initial information or abnormal changes in emotions, it issues a warning.

[0298] In this way, this system can prevent fraud and nuisance calls with even greater accuracy by combining not only caller information but also an emotion engine that analyzes the caller's emotional state in real time.

[0299] The processing flow will be explained below.

[0300] Step 1:

[0301] The terminal detects the incoming call and notifies the server of the information.

[0302] Specific operation: The terminal receives an incoming call signal from the telephone line and notifies the server of this fact.

[0303] Step 2:

[0304] The server instructs the terminal to play back the automatic response message.

[0305] Specific operation: The server sends a pre-prepared message to the terminal, playing an audio message saying, "This is the AI ​​support desk. Please tell us your name and requirements."

[0306] Step 3:

[0307] The terminal records the caller's voice and transmits the voice data to the server.

[0308] Specific operation: When the caller answers and states their name and requirements, the voice is recorded and sent to the server in real time.

[0309] Step 4:

[0310] The server sends the voice data to a voice recognition engine and converts it into text data.

[0311] Specific operation: The server sends the received voice data to a cloud speech recognition service, which converts the voice into text.

[0312] Step 5:

[0313] The server analyzes the text, extracts information such as "who is calling," "who is it addressed to," and "what the requirements are," and generates a summary.

[0314] Specific operation: Analyzes text using natural language processing (NLP) algorithms to extract sender information, recipient information, and requirement information.

[0315] Step 6:

[0316] The server obtains the sender's attribute information from the database and calculates the reliability.

[0317] Specific operation: The server obtains attribute information such as the caller's phone number, region, company name, and past call history from a database, and calculates the reliability using a statistical model.

[0318] Step 7:

[0319] The server uses an emotion engine to extract emotion data from the voice data and analyze the caller's emotional state.

[0320] Specific operation: The server sends the voice data to an emotion recognition engine and extracts the caller's emotional data (joy, anger, sadness, etc.).

[0321] Step 8:

[0322] The server comprehensively evaluates the reliability and emotional data and gives a result.

[0323] Specific operation: Determine whether the confidence and emotion data exceed the set threshold (e.g., confidence 90%, emotion data within the normal range).

[0324] Step 9:

[0325] The server converts the results from text to speech and notifies the device.

[0326] Specific operation: The server sends the summarized text data to the speech synthesis engine and sends the generated audio file to the terminal.

[0327] Step 10:

[0328] The terminal plays the notification message to the user.

[0329] Specific operation: The terminal plays the message "This is a call from Mr. / Ms. XX. The request is XX" to the user.

[0330] Step 11:

[0331] The user checks the notification and decides whether to respond.

[0332] Specific operation: After checking the notification message, the user can choose to press the answer button or the reject button.

[0333] Step 12:

[0334] The terminal connects or rejects the call according to the user's instructions.

[0335] Specific operation: When the user presses the answer button, the terminal connects the call; when the user presses the reject button, the terminal ends the call.

[0336] Step 13:

[0337] The server analyzes the conversation content and emotional data during the call in real time.

[0338] Specific operation: After the call is connected, the server sends the voice data to the voice recognition engine and emotion recognition engine in real time to monitor the conversation content and emotional changes.

[0339] Step 14:

[0340] The server compares the initial information with the conversation content and emotional state, and issues a warning if there is a mismatch or if there is an invalid emotional change.

[0341] Specific operation: The server uses a matching algorithm to compare the initial summary with the conversation content and emotional changes, and if there is a mismatch, it issues a warning alert.

[0342] In this way, the system can provide high levels of security and efficiency by analyzing caller information and emotions in real time and managing calls based on trust and emotion data.

[0343] Example 2

[0344] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0345] Conventional telephone answering systems only performed voice recognition and credibility assessment of the caller based on attribute information, which meant they were unable to adequately prevent fraudulent and nuisance calls. It was also difficult to properly analyze the caller's emotional state and detect abnormalities during the call. This could expose the recipient to unnecessary stress and risk.

[0346] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes an artificial intelligence device that responds when a call is received, a means for converting caller information into text using voice recognition technology, a means for calculating the reliability of the call based on the caller's attribute information, a means for extracting and analyzing emotional data from the caller's voice, and a means for notifying the recipient of the call based on the reliability and emotional data. This allows for an advanced combination of caller reliability evaluation and emotional analysis, thereby improving the accuracy of detecting fraudulent and nuisance calls and ensuring the safety of recipients.

[0347] - "Artificial intelligence device" is a system that autonomously responds to incoming calls and collects and analyzes caller information.

[0348] "Voice recognition technology" is a technology that converts voice data from a caller into text data.

[0349] "Caller attribute information" refers to various data related to the caller (e.g., number, past call history, registration information, etc.).

[0350] The "means for calculating the reliability" is a mechanism for quantifying the reliability of a sender based on the sender's attribute information.

[0351] The "means for extracting emotional data" is a function that analyzes and extracts the emotional state (anger, sadness, joy, etc.) from the caller's voice.

[0352] "Call notification means" is a mechanism that notifies the recipient of the caller's information, trustworthiness, and emotional state.

[0353] "Analysis of conversation content during a call" is the process of analyzing the conversation between the caller and receiver in real time during a call to confirm whether it matches the initial information.

[0354] This invention is a system in which artificial intelligence (AI) acts as the primary point of contact for telephone calls and eliminates fraudulent and nuisance calls by analyzing the caller's voice. This system includes three elements: a server, a terminal, and a user, and performs advanced analysis by combining an emotion engine.

[0355] Hardware and Software Configuration

[0356] server:

[0357] The server functions as the core of the system, running the artificial intelligence device and emotion engine. It has the following functions:

[0358] Speech recognition technology: Uses the Google Cloud Speech-to-Text API to convert the caller's voice data into text.

[0359] Database: A database is used to store and reference caller attribute information and past call history.

[0360] Emotion Engine: Uses IBM Watson Tone Analyzer to extract and analyze emotional data from audio.

[0361] Rating algorithms: Executes algorithms to calculate the trustworthiness of callers and evaluate the appropriateness of calls.

[0362] Device:

[0363] The terminal is a device used by the user, which detects incoming calls, plays an automatic answering message, and receives notifications from the server and notifies the user.

[0364] Incoming call detection: Has the ability to detect incoming calls in real time via communication lines.

[0365] Auto-answer: Has a voice playback function to record and play specified messages.

[0366] Voice recording: Equipped with a microphone and storage to record the caller's voice and send it to the server.

[0367] User:

[0368] A user is a person who uses the system and is responsible for answering or rejecting incoming calls.

[0369] Receiving notification: The terminal receives a notification of the evaluation result from the server and processes the call based on that notification.

[0370] System operation example

[0371] When a user receives a call:

[0372] 1. Detecting an incoming call: The device detects an incoming call.

[0373] The phone line on your device will be activated and a ring tone will play.

[0374] 2. Notify the server: The terminal notifies the server of the incoming call.

[0375] The terminal sends an incoming call notification to the server using the UDP protocol.

[0376] 3. Playing an automated response message: The server instructs the terminal to play the message, "This is an AI support desk. Please tell us your name and requirements."

[0377] The device plays a pre-recorded message and records the caller's voice.

[0378] 4. Sending audio data: The device sends the recorded audio data to the server.

[0379] The audio file is uploaded to the server via an HTTP POST request.

[0380] 5. Speech recognition and text conversion: The server converts the voice data into text using the Google Cloud Speech-to-Text API.

[0381] The server converts the speech into text and performs analysis.

[0382] 6. Extraction of caller information: The server extracts important information from the text data.

[0383] The server uses natural language processing (NLP) algorithms to extract information such as names and requirements from the text.

[0384] 7. Calculating trustworthiness: The server queries the database and calculates the trustworthiness of the sender.

[0385] The server calculates the reliability based on past history and registration information.

[0386] 8. Emotion data analysis: The server uses IBM Watson Tone Analyzer to analyze the emotion data.

[0387] The emotion engine analyzes the sender's emotional state.

[0388] 9. Notification and call connection: The server notifies the user of the evaluation result, and the user can choose whether to connect the call.

[0389] The server generates an audio notification and sends it to the device, where the user can confirm and connect or reject the call.

[0390] 10. In-call monitoring: The server analyzes call content and emotion data in real time and issues an alert if it detects any abnormalities.

[0391] The server uses a real-time processing engine to analyze the streaming data and generate alerts as needed.

[0392] Example prompt sentence:

[0393] "Please explain the steps an AI-powered automated telephone answering system takes to analyze the voice data of incoming calls and assess the caller's trustworthiness and emotional state."

[0394] "Please explain the process of combining speech recognition technology and an emotion engine to analyze the caller's information and notify the user."

[0395] "Please explain how callers' voice data can be used to analyze their confidence and emotional data to prevent spam calls."

[0396] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0397] Step 1:

[0398] Incoming call detection and notification

[0399] The terminal detects an incoming call. As input, the terminal monitors the communication line and confirms that there is an incoming call. As output, the terminal notifies the server of the result of the incoming call detection.

[0400] Specific operation: The terminal plays a ring tone and sends an incoming call notification to the server using the UDP protocol.

[0401] Step 2:

[0402] Auto-response message playback and voice recording

[0403] The server instructs the device to play a message saying, "This is an AI support desk. Please tell us your name and request." As input, the server receives an incoming call notification. As output, the device plays an automated response message and records the caller's voice.

[0404] Specific operation: The device plays the specified message, records the caller's voice using the microphone, and creates an audio file.

[0405] Step 3:

[0406] Sending audio data

[0407] The device sends the recorded voice data of the caller to the server. As input, the device receives the recorded voice data. As output, the device uploads the voice data to the server via an HTTP POST request.

[0408] Specific operation: The device sends the audio file to the server via an HTTP POST request, setting the appropriate URL and payload.

[0409] Step 4:

[0410] Speech recognition to text

[0411] The server converts the voice data into text using speech recognition technology (Google Cloud Speech-to-Text API). As input, the server receives the voice data. As output, it generates text data.

[0412] Specific operation: The server calls the API and executes the process of sending voice data and receiving text data.

[0413] Step 5:

[0414] Extracting caller information

[0415] The server extracts information from the text data, such as "Who is calling?", "Who is the caller addressed to?", and "What is the purpose of the call?". As input, the server receives the text data. As output, it generates the extracted caller information as structured data.

[0416] What it does: The server uses natural language processing (NLP) algorithms to extract important information from the text and convert it into structured data, such as JSON format.

[0417] Step 6:

[0418] Calculating reliability

[0419] The server retrieves the sender's attribute information from the database and calculates the reliability. As input, the server retrieves the sender's information and queries the database. As output, it generates data that quantifies the reliability.

[0420] Specific operation: The server queries the database using an SQL query, obtains the sender's attribute information, and executes an algorithm to calculate the trustworthiness based on that information.

[0421] Step 7:

[0422] Emotion data extraction and analysis

[0423] The server uses an emotion engine (IBM Watson Tone Analyzer) to extract emotion data from the recorded voice. As input, the server receives the voice data. As output, it generates the extracted emotion data.

[0424] Specific operation: The server calls the emotion engine API and executes the process of sending voice data and receiving emotion data.

[0425] Step 8:

[0426] Call notification

[0427] The server notifies the user of the evaluation results of the reliability and emotion data. As input, it receives the evaluation results (reliability and emotion data). As output, it converts the evaluation result notification into voice and sends it to the terminal.

[0428] Specific operation: The server converts the evaluation results from text to speech, and generates and transmits a message to send it to the terminal.

[0429] Step 9:

[0430] User discretion and call connection or rejection

[0431] The user checks the notification and selects whether to connect or reject the call. As input, the user receives the notification. As output, the user's selection is input to the terminal.

[0432] Specific action: The user operates the device interface to choose to connect or reject the call.

[0433] Step 10:

[0434] In-call monitoring and anomaly detection

[0435] The server monitors the conversation content and emotion data during the call in real time to detect anomalies. As input, the server receives the audio stream during the call. As output, it generates anomaly detection results and issues warning alerts if necessary.

[0436] Specific operation: The server analyzes the streaming data using a real-time processing engine, compares it with the initial information, and if an abnormality is detected, generates and sends a warning message.

[0437] (Application example 2)

[0438] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0439] Currently, the number of special frauds and nuisance sales calls via telephone is increasing, causing anxiety among many call recipients. It is particularly difficult to distinguish dangerous calls when the caller's emotional state cannot be determined. Therefore, there is a need for a system that can assess not only the caller's trustworthiness but also their emotional state in real time, and prompt the call recipient to take appropriate action.

[0440] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes an artificial intelligence device that answers an incoming call, a means for converting the caller's voice into text using voice recognition technology, a means for calculating the reliability of the call based on the caller's attribute information, a means for extracting emotional data from the caller's voice and evaluating the caller's emotional state, and a means for notifying the call recipient based on the reliability and emotional state. This allows the recipient to understand the caller's reliability and emotional state and respond safely and efficiently.

[0441] An "artificial intelligence device" is an intelligent system that automatically answers calls and analyzes the caller's information.

[0442] "Speech recognition technology" is a technology that converts voice data into text and is used to understand the caller's voice.

[0443] "Reliability" is an index calculated based on the sender's attribute information, and is used to evaluate the sender's reliability.

[0444] "Emotion data" refers to data relating to emotions extracted from the caller's voice, and is used to evaluate the caller's emotional state.

[0445] "Emotional state" indicates the emotional state of the sender, determined based on the sender's emotional data.

[0446] A "call notification means" is a means for informing the recipient of the status of the call based on confidence and emotional state.

[0447] The "means for analyzing the content of the conversation" is a means for analyzing the conversation between the caller and the recipient during the call and checking whether it matches the initial information.

[0448] The "means for monitoring emotional changes in real time" is a means for monitoring emotional changes of a caller or a receiver in real time during a call.

[0449] "Attribute information" refers to data related to the caller's personal information and past call history.

[0450] "Past call history" refers to a record of previous calls between a caller and a recipient.

[0451] The system based on this invention can automatically answer incoming calls and filter out special fraud and nuisance sales calls by evaluating the trustworthiness and emotional state of the caller. Specific embodiments of this system are described in detail below.

[0452] System configuration

[0453] The system mainly consists of three elements: the server, the terminal, and the user. The configuration of each element is as follows:

[0454] Server: The server is the central component that processes and analyzes voice data. When a call comes in, it records the caller's voice, converts it into text using speech recognition technology, calculates the caller's credibility based on their attribute information, and uses an emotion engine to evaluate the caller's emotional state and transmits the results to the device.

[0455] Terminal: The terminal detects incoming calls, plays an auto-answer message in cooperation with the server, and provides information to the user based on notifications received from the server, and connects or rejects the call.

[0456] User: The user handles the call based on the trust and emotional state information provided by the system. The call is connected or rejected according to the user's instructions.

[0457] Program processing

[0458] The system's operations are carried out by the following means:

[0459] Server Processing

[0460] The server uses voice recognition technology and an emotion engine to analyze the caller information of the incoming call and evaluate their trustworthiness and emotional state. The main software used is as follows:

[0461] SpeechRecognition: Used to convert the caller's voice data into text.

[0462] EmotionRecognizer: Extracts emotion data from a caller's voice and uses it to assess their emotional state.

[0463] The server combines these pieces of software to process the data and performs the following steps:

[0464] 1. Record the incoming call audio data.

[0465] 2. Convert the recording into text using voice recognition technology.

[0466] 3. The sender's attribute information is obtained from the database and the reliability is calculated.

[0467] 4. Extract emotional data using an emotion engine and evaluate the emotional state.

[0468] Terminal handling

[0469] The device plays an automated answering message based on instructions from the server, sends recorded audio to the server, and upon receiving a notification, communicates confidence and emotional state information to the user and connects or rejects the call based on user input.

[0470] User operations

[0471] The user can choose whether to accept or reject the call based on the information provided by the device. During the call, the server monitors the conversation and the user's emotional state, and immediately issues an alert if any abnormalities are detected.

[0472] Specific examples

[0473] The sequence of events when a user answers a call is as follows:

[0474] 1. The device detects an incoming call and notifies the server.

[0475] 2. The terminal plays an automated response message, and after the caller responds, the voice is recorded and sent to the server.

[0476] 3. The server converts the recorded voice into text and summarizes the caller information. At the same time, it analyzes the emotional data using an emotion engine.

[0477] 4. The database is referenced to obtain the sender's attribute information, and the trustworthiness and emotional data are evaluated comprehensively.

[0478] 5. If the confidence and emotion data exceed a certain threshold, the notification is converted from text to speech and notified to the user again via the device.

[0479] 6. The user checks the notification and chooses whether to connect or reject the call. The device follows this choice.

[0480] 7. During the call, the server monitors the conversation content and emotional changes in real time and issues an alert if it detects any abnormalities.

[0481] Prompt Sentence Examples

[0482] User: Hello, can you tell me about this system?

[0483] AI: This system analyzes incoming calls, assesses the caller's trustworthiness and emotional state, and notifies the recipient, protecting them from fraud and unwanted sales calls.

[0484] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0485] Step 1:

[0486] The terminal detects an incoming call and notifies the server. The input is the incoming call signal, and the output is an incoming call notification to the server. In concrete terms, the terminal detects that a call has come in and sends that information to the server.

[0487] Step 2:

[0488] The terminal plays an automated response message and records the caller's voice. The input is an instruction from the server to play the automated response message, and the output is the recorded voice data of the caller. Specifically, the terminal plays the message "This is the AI ​​support desk. Please tell us your name and request," and records the caller's response.

[0489] Step 3:

[0490] The server receives the recorded voice and converts it into text using speech recognition technology. The input is the caller's recorded voice data, and the output is the caller's text information. Specifically, the server uses the SpeechRecognition library to convert the voice data into text.

[0491] Step 4:

[0492] The server obtains attribute information from the sender's text information and calculates the reliability. The input is the sender's text information, and the output is the calculated reliability. Specifically, the server refers to the database to obtain the sender's attribute information, and calculates the reliability based on that information.

[0493] Step 5:

[0494] The server uses an emotion engine to extract emotional data from the caller's voice and evaluate their emotional state. The input is the caller's recorded voice data and text information, and the output is the evaluated emotional state. Specifically, the server uses the EmotionRecognizer library to extract emotional data from the voice data and evaluate the emotional state.

[0495] Step 6:

[0496] The server sends a call notification to the recipient based on the trust and emotional state. The input is the calculated trust and the evaluated emotional state, and the output is a notification to the recipient. Specifically, the server comprehensively evaluates the trust and emotional state and sends a notification to the recipient via SMS using Twilio.

[0497] Step 7:

[0498] The recipient checks the notification content and chooses whether to connect or reject the call. The input is the notification content received by the recipient, and the output is an instruction to connect or reject the call. Specifically, the recipient checks the notification content (trustworthiness and emotional state) and instructs the terminal to connect or reject the call.

[0499] Step 8:

[0500] During a call, the server monitors the conversation content and emotional changes in real time. The input is the voice data during the call, and the output is the monitoring results. Specifically, the server analyzes the call content, detects discrepancies with the initial information, or abnormal changes in emotions, and issues a warning.

[0501] Step 9:

[0502] When the server issues a warning, it notifies the recipient of the warning and terminates the call if necessary. The input is the warning issued by the server, and the output is the warning notification to the recipient and an instruction to end the call. Specifically, the server sends the warning content to the recipient via SMS and terminates the call according to the recipient's instructions.

[0503] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0504] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0505] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0506] [Second embodiment]

[0507] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0508] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0509] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0510] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0511] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0512] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0513] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0514] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0515] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0516] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0517] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0518] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0519] This invention is a system that uses artificial intelligence (AI) to handle primary telephone inquiries, eliminating special frauds and nuisance sales calls and realizing safe and efficient telephone response. Specific embodiments are described below.

[0520] System configuration

[0521] This system consists of three elements: a server, a terminal, and a user.

[0522] Server: The core part of the system where AI is executed. It analyzes caller information, calculates reliability, decides whether to notify the caller, and analyzes the content of the conversation during the call.

[0523] Terminal: A device used by a user. It detects incoming calls, plays an automatic answering message, and receives notifications from the server and notifies the user.

[0524] User: The person who uses this system, such as an elderly person, a home resident, or a company employee. The user receives notifications through the terminal and answers the call as needed.

[0525] Program processing

[0526] 1. Receiving calls and initial response

[0527] The server detects an incoming call via the device and instructs the device to play an automated response message: "This is an AI support desk. Please tell us your name and request," and records the caller's voice.

[0528] 2. Collection and analysis of caller information

[0529] The recorded voice data of the caller is sent to a server, which uses speech recognition technology to convert this voice data into text, extracting and summarizing who is calling, who the call is addressed to, and what the request is.

[0530] 3. Calculating reliability

[0531] The server retrieves the caller's attribute information (such as phone number, area, company name, and past call history) from a database and calculates the caller's trustworthiness based on this information. If the caller has withheld their number, the trustworthiness is set significantly lower.

[0532] 4. Notifications and call connections

[0533] If the reliability exceeds a certain threshold (e.g., 90%), the server converts the result from text to speech and notifies the user via the device. The user can then confirm the content of the notification and choose to accept or reject the call. The device will then connect or reject the call according to the user's instructions.

[0534] 5. Monitoring and updating attribute information during a call

[0535] During the call, the server analyzes the conversation in real time to check whether it matches the initial information. If there is a mismatch, the server issues a warning and updates the attribute information in the database.

[0536] Specific examples

[0537] When an elderly person answers a phone call

[0538] 1. The device detects an incoming call and notifies the server.

[0539] 2. The device plays an automated answering message and records the voice of the caller, who identifies himself as "Yamada Taro." The recording data is sent to the server.

[0540] 3. The server converts the voice data into text and summarizes it as "Caller: Yamada Taro, Recipient: Grandpa, Purpose: Check status."

[0541] 4. The server inspects the attribute information and calculates the reliability as 99%.

[0542] 5. The user (grandfather) is notified via the terminal that "Yamada Taro is calling to check on your status."

[0543] 6. The user presses the answer button to connect the call.

[0544] 7. During the call, the server monitors the conversation and verifies that it matches.

[0545] When a company receives nuisance calls

[0546] 1. The device detects an incoming call and notifies the server.

[0547] 2. The device plays an automated answering message and records the voice of the anonymous caller.

[0548] 3. The recorded data is sent to a server, where it is converted into text. The summary is "Caller: Anonymous, Destination: Unknown, Requirement: Sales."

[0549] 4. The server checks the attribute information and calculates the reliability as 10%.

[0550] 5. The server decides not to notify and the terminal does not connect the call.

[0551] In this way, the system can automatically answer calls under certain conditions, notifying and connecting the call recipient only if the call is trustworthy, effectively filtering out fraudulent and nuisance calls.

[0552] The processing flow will be explained below.

[0553] Step 1:

[0554] The terminal detects the incoming call and notifies the server of the information.

[0555] Specific operation: When the terminal receives an incoming call signal from the telephone line, it notifies the server of the incoming call.

[0556] Step 2:

[0557] The server instructs the terminal to play back the automatic response message.

[0558] Specific operation: The server sends a pre-prepared response message to the device and commands the device to play it. The message played is, "This is the AI ​​support desk. Please tell us your name and requirements."

[0559] Step 3:

[0560] The terminal records the caller's voice and transmits the voice data to the server.

[0561] Specific operation: When the caller answers and states their name and requirements, the terminal records the voice and transmits the data to the server in real time.

[0562] Step 4:

[0563] The server sends the voice data to a voice recognition engine and converts it into text data.

[0564] Specific operation: The server sends the received voice data to a cloud-based voice recognition service, converts the caller's speech into text, and receives it.

[0565] Step 5:

[0566] The server analyzes the text, extracts information such as "who is calling," "who is it addressed to," and "what the requirements are," and generates a summary.

[0567] Specific operation: Analyzes text using natural language processing (NLP) algorithms to obtain sender information, recipient information, and requirement information.

[0568] Step 6:

[0569] The server obtains the sender's attribute information from the database and calculates the reliability.

[0570] Specific operation: The server retrieves data such as the caller's phone number, area, company name, and past call history from a database, and calculates its reliability using a statistical model.

[0571] Step 7:

[0572] The server determines whether the confidence level exceeds a set threshold (e.g., 90%) and issues a result.

[0573] Specific operation: If the reliability is 90% or more, it is determined to be a legitimate call and proceeds to the next step. If it is less than 90%, the call is not notified and the call is terminated.

[0574] Step 8:

[0575] The server converts the results from text to speech and notifies the device.

[0576] Specific operation: The server sends the summarized text data to a speech synthesis engine to generate an audio file, which is then sent to the terminal for playback.

[0577] Step 9:

[0578] The terminal plays the notification message to the user.

[0579] Specific operation: The terminal plays the message "This is a call from Mr. / Ms. XX. The request is XX" to the user.

[0580] Step 10:

[0581] The user checks the notification and decides whether to respond.

[0582] Specific operation: After checking the notification message, the user can choose to press the answer button or the reject button.

[0583] Step 11:

[0584] The terminal connects or rejects the call according to the user's instructions.

[0585] Specific behavior: If the user presses the answer button, the call will be connected. If the user presses the reject button, the call will be ended.

[0586] Step 12:

[0587] The server analyzes the content of the call in real time and checks whether it matches the initial information.

[0588] Specific operation: The server analyzes the contents of the call using voice recognition and NLP, and if any content differs from the summary, it issues an alert.

[0589] In this way, the system reduces the risk of fraud and unwanted calls and efficiently directs only necessary calls.

[0590] Example 1

[0591] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0592] In modern times, telephone fraud and nuisance sales calls are on the rise. These calls are a major problem, especially for the elderly and businesses, and there is a need for methods to prevent these attacks. However, conventional telephone answering systems are unable to adequately address these issues. An effective system is needed to automatically filter out unreliable callers and notify recipients of only reliable calls.

[0593] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0594] In this invention, the server includes an artificial intelligence device that responds to incoming calls, a means for converting caller information into text using voice recognition technology, a means for calculating the reliability of the call based on the caller's attribute information, and a means for notifying the recipient of the call when the reliability exceeds a certain threshold. This makes it possible to automatically filter out calls from unreliable callers and efficiently deliver only reliable calls to the recipient. Furthermore, by providing a means for monitoring the content of conversations during calls in real time and issuing an alert if there is a discrepancy, further protection from fraud and nuisance calls can be achieved. The reliability calculation uses a machine learning algorithm, improving the accuracy and reliability of the system.

[0595] An "artificial intelligence device that answers incoming calls" is a device that detects an incoming call and automatically plays a response message.

[0596] "Means of converting caller information into text using voice recognition technology" refers to technology for analyzing recorded voice data and converting it into text data.

[0597] The "means for calculating the reliability of a call based on the caller's attribute information" is a means for evaluating attribute information such as the caller's telephone number, region, company name, and past call history, and calculating its reliability.

[0598] The "means for notifying the recipient of a call when the reliability exceeds a certain threshold" refers to a means for notifying the recipient of a call when the calculated reliability exceeds a set threshold.

[0599] "Means for monitoring the content of conversations during calls in real time and issuing an alert if there is a discrepancy" refers to a means for constantly monitoring the content of conversations during calls and issuing an alert if a discrepancy is detected between the information obtained the first time and the content.

[0600] A "machine learning algorithm" is a computer algorithm that learns patterns from data and uses them to make future predictions and classifications.

[0601] This invention is a system that uses artificial intelligence (AI) to handle the primary telephone contact point, eliminating special frauds and nuisance sales calls and realizing safe and efficient telephone response. This system consists of three elements: a server, a terminal, and a user. Specific embodiments are described below.

[0602] System configuration

[0603] 1. Server

[0604] This is the core part of the system where the AI ​​runs. The server is responsible for detecting incoming calls, voice recognition, calculating reliability, and monitoring call content in real time. It mainly uses the following software:

[0605] Google Speech-to-Text API or IBM Watson Speech to Text API: For speech recognition technology.

[0606] Amazon Polly or Google Cloud Text-to-Speech: For converting text to speech.

[0607] scikit-learn or XGBoost: For implementing machine learning algorithms.

[0608] 2. Terminal

[0609] A device used by a user. It detects incoming calls, plays an automatic answering message, and sends the audio data to a server. It also receives notifications from the server and conveys them to the user. Specifically, it is a smartphone or dedicated telephone terminal that the user owns.

[0610] 3. Users

[0611] The person using the system can be an elderly person, a home resident, a company employee, etc. The user receives a notification through the terminal and can answer the call if necessary.

[0612] Specific operation of the system

[0613] 1. Receiving calls and initial response

[0614] The device detects an incoming call and sends the detected call information to the server.

[0615] The server instructs the terminal to play an automatic response message.

[0616] The device plays a message saying, "This is an AI support desk. Please tell us your name and request," and records the caller's voice.

[0617] 2. Collection and analysis of caller information

[0618] The recorded audio data is sent from the terminal to the server.

[0619] The server uses voice recognition technology to convert the voice data into text, extracting information such as "who is calling," "who is it addressed to," and "what the requirement is."

[0620] 3. Calculating reliability

[0621] The server retrieves the caller's attribute information from a database, including phone number, area, company name, and past call history.

[0622] Calculate the trustworthiness of the sender using a machine learning algorithm, such as the Random Forest or XGBoost algorithms from scikit-learn.

[0623] 4. Notifications and call connections

[0624] If the reliability exceeds a set threshold, the server converts the text into speech and sends the speech data to the terminal.

[0625] The terminal plays the notification content to the user, who can then choose to respond or reject it.

[0626] If the user chooses to answer, the terminal connects the call.

[0627] 5. Monitoring and updating attribute information during a call

[0628] During the call, the server analyzes the conversation in real time to see if it matches the initial information.

[0629] If there is no match, the server issues a warning and notifies the user via the terminal.

[0630] The attribute information in the database is updated as needed.

[0631] Specific examples

[0632] When an elderly person answers a phone call

[0633] 1. The device detects an incoming call and notifies the server.

[0634] 2. The device plays an automated response message and records the caller's voice.

[0635] 3. The recorded data is sent to the server, where it is converted into text. The text is summarized as "Caller: XXX, Recipient: Grandpa, Message: Status check."

[0636] 4. The server checks the attribute information and calculates the reliability.

[0637] 5. Since the server has a high reliability, it sends the result to the terminal and notifies the user, "Mr. / Ms. XXX is calling to check on your status."

[0638] 6. The user presses the answer button to connect the call.

[0639] 7. During the call, the server monitors the conversation to ensure it matches.

[0640] When a company receives nuisance calls

[0641] 1. The device detects an incoming call and notifies the server.

[0642] 2. The device plays an automated response message and records the caller's voice.

[0643] 3. The recorded data is sent to a server, where it is converted into text. The text is summarized as "Caller: Anonymous, Destination: Unknown, Requirement: Sales."

[0644] 4. The server checks the attribute information and calculates the reliability.

[0645] 5. The server decides not to notify and the terminal does not connect the call.

[0646] Example prompt sentence:

[0647] "Please tell me the specific steps for a system that uses AI to prevent nuisance and fraudulent calls when elderly people receive phone calls."

[0648] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0649] Step 1:

[0650] The device detects an incoming call and sends the incoming call information to the server. The input includes the phone number and the time of the call. The device collects this data and sends it to the server to notify the call.

[0651] Specific operation: When the phone rings, the device's sensor reacts, captures the incoming call information (phone number and time) in the internal memory, and immediately transfers it to the server.

[0652] Step 2:

[0653] The server receives the incoming call information and sends an instruction to the terminal to play the automatic response message. The server receives the incoming call information from the terminal as input. Based on the received information, the server generates a response message and sends that instruction to the terminal.

[0654] Specific operation: The server analyzes the received telephone number and time information, and sends a command to the terminal to play the automatic response message.

[0655] Step 3:

[0656] The device plays a message saying, "This is an AI support desk. Please tell us your name and requirements," and records the caller's voice. As input, it receives a playback instruction from the server. The device receives the playback instruction and starts recording the voice while playing the specified message to the caller.

[0657] Specific operation: The device's speaker plays the message according to the instructions, the microphone records the caller's voice, and the recorded voice data is stored in temporary memory.

[0658] Step 4:

[0659] The recorded voice data is sent from the device to the server. As input, the recorded voice data is retrieved from the device's memory. The device uploads the voice data to the server.

[0660] Specific operation: To send the audio file stored in the temporary memory to the server, the terminal divides the data into packets and transmits them over the network.

[0661] Step 5:

[0662] The server uses voice recognition technology to convert the voice data into text and extracts information such as "Who is calling?", "Who is the caller?", and "What is the requirement?". The server receives the voice data sent from the device as input. The server uses voice recognition software such as the Google Speech-to-Text API to convert the voice data into text and extract the necessary information.

[0663] Specific operation: Voice data is sent to a voice recognition API, and the returned text is analyzed to extract and summarize important information (sender, recipient, requirements).

[0664] Step 6:

[0665] The server retrieves the caller's attribute information from the database and calculates the caller's reliability using a machine learning algorithm. The input is the text data obtained by speech recognition and the caller information in the database. The server uses, for example, a scikit-learn algorithm to calculate the reliability.

[0666] Specific operation: A database query is issued to obtain the caller's attribute information, which is then passed as input to a machine learning model to calculate a confidence score.

[0667] Step 7:

[0668] If the confidence score exceeds a set threshold, the server sends the result to the device and instructs it to play a notification message to the user.The server uses the calculated confidence score as input.The server converts the text to speech and sends it to the device.

[0669] Specific operation: After converting text to speech, the speech data is sent to the terminal, and the terminal plays the specified notification message to the user.

[0670] Step 8:

[0671] The user checks the notification message and chooses whether to respond or reject it. The notification is received from the device as input. The user selects a response by operating the device's buttons or touch screen.

[0672] Specific operation: The device detects the user's selection and sends that information to the server.

[0673] Step 9:

[0674] The terminal connects or rejects the call based on the user's instruction. The terminal receives the user's selection as input. The terminal performs the call connection operation based on the selection.

[0675] Specific operation: The terminal sends out a call connection signal or a call rejection signal to control the telephone line appropriately.

[0676] Step 10:

[0677] During the call, the server analyzes the conversation in real time to check whether it matches the initial information. The input includes the voice data from the call. The server again uses voice recognition technology to convert it into text data and verify the information.

[0678] What it does: Continuously samples the audio during a call, transcribes it, compares it with the initial data, and generates a warning if there is a mismatch.

[0679] Step 11:

[0680] If a mismatch occurs, the server issues a warning to the device and updates the attribute information in the database. The mismatch detection information is used as input. The server adds new information to the database and updates it.

[0681] Specific operation: Add a new entry to the database based on the detected discrepancy and send a warning signal to the terminal.

[0682] (Application example 1)

[0683] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0684] In today's world, with the increase in special frauds and nuisance sales calls, it is important for call recipients to be protected from these fraudulent calls. Senior citizens and businesses are particularly at risk of business disruption and financial loss from these fraudulent and nuisance calls. Traditional manual responses are insufficient, and more efficient and reliable countermeasures are needed.

[0685] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0686] In this invention, the server includes an artificial intelligence device that responds to incoming calls, a means for converting caller information into text using voice recognition technology, and a means for calculating the reliability of the call based on the caller's attribute information. This makes it possible to eliminate special frauds and nuisance sales calls and notify the recipient of only reliable calls.

[0687] An "artificial intelligence device that answers incoming calls" is a device that has the function of detecting an incoming call and automatically playing a response message.

[0688] "Means of converting to text using voice recognition technology" refers to technology that analyzes recorded voice data and extracts and converts the caller's voice into text.

[0689] The "means for calculating trustworthiness" is a technology that quantifies the trustworthiness of a caller based on the caller's attribute information (telephone number, region, company name, past call history, etc.).

[0690] The "means for notifying the recipient of a call" is a technology that has the function of notifying the recipient of a highly reliable call based on the calculated reliability.

[0691] The "means for transmitting notification content to the recipient by voice synthesis" is a technology for transmitting notification content generated by the server to the recipient as voice using voice synthesis technology.

[0692] "Means for analyzing the content of conversations during a call" refers to technology that monitors the conversation between the caller and recipient in real time during a call and checks whether it matches information obtained in advance.

[0693] "Means for storing caller attribute information in a database" refers to technology that has the function of recording and storing information such as caller characteristics and call history in a database.

[0694] "Means for adjusting reliability based on past call history" refers to a technology that analyzes accumulated call data and updates and adjusts the reliability of callers in real time.

[0695] "Means for providing response options" refers to a technique that presents a reliable call so that the recipient can choose whether to respond.

[0696] "Means for automatically disconnecting a call" refers to technology that has the function of automatically terminating a call when an unreliable call is detected.

[0697] MODE FOR CARRYING OUT THE INVENTION

[0698] System configuration

[0699] This invention is a system that uses AI to handle the primary telephone call counter and eliminate special frauds and nuisance sales calls. The system consists of three elements: a server, a terminal, and a user.

[0700] 1. Server: This is the core part of the system, and it analyzes caller information, calculates reliability, decides whether to notify, and analyzes the content of conversations during calls. The software used includes Python, speech_recognition, pyttsx3, and requests.

[0701] 2. Terminal: A device used by a user. It detects incoming calls, plays an automatic answering message, and receives notifications from the server and notifies the user.

[0702] 3. User: The person who uses this system, such as an elderly person, a home resident, or a company employee. The user receives notifications through the terminal and answers the call as needed.

[0703] System Operation

[0704] Receiving and initial phone calls

[0705] The server detects an incoming call via the device and instructs the device to play an automated answering message: "This is an AI support desk. Please tell us your name and request," and records the caller's voice.

[0706] Collection and analysis of caller information

[0707] The recorded voice data of the caller is sent to a server, which uses speech recognition technology to convert the data into text, extracting and summarizing who is calling, who the call is addressed to, and what the request is.

[0708] Calculating reliability

[0709] The server retrieves the caller's attribute information (such as phone number, area, company name, and past call history) from a database and calculates the caller's trustworthiness based on this information. If the caller has withheld their number, the trustworthiness is set significantly lower.

[0710] Notifications and call connections

[0711] If the reliability exceeds a certain threshold (e.g., 90%), the server converts the result from text to speech and notifies the user via the device. The user can then confirm the content of the notification and choose whether to answer or reject the call. The device will then connect or reject the call according to the user's instructions.

[0712] Monitoring and updating attribute information during a call

[0713] During the call, the server analyzes the conversation in real time to check whether it matches the initial information. If there is a mismatch, the server issues a warning and updates the attribute information in the database.

[0714] Specific examples

[0715] When an elderly person answers a phone call

[0716] 1. The device detects an incoming call and notifies the server.

[0717] 2. The device plays an automated response message and records the caller's voice. The recording is then sent to the server.

[0718] 3. The server converts the voice data into text and summarizes it as "Sender: Mr. X, Address: Recipient, Request: Status check."

[0719] 4. The server checks the attribute information and calculates the reliability.

[0720] 5. The user is notified via the terminal that "Mr. X is calling to check on your status."

[0721] 6. The user presses the answer button to connect the call.

[0722] 7. During the call, the server monitors the conversation and verifies that it matches.

[0723] When a company receives nuisance calls

[0724] 1. The device detects an incoming call and notifies the server.

[0725] 2. The device plays an automated answering message and records the voice of the anonymous caller.

[0726] 3. The recorded data is sent to the server, where it is converted into text. The text is summarized as "Caller: Anonymous, Address: Recipient, Requirement: Sales."

[0727] 4. The server checks the attribute information and calculates the reliability.

[0728] 5. Due to low reliability, the server decides not to notify and the device does not connect the call.

[0729] Prompt Sentence Examples

[0730] "We use an AI model that analyzes audio files, identifies the caller's name, destination, and requirements, and calculates the reliability. For example, the following is from: Caller: Mr. X, Destination: Recipient, Requirements: Status check."

[0731] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0732] Step 1:

[0733] The server detects an incoming call via the terminal. It receives the incoming call signal as input and sends an instruction to the terminal to play an automatic answering message as output. Specifically, the terminal's microphone is enabled and goes into recording mode.

[0734] Step 2:

[0735] The device plays an automated answering message saying, "This is an AI support desk. Please tell us your name and request," and records the caller's voice. It receives incoming call signals and playback instructions as input, generates voice data as output, and sends it to the server. Specifically, the message is played from the speaker, and the caller's voice is recorded by the microphone.

[0736] Step 3:

[0737] The server receives the recorded voice data and converts it into text using speech recognition technology. It receives voice data as input and generates text data as output. Specifically, it analyzes the voice data using the speech_recognition library and converts the caller's name and requirements into text.

[0738] Step 4:

[0739] The server analyzes the text data and extracts and summarizes information such as "who is calling," "who is the caller," and "what are the requirements?" It receives text data as input and generates summary data as output. Specifically, it uses a generative AI model to extract the necessary information from the text.

[0740] Step 5:

[0741] The server retrieves the caller's attribute information from a database and calculates the reliability. It takes the caller's number and past call history as input and generates a reliability score as output. Specifically, it executes a database query and calculates the reliability based on the obtained data.

[0742] Step 6:

[0743] The server determines whether the confidence score exceeds a certain threshold (for example, 90%) and notifies the device. It receives the confidence score as input and converts the notification content into audio as output, which is then sent to the device. Specifically, it uses the pyttsx3 library to convert text to audio.

[0744] Step 7:

[0745] The device receives the notification from the server and conveys the notification content to the recipient. The recipient checks the notification and chooses whether to answer or reject the call. The device receives the notification content as input and sends the recipient's choice to the server as output. Specific operations include playing the notification content from a speaker and displaying an interface for receiving user input.

[0746] Step 8:

[0747] If the user chooses to answer the call, the terminal connects the call by receiving the user's selection as input and generating a call connection signal as output, specifically by opening a line for sending and receiving the call's voice data in real time.

[0748] Step 9:

[0749] During the call, the server analyzes the conversation in real time to check whether it matches the initial information. It receives the call audio data as input and generates the analysis results as output. Specifically, it uses the speech_recognition library again to convert the acquired audio data into text and compare it with the initial information.

[0750] Step 10:

[0751] If a mismatch occurs in the call content, the server issues a warning and updates the attribute information in the database. It receives the analysis results as input and generates updated attribute information as output. Specifically, it displays a mismatch warning on the terminal and executes an update query on the database.

[0752] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0753] This invention is a system that uses artificial intelligence (AI) as the primary point of contact for telephone calls, and combines it with an emotion engine that recognizes the user's emotions to eliminate special frauds and nuisance sales calls, providing greater safety and efficiency. Specific embodiments are described in detail below.

[0754] System configuration

[0755] This system consists of three elements: a server, a terminal, and a user, and is further combined with an emotion engine to perform advanced analysis.

[0756] Server: The core of the system, where the AI ​​and emotion engine run. It analyzes caller information, calculates reliability, decides whether to notify the user, and analyzes the content and emotions of the conversation during the call.

[0757] Terminal: A device used by a user. It detects incoming calls, plays an automatic answering message, and receives notifications from the server and notifies the user.

[0758] User: The person who uses this system, such as an elderly person, a home resident, or a company employee. The user receives notifications through the terminal and answers the call as needed.

[0759] Program processing

[0760] 1. Receiving calls and initial response

[0761] The server detects the incoming call via the terminal and instructs the terminal to play an automated response message: "This is an AI support desk. Please tell us your name and request," and records the caller's voice.

[0762] 2. Collection and analysis of caller information

[0763] The recorded voice data of the caller is sent to a server, which uses speech recognition technology to convert this voice data into text, extracting and summarizing who is calling, who the call is addressed to, and what the request is.

[0764] 3. Calculating reliability

[0765] The server retrieves the caller's attribute information from the database and calculates the caller's trustworthiness based on this information. If the caller's number is withheld, the trustworthiness is set significantly lower.

[0766] 4. Analysis by Emotion Engine

[0767] The server uses an emotion engine to extract emotional data from the caller's recorded voice, analyzes the emotional data, and understands the caller's emotional state.

[0768] 5. Notifications and call connections

[0769] The server comprehensively evaluates the reliability and emotional data, and if the result exceeds a certain threshold, converts the text to speech and notifies the user via the device. The user can then review the notification and choose to accept or reject the call. The device will then connect or reject the call according to the user's instructions.

[0770] 6. Call monitoring and sentiment analysis

[0771] During a call, the server analyzes the conversation content and emotional data in real time. It issues alerts based on whether the initial information matches the conversation content and changes in emotion. This allows it to quickly detect suspicious activity and issue a warning alert if necessary.

[0772] Specific examples

[0773] Flow when a user receives a call

[0774] 1. The device detects an incoming call and notifies the server.

[0775] 2. The terminal plays an automated response message, and after the caller responds, the voice is recorded and sent to the server.

[0776] 3. The server converts the recorded voice into text and summarizes the caller information. At the same time, it analyzes the emotional data using an emotion engine.

[0777] 4. The database is referenced to obtain the sender's attribute information, and the trustworthiness and emotional data are evaluated comprehensively.

[0778] 5. If the confidence and emotion data exceed a certain threshold, the notification is converted from text to speech and notified to the user again via the device.

[0779] 6. The user checks the notification and chooses whether to connect or reject the call. The device follows this choice.

[0780] 7. During the call, the server monitors the conversation content and emotional changes in real time. If it detects any discrepancies with the initial information or abnormal changes in emotions, it issues a warning.

[0781] In this way, this system can prevent fraud and nuisance calls with even greater accuracy by combining not only caller information but also an emotion engine that analyzes the caller's emotional state in real time.

[0782] The processing flow will be explained below.

[0783] Step 1:

[0784] The terminal detects the incoming call and notifies the server of the information.

[0785] Specific operation: The terminal receives an incoming call signal from the telephone line and notifies the server of this fact.

[0786] Step 2:

[0787] The server instructs the terminal to play back the automatic response message.

[0788] Specific operation: The server sends a pre-prepared message to the terminal, playing an audio message saying, "This is the AI ​​support desk. Please tell us your name and requirements."

[0789] Step 3:

[0790] The terminal records the caller's voice and transmits the voice data to the server.

[0791] Specific operation: When the caller answers and states their name and requirements, the voice is recorded and sent to the server in real time.

[0792] Step 4:

[0793] The server sends the voice data to a voice recognition engine and converts it into text data.

[0794] Specific operation: The server sends the received voice data to a cloud speech recognition service, which converts the voice into text.

[0795] Step 5:

[0796] The server analyzes the text, extracts information such as "who is calling," "who is it addressed to," and "what the requirements are," and generates a summary.

[0797] Specific operation: Analyzes text using natural language processing (NLP) algorithms to extract sender information, recipient information, and requirement information.

[0798] Step 6:

[0799] The server obtains the sender's attribute information from the database and calculates the reliability.

[0800] Specific operation: The server obtains attribute information such as the caller's phone number, region, company name, and past call history from a database, and calculates the reliability using a statistical model.

[0801] Step 7:

[0802] The server uses an emotion engine to extract emotion data from the voice data and analyze the caller's emotional state.

[0803] Specific operation: The server sends the voice data to an emotion recognition engine and extracts the caller's emotional data (joy, anger, sadness, etc.).

[0804] Step 8:

[0805] The server comprehensively evaluates the reliability and emotional data and gives a result.

[0806] Specific operation: Determine whether the confidence and emotion data exceed the set threshold (e.g., confidence 90%, emotion data within the normal range).

[0807] Step 9:

[0808] The server converts the results from text to speech and notifies the device.

[0809] Specific operation: The server sends the summarized text data to the speech synthesis engine and sends the generated audio file to the terminal.

[0810] Step 10:

[0811] The terminal plays the notification message to the user.

[0812] Specific operation: The terminal plays the message "This is a call from Mr. / Ms. XX. The request is XX" to the user.

[0813] Step 11:

[0814] The user checks the notification and decides whether to respond.

[0815] Specific operation: After checking the notification message, the user can choose to press the answer button or the reject button.

[0816] Step 12:

[0817] The terminal connects or rejects the call according to the user's instructions.

[0818] Specific operation: When the user presses the answer button, the terminal connects the call; when the user presses the reject button, the terminal ends the call.

[0819] Step 13:

[0820] The server analyzes the conversation content and emotional data during the call in real time.

[0821] Specific operation: After the call is connected, the server sends the voice data to the voice recognition engine and emotion recognition engine in real time to monitor the conversation content and emotional changes.

[0822] Step 14:

[0823] The server compares the initial information with the conversation content and emotional state, and issues a warning if there is a mismatch or if there is an invalid emotional change.

[0824] Specific operation: The server uses a matching algorithm to compare the initial summary with the conversation content and emotional changes, and if there is a mismatch, it issues a warning alert.

[0825] In this way, the system can provide high levels of security and efficiency by analyzing caller information and emotions in real time and managing calls based on trust and emotion data.

[0826] Example 2

[0827] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0828] Conventional telephone answering systems only performed voice recognition and credibility assessment of the caller based on attribute information, which meant they were unable to adequately prevent fraudulent and nuisance calls. It was also difficult to properly analyze the caller's emotional state and detect abnormalities during the call. This could expose the recipient to unnecessary stress and risk.

[0829] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes an artificial intelligence device that responds when a call is received, a means for converting caller information into text using voice recognition technology, a means for calculating the reliability of the call based on the caller's attribute information, a means for extracting and analyzing emotional data from the caller's voice, and a means for notifying the recipient of the call based on the reliability and emotional data. This allows for an advanced combination of caller reliability evaluation and emotional analysis, thereby improving the accuracy of detecting fraudulent and nuisance calls and ensuring the safety of recipients.

[0830] - "Artificial intelligence device" is a system that autonomously responds to incoming calls and collects and analyzes caller information.

[0831] "Voice recognition technology" is a technology that converts voice data from a caller into text data.

[0832] "Caller attribute information" refers to various data related to the caller (e.g., number, past call history, registration information, etc.).

[0833] The "means for calculating the reliability" is a mechanism for quantifying the reliability of a sender based on the sender's attribute information.

[0834] The "means for extracting emotional data" is a function that analyzes and extracts the emotional state (anger, sadness, joy, etc.) from the caller's voice.

[0835] "Call notification means" is a mechanism that notifies the recipient of the caller's information, trustworthiness, and emotional state.

[0836] "Analysis of conversation content during a call" is the process of analyzing the conversation between the caller and receiver in real time during a call to confirm whether it matches the initial information.

[0837] This invention is a system in which artificial intelligence (AI) acts as the primary point of contact for telephone calls and eliminates fraudulent and nuisance calls by analyzing the caller's voice. This system includes three elements: a server, a terminal, and a user, and performs advanced analysis by combining an emotion engine.

[0838] Hardware and Software Configuration

[0839] server:

[0840] The server functions as the core of the system, running the artificial intelligence device and emotion engine. It has the following functions:

[0841] Speech recognition technology: Uses the Google Cloud Speech-to-Text API to convert the caller's voice data into text.

[0842] Database: A database is used to store and reference caller attribute information and past call history.

[0843] Emotion Engine: Uses IBM Watson Tone Analyzer to extract and analyze emotional data from audio.

[0844] Rating algorithms: Executes algorithms to calculate the trustworthiness of callers and evaluate the appropriateness of calls.

[0845] Device:

[0846] The terminal is a device used by the user, which detects incoming calls, plays an automatic answering message, and receives notifications from the server and notifies the user.

[0847] Incoming call detection: Has the ability to detect incoming calls in real time via communication lines.

[0848] Auto-answer: Has a voice playback function to record and play specified messages.

[0849] Voice recording: Equipped with a microphone and storage to record the caller's voice and send it to the server.

[0850] User:

[0851] A user is a person who uses the system and is responsible for answering or rejecting incoming calls.

[0852] Receiving notification: The terminal receives a notification of the evaluation result from the server and processes the call based on that notification.

[0853] System operation example

[0854] When a user receives a call:

[0855] 1. Detecting an incoming call: The device detects an incoming call.

[0856] The phone line on your device will be activated and a ring tone will play.

[0857] 2. Notify the server: The terminal notifies the server of the incoming call.

[0858] The terminal sends an incoming call notification to the server using the UDP protocol.

[0859] 3. Playing an automated response message: The server instructs the terminal to play the message, "This is an AI support desk. Please tell us your name and requirements."

[0860] The device plays a pre-recorded message and records the caller's voice.

[0861] 4. Sending audio data: The device sends the recorded audio data to the server.

[0862] The audio file is uploaded to the server via an HTTP POST request.

[0863] 5. Speech recognition and text conversion: The server converts the voice data into text using the Google Cloud Speech-to-Text API.

[0864] The server converts the speech into text and performs analysis.

[0865] 6. Extraction of caller information: The server extracts important information from the text data.

[0866] The server uses natural language processing (NLP) algorithms to extract information such as names and requirements from the text.

[0867] 7. Calculating trustworthiness: The server queries the database and calculates the trustworthiness of the sender.

[0868] The server calculates the reliability based on past history and registration information.

[0869] 8. Emotion data analysis: The server uses IBM Watson Tone Analyzer to analyze the emotion data.

[0870] The emotion engine analyzes the sender's emotional state.

[0871] 9. Notification and call connection: The server notifies the user of the evaluation result, and the user can choose whether to connect the call.

[0872] The server generates an audio notification and sends it to the device, where the user can confirm and connect or reject the call.

[0873] 10. In-call monitoring: The server analyzes call content and emotion data in real time and issues an alert if it detects any abnormalities.

[0874] The server uses a real-time processing engine to analyze the streaming data and generate alerts as needed.

[0875] Example prompt sentence:

[0876] "Please explain the steps an AI-powered automated telephone answering system takes to analyze the voice data of incoming calls and assess the caller's trustworthiness and emotional state."

[0877] "Please explain the process of combining speech recognition technology and an emotion engine to analyze the caller's information and notify the user."

[0878] "Please explain how callers' voice data can be used to analyze their confidence and emotional data to prevent spam calls."

[0879] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0880] Step 1:

[0881] Incoming call detection and notification

[0882] The terminal detects an incoming call. As input, the terminal monitors the communication line and confirms that there is an incoming call. As output, the terminal notifies the server of the result of the incoming call detection.

[0883] Specific operation: The terminal plays a ring tone and sends an incoming call notification to the server using the UDP protocol.

[0884] Step 2:

[0885] Auto-response message playback and voice recording

[0886] The server instructs the device to play a message saying, "This is an AI support desk. Please tell us your name and request." As input, the server receives an incoming call notification. As output, the device plays an automated response message and records the caller's voice.

[0887] Specific operation: The device plays the specified message, records the caller's voice using the microphone, and creates an audio file.

[0888] Step 3:

[0889] Sending audio data

[0890] The device sends the recorded voice data of the caller to the server. As input, the device receives the recorded voice data. As output, the device uploads the voice data to the server via an HTTP POST request.

[0891] Specific operation: The device sends the audio file to the server via an HTTP POST request, setting the appropriate URL and payload.

[0892] Step 4:

[0893] Speech recognition to text

[0894] The server converts the voice data into text using speech recognition technology (Google Cloud Speech-to-Text API). As input, the server receives the voice data. As output, it generates text data.

[0895] Specific operation: The server calls the API and executes the process of sending voice data and receiving text data.

[0896] Step 5:

[0897] Extracting caller information

[0898] The server extracts information from the text data, such as "Who is calling?", "Who is the caller addressed to?", and "What is the purpose of the call?". As input, the server receives the text data. As output, it generates the extracted caller information as structured data.

[0899] What it does: The server uses natural language processing (NLP) algorithms to extract important information from the text and convert it into structured data, such as JSON format.

[0900] Step 6:

[0901] Calculating reliability

[0902] The server retrieves the sender's attribute information from the database and calculates the reliability. As input, the server retrieves the sender's information and queries the database. As output, it generates data that quantifies the reliability.

[0903] Specific operation: The server queries the database using an SQL query, obtains the sender's attribute information, and executes an algorithm to calculate the trustworthiness based on that information.

[0904] Step 7:

[0905] Emotion data extraction and analysis

[0906] The server uses an emotion engine (IBM Watson Tone Analyzer) to extract emotion data from the recorded voice. As input, the server receives the voice data. As output, it generates the extracted emotion data.

[0907] Specific operation: The server calls the emotion engine API and executes the process of sending voice data and receiving emotion data.

[0908] Step 8:

[0909] Call notification

[0910] The server notifies the user of the evaluation results of the reliability and emotion data. As input, it receives the evaluation results (reliability and emotion data). As output, it converts the evaluation result notification into voice and sends it to the terminal.

[0911] Specific operation: The server converts the evaluation results from text to speech, and generates and transmits a message to send it to the terminal.

[0912] Step 9:

[0913] User discretion and call connection or rejection

[0914] The user checks the notification and selects whether to connect or reject the call. As input, the user receives the notification. As output, the user's selection is input to the terminal.

[0915] Specific action: The user operates the device interface to choose to connect or reject the call.

[0916] Step 10:

[0917] In-call monitoring and anomaly detection

[0918] The server monitors the conversation content and emotion data during the call in real time to detect anomalies. As input, the server receives the audio stream during the call. As output, it generates anomaly detection results and issues warning alerts if necessary.

[0919] Specific operation: The server analyzes the streaming data using a real-time processing engine, compares it with the initial information, and if an abnormality is detected, generates and sends a warning message.

[0920] (Application example 2)

[0921] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0922] Currently, the number of special frauds and nuisance sales calls via telephone is increasing, causing anxiety among many call recipients. It is particularly difficult to distinguish dangerous calls when the caller's emotional state cannot be determined. Therefore, there is a need for a system that can assess not only the caller's trustworthiness but also their emotional state in real time, and prompt the call recipient to take appropriate action.

[0923] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes an artificial intelligence device that answers an incoming call, a means for converting the caller's voice into text using voice recognition technology, a means for calculating the reliability of the call based on the caller's attribute information, a means for extracting emotional data from the caller's voice and evaluating the caller's emotional state, and a means for notifying the call recipient based on the reliability and emotional state. This allows the recipient to understand the caller's reliability and emotional state and respond safely and efficiently.

[0924] An "artificial intelligence device" is an intelligent system that automatically answers calls and analyzes the caller's information.

[0925] "Speech recognition technology" is a technology that converts voice data into text and is used to understand the caller's voice.

[0926] "Reliability" is an index calculated based on the sender's attribute information, and is used to evaluate the sender's reliability.

[0927] "Emotion data" refers to data relating to emotions extracted from the caller's voice, and is used to evaluate the caller's emotional state.

[0928] "Emotional state" indicates the emotional state of the sender, determined based on the sender's emotional data.

[0929] A "call notification means" is a means for informing the recipient of the status of the call based on confidence and emotional state.

[0930] The "means for analyzing the content of the conversation" is a means for analyzing the conversation between the caller and the recipient during the call and checking whether it matches the initial information.

[0931] The "means for monitoring emotional changes in real time" is a means for monitoring emotional changes of a caller or a receiver in real time during a call.

[0932] "Attribute information" refers to data related to the caller's personal information and past call history.

[0933] "Past call history" refers to a record of previous calls between a caller and a recipient.

[0934] The system based on this invention can automatically answer incoming calls and filter out special fraud and nuisance sales calls by evaluating the trustworthiness and emotional state of the caller. Specific embodiments of this system are described in detail below.

[0935] System configuration

[0936] The system mainly consists of three elements: the server, the terminal, and the user. The configuration of each element is as follows:

[0937] Server: The server is the central component that processes and analyzes voice data. When a call comes in, it records the caller's voice, converts it into text using speech recognition technology, calculates the caller's credibility based on their attribute information, and uses an emotion engine to evaluate the caller's emotional state and transmits the results to the device.

[0938] Terminal: The terminal detects incoming calls, plays an auto-answer message in cooperation with the server, and provides information to the user based on notifications received from the server, and connects or rejects the call.

[0939] User: The user handles the call based on the trust and emotional state information provided by the system. The call is connected or rejected according to the user's instructions.

[0940] Program processing

[0941] The system's operations are carried out by the following means:

[0942] Server Processing

[0943] The server uses voice recognition technology and an emotion engine to analyze the caller information of the incoming call and evaluate their trustworthiness and emotional state. The main software used is as follows:

[0944] SpeechRecognition: Used to convert the caller's voice data into text.

[0945] EmotionRecognizer: Extracts emotion data from a caller's voice and uses it to assess their emotional state.

[0946] The server combines these pieces of software to process the data and performs the following steps:

[0947] 1. Record the incoming call audio data.

[0948] 2. Convert the recording into text using voice recognition technology.

[0949] 3. The sender's attribute information is obtained from the database and the reliability is calculated.

[0950] 4. Extract emotional data using an emotion engine and evaluate the emotional state.

[0951] Terminal handling

[0952] The device plays an automated answering message based on instructions from the server, sends recorded audio to the server, and upon receiving a notification, communicates confidence and emotional state information to the user and connects or rejects the call based on user input.

[0953] User operations

[0954] The user can choose whether to accept or reject the call based on the information provided by the device. During the call, the server monitors the conversation and the user's emotional state, and immediately issues an alert if any abnormalities are detected.

[0955] Specific examples

[0956] The sequence of events when a user answers a call is as follows:

[0957] 1. The device detects an incoming call and notifies the server.

[0958] 2. The terminal plays an automated response message, and after the caller responds, the voice is recorded and sent to the server.

[0959] 3. The server converts the recorded voice into text and summarizes the caller information. At the same time, it analyzes the emotional data using an emotion engine.

[0960] 4. The database is referenced to obtain the sender's attribute information, and the trustworthiness and emotional data are evaluated comprehensively.

[0961] 5. If the confidence and emotion data exceed a certain threshold, the notification is converted from text to speech and notified to the user again via the device.

[0962] 6. The user checks the notification and chooses whether to connect or reject the call. The device follows this choice.

[0963] 7. During the call, the server monitors the conversation content and emotional changes in real time and issues an alert if it detects any abnormalities.

[0964] Prompt Sentence Examples

[0965] User: Hello, can you tell me about this system?

[0966] AI: This system analyzes incoming calls, assesses the caller's trustworthiness and emotional state, and notifies the recipient, protecting them from fraud and unwanted sales calls.

[0967] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0968] Step 1:

[0969] The terminal detects an incoming call and notifies the server. The input is the incoming call signal, and the output is an incoming call notification to the server. In concrete terms, the terminal detects that a call has come in and sends that information to the server.

[0970] Step 2:

[0971] The terminal plays an automated response message and records the caller's voice. The input is an instruction from the server to play the automated response message, and the output is the recorded voice data of the caller. Specifically, the terminal plays the message "This is the AI ​​support desk. Please tell us your name and request," and records the caller's response.

[0972] Step 3:

[0973] The server receives the recorded voice and converts it into text using speech recognition technology. The input is the caller's recorded voice data, and the output is the caller's text information. Specifically, the server uses the SpeechRecognition library to convert the voice data into text.

[0974] Step 4:

[0975] The server obtains attribute information from the sender's text information and calculates the reliability. The input is the sender's text information, and the output is the calculated reliability. Specifically, the server refers to the database to obtain the sender's attribute information, and calculates the reliability based on that information.

[0976] Step 5:

[0977] The server uses an emotion engine to extract emotional data from the caller's voice and evaluate their emotional state. The input is the caller's recorded voice data and text information, and the output is the evaluated emotional state. Specifically, the server uses the EmotionRecognizer library to extract emotional data from the voice data and evaluate the emotional state.

[0978] Step 6:

[0979] The server sends a call notification to the recipient based on the trust and emotional state. The input is the calculated trust and the evaluated emotional state, and the output is a notification to the recipient. Specifically, the server comprehensively evaluates the trust and emotional state and sends a notification to the recipient via SMS using Twilio.

[0980] Step 7:

[0981] The recipient checks the notification content and chooses whether to connect or reject the call. The input is the notification content received by the recipient, and the output is an instruction to connect or reject the call. Specifically, the recipient checks the notification content (trustworthiness and emotional state) and instructs the terminal to connect or reject the call.

[0982] Step 8:

[0983] During a call, the server monitors the conversation content and emotional changes in real time. The input is the voice data during the call, and the output is the monitoring results. Specifically, the server analyzes the call content, detects discrepancies with the initial information, or abnormal changes in emotions, and issues a warning.

[0984] Step 9:

[0985] When the server issues a warning, it notifies the recipient of the warning and terminates the call if necessary. The input is the warning issued by the server, and the output is the warning notification to the recipient and an instruction to end the call. Specifically, the server sends the warning content to the recipient via SMS and terminates the call according to the recipient's instructions.

[0986] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0987] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0988] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0989] [Third embodiment]

[0990] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0991] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0992] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0993] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0994] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0995] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0996] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0997] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0998] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0999] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1000] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1001] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1002] This invention is a system that uses artificial intelligence (AI) to handle primary telephone inquiries, eliminating special frauds and nuisance sales calls and realizing safe and efficient telephone response. Specific embodiments are described below.

[1003] System configuration

[1004] This system consists of three elements: a server, a terminal, and a user.

[1005] Server: The core part of the system where AI is executed. It analyzes caller information, calculates reliability, decides whether to notify the caller, and analyzes the content of the conversation during the call.

[1006] Terminal: A device used by a user. It detects incoming calls, plays an automatic answering message, and receives notifications from the server and notifies the user.

[1007] User: The person who uses this system, such as an elderly person, a home resident, or a company employee. The user receives notifications through the terminal and answers the call as needed.

[1008] Program processing

[1009] 1. Receiving calls and initial response

[1010] The server detects an incoming call via the device and instructs the device to play an automated response message: "This is an AI support desk. Please tell us your name and request," and records the caller's voice.

[1011] 2. Collection and analysis of caller information

[1012] The recorded voice data of the caller is sent to a server, which uses speech recognition technology to convert this voice data into text, extracting and summarizing who is calling, who the call is addressed to, and what the request is.

[1013] 3. Calculating reliability

[1014] The server retrieves the caller's attribute information (such as phone number, area, company name, and past call history) from a database and calculates the caller's trustworthiness based on this information. If the caller has withheld their number, the trustworthiness is set significantly lower.

[1015] 4. Notifications and call connections

[1016] If the reliability exceeds a certain threshold (e.g., 90%), the server converts the result from text to speech and notifies the user via the device. The user can then confirm the content of the notification and choose to accept or reject the call. The device will then connect or reject the call according to the user's instructions.

[1017] 5. Monitoring and updating attribute information during a call

[1018] During the call, the server analyzes the conversation in real time to check whether it matches the initial information. If there is a mismatch, the server issues a warning and updates the attribute information in the database.

[1019] Specific examples

[1020] When an elderly person answers a phone call

[1021] 1. The device detects an incoming call and notifies the server.

[1022] 2. The device plays an automated answering message and records the voice of the caller, who identifies himself as "Yamada Taro." The recording data is sent to the server.

[1023] 3. The server converts the voice data into text and summarizes it as "Caller: Yamada Taro, Recipient: Grandpa, Purpose: Check status."

[1024] 4. The server inspects the attribute information and calculates the reliability as 99%.

[1025] 5. The user (grandfather) is notified via the terminal that "Yamada Taro is calling to check on your status."

[1026] 6. The user presses the answer button to connect the call.

[1027] 7. During the call, the server monitors the conversation and verifies that it matches.

[1028] When a company receives nuisance calls

[1029] 1. The device detects an incoming call and notifies the server.

[1030] 2. The device plays an automated answering message and records the voice of the anonymous caller.

[1031] 3. The recorded data is sent to a server, where it is converted into text. The summary is "Caller: Anonymous, Destination: Unknown, Requirement: Sales."

[1032] 4. The server checks the attribute information and calculates the reliability as 10%.

[1033] 5. The server decides not to notify and the terminal does not connect the call.

[1034] In this way, the system can automatically answer calls under certain conditions, notifying and connecting the call recipient only if the call is trustworthy, effectively filtering out fraudulent and nuisance calls.

[1035] The processing flow will be explained below.

[1036] Step 1:

[1037] The terminal detects the incoming call and notifies the server of the information.

[1038] Specific operation: When the terminal receives an incoming call signal from the telephone line, it notifies the server of the incoming call.

[1039] Step 2:

[1040] The server instructs the terminal to play back the automatic response message.

[1041] Specific operation: The server sends a pre-prepared response message to the device and commands the device to play it. The message played is, "This is the AI ​​support desk. Please tell us your name and requirements."

[1042] Step 3:

[1043] The terminal records the caller's voice and transmits the voice data to the server.

[1044] Specific operation: When the caller answers and states their name and requirements, the terminal records the voice and transmits the data to the server in real time.

[1045] Step 4:

[1046] The server sends the voice data to a voice recognition engine and converts it into text data.

[1047] Specific operation: The server sends the received voice data to a cloud-based voice recognition service, converts the caller's speech into text, and receives it.

[1048] Step 5:

[1049] The server analyzes the text, extracts information such as "who is calling," "who is it addressed to," and "what the requirements are," and generates a summary.

[1050] Specific operation: Analyzes text using natural language processing (NLP) algorithms to obtain sender information, recipient information, and requirement information.

[1051] Step 6:

[1052] The server obtains the sender's attribute information from the database and calculates the reliability.

[1053] Specific operation: The server retrieves data such as the caller's phone number, area, company name, and past call history from a database, and calculates its reliability using a statistical model.

[1054] Step 7:

[1055] The server determines whether the confidence level exceeds a set threshold (e.g., 90%) and issues a result.

[1056] Specific operation: If the reliability is 90% or more, it is determined to be a legitimate call and proceeds to the next step. If it is less than 90%, the call is not notified and the call is terminated.

[1057] Step 8:

[1058] The server converts the results from text to speech and notifies the device.

[1059] Specific operation: The server sends the summarized text data to a speech synthesis engine to generate an audio file, which is then sent to the terminal for playback.

[1060] Step 9:

[1061] The terminal plays the notification message to the user.

[1062] Specific operation: The terminal plays the message "This is a call from Mr. / Ms. XX. The request is XX" to the user.

[1063] Step 10:

[1064] The user checks the notification and decides whether to respond.

[1065] Specific operation: After checking the notification message, the user can choose to press the answer button or the reject button.

[1066] Step 11:

[1067] The terminal connects or rejects the call according to the user's instructions.

[1068] Specific behavior: If the user presses the answer button, the call will be connected. If the user presses the reject button, the call will be ended.

[1069] Step 12:

[1070] The server analyzes the content of the call in real time and checks whether it matches the initial information.

[1071] Specific operation: The server analyzes the contents of the call using voice recognition and NLP, and if any content differs from the summary, it issues an alert.

[1072] In this way, the system reduces the risk of fraud and unwanted calls and efficiently directs only necessary calls.

[1073] Example 1

[1074] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1075] In modern times, telephone fraud and nuisance sales calls are on the rise. These calls are a major problem, especially for the elderly and businesses, and there is a need for methods to prevent these attacks. However, conventional telephone answering systems are unable to adequately address these issues. An effective system is needed to automatically filter out unreliable callers and notify recipients of only reliable calls.

[1076] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1077] In this invention, the server includes an artificial intelligence device that responds to incoming calls, a means for converting caller information into text using voice recognition technology, a means for calculating the reliability of the call based on the caller's attribute information, and a means for notifying the recipient of the call when the reliability exceeds a certain threshold. This makes it possible to automatically filter out calls from unreliable callers and efficiently deliver only reliable calls to the recipient. Furthermore, by providing a means for monitoring the content of conversations during calls in real time and issuing an alert if there is a discrepancy, further protection from fraud and nuisance calls can be achieved. The reliability calculation uses a machine learning algorithm, improving the accuracy and reliability of the system.

[1078] An "artificial intelligence device that answers incoming calls" is a device that detects an incoming call and automatically plays a response message.

[1079] "Means of converting caller information into text using voice recognition technology" refers to technology for analyzing recorded voice data and converting it into text data.

[1080] The "means for calculating the reliability of a call based on the caller's attribute information" is a means for evaluating attribute information such as the caller's telephone number, region, company name, and past call history, and calculating its reliability.

[1081] The "means for notifying the recipient of a call when the reliability exceeds a certain threshold" refers to a means for notifying the recipient of a call when the calculated reliability exceeds a set threshold.

[1082] "Means for monitoring the content of conversations during calls in real time and issuing an alert if there is a discrepancy" refers to a means for constantly monitoring the content of conversations during calls and issuing an alert if a discrepancy is detected between the information obtained the first time and the content.

[1083] A "machine learning algorithm" is a computer algorithm that learns patterns from data and uses them to make future predictions and classifications.

[1084] This invention is a system that uses artificial intelligence (AI) to handle the primary telephone contact point, eliminating special frauds and nuisance sales calls and realizing safe and efficient telephone response. This system consists of three elements: a server, a terminal, and a user. Specific embodiments are described below.

[1085] System configuration

[1086] 1. Server

[1087] This is the core part of the system where the AI ​​runs. The server is responsible for detecting incoming calls, voice recognition, calculating reliability, and monitoring call content in real time. It mainly uses the following software:

[1088] Google Speech-to-Text API or IBM Watson Speech to Text API: For speech recognition technology.

[1089] Amazon Polly or Google Cloud Text-to-Speech: For converting text to speech.

[1090] scikit-learn or XGBoost: For implementing machine learning algorithms.

[1091] 2. Terminal

[1092] A device used by a user. It detects incoming calls, plays an automatic answering message, and sends the audio data to a server. It also receives notifications from the server and conveys them to the user. Specifically, it is a smartphone or dedicated telephone terminal that the user owns.

[1093] 3. Users

[1094] The person using the system can be an elderly person, a home resident, a company employee, etc. The user receives a notification through the terminal and can answer the call if necessary.

[1095] Specific operation of the system

[1096] 1. Receiving calls and initial response

[1097] The device detects an incoming call and sends the detected call information to the server.

[1098] The server instructs the terminal to play an automatic response message.

[1099] The device plays a message saying, "This is an AI support desk. Please tell us your name and request," and records the caller's voice.

[1100] 2. Collection and analysis of caller information

[1101] The recorded audio data is sent from the terminal to the server.

[1102] The server uses voice recognition technology to convert the voice data into text, extracting information such as "who is calling," "who is it addressed to," and "what the requirement is."

[1103] 3. Calculating reliability

[1104] The server retrieves the caller's attribute information from a database, including phone number, area, company name, and past call history.

[1105] Calculate the trustworthiness of the sender using a machine learning algorithm, such as the Random Forest or XGBoost algorithms from scikit-learn.

[1106] 4. Notifications and call connections

[1107] If the reliability exceeds a set threshold, the server converts the text into speech and sends the speech data to the terminal.

[1108] The terminal plays the notification content to the user, who can then choose to respond or reject it.

[1109] If the user chooses to answer, the terminal connects the call.

[1110] 5. Monitoring and updating attribute information during a call

[1111] During the call, the server analyzes the conversation in real time to see if it matches the initial information.

[1112] If there is no match, the server issues a warning and notifies the user via the terminal.

[1113] The attribute information in the database is updated as needed.

[1114] Specific examples

[1115] When an elderly person answers a phone call

[1116] 1. The device detects an incoming call and notifies the server.

[1117] 2. The device plays an automated response message and records the caller's voice.

[1118] 3. The recorded data is sent to the server, where it is converted into text. The text is summarized as "Caller: XXX, Recipient: Grandpa, Message: Status check."

[1119] 4. The server checks the attribute information and calculates the reliability.

[1120] 5. Since the server has a high reliability, it sends the result to the terminal and notifies the user, "Mr. / Ms. XXX is calling to check on your status."

[1121] 6. The user presses the answer button to connect the call.

[1122] 7. During the call, the server monitors the conversation to ensure it matches.

[1123] When a company receives nuisance calls

[1124] 1. The device detects an incoming call and notifies the server.

[1125] 2. The device plays an automated response message and records the caller's voice.

[1126] 3. The recorded data is sent to a server, where it is converted into text. The text is summarized as "Caller: Anonymous, Destination: Unknown, Requirement: Sales."

[1127] 4. The server checks the attribute information and calculates the reliability.

[1128] 5. The server decides not to notify and the terminal does not connect the call.

[1129] Example prompt sentence:

[1130] "Please tell me the specific steps for a system that uses AI to prevent nuisance and fraudulent calls when elderly people receive phone calls."

[1131] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1132] Step 1:

[1133] The device detects an incoming call and sends the incoming call information to the server. The input includes the phone number and the time of the call. The device collects this data and sends it to the server to notify the call.

[1134] Specific operation: When the phone rings, the device's sensor reacts, captures the incoming call information (phone number and time) in the internal memory, and immediately transfers it to the server.

[1135] Step 2:

[1136] The server receives the incoming call information and sends an instruction to the terminal to play the automatic response message. The server receives the incoming call information from the terminal as input. Based on the received information, the server generates a response message and sends that instruction to the terminal.

[1137] Specific operation: The server analyzes the received telephone number and time information, and sends a command to the terminal to play the automatic response message.

[1138] Step 3:

[1139] The device plays a message saying, "This is an AI support desk. Please tell us your name and requirements," and records the caller's voice. As input, it receives a playback instruction from the server. The device receives the playback instruction and starts recording the voice while playing the specified message to the caller.

[1140] Specific operation: The device's speaker plays the message according to the instructions, the microphone records the caller's voice, and the recorded voice data is stored in temporary memory.

[1141] Step 4:

[1142] The recorded voice data is sent from the device to the server. As input, the recorded voice data is retrieved from the device's memory. The device uploads the voice data to the server.

[1143] Specific operation: To send the audio file stored in the temporary memory to the server, the terminal divides the data into packets and transmits them over the network.

[1144] Step 5:

[1145] The server uses voice recognition technology to convert the voice data into text and extracts information such as "Who is calling?", "Who is the caller?", and "What is the requirement?". The server receives the voice data sent from the device as input. The server uses voice recognition software such as the Google Speech-to-Text API to convert the voice data into text and extract the necessary information.

[1146] Specific operation: Voice data is sent to a voice recognition API, and the returned text is analyzed to extract and summarize important information (sender, recipient, requirements).

[1147] Step 6:

[1148] The server retrieves the caller's attribute information from the database and calculates the caller's reliability using a machine learning algorithm. The input is the text data obtained by speech recognition and the caller information in the database. The server uses, for example, a scikit-learn algorithm to calculate the reliability.

[1149] Specific operation: A database query is issued to obtain the caller's attribute information, which is then passed as input to a machine learning model to calculate a confidence score.

[1150] Step 7:

[1151] If the confidence score exceeds a set threshold, the server sends the result to the device and instructs it to play a notification message to the user.The server uses the calculated confidence score as input.The server converts the text to speech and sends it to the device.

[1152] Specific operation: After converting text to speech, the speech data is sent to the terminal, and the terminal plays the specified notification message to the user.

[1153] Step 8:

[1154] The user checks the notification message and chooses whether to respond or reject it. The notification is received from the device as input. The user selects a response by operating the device's buttons or touch screen.

[1155] Specific operation: The device detects the user's selection and sends that information to the server.

[1156] Step 9:

[1157] The terminal connects or rejects the call based on the user's instruction. The terminal receives the user's selection as input. The terminal performs the call connection operation based on the selection.

[1158] Specific operation: The terminal sends out a call connection signal or a call rejection signal to control the telephone line appropriately.

[1159] Step 10:

[1160] During the call, the server analyzes the conversation in real time to check whether it matches the initial information. The input includes the voice data from the call. The server again uses voice recognition technology to convert it into text data and verify the information.

[1161] What it does: Continuously samples the audio during a call, transcribes it, compares it with the initial data, and generates a warning if there is a mismatch.

[1162] Step 11:

[1163] If a mismatch occurs, the server issues a warning to the device and updates the attribute information in the database. The mismatch detection information is used as input. The server adds new information to the database and updates it.

[1164] Specific operation: Add a new entry to the database based on the detected discrepancy and send a warning signal to the terminal.

[1165] (Application example 1)

[1166] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1167] In today's world, with the increase in special frauds and nuisance sales calls, it is important for call recipients to be protected from these fraudulent calls. Senior citizens and businesses are particularly at risk of business disruption and financial loss from these fraudulent and nuisance calls. Traditional manual responses are insufficient, and more efficient and reliable countermeasures are needed.

[1168] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1169] In this invention, the server includes an artificial intelligence device that responds to incoming calls, a means for converting caller information into text using voice recognition technology, and a means for calculating the reliability of the call based on the caller's attribute information. This makes it possible to eliminate special frauds and nuisance sales calls and notify the recipient of only reliable calls.

[1170] An "artificial intelligence device that answers incoming calls" is a device that has the function of detecting an incoming call and automatically playing a response message.

[1171] "Means of converting to text using voice recognition technology" refers to technology that analyzes recorded voice data and extracts and converts the caller's voice into text.

[1172] The "means for calculating trustworthiness" is a technology that quantifies the trustworthiness of a caller based on the caller's attribute information (telephone number, region, company name, past call history, etc.).

[1173] The "means for notifying the recipient of a call" is a technology that has the function of notifying the recipient of a highly reliable call based on the calculated reliability.

[1174] The "means for transmitting notification content to the recipient by voice synthesis" is a technology for transmitting notification content generated by the server to the recipient as voice using voice synthesis technology.

[1175] "Means for analyzing the content of conversations during a call" refers to technology that monitors the conversation between the caller and recipient in real time during a call and checks whether it matches information obtained in advance.

[1176] "Means for storing caller attribute information in a database" refers to technology that has the function of recording and storing information such as caller characteristics and call history in a database.

[1177] "Means for adjusting reliability based on past call history" refers to a technology that analyzes accumulated call data and updates and adjusts the reliability of callers in real time.

[1178] "Means for providing response options" refers to a technique that presents a reliable call so that the recipient can choose whether to respond.

[1179] "Means for automatically disconnecting a call" refers to technology that has the function of automatically terminating a call when an unreliable call is detected.

[1180] MODE FOR CARRYING OUT THE INVENTION

[1181] System configuration

[1182] This invention is a system that uses AI to handle the primary telephone call counter and eliminate special frauds and nuisance sales calls. The system consists of three elements: a server, a terminal, and a user.

[1183] 1. Server: This is the core part of the system, and it analyzes caller information, calculates reliability, decides whether to notify, and analyzes the content of conversations during calls. The software used includes Python, speech_recognition, pyttsx3, and requests.

[1184] 2. Terminal: A device used by a user. It detects incoming calls, plays an automatic answering message, and receives notifications from the server and notifies the user.

[1185] 3. User: The person who uses this system, such as an elderly person, a home resident, or a company employee. The user receives notifications through the terminal and answers the call as needed.

[1186] System Operation

[1187] Receiving and initial phone calls

[1188] The server detects an incoming call via the device and instructs the device to play an automated answering message: "This is an AI support desk. Please tell us your name and request," and records the caller's voice.

[1189] Collection and analysis of caller information

[1190] The recorded voice data of the caller is sent to a server, which uses speech recognition technology to convert the data into text, extracting and summarizing who is calling, who the call is addressed to, and what the request is.

[1191] Calculating reliability

[1192] The server retrieves the caller's attribute information (such as phone number, area, company name, and past call history) from a database and calculates the caller's trustworthiness based on this information. If the caller has withheld their number, the trustworthiness is set significantly lower.

[1193] Notifications and call connections

[1194] If the reliability exceeds a certain threshold (e.g., 90%), the server converts the result from text to speech and notifies the user via the device. The user can then confirm the content of the notification and choose whether to answer or reject the call. The device will then connect or reject the call according to the user's instructions.

[1195] Monitoring and updating attribute information during a call

[1196] During the call, the server analyzes the conversation in real time to check whether it matches the initial information. If there is a mismatch, the server issues a warning and updates the attribute information in the database.

[1197] Specific examples

[1198] When an elderly person answers a phone call

[1199] 1. The device detects an incoming call and notifies the server.

[1200] 2. The device plays an automated response message and records the caller's voice. The recording is then sent to the server.

[1201] 3. The server converts the voice data into text and summarizes it as "Sender: Mr. X, Address: Recipient, Request: Status check."

[1202] 4. The server checks the attribute information and calculates the reliability.

[1203] 5. The user is notified via the terminal that "Mr. X is calling to check on your status."

[1204] 6. The user presses the answer button to connect the call.

[1205] 7. During the call, the server monitors the conversation and verifies that it matches.

[1206] When a company receives nuisance calls

[1207] 1. The device detects an incoming call and notifies the server.

[1208] 2. The device plays an automated answering message and records the voice of the anonymous caller.

[1209] 3. The recorded data is sent to the server, where it is converted into text. The text is summarized as "Caller: Anonymous, Address: Recipient, Requirement: Sales."

[1210] 4. The server checks the attribute information and calculates the reliability.

[1211] 5. Due to low reliability, the server decides not to notify and the device does not connect the call.

[1212] Prompt Sentence Examples

[1213] "We use an AI model that analyzes audio files, identifies the caller's name, destination, and requirements, and calculates the reliability. For example, the following is from: Caller: Mr. X, Destination: Recipient, Requirements: Status check."

[1214] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1215] Step 1:

[1216] The server detects an incoming call via the terminal. It receives the incoming call signal as input and sends an instruction to the terminal to play an automatic answering message as output. Specifically, the terminal's microphone is enabled and goes into recording mode.

[1217] Step 2:

[1218] The device plays an automated answering message saying, "This is an AI support desk. Please tell us your name and request," and records the caller's voice. It receives incoming call signals and playback instructions as input, generates voice data as output, and sends it to the server. Specifically, the message is played from the speaker, and the caller's voice is recorded by the microphone.

[1219] Step 3:

[1220] The server receives the recorded voice data and converts it into text using speech recognition technology. It receives voice data as input and generates text data as output. Specifically, it analyzes the voice data using the speech_recognition library and converts the caller's name and requirements into text.

[1221] Step 4:

[1222] The server analyzes the text data and extracts and summarizes information such as "who is calling," "who is the caller," and "what are the requirements?" It receives text data as input and generates summary data as output. Specifically, it uses a generative AI model to extract the necessary information from the text.

[1223] Step 5:

[1224] The server retrieves the caller's attribute information from a database and calculates the reliability. It takes the caller's number and past call history as input and generates a reliability score as output. Specifically, it executes a database query and calculates the reliability based on the obtained data.

[1225] Step 6:

[1226] The server determines whether the confidence score exceeds a certain threshold (for example, 90%) and notifies the device. It receives the confidence score as input and converts the notification content into audio as output, which is then sent to the device. Specifically, it uses the pyttsx3 library to convert text to audio.

[1227] Step 7:

[1228] The device receives the notification from the server and conveys the notification content to the recipient. The recipient checks the notification and chooses whether to answer or reject the call. The device receives the notification content as input and sends the recipient's choice to the server as output. Specific operations include playing the notification content from a speaker and displaying an interface for receiving user input.

[1229] Step 8:

[1230] If the user chooses to answer the call, the terminal connects the call by receiving the user's selection as input and generating a call connection signal as output, specifically by opening a line for sending and receiving the call's voice data in real time.

[1231] Step 9:

[1232] During the call, the server analyzes the conversation in real time to check whether it matches the initial information. It receives the call audio data as input and generates the analysis results as output. Specifically, it uses the speech_recognition library again to convert the acquired audio data into text and compare it with the initial information.

[1233] Step 10:

[1234] If a mismatch occurs in the call content, the server issues a warning and updates the attribute information in the database. It receives the analysis results as input and generates updated attribute information as output. Specifically, it displays a mismatch warning on the terminal and executes an update query on the database.

[1235] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1236] This invention is a system that uses artificial intelligence (AI) as the primary point of contact for telephone calls, and combines it with an emotion engine that recognizes the user's emotions to eliminate special frauds and nuisance sales calls, providing greater safety and efficiency. Specific embodiments are described in detail below.

[1237] System configuration

[1238] This system consists of three elements: a server, a terminal, and a user, and is further combined with an emotion engine to perform advanced analysis.

[1239] Server: The core of the system, where the AI ​​and emotion engine run. It analyzes caller information, calculates reliability, decides whether to notify the user, and analyzes the content and emotions of the conversation during the call.

[1240] Terminal: A device used by a user. It detects incoming calls, plays an automatic answering message, and receives notifications from the server and notifies the user.

[1241] User: The person who uses this system, such as an elderly person, a home resident, or a company employee. The user receives notifications through the terminal and answers the call as needed.

[1242] Program processing

[1243] 1. Receiving calls and initial response

[1244] The server detects the incoming call via the terminal and instructs the terminal to play an automated response message: "This is an AI support desk. Please tell us your name and request," and records the caller's voice.

[1245] 2. Collection and analysis of caller information

[1246] The recorded voice data of the caller is sent to a server, which uses speech recognition technology to convert this voice data into text, extracting and summarizing who is calling, who the call is addressed to, and what the request is.

[1247] 3. Calculating reliability

[1248] The server retrieves the caller's attribute information from the database and calculates the caller's trustworthiness based on this information. If the caller's number is withheld, the trustworthiness is set significantly lower.

[1249] 4. Analysis by Emotion Engine

[1250] The server uses an emotion engine to extract emotional data from the caller's recorded voice, analyzes the emotional data, and understands the caller's emotional state.

[1251] 5. Notifications and call connections

[1252] The server comprehensively evaluates the reliability and emotional data, and if the result exceeds a certain threshold, converts the text to speech and notifies the user via the device. The user can then review the notification and choose to accept or reject the call. The device will then connect or reject the call according to the user's instructions.

[1253] 6. Call monitoring and sentiment analysis

[1254] During a call, the server analyzes the conversation content and emotional data in real time. It issues alerts based on whether the initial information matches the conversation content and changes in emotion. This allows it to quickly detect suspicious activity and issue a warning alert if necessary.

[1255] Specific examples

[1256] Flow when a user receives a call

[1257] 1. The device detects an incoming call and notifies the server.

[1258] 2. The terminal plays an automated response message, and after the caller responds, the voice is recorded and sent to the server.

[1259] 3. The server converts the recorded voice into text and summarizes the caller information. At the same time, it analyzes the emotional data using an emotion engine.

[1260] 4. The database is referenced to obtain the sender's attribute information, and the trustworthiness and emotional data are evaluated comprehensively.

[1261] 5. If the confidence and emotion data exceed a certain threshold, the notification is converted from text to speech and notified to the user again via the device.

[1262] 6. The user checks the notification and chooses whether to connect or reject the call. The device follows this choice.

[1263] 7. During the call, the server monitors the conversation content and emotional changes in real time. If it detects any discrepancies with the initial information or abnormal changes in emotions, it issues a warning.

[1264] In this way, this system can prevent fraud and nuisance calls with even greater accuracy by combining not only caller information but also an emotion engine that analyzes the caller's emotional state in real time.

[1265] The processing flow will be explained below.

[1266] Step 1:

[1267] The terminal detects the incoming call and notifies the server of the information.

[1268] Specific operation: The terminal receives an incoming call signal from the telephone line and notifies the server of this fact.

[1269] Step 2:

[1270] The server instructs the terminal to play back the automatic response message.

[1271] Specific operation: The server sends a pre-prepared message to the terminal, playing an audio message saying, "This is the AI ​​support desk. Please tell us your name and requirements."

[1272] Step 3:

[1273] The terminal records the caller's voice and transmits the voice data to the server.

[1274] Specific operation: When the caller answers and states their name and requirements, the voice is recorded and sent to the server in real time.

[1275] Step 4:

[1276] The server sends the voice data to a voice recognition engine and converts it into text data.

[1277] Specific operation: The server sends the received voice data to a cloud speech recognition service, which converts the voice into text.

[1278] Step 5:

[1279] The server analyzes the text, extracts information such as "who is calling," "who is it addressed to," and "what the requirements are," and generates a summary.

[1280] Specific operation: Analyzes text using natural language processing (NLP) algorithms to extract sender information, recipient information, and requirement information.

[1281] Step 6:

[1282] The server obtains the sender's attribute information from the database and calculates the reliability.

[1283] Specific operation: The server obtains attribute information such as the caller's phone number, region, company name, and past call history from a database, and calculates the reliability using a statistical model.

[1284] Step 7:

[1285] The server uses an emotion engine to extract emotion data from the voice data and analyze the caller's emotional state.

[1286] Specific operation: The server sends the voice data to an emotion recognition engine and extracts the caller's emotional data (joy, anger, sadness, etc.).

[1287] Step 8:

[1288] The server comprehensively evaluates the reliability and emotional data and gives a result.

[1289] Specific operation: Determine whether the confidence and emotion data exceed the set threshold (e.g., confidence 90%, emotion data within the normal range).

[1290] Step 9:

[1291] The server converts the results from text to speech and notifies the device.

[1292] Specific operation: The server sends the summarized text data to the speech synthesis engine and sends the generated audio file to the terminal.

[1293] Step 10:

[1294] The terminal plays the notification message to the user.

[1295] Specific operation: The terminal plays the message "This is a call from Mr. / Ms. XX. The request is XX" to the user.

[1296] Step 11:

[1297] The user checks the notification and decides whether to respond.

[1298] Specific operation: After checking the notification message, the user can choose to press the answer button or the reject button.

[1299] Step 12:

[1300] The terminal connects or rejects the call according to the user's instructions.

[1301] Specific operation: When the user presses the answer button, the terminal connects the call; when the user presses the reject button, the terminal ends the call.

[1302] Step 13:

[1303] The server analyzes the conversation content and emotional data during the call in real time.

[1304] Specific operation: After the call is connected, the server sends the voice data to the voice recognition engine and emotion recognition engine in real time to monitor the conversation content and emotional changes.

[1305] Step 14:

[1306] The server compares the initial information with the conversation content and emotional state, and issues a warning if there is a mismatch or if there is an invalid emotional change.

[1307] Specific operation: The server uses a matching algorithm to compare the initial summary with the conversation content and emotional changes, and if there is a mismatch, it issues a warning alert.

[1308] In this way, the system can provide high levels of security and efficiency by analyzing caller information and emotions in real time and managing calls based on trust and emotion data.

[1309] Example 2

[1310] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1311] Conventional telephone answering systems only performed voice recognition and credibility assessment of the caller based on attribute information, which meant they were unable to adequately prevent fraudulent and nuisance calls. It was also difficult to properly analyze the caller's emotional state and detect abnormalities during the call. This could expose the recipient to unnecessary stress and risk.

[1312] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes an artificial intelligence device that responds when a call is received, a means for converting caller information into text using voice recognition technology, a means for calculating the reliability of the call based on the caller's attribute information, a means for extracting and analyzing emotional data from the caller's voice, and a means for notifying the recipient of the call based on the reliability and emotional data. This allows for an advanced combination of caller reliability evaluation and emotional analysis, thereby improving the accuracy of detecting fraudulent and nuisance calls and ensuring the safety of recipients.

[1313] - "Artificial intelligence device" is a system that autonomously responds to incoming calls and collects and analyzes caller information.

[1314] "Voice recognition technology" is a technology that converts voice data from a caller into text data.

[1315] "Caller attribute information" refers to various data related to the caller (e.g., number, past call history, registration information, etc.).

[1316] The "means for calculating the reliability" is a mechanism for quantifying the reliability of a sender based on the sender's attribute information.

[1317] The "means for extracting emotional data" is a function that analyzes and extracts the emotional state (anger, sadness, joy, etc.) from the caller's voice.

[1318] "Call notification means" is a mechanism that notifies the recipient of the caller's information, trustworthiness, and emotional state.

[1319] "Analysis of conversation content during a call" is the process of analyzing the conversation between the caller and receiver in real time during a call to confirm whether it matches the initial information.

[1320] This invention is a system in which artificial intelligence (AI) acts as the primary point of contact for telephone calls and eliminates fraudulent and nuisance calls by analyzing the caller's voice. This system includes three elements: a server, a terminal, and a user, and performs advanced analysis by combining an emotion engine.

[1321] Hardware and Software Configuration

[1322] server:

[1323] The server functions as the core of the system, running the artificial intelligence device and emotion engine. It has the following functions:

[1324] Speech recognition technology: Uses the Google Cloud Speech-to-Text API to convert the caller's voice data into text.

[1325] Database: A database is used to store and reference caller attribute information and past call history.

[1326] Emotion Engine: Uses IBM Watson Tone Analyzer to extract and analyze emotional data from audio.

[1327] Rating algorithms: Executes algorithms to calculate the trustworthiness of callers and evaluate the appropriateness of calls.

[1328] Device:

[1329] The terminal is a device used by the user, which detects incoming calls, plays an automatic answering message, and receives notifications from the server and notifies the user.

[1330] Incoming call detection: Has the ability to detect incoming calls in real time via communication lines.

[1331] Auto-answer: Has a voice playback function to record and play specified messages.

[1332] Voice recording: Equipped with a microphone and storage to record the caller's voice and send it to the server.

[1333] User:

[1334] A user is a person who uses the system and is responsible for answering or rejecting incoming calls.

[1335] Receiving notification: The terminal receives a notification of the evaluation result from the server and processes the call based on that notification.

[1336] System operation example

[1337] When a user receives a call:

[1338] 1. Detecting an incoming call: The device detects an incoming call.

[1339] The phone line on your device will be activated and a ring tone will play.

[1340] 2. Notify the server: The terminal notifies the server of the incoming call.

[1341] The terminal sends an incoming call notification to the server using the UDP protocol.

[1342] 3. Playing an automated response message: The server instructs the terminal to play the message, "This is an AI support desk. Please tell us your name and requirements."

[1343] The device plays a pre-recorded message and records the caller's voice.

[1344] 4. Sending audio data: The device sends the recorded audio data to the server.

[1345] The audio file is uploaded to the server via an HTTP POST request.

[1346] 5. Speech recognition and text conversion: The server converts the voice data into text using the Google Cloud Speech-to-Text API.

[1347] The server converts the speech into text and performs analysis.

[1348] 6. Extraction of caller information: The server extracts important information from the text data.

[1349] The server uses natural language processing (NLP) algorithms to extract information such as names and requirements from the text.

[1350] 7. Calculating trustworthiness: The server queries the database and calculates the trustworthiness of the sender.

[1351] The server calculates the reliability based on past history and registration information.

[1352] 8. Emotion data analysis: The server uses IBM Watson Tone Analyzer to analyze the emotion data.

[1353] The emotion engine analyzes the sender's emotional state.

[1354] 9. Notification and call connection: The server notifies the user of the evaluation result, and the user can choose whether to connect the call.

[1355] The server generates an audio notification and sends it to the device, where the user can confirm and connect or reject the call.

[1356] 10. In-call monitoring: The server analyzes call content and emotion data in real time and issues an alert if it detects any abnormalities.

[1357] The server uses a real-time processing engine to analyze the streaming data and generate alerts as needed.

[1358] Example prompt sentence:

[1359] "Please explain the steps an AI-powered automated telephone answering system takes to analyze the voice data of incoming calls and assess the caller's trustworthiness and emotional state."

[1360] "Please explain the process of combining speech recognition technology and an emotion engine to analyze the caller's information and notify the user."

[1361] "Please explain how callers' voice data can be used to analyze their confidence and emotional data to prevent spam calls."

[1362] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1363] Step 1:

[1364] Incoming call detection and notification

[1365] The terminal detects an incoming call. As input, the terminal monitors the communication line and confirms that there is an incoming call. As output, the terminal notifies the server of the result of the incoming call detection.

[1366] Specific operation: The terminal plays a ring tone and sends an incoming call notification to the server using the UDP protocol.

[1367] Step 2:

[1368] Auto-response message playback and voice recording

[1369] The server instructs the device to play a message saying, "This is an AI support desk. Please tell us your name and request." As input, the server receives an incoming call notification. As output, the device plays an automated response message and records the caller's voice.

[1370] Specific operation: The device plays the specified message, records the caller's voice using the microphone, and creates an audio file.

[1371] Step 3:

[1372] Sending audio data

[1373] The device sends the recorded voice data of the caller to the server. As input, the device receives the recorded voice data. As output, the device uploads the voice data to the server via an HTTP POST request.

[1374] Specific operation: The device sends the audio file to the server via an HTTP POST request, setting the appropriate URL and payload.

[1375] Step 4:

[1376] Speech recognition to text

[1377] The server converts the voice data into text using speech recognition technology (Google Cloud Speech-to-Text API). As input, the server receives the voice data. As output, it generates text data.

[1378] Specific operation: The server calls the API and executes the process of sending voice data and receiving text data.

[1379] Step 5:

[1380] Extracting caller information

[1381] The server extracts information from the text data, such as "Who is calling?", "Who is the caller addressed to?", and "What is the purpose of the call?". As input, the server receives the text data. As output, it generates the extracted caller information as structured data.

[1382] What it does: The server uses natural language processing (NLP) algorithms to extract important information from the text and convert it into structured data, such as JSON format.

[1383] Step 6:

[1384] Calculating reliability

[1385] The server retrieves the sender's attribute information from the database and calculates the reliability. As input, the server retrieves the sender's information and queries the database. As output, it generates data that quantifies the reliability.

[1386] Specific operation: The server queries the database using an SQL query, obtains the sender's attribute information, and executes an algorithm to calculate the trustworthiness based on that information.

[1387] Step 7:

[1388] Emotion data extraction and analysis

[1389] The server uses an emotion engine (IBM Watson Tone Analyzer) to extract emotion data from the recorded voice. As input, the server receives the voice data. As output, it generates the extracted emotion data.

[1390] Specific operation: The server calls the emotion engine API and executes the process of sending voice data and receiving emotion data.

[1391] Step 8:

[1392] Call notification

[1393] The server notifies the user of the evaluation results of the reliability and emotion data. As input, it receives the evaluation results (reliability and emotion data). As output, it converts the evaluation result notification into voice and sends it to the terminal.

[1394] Specific operation: The server converts the evaluation results from text to speech, and generates and transmits a message to send it to the terminal.

[1395] Step 9:

[1396] User discretion and call connection or rejection

[1397] The user checks the notification and selects whether to connect or reject the call. As input, the user receives the notification. As output, the user's selection is input to the terminal.

[1398] Specific action: The user operates the device interface to choose to connect or reject the call.

[1399] Step 10:

[1400] In-call monitoring and anomaly detection

[1401] The server monitors the conversation content and emotion data during the call in real time to detect anomalies. As input, the server receives the audio stream during the call. As output, it generates anomaly detection results and issues warning alerts if necessary.

[1402] Specific operation: The server analyzes the streaming data using a real-time processing engine, compares it with the initial information, and if an abnormality is detected, generates and sends a warning message.

[1403] (Application example 2)

[1404] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1405] Currently, the number of special frauds and nuisance sales calls via telephone is increasing, causing anxiety among many call recipients. It is particularly difficult to distinguish dangerous calls when the caller's emotional state cannot be determined. Therefore, there is a need for a system that can assess not only the caller's trustworthiness but also their emotional state in real time, and prompt the call recipient to take appropriate action.

[1406] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes an artificial intelligence device that answers an incoming call, a means for converting the caller's voice into text using voice recognition technology, a means for calculating the reliability of the call based on the caller's attribute information, a means for extracting emotional data from the caller's voice and evaluating the caller's emotional state, and a means for notifying the call recipient based on the reliability and emotional state. This allows the recipient to understand the caller's reliability and emotional state and respond safely and efficiently.

[1407] An "artificial intelligence device" is an intelligent system that automatically answers calls and analyzes the caller's information.

[1408] "Speech recognition technology" is a technology that converts voice data into text and is used to understand the caller's voice.

[1409] "Reliability" is an index calculated based on the sender's attribute information, and is used to evaluate the sender's reliability.

[1410] "Emotion data" refers to data relating to emotions extracted from the caller's voice, and is used to evaluate the caller's emotional state.

[1411] "Emotional state" indicates the emotional state of the sender, determined based on the sender's emotional data.

[1412] A "call notification means" is a means for informing the recipient of the status of the call based on confidence and emotional state.

[1413] The "means for analyzing the content of the conversation" is a means for analyzing the conversation between the caller and the recipient during the call and checking whether it matches the initial information.

[1414] The "means for monitoring emotional changes in real time" is a means for monitoring emotional changes of a caller or a receiver in real time during a call.

[1415] "Attribute information" refers to data related to the caller's personal information and past call history.

[1416] "Past call history" refers to a record of previous calls between a caller and a recipient.

[1417] The system based on this invention can automatically answer incoming calls and filter out special fraud and nuisance sales calls by evaluating the trustworthiness and emotional state of the caller. Specific embodiments of this system are described in detail below.

[1418] System configuration

[1419] The system mainly consists of three elements: the server, the terminal, and the user. The configuration of each element is as follows:

[1420] Server: The server is the central component that processes and analyzes voice data. When a call comes in, it records the caller's voice, converts it into text using speech recognition technology, calculates the caller's credibility based on their attribute information, and uses an emotion engine to evaluate the caller's emotional state and transmits the results to the device.

[1421] Terminal: The terminal detects incoming calls, plays an auto-answer message in cooperation with the server, and provides information to the user based on notifications received from the server, and connects or rejects the call.

[1422] User: The user handles the call based on the trust and emotional state information provided by the system. The call is connected or rejected according to the user's instructions.

[1423] Program processing

[1424] The system's operations are carried out by the following means:

[1425] Server Processing

[1426] The server uses voice recognition technology and an emotion engine to analyze the caller information of the incoming call and evaluate their trustworthiness and emotional state. The main software used is as follows:

[1427] SpeechRecognition: Used to convert the caller's voice data into text.

[1428] EmotionRecognizer: Extracts emotion data from a caller's voice and uses it to assess their emotional state.

[1429] The server combines these pieces of software to process the data and performs the following steps:

[1430] 1. Record the incoming call audio data.

[1431] 2. Convert the recording into text using voice recognition technology.

[1432] 3. The sender's attribute information is obtained from the database and the reliability is calculated.

[1433] 4. Extract emotional data using an emotion engine and evaluate the emotional state.

[1434] Terminal handling

[1435] The device plays an automated answering message based on instructions from the server, sends recorded audio to the server, and upon receiving a notification, communicates confidence and emotional state information to the user and connects or rejects the call based on user input.

[1436] User operations

[1437] The user can choose whether to accept or reject the call based on the information provided by the device. During the call, the server monitors the conversation and the user's emotional state, and immediately issues an alert if any abnormalities are detected.

[1438] Specific examples

[1439] The sequence of events when a user answers a call is as follows:

[1440] 1. The device detects an incoming call and notifies the server.

[1441] 2. The terminal plays an automated response message, and after the caller responds, the voice is recorded and sent to the server.

[1442] 3. The server converts the recorded voice into text and summarizes the caller information. At the same time, it analyzes the emotional data using an emotion engine.

[1443] 4. The database is referenced to obtain the sender's attribute information, and the trustworthiness and emotional data are evaluated comprehensively.

[1444] 5. If the confidence and emotion data exceed a certain threshold, the notification is converted from text to speech and notified to the user again via the device.

[1445] 6. The user checks the notification and chooses whether to connect or reject the call. The device follows this choice.

[1446] 7. During the call, the server monitors the conversation content and emotional changes in real time and issues an alert if it detects any abnormalities.

[1447] Prompt Sentence Examples

[1448] User: Hello, can you tell me about this system?

[1449] AI: This system analyzes incoming calls, assesses the caller's trustworthiness and emotional state, and notifies the recipient, protecting them from fraud and unwanted sales calls.

[1450] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1451] Step 1:

[1452] The terminal detects an incoming call and notifies the server. The input is the incoming call signal, and the output is an incoming call notification to the server. In concrete terms, the terminal detects that a call has come in and sends that information to the server.

[1453] Step 2:

[1454] The terminal plays an automated response message and records the caller's voice. The input is an instruction from the server to play the automated response message, and the output is the recorded voice data of the caller. Specifically, the terminal plays the message "This is the AI ​​support desk. Please tell us your name and request," and records the caller's response.

[1455] Step 3:

[1456] The server receives the recorded voice and converts it into text using speech recognition technology. The input is the caller's recorded voice data, and the output is the caller's text information. Specifically, the server uses the SpeechRecognition library to convert the voice data into text.

[1457] Step 4:

[1458] The server obtains attribute information from the sender's text information and calculates the reliability. The input is the sender's text information, and the output is the calculated reliability. Specifically, the server refers to the database to obtain the sender's attribute information, and calculates the reliability based on that information.

[1459] Step 5:

[1460] The server uses an emotion engine to extract emotional data from the caller's voice and evaluate their emotional state. The input is the caller's recorded voice data and text information, and the output is the evaluated emotional state. Specifically, the server uses the EmotionRecognizer library to extract emotional data from the voice data and evaluate the emotional state.

[1461] Step 6:

[1462] The server sends a call notification to the recipient based on the trust and emotional state. The input is the calculated trust and the evaluated emotional state, and the output is a notification to the recipient. Specifically, the server comprehensively evaluates the trust and emotional state and sends a notification to the recipient via SMS using Twilio.

[1463] Step 7:

[1464] The recipient checks the notification content and chooses whether to connect or reject the call. The input is the notification content received by the recipient, and the output is an instruction to connect or reject the call. Specifically, the recipient checks the notification content (trustworthiness and emotional state) and instructs the terminal to connect or reject the call.

[1465] Step 8:

[1466] During a call, the server monitors the conversation content and emotional changes in real time. The input is the voice data during the call, and the output is the monitoring results. Specifically, the server analyzes the call content, detects discrepancies with the initial information, or abnormal changes in emotions, and issues a warning.

[1467] Step 9:

[1468] When the server issues a warning, it notifies the recipient of the warning and terminates the call if necessary. The input is the warning issued by the server, and the output is the warning notification to the recipient and an instruction to end the call. Specifically, the server sends the warning content to the recipient via SMS and terminates the call according to the recipient's instructions.

[1469] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1470] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1471] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1472] [Fourth embodiment]

[1473] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1474] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1475] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1476] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1477] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1478] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1479] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1480] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1481] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1482] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1483] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1484] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1485] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1486] This invention is a system that uses artificial intelligence (AI) to handle primary telephone inquiries, eliminating special frauds and nuisance sales calls and realizing safe and efficient telephone response. Specific embodiments are described below.

[1487] System configuration

[1488] This system consists of three elements: a server, a terminal, and a user.

[1489] Server: The core part of the system where AI is executed. It analyzes caller information, calculates reliability, decides whether to notify the caller, and analyzes the content of the conversation during the call.

[1490] Terminal: A device used by a user. It detects incoming calls, plays an automatic answering message, and receives notifications from the server and notifies the user.

[1491] User: The person who uses this system, such as an elderly person, a home resident, or a company employee. The user receives notifications through the terminal and answers the call as needed.

[1492] Program processing

[1493] 1. Receiving calls and initial response

[1494] The server detects an incoming call via the device and instructs the device to play an automated response message: "This is an AI support desk. Please tell us your name and request," and records the caller's voice.

[1495] 2. Collection and analysis of caller information

[1496] The recorded voice data of the caller is sent to a server, which uses speech recognition technology to convert this voice data into text, extracting and summarizing who is calling, who the call is addressed to, and what the request is.

[1497] 3. Calculating reliability

[1498] The server retrieves the caller's attribute information (such as phone number, area, company name, and past call history) from a database and calculates the caller's trustworthiness based on this information. If the caller has withheld their number, the trustworthiness is set significantly lower.

[1499] 4. Notifications and call connections

[1500] If the reliability exceeds a certain threshold (e.g., 90%), the server converts the result from text to speech and notifies the user via the device. The user can then confirm the content of the notification and choose to accept or reject the call. The device will then connect or reject the call according to the user's instructions.

[1501] 5. Monitoring and updating attribute information during a call

[1502] During the call, the server analyzes the conversation in real time to check whether it matches the initial information. If there is a mismatch, the server issues a warning and updates the attribute information in the database.

[1503] Specific examples

[1504] When an elderly person answers a phone call

[1505] 1. The device detects an incoming call and notifies the server.

[1506] 2. The device plays an automated answering message and records the voice of the caller, who identifies himself as "Yamada Taro." The recording data is sent to the server.

[1507] 3. The server converts the voice data into text and summarizes it as "Caller: Yamada Taro, Recipient: Grandpa, Purpose: Check status."

[1508] 4. The server inspects the attribute information and calculates the reliability as 99%.

[1509] 5. The user (grandfather) is notified via the terminal that "Yamada Taro is calling to check on your status."

[1510] 6. The user presses the answer button to connect the call.

[1511] 7. During the call, the server monitors the conversation and verifies that it matches.

[1512] When a company receives nuisance calls

[1513] 1. The device detects an incoming call and notifies the server.

[1514] 2. The device plays an automated answering message and records the voice of the anonymous caller.

[1515] 3. The recorded data is sent to a server, where it is converted into text. The summary is "Caller: Anonymous, Destination: Unknown, Requirement: Sales."

[1516] 4. The server checks the attribute information and calculates the reliability as 10%.

[1517] 5. The server decides not to notify and the terminal does not connect the call.

[1518] In this way, the system can automatically answer calls under certain conditions, notifying and connecting the call recipient only if the call is trustworthy, effectively filtering out fraudulent and nuisance calls.

[1519] The processing flow will be explained below.

[1520] Step 1:

[1521] The terminal detects the incoming call and notifies the server of the information.

[1522] Specific operation: When the terminal receives an incoming call signal from the telephone line, it notifies the server of the incoming call.

[1523] Step 2:

[1524] The server instructs the terminal to play back the automatic response message.

[1525] Specific operation: The server sends a pre-prepared response message to the device and commands the device to play it. The message played is, "This is the AI ​​support desk. Please tell us your name and requirements."

[1526] Step 3:

[1527] The terminal records the caller's voice and transmits the voice data to the server.

[1528] Specific operation: When the caller answers and states their name and requirements, the terminal records the voice and transmits the data to the server in real time.

[1529] Step 4:

[1530] The server sends the voice data to a voice recognition engine and converts it into text data.

[1531] Specific operation: The server sends the received voice data to a cloud-based voice recognition service, converts the caller's speech into text, and receives it.

[1532] Step 5:

[1533] The server analyzes the text, extracts information such as "who is calling," "who is it addressed to," and "what the requirements are," and generates a summary.

[1534] Specific operation: Analyzes text using natural language processing (NLP) algorithms to obtain sender information, recipient information, and requirement information.

[1535] Step 6:

[1536] The server obtains the sender's attribute information from the database and calculates the reliability.

[1537] Specific operation: The server retrieves data such as the caller's phone number, area, company name, and past call history from a database, and calculates its reliability using a statistical model.

[1538] Step 7:

[1539] The server determines whether the confidence level exceeds a set threshold (e.g., 90%) and issues a result.

[1540] Specific operation: If the reliability is 90% or more, it is determined to be a legitimate call and proceeds to the next step. If it is less than 90%, the call is not notified and the call is terminated.

[1541] Step 8:

[1542] The server converts the results from text to speech and notifies the device.

[1543] Specific operation: The server sends the summarized text data to a speech synthesis engine to generate an audio file, which is then sent to the terminal for playback.

[1544] Step 9:

[1545] The terminal plays the notification message to the user.

[1546] Specific operation: The terminal plays the message "This is a call from Mr. / Ms. XX. The request is XX" to the user.

[1547] Step 10:

[1548] The user checks the notification and decides whether to respond.

[1549] Specific operation: After checking the notification message, the user can choose to press the answer button or the reject button.

[1550] Step 11:

[1551] The terminal connects or rejects the call according to the user's instructions.

[1552] Specific behavior: If the user presses the answer button, the call will be connected. If the user presses the reject button, the call will be ended.

[1553] Step 12:

[1554] The server analyzes the content of the call in real time and checks whether it matches the initial information.

[1555] Specific operation: The server analyzes the contents of the call using voice recognition and NLP, and if any content differs from the summary, it issues an alert.

[1556] In this way, the system reduces the risk of fraud and unwanted calls and efficiently directs only necessary calls.

[1557] Example 1

[1558] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1559] In modern times, telephone fraud and nuisance sales calls are on the rise. These calls are a major problem, especially for the elderly and businesses, and there is a need for methods to prevent these attacks. However, conventional telephone answering systems are unable to adequately address these issues. An effective system is needed to automatically filter out unreliable callers and notify recipients of only reliable calls.

[1560] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1561] In this invention, the server includes an artificial intelligence device that responds to incoming calls, a means for converting caller information into text using voice recognition technology, a means for calculating the reliability of the call based on the caller's attribute information, and a means for notifying the recipient of the call when the reliability exceeds a certain threshold. This makes it possible to automatically filter out calls from unreliable callers and efficiently deliver only reliable calls to the recipient. Furthermore, by providing a means for monitoring the content of conversations during calls in real time and issuing an alert if there is a discrepancy, further protection from fraud and nuisance calls can be achieved. The reliability calculation uses a machine learning algorithm, improving the accuracy and reliability of the system.

[1562] An "artificial intelligence device that answers incoming calls" is a device that detects an incoming call and automatically plays a response message.

[1563] "Means of converting caller information into text using voice recognition technology" refers to technology for analyzing recorded voice data and converting it into text data.

[1564] The "means for calculating the reliability of a call based on the caller's attribute information" is a means for evaluating attribute information such as the caller's telephone number, region, company name, and past call history, and calculating its reliability.

[1565] The "means for notifying the recipient of a call when the reliability exceeds a certain threshold" refers to a means for notifying the recipient of a call when the calculated reliability exceeds a set threshold.

[1566] "Means for monitoring the content of conversations during calls in real time and issuing an alert if there is a discrepancy" refers to a means for constantly monitoring the content of conversations during calls and issuing an alert if a discrepancy is detected between the information obtained the first time and the content.

[1567] A "machine learning algorithm" is a computer algorithm that learns patterns from data and uses them to make future predictions and classifications.

[1568] This invention is a system that uses artificial intelligence (AI) to handle the primary telephone contact point, eliminating special frauds and nuisance sales calls and realizing safe and efficient telephone response. This system consists of three elements: a server, a terminal, and a user. Specific embodiments are described below.

[1569] System configuration

[1570] 1. Server

[1571] This is the core part of the system where the AI ​​runs. The server is responsible for detecting incoming calls, voice recognition, calculating reliability, and monitoring call content in real time. It mainly uses the following software:

[1572] Google Speech-to-Text API or IBM Watson Speech to Text API: For speech recognition technology.

[1573] Amazon Polly or Google Cloud Text-to-Speech: For converting text to speech.

[1574] scikit-learn or XGBoost: For implementing machine learning algorithms.

[1575] 2. Terminal

[1576] A device used by a user. It detects incoming calls, plays an automatic answering message, and sends the audio data to a server. It also receives notifications from the server and conveys them to the user. Specifically, it is a smartphone or dedicated telephone terminal that the user owns.

[1577] 3. Users

[1578] The person using the system can be an elderly person, a home resident, a company employee, etc. The user receives a notification through the terminal and can answer the call if necessary.

[1579] Specific operation of the system

[1580] 1. Receiving calls and initial response

[1581] The device detects an incoming call and sends the detected call information to the server.

[1582] The server instructs the terminal to play an automatic response message.

[1583] The device plays a message saying, "This is an AI support desk. Please tell us your name and request," and records the caller's voice.

[1584] 2. Collection and analysis of caller information

[1585] The recorded audio data is sent from the terminal to the server.

[1586] The server uses voice recognition technology to convert the voice data into text, extracting information such as "who is calling," "who is it addressed to," and "what the requirement is."

[1587] 3. Calculating reliability

[1588] The server retrieves the caller's attribute information from a database, including phone number, area, company name, and past call history.

[1589] Calculate the trustworthiness of the sender using a machine learning algorithm, such as the Random Forest or XGBoost algorithms from scikit-learn.

[1590] 4. Notifications and call connections

[1591] If the reliability exceeds a set threshold, the server converts the text into speech and sends the speech data to the terminal.

[1592] The terminal plays the notification content to the user, who can then choose to respond or reject it.

[1593] If the user chooses to answer, the terminal connects the call.

[1594] 5. Monitoring and updating attribute information during a call

[1595] During the call, the server analyzes the conversation in real time to see if it matches the initial information.

[1596] If there is no match, the server issues a warning and notifies the user via the terminal.

[1597] The attribute information in the database is updated as needed.

[1598] Specific examples

[1599] When an elderly person answers a phone call

[1600] 1. The device detects an incoming call and notifies the server.

[1601] 2. The device plays an automated response message and records the caller's voice.

[1602] 3. The recorded data is sent to the server, where it is converted into text. The text is summarized as "Caller: XXX, Recipient: Grandpa, Message: Status check."

[1603] 4. The server checks the attribute information and calculates the reliability.

[1604] 5. Since the server has a high reliability, it sends the result to the terminal and notifies the user, "Mr. / Ms. XXX is calling to check on your status."

[1605] 6. The user presses the answer button to connect the call.

[1606] 7. During the call, the server monitors the conversation to ensure it matches.

[1607] When a company receives nuisance calls

[1608] 1. The device detects an incoming call and notifies the server.

[1609] 2. The device plays an automated response message and records the caller's voice.

[1610] 3. The recorded data is sent to a server, where it is converted into text. The text is summarized as "Caller: Anonymous, Destination: Unknown, Requirement: Sales."

[1611] 4. The server checks the attribute information and calculates the reliability.

[1612] 5. The server decides not to notify and the terminal does not connect the call.

[1613] Example prompt sentence:

[1614] "Please tell me the specific steps for a system that uses AI to prevent nuisance and fraudulent calls when elderly people receive phone calls."

[1615] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1616] Step 1:

[1617] The device detects an incoming call and sends the incoming call information to the server. The input includes the phone number and the time of the call. The device collects this data and sends it to the server to notify the call.

[1618] Specific operation: When the phone rings, the device's sensor reacts, captures the incoming call information (phone number and time) in the internal memory, and immediately transfers it to the server.

[1619] Step 2:

[1620] The server receives the incoming call information and sends an instruction to the terminal to play the automatic response message. The server receives the incoming call information from the terminal as input. Based on the received information, the server generates a response message and sends that instruction to the terminal.

[1621] Specific operation: The server analyzes the received telephone number and time information, and sends a command to the terminal to play the automatic response message.

[1622] Step 3:

[1623] The device plays a message saying, "This is an AI support desk. Please tell us your name and requirements," and records the caller's voice. As input, it receives a playback instruction from the server. The device receives the playback instruction and starts recording the voice while playing the specified message to the caller.

[1624] Specific operation: The device's speaker plays the message according to the instructions, the microphone records the caller's voice, and the recorded voice data is stored in temporary memory.

[1625] Step 4:

[1626] The recorded voice data is sent from the device to the server. As input, the recorded voice data is retrieved from the device's memory. The device uploads the voice data to the server.

[1627] Specific operation: To send the audio file stored in the temporary memory to the server, the terminal divides the data into packets and transmits them over the network.

[1628] Step 5:

[1629] The server uses voice recognition technology to convert the voice data into text and extracts information such as "Who is calling?", "Who is the caller?", and "What is the requirement?". The server receives the voice data sent from the device as input. The server uses voice recognition software such as the Google Speech-to-Text API to convert the voice data into text and extract the necessary information.

[1630] Specific operation: Voice data is sent to a voice recognition API, and the returned text is analyzed to extract and summarize important information (sender, recipient, requirements).

[1631] Step 6:

[1632] The server retrieves the caller's attribute information from the database and calculates the caller's reliability using a machine learning algorithm. The input is the text data obtained by speech recognition and the caller information in the database. The server uses, for example, a scikit-learn algorithm to calculate the reliability.

[1633] Specific operation: A database query is issued to obtain the caller's attribute information, which is then passed as input to a machine learning model to calculate a confidence score.

[1634] Step 7:

[1635] If the confidence score exceeds a set threshold, the server sends the result to the device and instructs it to play a notification message to the user.The server uses the calculated confidence score as input.The server converts the text to speech and sends it to the device.

[1636] Specific operation: After converting text to speech, the speech data is sent to the terminal, and the terminal plays the specified notification message to the user.

[1637] Step 8:

[1638] The user checks the notification message and chooses whether to respond or reject it. The notification is received from the device as input. The user selects a response by operating the device's buttons or touch screen.

[1639] Specific operation: The device detects the user's selection and sends that information to the server.

[1640] Step 9:

[1641] The terminal connects or rejects the call based on the user's instruction. The terminal receives the user's selection as input. The terminal performs the call connection operation based on the selection.

[1642] Specific operation: The terminal sends out a call connection signal or a call rejection signal to control the telephone line appropriately.

[1643] Step 10:

[1644] During the call, the server analyzes the conversation in real time to check whether it matches the initial information. The input includes the voice data from the call. The server again uses voice recognition technology to convert it into text data and verify the information.

[1645] What it does: Continuously samples the audio during a call, transcribes it, compares it with the initial data, and generates a warning if there is a mismatch.

[1646] Step 11:

[1647] If a mismatch occurs, the server issues a warning to the device and updates the attribute information in the database. The mismatch detection information is used as input. The server adds new information to the database and updates it.

[1648] Specific operation: Add a new entry to the database based on the detected discrepancy and send a warning signal to the terminal.

[1649] (Application example 1)

[1650] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1651] In today's world, with the increase in special frauds and nuisance sales calls, it is important for call recipients to be protected from these fraudulent calls. Senior citizens and businesses are particularly at risk of business disruption and financial loss from these fraudulent and nuisance calls. Traditional manual responses are insufficient, and more efficient and reliable countermeasures are needed.

[1652] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1653] In this invention, the server includes an artificial intelligence device that responds to incoming calls, a means for converting caller information into text using voice recognition technology, and a means for calculating the reliability of the call based on the caller's attribute information. This makes it possible to eliminate special frauds and nuisance sales calls and notify the recipient of only reliable calls.

[1654] An "artificial intelligence device that answers incoming calls" is a device that has the function of detecting an incoming call and automatically playing a response message.

[1655] "Means of converting to text using voice recognition technology" refers to technology that analyzes recorded voice data and extracts and converts the caller's voice into text.

[1656] The "means for calculating trustworthiness" is a technology that quantifies the trustworthiness of a caller based on the caller's attribute information (telephone number, region, company name, past call history, etc.).

[1657] The "means for notifying the recipient of a call" is a technology that has the function of notifying the recipient of a highly reliable call based on the calculated reliability.

[1658] The "means for transmitting notification content to the recipient by voice synthesis" is a technology for transmitting notification content generated by the server to the recipient as voice using voice synthesis technology.

[1659] "Means for analyzing the content of conversations during a call" refers to technology that monitors the conversation between the caller and recipient in real time during a call and checks whether it matches information obtained in advance.

[1660] "Means for storing caller attribute information in a database" refers to technology that has the function of recording and storing information such as caller characteristics and call history in a database.

[1661] "Means for adjusting reliability based on past call history" refers to a technology that analyzes accumulated call data and updates and adjusts the reliability of callers in real time.

[1662] "Means for providing response options" refers to a technique that presents a reliable call so that the recipient can choose whether to respond.

[1663] "Means for automatically disconnecting a call" refers to technology that has the function of automatically terminating a call when an unreliable call is detected.

[1664] MODE FOR CARRYING OUT THE INVENTION

[1665] System configuration

[1666] This invention is a system that uses AI to handle the primary telephone call counter and eliminate special frauds and nuisance sales calls. The system consists of three elements: a server, a terminal, and a user.

[1667] 1. Server: This is the core part of the system, and it analyzes caller information, calculates reliability, decides whether to notify, and analyzes the content of conversations during calls. The software used includes Python, speech_recognition, pyttsx3, and requests.

[1668] 2. Terminal: A device used by a user. It detects incoming calls, plays an automatic answering message, and receives notifications from the server and notifies the user.

[1669] 3. User: The person who uses this system, such as an elderly person, a home resident, or a company employee. The user receives notifications through the terminal and answers the call as needed.

[1670] System Operation

[1671] Receiving and initial phone calls

[1672] The server detects an incoming call via the device and instructs the device to play an automated answering message: "This is an AI support desk. Please tell us your name and request," and records the caller's voice.

[1673] Collection and analysis of caller information

[1674] The recorded voice data of the caller is sent to a server, which uses speech recognition technology to convert the data into text, extracting and summarizing who is calling, who the call is addressed to, and what the request is.

[1675] Calculating reliability

[1676] The server retrieves the caller's attribute information (such as phone number, area, company name, and past call history) from a database and calculates the caller's trustworthiness based on this information. If the caller has withheld their number, the trustworthiness is set significantly lower.

[1677] Notifications and call connections

[1678] If the reliability exceeds a certain threshold (e.g., 90%), the server converts the result from text to speech and notifies the user via the device. The user can then confirm the content of the notification and choose whether to answer or reject the call. The device will then connect or reject the call according to the user's instructions.

[1679] Monitoring and updating attribute information during a call

[1680] During the call, the server analyzes the conversation in real time to check whether it matches the initial information. If there is a mismatch, the server issues a warning and updates the attribute information in the database.

[1681] Specific examples

[1682] When an elderly person answers a phone call

[1683] 1. The device detects an incoming call and notifies the server.

[1684] 2. The device plays an automated response message and records the caller's voice. The recording is then sent to the server.

[1685] 3. The server converts the voice data into text and summarizes it as "Sender: Mr. X, Address: Recipient, Request: Status check."

[1686] 4. The server checks the attribute information and calculates the reliability.

[1687] 5. The user is notified via the terminal that "Mr. X is calling to check on your status."

[1688] 6. The user presses the answer button to connect the call.

[1689] 7. During the call, the server monitors the conversation and verifies that it matches.

[1690] When a company receives nuisance calls

[1691] 1. The device detects an incoming call and notifies the server.

[1692] 2. The device plays an automated answering message and records the voice of the anonymous caller.

[1693] 3. The recorded data is sent to the server, where it is converted into text. The text is summarized as "Caller: Anonymous, Address: Recipient, Requirement: Sales."

[1694] 4. The server checks the attribute information and calculates the reliability.

[1695] 5. Due to low reliability, the server decides not to notify and the device does not connect the call.

[1696] Prompt Sentence Examples

[1697] "We use an AI model that analyzes audio files, identifies the caller's name, destination, and requirements, and calculates the reliability. For example, the following is from: Caller: Mr. X, Destination: Recipient, Requirements: Status check."

[1698] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1699] Step 1:

[1700] The server detects an incoming call via the terminal. It receives the incoming call signal as input and sends an instruction to the terminal to play an automatic answering message as output. Specifically, the terminal's microphone is enabled and goes into recording mode.

[1701] Step 2:

[1702] The device plays an automated answering message saying, "This is an AI support desk. Please tell us your name and request," and records the caller's voice. It receives incoming call signals and playback instructions as input, generates voice data as output, and sends it to the server. Specifically, the message is played from the speaker, and the caller's voice is recorded by the microphone.

[1703] Step 3:

[1704] The server receives the recorded voice data and converts it into text using speech recognition technology. It receives voice data as input and generates text data as output. Specifically, it analyzes the voice data using the speech_recognition library and converts the caller's name and requirements into text.

[1705] Step 4:

[1706] The server analyzes the text data and extracts and summarizes information such as "who is calling," "who is the caller," and "what are the requirements?" It receives text data as input and generates summary data as output. Specifically, it uses a generative AI model to extract the necessary information from the text.

[1707] Step 5:

[1708] The server retrieves the caller's attribute information from a database and calculates the reliability. It takes the caller's number and past call history as input and generates a reliability score as output. Specifically, it executes a database query and calculates the reliability based on the obtained data.

[1709] Step 6:

[1710] The server determines whether the confidence score exceeds a certain threshold (for example, 90%) and notifies the device. It receives the confidence score as input and converts the notification content into audio as output, which is then sent to the device. Specifically, it uses the pyttsx3 library to convert text to audio.

[1711] Step 7:

[1712] The device receives the notification from the server and conveys the notification content to the recipient. The recipient checks the notification and chooses whether to answer or reject the call. The device receives the notification content as input and sends the recipient's choice to the server as output. Specific operations include playing the notification content from a speaker and displaying an interface for receiving user input.

[1713] Step 8:

[1714] If the user chooses to answer the call, the terminal connects the call by receiving the user's selection as input and generating a call connection signal as output, specifically by opening a line for sending and receiving the call's voice data in real time.

[1715] Step 9:

[1716] During the call, the server analyzes the conversation in real time to check whether it matches the initial information. It receives the call audio data as input and generates the analysis results as output. Specifically, it uses the speech_recognition library again to convert the acquired audio data into text and compare it with the initial information.

[1717] Step 10:

[1718] If a mismatch occurs in the call content, the server issues a warning and updates the attribute information in the database. It receives the analysis results as input and generates updated attribute information as output. Specifically, it displays a mismatch warning on the terminal and executes an update query on the database.

[1719] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1720] This invention is a system that uses artificial intelligence (AI) as the primary point of contact for telephone calls, and combines it with an emotion engine that recognizes the user's emotions to eliminate special frauds and nuisance sales calls, providing greater safety and efficiency. Specific embodiments are described in detail below.

[1721] System configuration

[1722] This system consists of three elements: a server, a terminal, and a user, and is further combined with an emotion engine to perform advanced analysis.

[1723] Server: The core of the system, where the AI ​​and emotion engine run. It analyzes caller information, calculates reliability, decides whether to notify the user, and analyzes the content and emotions of the conversation during the call.

[1724] Terminal: A device used by a user. It detects incoming calls, plays an automatic answering message, and receives notifications from the server and notifies the user.

[1725] User: The person who uses this system, such as an elderly person, a home resident, or a company employee. The user receives notifications through the terminal and answers the call as needed.

[1726] Program processing

[1727] 1. Receiving calls and initial response

[1728] The server detects the incoming call via the terminal and instructs the terminal to play an automated response message: "This is an AI support desk. Please tell us your name and request," and records the caller's voice.

[1729] 2. Collection and analysis of caller information

[1730] The recorded voice data of the caller is sent to a server, which uses speech recognition technology to convert this voice data into text, extracting and summarizing who is calling, who the call is addressed to, and what the request is.

[1731] 3. Calculating reliability

[1732] The server retrieves the caller's attribute information from the database and calculates the caller's trustworthiness based on this information. If the caller's number is withheld, the trustworthiness is set significantly lower.

[1733] 4. Analysis by Emotion Engine

[1734] The server uses an emotion engine to extract emotional data from the caller's recorded voice, analyzes the emotional data, and understands the caller's emotional state.

[1735] 5. Notifications and call connections

[1736] The server comprehensively evaluates the reliability and emotional data, and if the result exceeds a certain threshold, converts the text to speech and notifies the user via the device. The user can then review the notification and choose to accept or reject the call. The device will then connect or reject the call according to the user's instructions.

[1737] 6. Call monitoring and sentiment analysis

[1738] During a call, the server analyzes the conversation content and emotional data in real time. It issues alerts based on whether the initial information matches the conversation content and changes in emotion. This allows it to quickly detect suspicious activity and issue a warning alert if necessary.

[1739] Specific examples

[1740] Flow when a user receives a call

[1741] 1. The device detects an incoming call and notifies the server.

[1742] 2. The terminal plays an automated response message, and after the caller responds, the voice is recorded and sent to the server.

[1743] 3. The server converts the recorded voice into text and summarizes the caller information. At the same time, it analyzes the emotional data using an emotion engine.

[1744] 4. The database is referenced to obtain the sender's attribute information, and the trustworthiness and emotional data are evaluated comprehensively.

[1745] 5. If the confidence and emotion data exceed a certain threshold, the notification is converted from text to speech and notified to the user again via the device.

[1746] 6. The user checks the notification and chooses whether to connect or reject the call. The device follows this choice.

[1747] 7. During the call, the server monitors the conversation content and emotional changes in real time. If it detects any discrepancies with the initial information or abnormal changes in emotions, it issues a warning.

[1748] In this way, this system can prevent fraud and nuisance calls with even greater accuracy by combining not only caller information but also an emotion engine that analyzes the caller's emotional state in real time.

[1749] The processing flow will be explained below.

[1750] Step 1:

[1751] The terminal detects the incoming call and notifies the server of the information.

[1752] Specific operation: The terminal receives an incoming call signal from the telephone line and notifies the server of this fact.

[1753] Step 2:

[1754] The server instructs the terminal to play back the automatic response message.

[1755] Specific operation: The server sends a pre-prepared message to the terminal, playing an audio message saying, "This is the AI ​​support desk. Please tell us your name and requirements."

[1756] Step 3:

[1757] The terminal records the caller's voice and transmits the voice data to the server.

[1758] Specific operation: When the caller answers and states their name and requirements, the voice is recorded and sent to the server in real time.

[1759] Step 4:

[1760] The server sends the voice data to a voice recognition engine and converts it into text data.

[1761] Specific operation: The server sends the received voice data to a cloud speech recognition service, which converts the voice into text.

[1762] Step 5:

[1763] The server analyzes the text, extracts information such as "who is calling," "who is it addressed to," and "what the requirements are," and generates a summary.

[1764] Specific operation: Analyzes text using natural language processing (NLP) algorithms to extract sender information, recipient information, and requirement information.

[1765] Step 6:

[1766] The server obtains the sender's attribute information from the database and calculates the reliability.

[1767] Specific operation: The server obtains attribute information such as the caller's phone number, region, company name, and past call history from a database, and calculates the reliability using a statistical model.

[1768] Step 7:

[1769] The server uses an emotion engine to extract emotion data from the voice data and analyze the caller's emotional state.

[1770] Specific operation: The server sends the voice data to an emotion recognition engine and extracts the caller's emotional data (joy, anger, sadness, etc.).

[1771] Step 8:

[1772] The server comprehensively evaluates the reliability and emotional data and gives a result.

[1773] Specific operation: Determine whether the confidence and emotion data exceed the set threshold (e.g., confidence 90%, emotion data within the normal range).

[1774] Step 9:

[1775] The server converts the results from text to speech and notifies the device.

[1776] Specific operation: The server sends the summarized text data to the speech synthesis engine and sends the generated audio file to the terminal.

[1777] Step 10:

[1778] The terminal plays the notification message to the user.

[1779] Specific operation: The terminal plays the message "This is a call from Mr. / Ms. XX. The request is XX" to the user.

[1780] Step 11:

[1781] The user checks the notification and decides whether to respond.

[1782] Specific operation: After checking the notification message, the user can choose to press the answer button or the reject button.

[1783] Step 12:

[1784] The terminal connects or rejects the call according to the user's instructions.

[1785] Specific operation: When the user presses the answer button, the terminal connects the call; when the user presses the reject button, the terminal ends the call.

[1786] Step 13:

[1787] The server analyzes the conversation content and emotional data during the call in real time.

[1788] Specific operation: After the call is connected, the server sends the voice data to the voice recognition engine and emotion recognition engine in real time to monitor the conversation content and emotional changes.

[1789] Step 14:

[1790] The server compares the initial information with the conversation content and emotional state, and issues a warning if there is a mismatch or if there is an invalid emotional change.

[1791] Specific operation: The server uses a matching algorithm to compare the initial summary with the conversation content and emotional changes, and if there is a mismatch, it issues a warning alert.

[1792] In this way, the system can provide high levels of security and efficiency by analyzing caller information and emotions in real time and managing calls based on trust and emotion data.

[1793] Example 2

[1794] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1795] Conventional telephone answering systems only performed voice recognition and credibility assessment of the caller based on attribute information, which meant they were unable to adequately prevent fraudulent and nuisance calls. It was also difficult to properly analyze the caller's emotional state and detect abnormalities during the call. This could expose the recipient to unnecessary stress and risk.

[1796] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes an artificial intelligence device that responds when a call is received, a means for converting caller information into text using voice recognition technology, a means for calculating the reliability of the call based on the caller's attribute information, a means for extracting and analyzing emotional data from the caller's voice, and a means for notifying the recipient of the call based on the reliability and emotional data. This allows for an advanced combination of caller reliability evaluation and emotional analysis, thereby improving the accuracy of detecting fraudulent and nuisance calls and ensuring the safety of recipients.

[1797] - "Artificial intelligence device" is a system that autonomously responds to incoming calls and collects and analyzes caller information.

[1798] "Voice recognition technology" is a technology that converts voice data from a caller into text data.

[1799] "Caller attribute information" refers to various data related to the caller (e.g., number, past call history, registration information, etc.).

[1800] The "means for calculating the reliability" is a mechanism for quantifying the reliability of a sender based on the sender's attribute information.

[1801] The "means for extracting emotional data" is a function that analyzes and extracts the emotional state (anger, sadness, joy, etc.) from the caller's voice.

[1802] "Call notification means" is a mechanism that notifies the recipient of the caller's information, trustworthiness, and emotional state.

[1803] "Analysis of conversation content during a call" is the process of analyzing the conversation between the caller and receiver in real time during a call to confirm whether it matches the initial information.

[1804] This invention is a system in which artificial intelligence (AI) acts as the primary point of contact for telephone calls and eliminates fraudulent and nuisance calls by analyzing the caller's voice. This system includes three elements: a server, a terminal, and a user, and performs advanced analysis by combining an emotion engine.

[1805] Hardware and Software Configuration

[1806] server:

[1807] The server functions as the core of the system, running the artificial intelligence device and emotion engine. It has the following functions:

[1808] Speech recognition technology: Uses the Google Cloud Speech-to-Text API to convert the caller's voice data into text.

[1809] Database: A database is used to store and reference caller attribute information and past call history.

[1810] Emotion Engine: Uses IBM Watson Tone Analyzer to extract and analyze emotional data from audio.

[1811] Rating algorithms: Executes algorithms to calculate the trustworthiness of callers and evaluate the appropriateness of calls.

[1812] Device:

[1813] The terminal is a device used by the user, which detects incoming calls, plays an automatic answering message, and receives notifications from the server and notifies the user.

[1814] Incoming call detection: Has the ability to detect incoming calls in real time via communication lines.

[1815] Auto-answer: Has a voice playback function to record and play specified messages.

[1816] Voice recording: Equipped with a microphone and storage to record the caller's voice and send it to the server.

[1817] User:

[1818] A user is a person who uses the system and is responsible for answering or rejecting incoming calls.

[1819] Receiving notification: The terminal receives a notification of the evaluation result from the server and processes the call based on that notification.

[1820] System operation example

[1821] When a user receives a call:

[1822] 1. Detecting an incoming call: The device detects an incoming call.

[1823] The phone line on your device will be activated and a ring tone will play.

[1824] 2. Notify the server: The terminal notifies the server of the incoming call.

[1825] The terminal sends an incoming call notification to the server using the UDP protocol.

[1826] 3. Playing an automated response message: The server instructs the terminal to play the message, "This is an AI support desk. Please tell us your name and requirements."

[1827] The device plays a pre-recorded message and records the caller's voice.

[1828] 4. Sending audio data: The device sends the recorded audio data to the server.

[1829] The audio file is uploaded to the server via an HTTP POST request.

[1830] 5. Speech recognition and text conversion: The server converts the voice data into text using the Google Cloud Speech-to-Text API.

[1831] The server converts the speech into text and performs analysis.

[1832] 6. Extraction of caller information: The server extracts important information from the text data.

[1833] The server uses natural language processing (NLP) algorithms to extract information such as names and requirements from the text.

[1834] 7. Calculating trustworthiness: The server queries the database and calculates the trustworthiness of the sender.

[1835] The server calculates the reliability based on past history and registration information.

[1836] 8. Emotion data analysis: The server uses IBM Watson Tone Analyzer to analyze the emotion data.

[1837] The emotion engine analyzes the sender's emotional state.

[1838] 9. Notification and call connection: The server notifies the user of the evaluation result, and the user can choose whether to connect the call.

[1839] The server generates an audio notification and sends it to the device, where the user can confirm and connect or reject the call.

[1840] 10. In-call monitoring: The server analyzes call content and emotion data in real time and issues an alert if it detects any abnormalities.

[1841] The server uses a real-time processing engine to analyze the streaming data and generate alerts as needed.

[1842] Example prompt sentence:

[1843] "Please explain the steps an AI-powered automated telephone answering system takes to analyze the voice data of incoming calls and assess the caller's trustworthiness and emotional state."

[1844] "Please explain the process of combining speech recognition technology and an emotion engine to analyze the caller's information and notify the user."

[1845] "Please explain how callers' voice data can be used to analyze their confidence and emotional data to prevent spam calls."

[1846] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1847] Step 1:

[1848] Incoming call detection and notification

[1849] The terminal detects an incoming call. As input, the terminal monitors the communication line and confirms that there is an incoming call. As output, the terminal notifies the server of the result of the incoming call detection.

[1850] Specific operation: The terminal plays a ring tone and sends an incoming call notification to the server using the UDP protocol.

[1851] Step 2:

[1852] Auto-response message playback and voice recording

[1853] The server instructs the device to play a message saying, "This is an AI support desk. Please tell us your name and request." As input, the server receives an incoming call notification. As output, the device plays an automated response message and records the caller's voice.

[1854] Specific operation: The device plays the specified message, records the caller's voice using the microphone, and creates an audio file.

[1855] Step 3:

[1856] Sending audio data

[1857] The device sends the recorded voice data of the caller to the server. As input, the device receives the recorded voice data. As output, the device uploads the voice data to the server via an HTTP POST request.

[1858] Specific operation: The device sends the audio file to the server via an HTTP POST request, setting the appropriate URL and payload.

[1859] Step 4:

[1860] Speech recognition to text

[1861] The server converts the voice data into text using speech recognition technology (Google Cloud Speech-to-Text API). As input, the server receives the voice data. As output, it generates text data.

[1862] Specific operation: The server calls the API and executes the process of sending voice data and receiving text data.

[1863] Step 5:

[1864] Extracting caller information

[1865] The server extracts information from the text data, such as "Who is calling?", "Who is the caller addressed to?", and "What is the purpose of the call?". As input, the server receives the text data. As output, it generates the extracted caller information as structured data.

[1866] What it does: The server uses natural language processing (NLP) algorithms to extract important information from the text and convert it into structured data, such as JSON format.

[1867] Step 6:

[1868] Calculating reliability

[1869] The server retrieves the sender's attribute information from the database and calculates the reliability. As input, the server retrieves the sender's information and queries the database. As output, it generates data that quantifies the reliability.

[1870] Specific operation: The server queries the database using an SQL query, obtains the sender's attribute information, and executes an algorithm to calculate the trustworthiness based on that information.

[1871] Step 7:

[1872] Emotion data extraction and analysis

[1873] The server uses an emotion engine (IBM Watson Tone Analyzer) to extract emotion data from the recorded voice. As input, the server receives the voice data. As output, it generates the extracted emotion data.

[1874] Specific operation: The server calls the emotion engine API and executes the process of sending voice data and receiving emotion data.

[1875] Step 8:

[1876] Call notification

[1877] The server notifies the user of the evaluation results of the reliability and emotion data. As input, it receives the evaluation results (reliability and emotion data). As output, it converts the evaluation result notification into voice and sends it to the terminal.

[1878] Specific operation: The server converts the evaluation results from text to speech, and generates and transmits a message to send it to the terminal.

[1879] Step 9:

[1880] User discretion and call connection or rejection

[1881] The user checks the notification and selects whether to connect or reject the call. As input, the user receives the notification. As output, the user's selection is input to the terminal.

[1882] Specific action: The user operates the device interface to choose to connect or reject the call.

[1883] Step 10:

[1884] In-call monitoring and anomaly detection

[1885] The server monitors the conversation content and emotion data during the call in real time to detect anomalies. As input, the server receives the audio stream during the call. As output, it generates anomaly detection results and issues warning alerts if necessary.

[1886] Specific operation: The server analyzes the streaming data using a real-time processing engine, compares it with the initial information, and if an abnormality is detected, generates and sends a warning message.

[1887] (Application example 2)

[1888] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1889] Currently, the number of special frauds and nuisance sales calls via telephone is increasing, causing anxiety among many call recipients. It is particularly difficult to distinguish dangerous calls when the caller's emotional state cannot be determined. Therefore, there is a need for a system that can assess not only the caller's trustworthiness but also their emotional state in real time, and prompt the call recipient to take appropriate action.

[1890] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes an artificial intelligence device that answers an incoming call, a means for converting the caller's voice into text using voice recognition technology, a means for calculating the reliability of the call based on the caller's attribute information, a means for extracting emotional data from the caller's voice and evaluating the caller's emotional state, and a means for notifying the call recipient based on the reliability and emotional state. This allows the recipient to understand the caller's reliability and emotional state and respond safely and efficiently.

[1891] An "artificial intelligence device" is an intelligent system that automatically answers calls and analyzes the caller's information.

[1892] "Speech recognition technology" is a technology that converts voice data into text and is used to understand the caller's voice.

[1893] "Reliability" is an index calculated based on the sender's attribute information, and is used to evaluate the sender's reliability.

[1894] "Emotion data" refers to data relating to emotions extracted from the caller's voice, and is used to evaluate the caller's emotional state.

[1895] "Emotional state" indicates the emotional state of the sender, determined based on the sender's emotional data.

[1896] A "call notification means" is a means for informing the recipient of the status of the call based on confidence and emotional state.

[1897] The "means for analyzing the content of the conversation" is a means for analyzing the conversation between the caller and the recipient during the call and checking whether it matches the initial information.

[1898] The "means for monitoring emotional changes in real time" is a means for monitoring emotional changes of a caller or a receiver in real time during a call.

[1899] "Attribute information" refers to data related to the caller's personal information and past call history.

[1900] "Past call history" refers to a record of previous calls between a caller and a recipient.

[1901] The system based on this invention can automatically answer incoming calls and filter out special fraud and nuisance sales calls by evaluating the trustworthiness and emotional state of the caller. Specific embodiments of this system are described in detail below.

[1902] System configuration

[1903] The system mainly consists of three elements: the server, the terminal, and the user. The configuration of each element is as follows:

[1904] Server: The server is the central component that processes and analyzes voice data. When a call comes in, it records the caller's voice, converts it into text using speech recognition technology, calculates the caller's credibility based on their attribute information, and uses an emotion engine to evaluate the caller's emotional state and transmits the results to the device.

[1905] Terminal: The terminal detects incoming calls, plays an auto-answer message in cooperation with the server, and provides information to the user based on notifications received from the server, and connects or rejects the call.

[1906] User: The user handles the call based on the trust and emotional state information provided by the system. The call is connected or rejected according to the user's instructions.

[1907] Program processing

[1908] The system's operations are carried out by the following means:

[1909] Server Processing

[1910] The server uses voice recognition technology and an emotion engine to analyze the caller information of the incoming call and evaluate their trustworthiness and emotional state. The main software used is as follows:

[1911] SpeechRecognition: Used to convert the caller's voice data into text.

[1912] EmotionRecognizer: Extracts emotion data from a caller's voice and uses it to assess their emotional state.

[1913] The server combines these pieces of software to process the data and performs the following steps:

[1914] 1. Record the incoming call audio data.

[1915] 2. Convert the recording into text using voice recognition technology.

[1916] 3. The sender's attribute information is obtained from the database and the reliability is calculated.

[1917] 4. Extract emotional data using an emotion engine and evaluate the emotional state.

[1918] Terminal handling

[1919] The device plays an automated answering message based on instructions from the server, sends recorded audio to the server, and upon receiving a notification, communicates confidence and emotional state information to the user and connects or rejects the call based on user input.

[1920] User operations

[1921] The user can choose whether to accept or reject the call based on the information provided by the device. During the call, the server monitors the conversation and the user's emotional state, and immediately issues an alert if any abnormalities are detected.

[1922] Specific examples

[1923] The sequence of events when a user answers a call is as follows:

[1924] 1. The device detects an incoming call and notifies the server.

[1925] 2. The terminal plays an automated response message, and after the caller responds, the voice is recorded and sent to the server.

[1926] 3. The server converts the recorded voice into text and summarizes the caller information. At the same time, it analyzes the emotional data using an emotion engine.

[1927] 4. The database is referenced to obtain the sender's attribute information, and the trustworthiness and emotional data are evaluated comprehensively.

[1928] 5. If the confidence and emotion data exceed a certain threshold, the notification is converted from text to speech and notified to the user again via the device.

[1929] 6. The user checks the notification and chooses whether to connect or reject the call. The device follows this choice.

[1930] 7. During the call, the server monitors the conversation content and emotional changes in real time and issues an alert if it detects any abnormalities.

[1931] Prompt Sentence Examples

[1932] User: Hello, can you tell me about this system?

[1933] AI: This system analyzes incoming calls, assesses the caller's trustworthiness and emotional state, and notifies the recipient, protecting them from fraud and unwanted sales calls.

[1934] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1935] Step 1:

[1936] The terminal detects an incoming call and notifies the server. The input is the incoming call signal, and the output is an incoming call notification to the server. In concrete terms, the terminal detects that a call has come in and sends that information to the server.

[1937] Step 2:

[1938] The terminal plays an automated response message and records the caller's voice. The input is an instruction from the server to play the automated response message, and the output is the recorded voice data of the caller. Specifically, the terminal plays the message "This is the AI ​​support desk. Please tell us your name and request," and records the caller's response.

[1939] Step 3:

[1940] The server receives the recorded voice and converts it into text using speech recognition technology. The input is the caller's recorded voice data, and the output is the caller's text information. Specifically, the server uses the SpeechRecognition library to convert the voice data into text.

[1941] Step 4:

[1942] The server obtains attribute information from the sender's text information and calculates the reliability. The input is the sender's text information, and the output is the calculated reliability. Specifically, the server refers to the database to obtain the sender's attribute information, and calculates the reliability based on that information.

[1943] Step 5:

[1944] The server uses an emotion engine to extract emotional data from the caller's voice and evaluate their emotional state. The input is the caller's recorded voice data and text information, and the output is the evaluated emotional state. Specifically, the server uses the EmotionRecognizer library to extract emotional data from the voice data and evaluate the emotional state.

[1945] Step 6:

[1946] The server sends a call notification to the recipient based on the trust and emotional state. The input is the calculated trust and the evaluated emotional state, and the output is a notification to the recipient. Specifically, the server comprehensively evaluates the trust and emotional state and sends a notification to the recipient via SMS using Twilio.

[1947] Step 7:

[1948] The recipient checks the notification content and chooses whether to connect or reject the call. The input is the notification content received by the recipient, and the output is an instruction to connect or reject the call. Specifically, the recipient checks the notification content (trustworthiness and emotional state) and instructs the terminal to connect or reject the call.

[1949] Step 8:

[1950] During a call, the server monitors the conversation content and emotional changes in real time. The input is the voice data during the call, and the output is the monitoring results. Specifically, the server analyzes the call content, detects discrepancies with the initial information, or abnormal changes in emotions, and issues a warning.

[1951] Step 9:

[1952] When the server issues a warning, it notifies the recipient of the warning and terminates the call if necessary. The input is the warning issued by the server, and the output is the warning notification to the recipient and an instruction to end the call. Specifically, the server sends the warning content to the recipient via SMS and terminates the call according to the recipient's instructions.

[1953] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1954] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1955] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1956] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1957] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple em...

Claims

1. An artificial intelligence device that answers calls when they come in, A means for converting the caller's information into text using voice recognition technology; A means for calculating the reliability of a call based on attribute information of a caller; means for notifying a recipient of the call based on the reliability; A system including:

2. 2. The system according to claim 1, further comprising means for analyzing the content of the conversation during the call and confirming whether it matches the initial information.

3. 2. The system according to claim 1, further comprising means for storing attribute information of a caller in a database and adjusting the reliability based on a past call history.

Images