system

Abstract

To provide a system.SOLUTION: The system includes: means for acquiring inquiry data including a text, voice, or an image which a user sent through a messenger application; means for analyzing the inquiry data and deriving information wanted by the user as the result of analysis; means for calling for information acquisition API and acquiring the information wanted by the user; and means for presenting the information acquired by using the notification function of the messenger application.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including a description and related instruction sentence regarding the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-180282 A Summary of the Invention [Problem to be solved by the invention]

[0004] Currently, there are many ways to obtain information, but they are scattered, making it difficult to obtain all information on one platform. In addition, for non-digital native users and elderly people, the operation is complicated and specific devices are required. [Means for solving the problem]

[0005] This invention provides a messenger app internet search tool that can be used with text, voice, and images, and makes it possible to obtain all information on one platform by covering information on weather, disasters, news, EC, finance, etc. In addition, to make it available to non-digital native users and elderly people, it supports AI voiceprint authentication and can be used on landlines. [Brief description of the drawings]

[0006] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Diagram 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. FIG. [Diagram 3] FIG. 11 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Diagram 5] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 13 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 2 is a sequence diagram showing a process flow of the data processing system in the first embodiment of the first form example. [Figure 12] FIG. 11 is a sequence diagram showing a process flow of the data processing system in the application example 1 of the first embodiment. [Figure 13] FIG. 11 is a sequence diagram showing a process flow of a data processing system in the second embodiment of the second embodiment. [Figure 14] FIG. 13 is a sequence diagram showing a process flow of the data processing system in application example 2 of embodiment 2. [Figure 15] FIG. 13 is a sequence diagram showing a process flow of a data processing system in a third embodiment of the present invention. [Figure 16] FIG. 13 is a sequence diagram showing a process flow of the data processing system in the third application example of the third embodiment. [Figure 17] FIG. 11 is a sequence diagram showing the flow of processing in the data processing system in the first embodiment of the first form example when an emotion engine is combined. [Figure 18] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Form Example 1 when combined with an emotion engine. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0008] do.

[0009] First, the terms used in the following description will be explained.

[0010] In the following embodiments, a signed processor (hereinafter simply referred to as a "processor") may be one arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be one type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)).

[0011] In the following embodiments, a signed RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by the processor.

[0012] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0013] In the following embodiments, a communication I / F (Interface) with a code is an interface including a communication processor and an antenna. The communication I / F controls communication between multiple computers. An example of a communication standard applied to the communication I / F is a wireless communication standard including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0014] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. In addition, in this specification, the same idea as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0015] [First embodiment]

[0016] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0017] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0018] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0019] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0020] The reception device 38 includes a touch panel 38A and a microphone 38B, and receives user input. The touch panel 38A detects contact with a pointer (e.g., a pen or a finger) to receive user input by the touch of the pointer. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0021] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (e.g., voice and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs voice according to instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a Complementary Metal-Oxide-Semiconductor (CMOS) image sensor or a Charge Coupled Device (CCD) image sensor.

[0022] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54.

[0023] FIG. 2 shows an example of main functions of the data processing device 12 and the smart device 14.

[0024] As shown in Fig. 2, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32. The specific process program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific process program 56 from the storage 32, and executes the read specific process program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific process program 56 executed on the RAM 30.

[0025] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0026] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores a reception output program 60. The reception output program 60 is used together with the specific processing program 56 by the data processing system 10. The processor 46 reads out the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0027] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0028] "Example 1"

[0029] The present invention provides a messenger app internet search tool that can use text, voice, and images. Specifically, when a user sends text, voice, or image through a messenger app, the system analyzes it and provides the required information. For example, when a user sends a text such as "What's the weather like tomorrow?", the system retrieves weather forecast information and provides it to the user.

[0030] "Example 2"

[0031] Furthermore, the present invention supports AI voiceprint authentication to enable use by non-digital native users and elderly people. Specifically, when a user requests information by voice from a landline phone, the system performs AI voiceprint authentication to authenticate the user. If authentication is successful, the information requested by the user is provided by voice. For example, when an elderly person requests by voice from a landline phone, "What's in the news today?", the system obtains the news information and provides it by voice.

[0032] The process flow of each embodiment will be described below.

[0033] "Example 1"

[0034] Step 1: User sends a text, voice or image via a messenger app.

[0035] Step 2: The system analyzes the transmitted text, audio and images.

[0036] Step 3: Based on the analysis result, the system obtains the required information. For example, if the user sends a text "What's the weather like tomorrow?", the system will obtain weather forecast information.

[0037] Step 4: Provide the retrieved information to the user.

[0038] "Example 2"

[0039] Step 1: User requests information by voice over a landline phone.

[0040] Step 2: The system analyzes the voice and performs AI voiceprint authentication.

[0041] Step 3: If the authentication is successful, the system retrieves the information requested by the user. For example, if an elderly person requests "What's the news today?" by voice, the system retrieves the news information.

[0042] Step 4: Provide the retrieved information to the user via voice.

[0043] Example 1

[0044] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal".

[0045] Conventional messenger apps are limited to sending text, voice, and images, making it difficult to provide users with the information they need quickly and accurately. In addition, these apps are difficult to use for non-digital native users and elderly people. Furthermore, voice recognition and information provision functions are lacking for elderly people who use landlines. To solve these issues, a system that analyzes data sent by users and provides the necessary information is required.

[0046] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0047] In this invention, the server includes a means for receiving data sent by a user through a messenger app, a means for using a natural language processing engine to analyze the received data, and a means for acquiring necessary information based on the analysis result, which makes it possible to analyze the text, voice, and image sent by the user and provide the necessary information quickly and accurately.

[0048] "Text" refers to textual information that a user sends through a messenger app.

[0049] "Audio" refers to the sound information that a user sends through a messenger app.

[0050] An "image" is any visual information that a user sends through a messenger app.

[0051] "Messenger App Internet Search Tool" is an application that allows users to send text, voice, and images, which are then analyzed to provide the necessary information.

[0052] "Information on weather, disasters, news, e-commerce, finance, etc." are a wide range of information categories that users desire.

[0053] A "smartphone" is a portable information terminal that can connect to the Internet and use applications.

[0054] A "server" is a computer system that receives data sent by a user, analyzes it, and provides the necessary information.

[0055] A "natural language processing engine" is software that analyzes text data and understands its meaning and intent.

[0056] The "analysis results" are information obtained by the natural language processing engine analyzing text data.

[0057] "Means of obtaining information" refers to the process of obtaining the necessary information from external information sources based on the analysis results.

[0058] "Means of returning information" is the process by which retrieved information is provided to the user.

[0059] "Non-digital native users" are users who are not familiar with digital technology.

[0060] "Artificial intelligence voiceprint authentication" is a technology that analyzes voice to recognize the speaker.

[0061] "Elderly people" are older users who may have difficulty using digital technology.

[0062] A "fixed-line telephone" is a telephone that is fixed and cannot be moved.

[0063] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0064] The present invention is a system that analyzes text, voice, and images sent by a user through a messenger app and provides necessary information. A specific embodiment of the system will be described below.

[0065] 1. System Configuration

[0066] The system consists of the following main components:

[0067] User device: The device through which a user uses the messenger app, such as a smartphone or landline phone.

[0068] Server: A computer system that receives data sent by users, analyzes it, and provides the required information.

[0069] Natural language processing engine: Software for analyzing text data and understanding its meaning and intent. Specifically, it uses natural language processing APIs.

[0070] Speech recognition software: Software for converting voice data into text, specifically using a speech recognition API.

[0071] Image recognition software: Software for analyzing image data. Specifically, it uses an image recognition API.

[0072] Information Acquisition API: An API for acquiring necessary information from external information sources based on the analysis results. Specifically, a weather forecast API is used.

[0073] 2. Program Processing

[0074] 1. User Input

[0075] User opens the Messenger app and sends a text, voice, or image.

[0076] Example: A user enters the text "What's the weather like tomorrow?" and presses the submit button.

[0077] 2. Receiving Data

[0078] The server receives data sent from the messenger app in real time and temporarily stores the received data in a database.

[0079] 3. Data Analysis

[0080] The server calls a natural language processing engine to analyze the data it receives. The API analyzes the text and understands the intent.

[0081] In the case of voice data, the server uses voice recognition software to convert the voice to text.

[0082] In the case of image data, the server uses image recognition software to analyze the image.

[0083] Example: The server sends the text "What's the weather like tomorrow?" to a natural language processing engine, and obtains the intent "I'm looking for a weather forecast" as the analysis result.

[0084] 4. Acquisition of information

[0085] Based on the analysis results, the server calls an information retrieval API and retrieves the specified information.

[0086] Example: Based on the analysis result of "What's the weather going to be tomorrow?", the server calls a weather forecast API and obtains the information "Tomorrow's weather will be sunny."

[0087] 5. Provision of information

[0088] The server returns the acquired information to the user. When returning the information, the notification function of the messenger app is used to display the information to the user.

[0089] Example: The server notifies the user that "tomorrow's weather will be sunny" via a messenger app and displays the information on the user's screen.

[0090] 3. Examples and prompts

[0091] Example 1: Text questions

[0092] User: "What's the weather like tomorrow?"

[0093] Server: Analyzes the text, retrieves weather forecast information, and replies to the user, "Tomorrow's weather will be sunny."

[0094] Example 2: Audio questions

[0095] User: Asks out loud, "What's the weather like tomorrow?"

[0096] Server: Converts the speech to text, analyzes it to obtain weather forecast information, and responds to the user, "Tomorrow's weather will be sunny."

[0097] Example 3: Image Question

[0098] User: Send weather related images.

[0099] Server: Analyzes the image, retrieves relevant weather forecast information, and responds to the user, "The weather in this area tomorrow will be sunny."

[0100] Example prompts for generative AI models

[0101] "If a user sends a text through your messenger app saying, 'What's the weather going to be like tomorrow?', please explain how you would get the weather forecast information and respond to the user."

[0102] In this way, a system can be created that analyzes the text, voice, and images sent by users through messenger apps and provides the necessary information.

[0103] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0104] Step 1:

[0105] User Input

[0106] How it works: A user opens a messaging app on their smartphone and sends a text, voice, or image.

[0107] Input: The user enters the text "What's the weather going to be like tomorrow?" and presses the submit button.

[0108] Output: The messenger app sends text data to the server.

[0109] Step 2:

[0110] Receiving data

[0111] How it works: The server receives inquiry data sent from the messenger app in real time, and temporarily stores the received data in a database.

[0112] Input: Text data sent from a messenger app: "What's the weather going to be like tomorrow?"

[0113] Output: The server prepares the received text data for analysis.

[0114] Step 3:

[0115] Analyzing data

[0116] How it works: The server calls a natural language processing engine to analyze the data it receives. The API analyzes the text and understands the intent. If it's voice data, the server uses voice recognition software to convert the speech to text. If it's image data, the server uses image recognition software to analyze the image.

[0117] Input: Text data received by the server: "What's the weather going to be like tomorrow?"

[0118] Output: The natural language processing engine returns the analysis result "You are looking for a weather forecast."

[0119] Step 4:

[0120] Obtaining information

[0121] Operation: Based on the analysis results, the server calls the information retrieval API and retrieves the specified information.

[0122] Input: Parsing result from the natural language processing engine: "I'm looking for a weather forecast."

[0123] Output: The information retrieval API returns the information "Tomorrow's weather will be sunny."

[0124] Step 5:

[0125] Provision of information

[0126] Behavior: The server returns the retrieved information to the user, using the notification feature of the messenger app to display the information to the user.

[0127] Input: Information from the information retrieval API: "Tomorrow's weather will be sunny."

[0128] Output: The server notifies the user that "tomorrow's weather will be sunny" through the messenger app and displays it on the user's screen.

[0129] In this way, a system can be created that analyzes the text, voice, and images sent by users through messenger apps and provides the necessary information.

[0130] (Application example 1)

[0131] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0132] Conventional messenger applications have the ability to provide information by analyzing text, voice, and images, but in autonomous vehicles, it is difficult to obtain information about the vehicle's status and surroundings in real time. In addition, there was a lack of functionality to provide information in response to user questions using voice or image recognition, which limited the user experience.

[0133] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means. In this invention, the server includes a means for providing a messenger app Internet search tool that can be used from text, voice, and image, a means for covering information on weather, disasters, news, EC, finance, etc., a means for enabling use from a smartphone messenger app, a means for acquiring the vehicle status and surrounding information in real time as an application installed in the autonomous driving vehicle, and a means for providing information in response to a user's question using voice recognition and image recognition. This makes it possible for a user to acquire necessary information in real time through text, voice, and image even in an autonomous driving vehicle.

[0134] "Text" is a means of expressing information using letters and symbols.

[0135] "Voice" is the means of recording and reproducing the human voice and other sounds as digital data.

[0136] An "image" is a means of recording and displaying visual information as digital data.

[0137] A "messenger app" is a software application that enables users to send and receive text, voice, and images.

[0138] An "Internet search tool" is software that searches for information on the Internet and provides it to users.

[0139] "Weather" is information that refers to the weather conditions in a particular area.

[0140] "Disaster" refers to information about damage caused by natural phenomena or man-made factors.

[0141] "News" is content intended to report current events or information.

[0142] "EC" is an abbreviation for electronic commerce, the act of buying and selling goods and services via the Internet.

[0143] "Finance" refers to information relating to the management, operation, and trading of funds.

[0144] A "smartphone" is a mobile phone that has Internet connectivity and a variety of applications.

[0145] An "autonomous vehicle" is a vehicle that drives autonomously using artificial intelligence and sensor technology.

[0146] "Real time" refers to a state in which processing and information provision are carried out immediately without delay.

[0147] "Speech recognition" is a technology that analyzes voice as digital data and converts it into text.

[0148] "Image recognition" is a technology that analyzes images and extracts specific information.

[0149] "User" refers to a person who uses a system or application.

[0150] A system for implementing this invention includes a means for providing a messenger app Internet search tool that can be used from text, voice, and images, a means for covering information on weather, disasters, news, e-commerce, finance, etc., a means for enabling use from a smartphone messenger app, a means for acquiring vehicle status and surrounding information in real time as an application to be installed in an autonomous vehicle, and a means for providing information in response to a user's questions using voice recognition and image recognition.

[0151] The server analyzes the text, voice, and images sent by the user and provides the necessary information. Specifically, when a user asks, "Where is the next gas station?", the server uses voice recognition technology to convert the voice into text, analyzes the text, and obtains the location information of the nearest gas station, which it then provides to the user.

[0152] To realize this system, the following hardware and software are used.

[0153] Hardware: Smartphones, microphones, cameras, self-driving vehicles

[0154] Software: Python, Requests library, SpeechRecognition library, Pillow library, Pytesseract library

[0155] The server uses the SpeechRecognition library for voice recognition and the Pytesseract library for image recognition, so that even if the user inputs an audio file or an image file, it can properly analyze it and provide information.

[0156] As a concrete example, consider the case where a user inputs a voice file saying "What's the weather tomorrow?" The server analyzes the voice file and converts it into text. It then uses WeatherAPI to get weather information for the specified location and provides it to the user.

[0157] Examples of prompt sentences to be input to a generative AI model include the following:

[0158] "The user has input an audio file 'weather_query.wav'. Write a program that parses this audio file and provides weather information."

[0159] In this way, users can obtain the information they need in real time through text, audio and images.

[0160] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0161] Step 1:

[0162] The user inputs text, an audio file, or an image file. The input data is sent to the server. Inputs include text messages, audio files (e.g., .wav format), and image files (e.g., .jpg format).

[0163] Step 2:

[0164] The server determines the format of the input data. Specifically, it analyzes the file extension and header information of the data to determine whether the input is text, audio, or image. The input is the data sent by the user, and the output is the data format information.

[0165] Step 3:

[0166] The server selects the appropriate analysis method depending on the format of the input data. If it is text, it proceeds with the analysis. If it is an audio file, it uses the SpeechRecognition library to convert the audio to text. If it is an image file, it uses the Pytesseract library to extract text from the image. The input is the format information and the original data, and the output is the analyzed text data.

[0167] Step 4:

[0168] The server performs natural language processing based on the analyzed text data to understand the user's question. Specifically, it tokenizes the text data and extracts keywords. The input is the analyzed text data, and the output is the extracted keywords.

[0169] Step 5:

[0170] The server retrieves the required information from the appropriate source based on the extracted keywords. For example, if the keyword "weather" is included, the Weather API is used to retrieve the weather information for the specified location. The input is the extracted keywords, and the output is the retrieved information.

[0171] Step 6:

[0172] The server formats the acquired information into a format for replying to the user. Specifically, it summarizes the information in text format and provides it to the user in an easy-to-understand format. The input is the acquired information, and the output is a formatted reply message.

[0173] Step 7:

[0174] The server sends the formatted reply message to the user, and the user receives the reply message through a messenger app. The input is the formatted reply message, and the output is the message sent to the user.

[0175] Through the above processing steps, the user can obtain the required information in real time through text, voice, and images.

[0176] Example 2

[0177] Next, a second embodiment of the second embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0178] Conventional information systems were often difficult to use for non-digital natives and the elderly. In addition, there were limited means of providing information by voice to users who used landlines, making it difficult for these users to easily obtain the latest information.

[0179] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for providing a messenger app Internet search tool that can be used from text, voice, and image, a means for covering information such as weather, disasters, news, electronic commerce, and finance, a means for enabling use from a messenger app of a mobile information terminal, a means for receiving voice data through a fixed telephone line, a means for analyzing the voice data and authenticating the user's voiceprint, a means for acquiring requested information when authentication is successful, a means for converting the acquired information into voice data, and a means for transmitting the voice data to the user through a fixed telephone line. This makes it possible for even users who are not digital natives or elderly people to easily acquire the latest information by voice using a fixed telephone line.

[0180] A "text, voice and image messaging app internet search tool" is an application that allows users to search for information and send and receive messages using text, voice and images.

[0181] "Information on weather, disasters, news, e-commerce, finance, etc." refers to a wide range of information that users need on a daily basis, such as meteorological information, information on natural disasters, the latest news, information on online shopping, and information on financial markets and banking.

[0182] A "mobile information terminal" is an electronic device, such as a smartphone or tablet, that is portable and can connect to the Internet.

[0183] A "fixed line" is a communication line for conducting voice communications over a traditional wired telephone network.

[0184] "Voice Data" means a digital representation of a User's voice.

[0185] An "AI voiceprint authentication system" is a system that uses artificial intelligence technology to identify individuals from voice data.

[0186] "Authentication" is the process by which a system verifies a user's identity and grants access.

[0187] A "data source" is an external source from which information is obtained.

[0188] "Speech synthesis software" refers to software for converting text data into voice data.

[0189] "Playing audio data" refers to the process of playing audio data as sound through an output device such as a speaker.

[0190] This invention provides an information provision system that can be used by non-digital native users and elderly people. In particular, we provide a system that allows users to request information by voice using a landline phone, authenticates the user using AI voiceprint authentication, and provides the requested information by voice.

[0191] Server Roles

[0192] The server receives voice data from the user via a landline. Specifically, when a user makes a voice request from a landline such as "What's in the news today?", the voice data is sent to the server. The server then sends the received voice data to an AI voiceprint authentication system, which analyzes the user's voiceprint. This authentication uses voice recognition software such as "Google (registered trademark) Cloud Speech-to-Text." If authentication is successful, the server accesses data sources such as "News APIs" and "RSS feeds" to obtain the information requested by the user. The acquired information is converted into voice data using voice synthesis software such as "Amazon Polly," and provided to the user via the landline.

[0193] Terminal Roles

[0194] The terminal functions as a landline phone and is the interface when the user requests information by voice. When the user requests "What's the news today?" by voice from the landline phone, the terminal transmits the voice data to the server. When the terminal receives a response from the server, it plays the voice data and provides the information to the user.

[0195] User Roles

[0196] A user requests information by voice using a landline phone. For example, when an elderly person requests "What's in the news today?" from a landline phone, the voice data is sent to the server. The server authenticates the user's voiceprint, retrieves the requested news information, and provides it by voice.

[0197] Examples

[0198] Example 1: Requesting News Information

[0199] The user makes a voice request from a landline saying, "What's the news today?" The server receives this voice data and authenticates the user using an AI voiceprint authentication system. If authentication is successful, the server retrieves the latest news information from the news API, converts it into voice data, and provides it to the user.

[0200] Example 2: Requesting weather information

[0201] The user makes a voice request from a landline saying, "What's the weather today?" The server receives this voice data and authenticates the user using an AI voiceprint authentication system. If authentication is successful, the server retrieves the latest weather information from the weather API, converts it into voice data, and provides it to the user.

[0202] Examples of prompt statements

[0203] "Please explain the process for a system that authenticates the voiceprint of a user requesting information via voice over a landline phone and then provides the requested information via voice."

[0204] By inputting this prompt sentence into a generative AI model, it is possible to generate a sentence that explains the system's processing in natural language.

[0205] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0206] Step 1:

[0207] The server receives voice data from the user via a fixed telephone line.

[0208] Input: Voice data from users calling from landlines

[0209] Output: Audio data stored on the server

[0210] Specific operation: When a user voice requests "What's the news today?" from a landline phone, the voice data is sent to the server.

[0211] Step 2:

[0212] The server sends the received voice data to the AI ​​voiceprint authentication system and analyzes the user's voiceprint.

[0213] Input: Received audio data

[0214] Output: Analyzed voiceprint data

[0215] Specific operations: Voice data is converted into text using voice recognition software such as "Google Cloud Speech-to-Text," and voiceprint authentication is then performed based on the text data.

[0216] Step 3:

[0217] The server authenticates the user based on the analyzed voiceprint data.

[0218] Input: Analyzed voiceprint data

[0219] Output: Authentication result (success or failure)

[0220] Specific operation: The voiceprint data is compared with pre-registered voiceprint data, and if there is a match, the authentication is successful.

[0221] Step 4:

[0222] If authentication is successful, the server contacts the appropriate data source to obtain the information requested by the user.

[0223] Input: Authentication result (success) and user request

[0224] Output: Retrieved information data

[0225] Specific operation: Obtain the latest news information from "News API", "RSS feed", etc.

[0226] Step 5:

[0227] The server converts the acquired information into audio data.

[0228] Input: Acquired information data

[0229] Output: Audio data

[0230] Specific operation: Converts text data into speech data using speech synthesis software such as "Amazon Polly".

[0231] Step 6:

[0232] The server transmits the generated voice data to the user via a fixed telephone line.

[0233] Input: Generated audio data

[0234] Output: Voice data sent to the user's landline phone

[0235] What it does: It converts voice data into a format suitable for landline telephone lines and sends it to the user's landline.

[0236] Step 7:

[0237] The terminal receives the audio data transmitted from the server and plays it back.

[0238] Input: Audio data sent from the server

[0239] Output: Played audio information

[0240] Specific operation: Play audio data through the landline phone speaker to provide information to the user.

[0241] (Application example 2)

[0242] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0243] Modern digital services are designed primarily for young people and digital natives, and are often difficult to use for the elderly and users who are not familiar with digital technology. Food delivery services also require the use of smartphones and the Internet, which makes them difficult for the elderly to use. Furthermore, although the number of services using voice recognition technology is increasing, the accuracy and security of user authentication may not be sufficient.

[0244] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0245] In this invention, the server includes a means for providing a messenger app internet search tool that can be used from text, voice, and image, a means for covering information on weather, disasters, news, EC, finance, etc., a means for enabling use from a smartphone messenger app, and a means for accepting voice orders from a landline in a food delivery service, a means for authenticating a user by AI voiceprint authentication and providing recommended menu items taking into consideration past order history and preferences, and a means for confirming the order details by voice and finally confirming the order. This enables even elderly people and users who are not familiar with digital to use food delivery services safely and easily using a landline.

[0246] A "text, voice and image messaging app internet search tool" is an application that allows users to search for information and send and receive messages using text, voice and images.

[0247] "Means of covering information on weather, disasters, news, e-commerce, finance, etc." refers to a function that comprehensively provides various types of information, such as weather forecasts, disaster information, the latest news, e-commerce information, and financial information.

[0248] "Means enabling use from a messenger app on a smartphone" refers to a function that enables users to use the service through a messenger app running on their smartphone.

[0249] "A means for accepting voice orders from a landline telephone in a food delivery service" refers to a function that enables a user to place a food delivery order by voice using a landline telephone.

[0250] "A means of authenticating users through AI voiceprint recognition and providing recommended menu items taking into consideration their past order history and preferences" refers to a function that identifies users through voiceprint recognition technology using artificial intelligence, and presents recommended menu items based on the user's past order history and preferences.

[0251] "A means for confirming order details by voice and finally confirming the order" is a function that supports the process in which a user confirms the order details by voice and finally confirms the order.

[0252] A system for carrying out the present invention has the following configuration: The server includes means for providing a messenger app internet search tool that can be used from text, voice, and image, means for covering information on weather, disasters, news, EC, finance, etc., means for enabling use from a smartphone messenger app, means for accepting voice orders from a landline phone in a food delivery service, means for authenticating a user by AI voiceprint authentication and providing recommended menu items taking into consideration past order history and preferences, and means for confirming the order details by voice and finally confirming the order.

[0253] Hardware and Software Configuration

[0254] The server uses a voice recognition library (e.g., speech_recognition), a voice synthesis library (e.g., pyttsx3), an AI voiceprint authentication module (e.g., voice_authentication), and a food delivery service API (e.g., food_delivery_service). The user uses a landline phone to place an order by voice. The server captures the voice using a microphone and outputs the voice using a speaker.

[0255] Data processing and calculation

[0256] The server processes the data in the following steps:

[0257] 1. When a user requests information by voice from a landline phone, the server picks up the voice through a microphone.

[0258] 2. Use a speech recognition library to convert the captured audio into text.

[0259] 3. Use the AI ​​voiceprint recognition module to authenticate the user's voiceprint.

[0260] 4. If authentication is successful, use the food delivery service API to get menu recommendations that take into account past ordering history and user preferences.

[0261] 5. The recommended menu is converted into voice using a speech synthesis library and provided to the user through the speaker.

[0262] 6. The user confirms the order details by voice and finally confirms the order.

[0263] Examples

[0264] For example, when a user makes a voice request from a landline saying, "What's the recommendation today?", the server captures the voice and converts it into text using a voice recognition library. Next, the AI ​​voiceprint authentication module authenticates the user and obtains menu recommendations using the food delivery service API. The obtained menu is converted into speech using a speech synthesis library and provided to the user through the speaker. When the user confirms by voice, "I'd like to order a sushi set," the server confirms the order.

[0265] Examples of prompt statements

[0266] User: Hello, Food Delivery Assistant, what's your name?

[0267] System: This is Tanaka.

[0268] User: Hello Tanaka-san. Please place your order.

[0269] System: What's your recommendation today?

[0270] User: Today's recommendation is the sushi set. Would you like to order the sushi set?

[0271] System: Yes.

[0272] User: Your order has been received, thank you.

[0273] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0274] Step 1:

[0275] A user requests information by voice from a landline phone. The server captures the user's voice through a microphone. The input is the user's voice and the output is voice data.

[0276] Step 2:

[0277] The server converts the acquired voice data into text using a voice recognition library (e.g., speech_recognition). The input is voice data, and the output is text data.

[0278] Step 3:

[0279] The server uses an AI voiceprint authentication module (e.g. voice_authentication) to authenticate the user's voiceprint from the text data. The input is the text data, and the output is the authentication result. If the authentication is successful, proceed to the next step.

[0280] Step 4:

[0281] The server uses a food delivery service API (e.g., food_delivery_service) to obtain recommended menu items that take into account the user's past order history and preferences. The input is the user authentication result and past order history data, and the output is the recommended menu items.

[0282] Step 5:

[0283] The server converts the text data of the recommended menu into speech using a speech synthesis library (e.g., pyttsx3). The input is the text data of the recommended menu, and the output is speech data.

[0284] Step 6:

[0285] The server provides the user with a recommended menu by voice through a speaker. The input is voice data, and the output is voice output to the user.

[0286] Step 7:

[0287] The user confirms the order details by voice on the landline phone. The server again picks up the user's voice through the microphone. The input is the user's voice, and the output is voice data.

[0288] Step 8:

[0289] The server converts the acquired voice data into text using a voice recognition library. The input is voice data and the output is text data.

[0290] Step 9:

[0291] The server analyzes the text data and confirms the order details. The input is the text data, and the output is the confirmation result of the order details.

[0292] Step 10:

[0293] The server confirms the order details and confirms the order using the food delivery service API. The input is the confirmation result of the order details, and the output is a notification that the order has been confirmed.

[0294] Step 11:

[0295] The server converts the order confirmation notification into speech using a speech synthesis library. The input is the order confirmation notification, and the output is speech data.

[0296] Step 12:

[0297] The server notifies the user of the order confirmation by voice through the speaker. The input is voice data, and the output is voice output to the user.

[0298] Furthermore, an emotion engine that estimates the emotion of the user may be combined. That is, the identification processing unit 290 may estimate the emotion of the user using the emotion identification model 59, and perform identification processing using the emotion of the user.

[0299] "Example 1"

[0300] In one embodiment of the present invention, an emotion engine that recognizes the user's emotions is incorporated. This emotion engine analyzes the user's emotions from text, voice, and images sent by the user through the messenger app. For example, if a user sends the text "Today is the worst," the emotion engine recognizes from this text that the user is feeling negative emotions. Also, if a user sends an image of a smiling face, the emotion engine recognizes from this image that the user is feeling positive emotions. In this way, the emotion engine can accurately grasp the user's emotions and provide information and services according to the emotions.

[0301] "Example 2"

[0302] In another embodiment of the present invention, an emotion engine is incorporated as a means of supporting users who are not digital natives. This emotion engine analyzes emotions from the user's voice. For example, if an elderly person makes a voice request from a landline, "What's the news today?", the emotion engine analyzes the user's emotions from the tone and strength of the voice. In this way, the emotion engine can accurately grasp the user's emotions and provide information and services accordingly.

[0303] "Example 5"

[0304] In a further embodiment of the present invention, an emotion engine is incorporated as a means that supports AI voiceprint authentication and can be used by elderly people even on landlines. This emotion engine analyzes emotions from the user's voice. For example, if an elderly person makes a voice request on a landline such as "What's the weather today?", the emotion engine analyzes the user's emotions from the tone and strength of the voice. In this way, the emotion engine can accurately grasp the user's emotions and provide information and services accordingly.

[0305] The process flow of each embodiment will be described below.

[0306] "Example 1"

[0307] Step 1: User sends a text, voice or image via a messenger app.

[0308] Step 2: The emotion engine analyzes the user's emotions from the submitted text, voice, and images.

[0309] Step 3: The system provides information and services based on the emotions analyzed by the emotion engine.

[0310] "Example 2"

[0311] Step 1: A non-digital native user makes a request via voice.

[0312] Step 2: The emotion engine analyzes the emotion from the user's voice.

[0313] Step 3: The system provides information and services based on the emotions analyzed by the emotion engine.

[0314] "Example 5"

[0315] Step 1: The senior makes a voice request from a landline phone.

[0316] Step 2: The emotion engine analyzes the emotion from the user's voice.

[0317] Step 3: The system provides information and services based on the emotions analyzed by the emotion engine.

[0318] Example 1

[0319] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal".

[0320] Conventional messenger apps lack the ability to analyze text, voice, and image messages sent by users and provide the necessary information. They also lack the ability to recognize users' emotions and respond appropriately, limiting the user experience. Furthermore, these systems are difficult to use for non-digital native users and the elderly.

[0321] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0322] In this invention, the server includes a means for providing a messenger app Internet search tool that can be used from text, voice, and image, a means for covering information on weather, disasters, news, electronic commerce, finance, etc., a means for enabling use from a messenger app on a mobile information terminal, a means for analyzing a user's text, voice, and image, a means for acquiring necessary information from the Internet based on the analysis result, a means including an emotion engine for recognizing the user's emotion, and a means for providing appropriate information to the user based on the analysis result and the acquired information. This makes it possible to not only analyze data sent by the user and provide necessary information, but also recognize the user's emotion and take appropriate action.

[0323] "Text" refers to textual information that a user sends through a messenger app.

[0324] "Audio" refers to sound information that a user sends through a messenger app.

[0325] An "image" is visual information that a user sends through a messenger app.

[0326] A "messenger app internet search tool" is a tool that analyzes data sent through a messenger app and retrieves and provides necessary information from the internet.

[0327] "Weather" is information about the weather.

[0328] "Disasters" refers to information about natural and man-made disasters.

[0329] "News" is reporting on current events or information.

[0330] "Electronic commerce" means commercial transactions conducted over the Internet.

[0331] "Finance" refers to information regarding the operation and management of funds.

[0332] A "portable information terminal" is a portable information processing device such as a smartphone or tablet.

[0333] "Analysis" is the process of analyzing the transmitted data to make sense of it.

[0334] "Obtaining from the Internet" refers to searching for and obtaining the necessary information via the Internet.

[0335] An "emotion engine" is software for analyzing emotions from data sent by users.

[0336] "Providing appropriate information" refers to responding with information that is useful to the user based on the analysis results and the information obtained.

[0337] This invention is a system that analyzes text, voice, and images sent by users through messenger apps and provides necessary information. It also incorporates an emotion engine that recognizes the user's emotions.

[0338] Hardware and Software Used

[0339] Hardware: Mobile information terminals (smartphones, tablets), servers

[0340] Software: Messenger app, Google Cloud Natural Language API, Amazon Rekognition, OpenWeatherMap API, IBM Watson(R) Tone Analyzer

[0341] Data processing and calculation

[0342] 1. User Input

[0343] Users use messenger apps on their mobile devices to send text, voice, and images.

[0344] Example: A user enters the text "What's the weather like tomorrow?" and presses the submit button.

[0345] 2. Data transfer

[0346] The terminal transfers the data sent by the user to a server via the Internet.

[0347] What happens: The messenger app sends data to the server as an HTTP request.

[0348] 3. Data Analysis

[0349] The server analyzes the received text, audio, and images.

[0350] The analysis uses Google Cloud Natural Language API to analyze text and Amazon Rekognition to analyze images.

[0351] Example: Parsing the text "What's the weather like tomorrow?" to recognize that it is a request for weather forecast information.

[0352] 4. Acquisition of information

[0353] The server retrieves the necessary information from the Internet based on the analysis results.

[0354] Specific software: Obtains weather forecast information using the OpenWeatherMap API.

[0355] Example: In response to the request "What's the weather going to be like tomorrow?", weather forecast information is retrieved from the OpenWeatherMap API.

[0356] 5. Emotion Recognition

[0357] The server uses an emotion engine to analyze emotions from the user's text, voice, and images.

[0358] Specific software: Analyzing the sentiment of text using IBM Watson Tone Analyzer.

[0359] Example: If a user sends the text "Today is the worst", the sentiment engine will recognize the negative sentiment.

[0360] 6. Provision of information

[0361] The server provides the user with appropriate information based on the analysis results and the acquired information.

[0362] Specific behavior: The server generates the text "Tomorrow's weather will be sunny" and returns it to the user.

[0363] Example: If the user asks "What's the weather going to be like tomorrow?", the server responds with "The weather will be sunny tomorrow."

[0364] Examples of prompt statements

[0365] "Please explain the system that analyzes the text, voice, and images sent by the user through the messenger app and provides the necessary information. Please also specify the names of the specific software and APIs."

[0366] In this way, the present invention is a system that not only analyzes data sent by a user and provides necessary information, but also recognizes the user's emotions and takes appropriate action.

[0367] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0368] Step 1:

[0369] Users send text, voice, and images through messenger apps

[0370] Input: A user inputs text, voice, or images into a messenger app.

[0371] Specific behavior: The user enters the text "What's the weather like tomorrow?" and presses the send button.

[0372] Output: The messenger app prepares to send the input data.

[0373] Step 2:

[0374] The device forwards the transmitted data to the server.

[0375] Input: The data sent when the user presses the submit button.

[0376] What happens: The messenger app sends text data to the server as an HTTP request.

[0377] Output: The server receives the data.

[0378] Step 3:

[0379] The server analyzes the data it receives

[0380] Input: Text, audio, and image data sent from the device.

[0381] What happens next: The server uses the Google Cloud Natural Language API to parse the text and Amazon Rekognition to analyze the images.

[0382] Output: The analysis results include the user's request and the image content.

[0383] Example: Parsing the text "What's the weather like tomorrow?" to recognize that it is a request for weather forecast information.

[0384] Step 4:

[0385] The server retrieves the necessary information from the Internet.

[0386] Input: The user's request obtained as a result of analysis.

[0387] Specific behavior: The server retrieves weather forecast information using the OpenWeatherMap API.

[0388] Output: The desired information (e.g. weather forecast information) is obtained.

[0389] Example: In response to the request "What's the weather going to be like tomorrow?", weather forecast information is retrieved from the OpenWeatherMap API.

[0390] Step 5:

[0391] The server uses an emotion engine to analyze the user's emotions.

[0392] Input: Text, audio, and image data submitted by the user.

[0393] What happens: The server uses IBM Watson Tone Analyzer to analyze the sentiment of the text.

[0394] Output: The user's emotional state (e.g. positive, negative).

[0395] Example: If a user sends the text "Today is the worst", the sentiment engine will recognize the negative sentiment.

[0396] Step 6:

[0397] The server provides the user with appropriate information based on the analysis results and the acquired information.

[0398] Input: Analysis results and retrieved information.

[0399] Specific behavior: The server generates the text "Tomorrow's weather will be sunny" and returns it to the user.

[0400] Output: The user is provided with the appropriate information.

[0401] Example: If the user asks "What's the weather going to be like tomorrow?", the server responds with "The weather will be sunny tomorrow."

[0402] (Application example 1)

[0403] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0404] Conventional messenger apps only had limited functionality for analyzing the content of text, voice, and images sent by users and providing appropriate information. In addition, they lacked functionality for analyzing user emotions and taking appropriate action in emergencies, making them difficult to use in situations that required the provision of security-related information or emergency responses. Furthermore, their response to users who are not digital natives and the elderly was insufficient.

[0405] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0406] In this invention, the server includes a means for providing a messenger app Internet search tool that can be used from text, voice, and image, a means for covering information on weather, disasters, news, EC, finance, etc., a means for enabling use from a smartphone messenger app, a means for analyzing user emotions and taking appropriate action in an emergency, and a means for providing information on security. This makes it possible to analyze the text, voice, and image sent by the user and provide appropriate information, and in particular to provide information on security and respond quickly to emergencies. In addition, support for users who are not digital natives and the elderly is also strengthened.

[0407] A "messenger app internet search tool that can be used with text, voice, and images" is a messenger app that has the function of analyzing the text, voice, and images sent by the user and searching for and providing information on the internet.

[0408] "Means of covering information on weather, disasters, news, e-commerce, finance, etc." refers to the function of comprehensively providing various types of information, such as weather forecasts, disaster information, the latest news, information on e-commerce, and financial information.

[0409] "Means enabling access from a smartphone's messenger app" refers to a function that enables users to access various information through a messenger app running on their smartphone.

[0410] "A means of analyzing user emotions and taking appropriate action in an emergency" refers to a function that analyzes emotions from the text, voice, and images sent by the user, and takes appropriate action based on those emotions in the event of an emergency.

[0411] "Means of providing security-related information" refers to the function of analyzing information sent by users and providing appropriate security-related information and countermeasures.

[0412] "Measures to accommodate non-digital native users" refers to functions and interfaces that make it easy to use even for users who are not familiar with digital technology.

[0413] "A means that supports AI voiceprint authentication and allows elderly people to use the system via landline telephone" refers to a function that uses voiceprint authentication technology using artificial intelligence to enable elderly people to use the system via landline telephone.

[0414] The system for implementing this invention has the following configuration: First, a user sends text, voice, and images through a messenger app on their smartphone. These data are sent to a server, which then analyzes each data.

[0415] The server uses natural language processing libraries (e.g. TextBlob), speech recognition libraries (e.g. Google Speech-to-Text API), and image analysis libraries (e.g. PIL) for text, voice, and image analysis to extract the necessary information from the data submitted by the user and generate an appropriate response.

[0416] In addition, the server uses an emotion analysis engine to analyze the user's emotions. This emotion analysis engine determines the user's emotions from text, voice, and images, and takes appropriate action in the event of an emergency. For example, if the user sends a voice message saying "help," the server will send a notification to the emergency contact.

[0417] In order to provide information on security, the server analyzes the information sent by the user and provides appropriate security information and countermeasures. For example, if a user sends an image of a suspicious person, the server analyzes the image and notifies the police.

[0418] The system also takes into consideration users who are not digital natives and the elderly. The server uses AI voiceprint authentication technology to allow elderly people to use the system through a landline, making it easy for users who are not familiar with digital technology to use the system.

[0419] For example, if a user sends a text saying "I forgot my house keys," the server will analyze the text, search for nearby locksmiths and provide their contact information. If a user sends a text saying "I'm scared," the emotion analysis engine will recognize the negative emotion and take emergency action.

[0420] Examples of prompts to input to a generative AI model include:

[0421] If a user sends a text saying "I forgot my house keys", the sentiment engine should parse the user sentiment from this text and generate a response to search for a nearby locksmith.

[0422] In this way, a system can be realized that analyzes the text, voice, and images sent by users and provides appropriate information and takes emergency measures.

[0423] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0424] Step 1:

[0425] A user sends text, voice, and images through a messenger app on a smartphone. The input is the text, voice, and image data sent by the user, and the output is the transmission of these data to a server. A specific operation is when a user enters the text "I forgot my house key" and presses the send button.

[0426] Step 2:

[0427] The server analyzes the text, voice, and image data it receives. The input is the data sent by the user, and the output is the analysis result. Specifically, the server analyzes the text using a natural language processing library (e.g. TextBlob), converts the voice to text using a speech recognition library (e.g. Google Speech-to-Text API), and analyzes the image using an image analysis library (e.g. PIL).

[0428] Step 3:

[0429] The server analyzes the user's emotions using the emotion analysis engine. The input is the text, voice, and image data analyzed in step 2, and the output is the user's emotional state. In concrete terms, the server uses the emotion analysis engine to determine the user's emotions from the text and voice, and recognizes neutral emotions from the text, for example, "I forgot my house keys."

[0430] Step 4:

[0431] The server provides appropriate information based on the analysis results. The input is the analysis results of steps 2 and 3, and the output is the information provided to the user. In concrete terms, the server searches for a nearby locksmith for the text "I forgot my house keys" and provides the user with their contact information.

[0432] Step 5:

[0433] The server provides security information. The input is the data sent by the user and the analysis results, and the output is security information and countermeasures. In concrete terms, when a user sends an image of a suspicious person, the server analyzes the image and notifies the police.

[0434] Step 6:

[0435] The server takes appropriate action in case of emergency. The input is the result of the emotion analysis engine, and the output is the execution of emergency response. Specifically, when the user voices "help," the server sends a notification to the emergency contact.

[0436] Step 7:

[0437] The server supports users who are not digital natives and the elderly. The input is the user's usage status and voice data, and the output is a simple operation interface and voice authentication results. In concrete terms, the server uses AI voiceprint authentication technology to enable the elderly to use the system through a landline phone.

[0438] In this way, a system is realized that analyzes the text, voice, and images sent by the user and provides appropriate information and responds to emergencies.

[0439] Example 2

[0440] Next, a second embodiment of the second embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0441] Modern information provision systems are often difficult to use for non-digital native users and the elderly. In addition, there is a lack of means to provide appropriate information through voice recognition and emotion analysis to users who use landlines. For this reason, there is a demand for information provision systems that improve user convenience and are easy to use, especially for non-digital native users and the elderly.

[0442] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0443] In this invention, the server includes a means for providing a messenger app internet search tool that can be used from text, voice, and image, a means for covering information such as weather, disasters, news, e-commerce, and finance, a means for enabling use from a messenger app of a communication terminal, a means for receiving voice data through a landline and authenticating a user using an AI voiceprint authentication system, a means for analyzing the contents of a user's request and acquiring necessary information if authentication is successful, a means for converting the acquired information into voice data using a voice synthesis system and providing it to the user through a landline, and a means for analyzing emotions from the user's voice and providing information and services accordingly. This allows even users who are not digital natives or the elderly to easily acquire information through a landline and receive appropriate services.

[0444] A "text, voice and image messaging app internet search tool" is an application that allows users to search for information and send and receive messages using text, voice and images.

[0445] "Information on weather, disasters, news, e-commerce, finance, etc." refers to meteorological information, information on natural disasters, the latest news, information on online shopping, and information on financial markets and transactions.

[0446] A "communication terminal" is a device for communicating via the Internet or telephone lines, including smartphones and landline phones.

[0447] A "fixed line" is a wired communications line for carrying analog or digital telephone communications.

[0448] "Voice Data" means data that has been converted into digital form from a User's voice.

[0449] The "AI voiceprint authentication system" is an artificial intelligence system that analyzes the user's voice and authenticates individuals based on their voiceprints.

[0450] "User authentication" is the process by which a system verifies a user's identity and grants access rights.

[0451] "Request analysis" is the process of understanding the user's intent and the information they are looking for from the voice or text they provide.

[0452] "Obtaining information" means collecting the necessary data or information from external databases or APIs based on user requests.

[0453] A "voice synthesis system" is a technology that converts text data into voice data.

[0454] "Emotion analysis" is the process of determining the user's emotional state from their voice and taking appropriate action based on that information.

[0455] "Provision of information and services" refers to providing users with necessary information or specific services.

[0456] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0457] This invention provides an information provision system that can be used by non-digital native users and elderly people. Specifically, this system uses AI voiceprint authentication and a voice synthesis system to provide information to users who request information by voice using a landline phone.

[0458] Server Roles

[0459] The server receives voice requests from users through landline telephone lines. The received voice data is converted into digital format using a voice modem. The server then sends this voice data to an AI voiceprint authentication system to authenticate the user. This AI voiceprint authentication system uses common voice recognition software (e.g., voice recognition API). If authentication is successful, the server analyzes the content of the user's request and obtains the required information. For example, to obtain news information, the server uses a news API to obtain the latest news data. The server then converts the obtained information into voice data using a voice synthesis system (e.g., voice synthesis API) and provides it to the user through the landline telephone line.

[0460] Terminal Roles

[0461] The terminal receives the user's voice request via a landline telephone line. The received voice data is sent to a voice recognition module in the terminal and converted into text data. This voice recognition module uses general voice recognition software (e.g., voice recognition API). The converted text data is sent to a server, where the user is authenticated by an AI voiceprint authentication system. If authentication is successful, the terminal converts the information obtained from the server into voice data using a voice synthesis system (e.g., voice synthesis API) and provides it to the user via the landline telephone line.

[0462] User Roles

[0463] A user requests information by voice using a landline. For example, the user may make a request such as, "What's in the news today?" This voice request is sent to the server via the landline. The server authenticates the user using an AI voiceprint authentication system, and if authentication is successful, retrieves the information requested by the user. The retrieved information is converted into voice data using a voice synthesis system and provided to the user via the landline. The user can obtain the required information by listening to the provided voice information.

[0464] Examples

[0465] Example 1: Requesting news information

[0466] The user makes a voice request from a landline saying, "What's in the news today?" The server receives this voice data and authenticates the user using an AI voiceprint authentication system (e.g., voice recognition API). If authentication is successful, the server uses a news API to obtain the latest news data. The obtained news data is converted into voice data using a voice synthesis system (e.g., voice synthesis API) and provided to the user via the landline.

[0467] Example 2: Requesting weather information

[0468] A user makes a voice request from a landline saying, "What's the weather today?" The server receives this voice data and authenticates the user using an AI voiceprint authentication system (e.g., voice recognition API). If authentication is successful, the server uses a weather API to obtain the latest weather data. The obtained weather data is converted into voice data using a voice synthesis system (e.g., voice synthesis API) and provided to the user via the landline.

[0469] Examples of prompt statements

[0470] "Please explain in detail from the perspectives of the server, the terminal, and the user the process of a system that uses AI voiceprint authentication and a voice synthesis system to provide information to users who request information by voice from a landline phone. Please also add specific actions for each processing step."

[0471] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0472] Step 1:

[0473] The server receives voice requests from users over landline telephone lines.

[0474] Input: Voice data from a landline

[0475] Specific operation: The server uses a voice modem to convert analog voice signals into digital voice data.

[0476] Output: Digital audio data

[0477] Step 2:

[0478] The server sends the received voice data to the AI ​​voiceprint authentication system to authenticate the user.

[0479] Input: Digital audio data

[0480] Specific operation: The server calls the voice recognition API and converts the voice data into text data, which is then sent to the AI ​​voiceprint authentication system.

[0481] Output: Authentication result

[0482] Step 3:

[0483] The server receives the authentication results from the AI ​​voiceprint authentication system and authenticates the user.

[0484] Input: Authentication result

[0485] Specific operation: The server analyzes the authentication results returned by the AI ​​voiceprint authentication system and determines whether the authentication was successful.

[0486] Output: Authentication success or failure flag.

[0487] Step 4:

[0488] If the authentication is successful, the server analyzes the user's request and obtains the necessary information.

[0489] Input: Authentication success flag, voice data

[0490] Specific operation: The server uses a speech recognition API to convert the voice data into text data and analyzes the user's request. Based on the analysis results, the server retrieves the required information from external databases such as news APIs and weather APIs.

[0491] Output: Acquired information data

[0492] Step 5:

[0493] The server converts the acquired information into voice data using a voice synthesis system.

[0494] Input: Acquired information data

[0495] Specific operation: The server uses a speech synthesis API to convert the text data into speech data.

[0496] Output: Audio data

[0497] Step 6:

[0498] The server provides the generated voice data to the user via a fixed telephone line.

[0499] Input: Audio data

[0500] What it does: The server uses a voice modem to convert the digital voice data into an analog voice signal and transmit it to the user over a landline.

[0501] Output: Audio information provided to the user

[0502] Step 7:

[0503] A user requests information by voice using a landline telephone.

[0504] Input: User's voice request

[0505] Specific actions: A user picks up a landline phone receiver and speaks a request such as "What's in the news today?"

[0506] Output: Voice data sent over a landline

[0507] Step 8:

[0508] The user obtains the required information by listening to the provided audio information.

[0509] Input: Voice information provided via a landline telephone line

[0510] Specific operation: The user listens to news and weather information provided through a landline telephone receiver and obtains the required information.

[0511] Output: Information the user gets

[0512] (Application example 2)

[0513] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0514] When non-digital native users and the elderly use landlines to obtain information, there are problems such as complex operations and difficulty in authentication. In addition, there is an issue that the user experience is not improved because appropriate information is not provided according to the user's emotions.

[0515] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0516] In this invention, the server includes a means for providing a messenger app internet search tool that can be used from text, voice, and image, a means for covering information on weather, disasters, news, EC, finance, etc., a means for enabling use from a messenger app on a smartphone, a means for requesting information by voice from a landline, a means for performing AI voiceprint authentication, a means including an emotion engine that analyzes emotions from the user's voice, and a means for providing information according to emotions. This allows even users who are not digital natives or the elderly to easily obtain information using a landline and provide appropriate information according to emotions.

[0517] A "text, voice and image enabled messenger app internet search tool" is a tool that allows users to search for information using text, voice and images and is accessible through a messenger app.

[0518] "Means that cover a wide range of information, including weather, disasters, news, e-commerce, and finance" refers to means that comprehensively provide a wide range of information, such as weather forecasts, disaster information, the latest news, e-commerce information, and financial information.

[0519] "Means enabling use from a messenger app on a smartphone" refers to means that enable information to be obtained or services to be used via a messenger app installed on a smartphone.

[0520] A "means for requesting information by voice from a landline telephone" is a means by which a user can request information by voice using a landline telephone.

[0521] "Means for performing AI voiceprint authentication" refers to a means for using artificial intelligence to authenticate a user's voiceprint and confirm the user's identity.

[0522] "Means including an emotion engine that analyzes emotions from a user's voice" refers to means including an emotion engine that can analyze user's voice data and read emotions from the voice.

[0523] A "means for providing information according to emotions" is a means for providing appropriate information according to the user's emotional state.

[0524] A system for carrying out this invention has the following configuration: The server includes a means for providing a messenger app internet search tool that can be used from text, voice, and image, a means for covering information on weather, disasters, news, EC, finance, etc., a means for enabling use from a messenger app on a smartphone, a means for requesting information by voice from a landline phone, a means for performing AI voiceprint authentication, a means including an emotion engine that analyzes emotions from the user's voice, and a means for providing information according to emotions.

[0525] Hardware and Software Used

[0526] Hardware: Smartphones, landlines

[0527] Software: Python, speech_recognition library, pyttsx3 library, custom emotion_recognition module, voice_authentication module

[0528] Data processing and calculation

[0529] The server first receives voice input from a landline phone. This voice data is converted to text using the speech_recognition library. Then, it authenticates the user's voiceprint from the voice data using the voice_authentication module. If authentication is successful, it analyzes the user's emotions from the voice data using the emotion_recognition module.

[0530] Based on the analyzed emotions, the server selects appropriate information and provides it by voice using the pyttsx3 library. This series of processes enables even non-digital native users and elderly people to easily obtain information using a landline phone and receive appropriate information according to their emotions.

[0531] Examples

[0532] For example, if a user makes a voice request from a landline saying, "What is the security situation at home?", the server will perform voice authentication and emotion analysis and respond by voice with, "The current security situation is normal." In this way, the user can obtain information with peace of mind.

[0533] Examples of prompt statements

[0534] When a user makes a voice request from a landline phone saying, "What is the security status at home?", the system should perform voice recognition and emotion analysis and provide emotion-based security information via voice.

[0535] This invention makes it possible for even non-digital native users and elderly people to easily obtain information using a landline phone and to be provided with appropriate information according to their emotions.

[0536] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0537] Step 1:

[0538] A user requests information by voice from a landline phone. The user uses a landline phone to make a request by voice, for example, "What is the security situation at my home?" This voice data is input into the system.

[0539] Step 2:

[0540] The server receives the voice data and converts the voice to text using the speech_recognition library. The input is voice data and the output is text data. This conversion converts the voice data into analyzable text data.

[0541] Step 3:

[0542] The server uses the voice_authentication module to authenticate the user's voiceprint from the voice data corresponding to the text data. The input is the voice data, and the output is the authentication result. This authentication confirms the user's identity.

[0543] Step 4:

[0544] If the authentication is successful, the server uses the emotion_recognition module to analyze the user's emotions from the voice data. The input is the voice data, and the output is the emotion data. This analysis allows the user's emotional state to be understood.

[0545] Step 5:

[0546] The server selects appropriate information based on the analyzed emotion data. For example, if the user expresses anxiety, the server selects reassuring information such as "The current security situation is normal." The input is emotion data, and the output is the selected information.

[0547] Step 6:

[0548] The server converts the selected information into audio using the pyttsx3 library and provides it to the user through a landline phone. The input is the selected information and the output is audio data. This conversion allows the user to receive the information by audio.

[0549] Step 7:

[0550] The user receives the voice information provided via a landline phone and obtains the information he or she needs, allowing the user to obtain information with peace of mind.

[0551] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0552] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0553] Another example of generative AI is Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) are mentioned.

[0554] In the above embodiment, an example was given in which the specific process was performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0555] [Second embodiment]

[0556] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0557] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0558] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0559] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0560] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[0561] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[0562] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[0563] Fig. 4 shows an example of main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0564] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0565] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0566] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0567] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0568] "Example 1"

[0569] The present invention provides a messenger app internet search tool that can use text, voice, and images. Specifically, when a user sends text, voice, or image through a messenger app, the system analyzes it and provides the required information. For example, when a user sends a text such as "What's the weather like tomorrow?", the system retrieves weather forecast information and provides it to the user.

[0570] "Example 2"

[0571] Furthermore, the present invention supports AI voiceprint authentication to enable use by non-digital native users and elderly people. Specifically, when a user requests information by voice from a landline phone, the system performs AI voiceprint authentication to authenticate the user. If authentication is successful, the information requested by the user is provided by voice. For example, when an elderly person requests by voice from a landline phone, "What's in the news today?", the system obtains the news information and provides it by voice.

[0572] The process flow of each embodiment will be described below.

[0573] "Example 1"

[0574] Step 1: User sends a text, voice or image via a messenger app.

[0575] Step 2: The system analyzes the transmitted text, audio and images.

[0576] Step 3: Based on the analysis result, the system obtains the required information. For example, if the user sends a text "What's the weather like tomorrow?", the system will obtain weather forecast information.

[0577] Step 4: Provide the retrieved information to the user.

[0578] "Example 2"

[0579] Step 1: User requests information by voice over a landline phone.

[0580] Step 2: The system analyzes the voice and performs AI voiceprint authentication.

[0581] Step 3: If the authentication is successful, the system retrieves the information requested by the user. For example, if an elderly person requests "What's the news today?" by voice, the system retrieves the news information.

[0582] Step 4: Provide the retrieved information to the user via voice.

[0583] Example 1

[0584] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0585] Conventional messenger apps are limited to sending text, voice, and images, making it difficult to provide users with the information they need quickly and accurately. In addition, these apps are difficult to use for non-digital native users and elderly people. Furthermore, voice recognition and information provision functions are lacking for elderly people who use landlines. To solve these issues, a system that analyzes data sent by users and provides the necessary information is required.

[0586] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0587] In this invention, the server includes a means for receiving data sent by a user through a messenger app, a means for using a natural language processing engine to analyze the received data, and a means for acquiring necessary information based on the analysis result, which makes it possible to analyze the text, voice, and image sent by the user and provide the necessary information quickly and accurately.

[0588] "Text" refers to textual information that a user sends through a messenger app.

[0589] "Audio" refers to the sound information that a user sends through a messenger app.

[0590] An "image" is any visual information that a user sends through a messenger app.

[0591] "Messenger App Internet Search Tool" is an application that allows users to send text, voice, and images, which are then analyzed to provide the necessary information.

[0592] "Information on weather, disasters, news, e-commerce, finance, etc." are a wide range of information categories that users desire.

[0593] A "smartphone" is a portable information terminal that can connect to the Internet and use applications.

[0594] A "server" is a computer system that receives data sent by a user, analyzes it, and provides the necessary information.

[0595] A "natural language processing engine" is software that analyzes text data and understands its meaning and intent.

[0596] The "analysis results" are information obtained by the natural language processing engine analyzing text data.

[0597] "Means of obtaining information" refers to the process of obtaining the necessary information from external information sources based on the analysis results.

[0598] "Means of returning information" is the process by which retrieved information is provided to the user.

[0599] "Non-digital native users" are users who are not familiar with digital technology.

[0600] "Artificial intelligence voiceprint authentication" is a technology that analyzes voice to recognize the speaker.

[0601] "Elderly people" are older users who may have difficulty using digital technology.

[0602] A "fixed-line telephone" is a telephone that is fixed and cannot be moved.

[0603] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0604] The present invention is a system that analyzes text, voice, and images sent by a user through a messenger app and provides necessary information. A specific embodiment of the system will be described below.

[0605] 1. System Configuration

[0606] The system consists of the following main components:

[0607] User device: The device through which a user uses the messenger app, such as a smartphone or landline phone.

[0608] Server: A computer system that receives data sent by users, analyzes it, and provides the required information.

[0609] Natural language processing engine: Software for analyzing text data and understanding its meaning and intent. Specifically, it uses natural language processing APIs.

[0610] Speech recognition software: Software for converting voice data into text, specifically using a speech recognition API.

[0611] Image recognition software: Software for analyzing image data. Specifically, it uses an image recognition API.

[0612] Information Acquisition API: An API for acquiring necessary information from external information sources based on the analysis results. Specifically, a weather forecast API is used.

[0613] 2. Program Processing

[0614] 1. User Input

[0615] User opens the Messenger app and sends a text, voice, or image.

[0616] Example: A user enters the text "What's the weather like tomorrow?" and presses the submit button.

[0617] 2. Receiving Data

[0618] The server receives data sent from the messenger app in real time and temporarily stores the received data in a database.

[0619] 3. Data Analysis

[0620] The server calls a natural language processing engine to analyze the data it receives. The API analyzes the text and understands the intent.

[0621] In the case of voice data, the server uses voice recognition software to convert the voice to text.

[0622] In the case of image data, the server uses image recognition software to analyze the image.

[0623] Example: The server sends the text "What's the weather like tomorrow?" to a natural language processing engine, and obtains the intent "I'm looking for a weather forecast" as the analysis result.

[0624] 4. Acquisition of information

[0625] Based on the analysis results, the server calls an information retrieval API and retrieves the specified information.

[0626] Example: Based on the analysis result of "What's the weather going to be tomorrow?", the server calls a weather forecast API and obtains the information "Tomorrow's weather will be sunny."

[0627] 5. Provision of information

[0628] The server returns the acquired information to the user. When returning the information, the notification function of the messenger app is used to display the information to the user.

[0629] Example: The server notifies the user that "tomorrow's weather will be sunny" via a messenger app and displays the information on the user's screen.

[0630] 3. Examples and prompts

[0631] Example 1: Text questions

[0632] User: "What's the weather like tomorrow?"

[0633] Server: Analyzes the text, retrieves weather forecast information, and replies to the user, "Tomorrow's weather will be sunny."

[0634] Example 2: Audio questions

[0635] User: Asks out loud, "What's the weather like tomorrow?"

[0636] Server: Converts the speech to text, analyzes it to obtain weather forecast information, and responds to the user, "Tomorrow's weather will be sunny."

[0637] Example 3: Image Question

[0638] User: Send weather related images.

[0639] Server: Analyzes the image, retrieves relevant weather forecast information, and responds to the user, "The weather in this area tomorrow will be sunny."

[0640] Example prompts for generative AI models

[0641] "If a user sends a text through your messenger app saying, 'What's the weather going to be like tomorrow?', please explain how you would get the weather forecast information and respond to the user."

[0642] In this way, a system can be created that analyzes the text, voice, and images sent by users through messenger apps and provides the necessary information.

[0643] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0644] Step 1:

[0645] User Input

[0646] How it works: A user opens a messaging app on their smartphone and sends a text, voice, or image.

[0647] Input: The user enters the text "What's the weather going to be like tomorrow?" and presses the submit button.

[0648] Output: The messenger app sends text data to the server.

[0649] Step 2:

[0650] Receiving data

[0651] How it works: The server receives data sent from the messenger app in real time, and temporarily stores it in a database.

[0652] Input: Text data sent from a messenger app: "What's the weather going to be like tomorrow?"

[0653] Output: The server prepares the received text data for analysis.

[0654] Step 3:

[0655] Analyzing data

[0656] How it works: The server calls a natural language processing engine to analyze the data it receives. The API analyzes the text and understands the intent. If it's voice data, the server uses voice recognition software to convert the speech to text. If it's image data, the server uses image recognition software to analyze the image.

[0657] Input: Text data received by the server: "What's the weather going to be like tomorrow?"

[0658] Output: The natural language processing engine returns the analysis result "You are looking for a weather forecast."

[0659] Step 4:

[0660] Obtaining information

[0661] Operation: Based on the analysis results, the server calls the information retrieval API and retrieves the specified information.

[0662] Input: Parsing result from the natural language processing engine: "I'm looking for a weather forecast."

[0663] Output: The information retrieval API returns the information "Tomorrow's weather will be sunny."

[0664] Step 5:

[0665] Provision of information

[0666] Behavior: The server returns the retrieved information to the user, using the notification feature of the messenger app to display the information to the user.

[0667] Input: Information from the information retrieval API: "Tomorrow's weather will be sunny."

[0668] Output: The server notifies the user that "tomorrow's weather will be sunny" through the messenger app and displays it on the user's screen.

[0669] In this way, a system can be created that analyzes the text, voice, and images sent by users through messenger apps and provides the necessary information.

[0670] (Application example 1)

[0671] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0672] Conventional messenger applications have the ability to provide information by analyzing text, voice, and images, but in autonomous vehicles, it is difficult to obtain information about the vehicle's status and surroundings in real time. In addition, there was a lack of functionality to provide information in response to user questions using voice or image recognition, which limited the user experience.

[0673] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means. In this invention, the server includes a means for providing a messenger app Internet search tool that can be used from text, voice, and image, a means for covering information on weather, disasters, news, EC, finance, etc., a means for enabling use from a smartphone messenger app, a means for acquiring the vehicle status and surrounding information in real time as an application installed in the autonomous driving vehicle, and a means for providing information in response to a user's question using voice recognition and image recognition. This makes it possible for a user to acquire necessary information in real time through text, voice, and image even in an autonomous driving vehicle.

[0674] "Text" is a means of expressing information using letters and symbols.

[0675] "Voice" is the means of recording and reproducing the human voice and other sounds as digital data.

[0676] An "image" is a means of recording and displaying visual information as digital data.

[0677] A "messenger app" is a software application that enables users to send and receive text, voice, and images.

[0678] An "Internet search tool" is software that searches for information on the Internet and provides it to users.

[0679] "Weather" is information that refers to the weather conditions in a particular area.

[0680] "Disaster" refers to information about damage caused by natural phenomena or man-made factors.

[0681] "News" is content intended to report current events or information.

[0682] "EC" is an abbreviation for electronic commerce, the act of buying and selling goods and services via the Internet.

[0683] "Finance" refers to information relating to the management, operation, and trading of funds.

[0684] A "smartphone" is a mobile phone that has Internet connectivity and a variety of applications.

[0685] An "autonomous vehicle" is a vehicle that drives autonomously using artificial intelligence and sensor technology.

[0686] "Real time" refers to a state in which processing and information provision are carried out immediately without delay.

[0687] "Speech recognition" is a technology that analyzes voice as digital data and converts it into text.

[0688] "Image recognition" is a technology that analyzes images and extracts specific information.

[0689] "User" refers to a person who uses a system or application.

[0690] A system for implementing this invention includes a means for providing a messenger app Internet search tool that can be used from text, voice, and images, a means for covering information on weather, disasters, news, e-commerce, finance, etc., a means for enabling use from a smartphone messenger app, a means for acquiring vehicle status and surrounding information in real time as an application to be installed in an autonomous vehicle, and a means for providing information in response to a user's questions using voice recognition and image recognition.

[0691] The server analyzes the text, voice, and images sent by the user and provides the necessary information. Specifically, when a user asks, "Where is the next gas station?", the server uses voice recognition technology to convert the voice into text, analyzes the text, and obtains the location information of the nearest gas station, which it then provides to the user.

[0692] To realize this system, the following hardware and software are used.

[0693] Hardware: Smartphones, microphones, cameras, self-driving vehicles

[0694] Software: Python, Requests library, SpeechRecognition library, Pillow library, Pytesseract library

[0695] The server uses the SpeechRecognition library for voice recognition and the Pytesseract library for image recognition, so that even if the user inputs an audio file or an image file, it can properly analyze it and provide information.

[0696] As a concrete example, consider the case where a user inputs a voice file saying "What's the weather tomorrow?" The server analyzes the voice file and converts it into text. It then uses WeatherAPI to get weather information for the specified location and provides it to the user.

[0697] Examples of prompt sentences to be input to a generative AI model include the following:

[0698] "The user has input an audio file 'weather_query.wav'. Write a program that parses this audio file and provides weather information."

[0699] In this way, users can obtain the information they need in real time through text, audio and images.

[0700] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0701] Step 1:

[0702] The user inputs text, an audio file, or an image file. The input data is sent to the server. Inputs include text messages, audio files (e.g., .wav format), and image files (e.g., .jpg format).

[0703] Step 2:

[0704] The server determines the format of the input data. Specifically, it analyzes the file extension and header information of the data to determine whether the input is text, audio, or image. The input is the data sent by the user, and the output is the data format information.

[0705] Step 3:

[0706] The server selects the appropriate analysis method depending on the format of the input data. If it is text, it proceeds with the analysis. If it is an audio file, it uses the SpeechRecognition library to convert the audio to text. If it is an image file, it uses the Pytesseract library to extract text from the image. The input is the format information and the original data, and the output is the analyzed text data.

[0707] Step 4:

[0708] The server performs natural language processing based on the analyzed text data to understand the user's question. Specifically, it tokenizes the text data and extracts keywords. The input is the analyzed text data, and the output is the extracted keywords.

[0709] Step 5:

[0710] The server retrieves the required information from the appropriate source based on the extracted keywords. For example, if the keyword "weather" is included, the Weather API is used to retrieve the weather information for the specified location. The input is the extracted keywords, and the output is the retrieved information.

[0711] Step 6:

[0712] The server formats the acquired information into a format for replying to the user. Specifically, it summarizes the information in text format and provides it to the user in an easy-to-understand format. The input is the acquired information, and the output is a formatted reply message.

[0713] Step 7:

[0714] The server sends the formatted reply message to the user, and the user receives the reply message through a messenger app. The input is the formatted reply message, and the output is the message sent to the user.

[0715] Through the above processing steps, the user can obtain the required information in real time through text, voice, and images.

[0716] Example 2

[0717] Next, a description will be given of Example 2 of the embodiment 2. In the following description, the data processing device 12 is referred to as a "server" and the smart glasses 214 are referred to as a "terminal".

[0718] Conventional information systems were often difficult to use for non-digital natives and the elderly. In addition, there were limited means of providing information by voice to users who used landlines, making it difficult for these users to easily obtain the latest information.

[0719] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for providing a messenger app Internet search tool that can be used from text, voice, and image, a means for covering information such as weather, disasters, news, electronic commerce, and finance, a means for enabling use from a messenger app of a mobile information terminal, a means for receiving voice data through a fixed telephone line, a means for analyzing the voice data and authenticating the user's voiceprint, a means for acquiring requested information when authentication is successful, a means for converting the acquired information into voice data, and a means for transmitting the voice data to the user through a fixed telephone line. This makes it possible for even users who are not digital natives or elderly people to easily acquire the latest information by voice using a fixed telephone line.

[0720] A "text, voice and image messaging app internet search tool" is an application that allows users to search for information and send and receive messages using text, voice and images.

[0721] "Information on weather, disasters, news, e-commerce, finance, etc." refers to a wide range of information that users need on a daily basis, such as meteorological information, information on natural disasters, the latest news, information on online shopping, and information on financial markets and banking.

[0722] A "mobile information terminal" is an electronic device, such as a smartphone or tablet, that is portable and can connect to the Internet.

[0723] A "fixed line" is a communication line for conducting voice communications over a traditional wired telephone network.

[0724] "Voice Data" means a digital representation of a User's voice.

[0725] An "AI voiceprint authentication system" is a system that uses artificial intelligence technology to identify individuals from voice data.

[0726] "Authentication" is the process by which a system verifies a user's identity and grants access.

[0727] A "data source" is an external source from which information is obtained.

[0728] "Speech synthesis software" refers to software for converting text data into voice data.

[0729] "Playing audio data" refers to the process of playing audio data as sound through an output device such as a speaker.

[0730] This invention provides an information provision system that can be used by non-digital native users and elderly people. In particular, we provide a system that allows users to request information by voice using a landline phone, authenticates the user using AI voiceprint authentication, and provides the requested information by voice.

[0731] Server Roles

[0732] The server receives voice data from the user via a landline. Specifically, when a user makes a voice request from a landline such as "What's in the news today?", the voice data is sent to the server. The server then sends the received voice data to an AI voiceprint authentication system, which analyzes the user's voiceprint. This authentication uses voice recognition software such as "Google Cloud Speech-to-Text." If authentication is successful, the server accesses data sources such as "News APIs" and "RSS feeds" to obtain the information requested by the user. The obtained information is converted into voice data using voice synthesis software such as "Amazon Polly" and provided to the user via the landline.

[0733] Terminal Roles

[0734] The terminal functions as a landline phone and is the interface when the user requests information by voice. When the user requests "What's the news today?" by voice from the landline phone, the terminal transmits the voice data to the server. When the terminal receives a response from the server, it plays the voice data and provides the information to the user.

[0735] User Roles

[0736] A user requests information by voice using a landline phone. For example, when an elderly person requests "What's in the news today?" from a landline phone, the voice data is sent to the server. The server authenticates the user's voiceprint, retrieves the requested news information, and provides it by voice.

[0737] Examples

[0738] Example 1: Requesting News Information

[0739] The user makes a voice request from a landline saying, "What's the news today?" The server receives this voice data and authenticates the user using an AI voiceprint authentication system. If authentication is successful, the server retrieves the latest news information from the news API, converts it into voice data, and provides it to the user.

[0740] Example 2: Requesting weather information

[0741] The user makes a voice request from a landline saying, "What's the weather today?" The server receives this voice data and authenticates the user using an AI voiceprint authentication system. If authentication is successful, the server retrieves the latest weather information from the weather API, converts it into voice data, and provides it to the user.

[0742] Examples of prompt statements

[0743] "Please explain the process for a system that authenticates the voiceprint of a user requesting information via voice over a landline phone and then provides the requested information via voice."

[0744] By inputting this prompt sentence into a generative AI model, it is possible to generate a sentence that explains the system's processing in natural language.

[0745] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0746] Step 1:

[0747] The server receives voice data from the user via a fixed telephone line.

[0748] Input: Voice data from users calling from landlines

[0749] Output: Audio data stored on the server

[0750] Specific operation: When a user voice requests "What's the news today?" from a landline phone, the voice data is sent to the server.

[0751] Step 2:

[0752] The server sends the received voice data to the AI ​​voiceprint authentication system and analyzes the user's voiceprint.

[0753] Input: Received audio data

[0754] Output: Analyzed voiceprint data

[0755] Specific operations: Voice data is converted into text using voice recognition software such as "Google Cloud Speech-to-Text," and voiceprint authentication is then performed based on the text data.

[0756] Step 3:

[0757] The server authenticates the user based on the analyzed voiceprint data.

[0758] Input: Analyzed voiceprint data

[0759] Output: Authentication result (success or failure)

[0760] Specific operation: The voiceprint data is compared with pre-registered voiceprint data, and if there is a match, the authentication is successful.

[0761] Step 4:

[0762] If authentication is successful, the server contacts the appropriate data source to obtain the information requested by the user.

[0763] Input: Authentication result (success) and user request

[0764] Output: Retrieved information data

[0765] Specific operation: Obtain the latest news information from "News API", "RSS feed", etc.

[0766] Step 5:

[0767] The server converts the acquired information into audio data.

[0768] Input: Acquired information data

[0769] Output: Audio data

[0770] Specific operation: Converts text data into speech data using speech synthesis software such as "Amazon Polly".

[0771] Step 6:

[0772] The server transmits the generated voice data to the user via a fixed telephone line.

[0773] Input: Generated audio data

[0774] Output: Voice data sent to the user's landline phone

[0775] What it does: It converts voice data into a format suitable for landline telephone lines and sends it to the user's landline.

[0776] Step 7:

[0777] The terminal receives the audio data transmitted from the server and plays it back.

[0778] Input: Audio data sent from the server

[0779] Output: Played audio information

[0780] Specific operation: Play audio data through the landline phone speaker to provide information to the user.

[0781] (Application example 2)

[0782] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the smart glasses 214 will be referred to as a "terminal".

[0783] Modern digital services are designed primarily for young people and digital natives, and are often difficult to use for the elderly and users who are not familiar with digital technology. Food delivery services also require the use of smartphones and the Internet, which makes them difficult for the elderly to use. Furthermore, although the number of services using voice recognition technology is increasing, the accuracy and security of user authentication may not be sufficient.

[0784] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0785] In this invention, the server includes a means for providing a messenger app internet search tool that can be used from text, voice, and image, a means for covering information on weather, disasters, news, EC, finance, etc., a means for enabling use from a smartphone messenger app, and a means for accepting voice orders from a landline in a food delivery service, a means for authenticating a user by AI voiceprint authentication and providing recommended menu items taking into consideration past order history and preferences, and a means for confirming the order details by voice and finally confirming the order. This enables even elderly people and users who are not familiar with digital to use food delivery services safely and easily using a landline.

[0786] A "text, voice and image messaging app internet search tool" is an application that allows users to search for information and send and receive messages using text, voice and images.

[0787] "Means of covering information on weather, disasters, news, e-commerce, finance, etc." refers to a function that comprehensively provides various types of information, such as weather forecasts, disaster information, the latest news, e-commerce information, and financial information.

[0788] "Means enabling use from a messenger app on a smartphone" refers to a function that enables users to use the service through a messenger app running on their smartphone.

[0789] "A means for accepting voice orders from a landline telephone in a food delivery service" refers to a function that enables a user to place a food delivery order by voice using a landline telephone.

[0790] "A means of authenticating users through AI voiceprint recognition and providing recommended menu items taking into consideration their past order history and preferences" refers to a function that identifies users through voiceprint recognition technology using artificial intelligence, and presents recommended menu items based on the user's past order history and preferences.

[0791] "A means for confirming order details by voice and finally confirming the order" is a function that supports the process in which a user confirms the order details by voice and finally confirms the order.

[0792] A system for carrying out the present invention has the following configuration: The server includes means for providing a messenger app internet search tool that can be used from text, voice, and image, means for covering information on weather, disasters, news, EC, finance, etc., means for enabling use from a smartphone messenger app, means for accepting voice orders from a landline phone in a food delivery service, means for authenticating a user by AI voiceprint authentication and providing recommended menu items taking into consideration past order history and preferences, and means for confirming the order details by voice and finally confirming the order.

[0793] Hardware and Software Configuration

[0794] The server uses a voice recognition library (e.g., speech_recognition), a voice synthesis library (e.g., pyttsx3), an AI voiceprint authentication module (e.g., voice_authentication), and a food delivery service API (e.g., food_delivery_service). The user uses a landline phone to place an order by voice. The server captures the voice using a microphone and outputs the voice using a speaker.

[0795] Data processing and calculation

[0796] The server processes the data in the following steps:

[0797] 1. When a user requests information by voice from a landline phone, the server picks up the voice through a microphone.

[0798] 2. Use a speech recognition library to convert the captured audio into text.

[0799] 3. Use the AI ​​voiceprint recognition module to authenticate the user's voiceprint.

[0800] 4. If authentication is successful, use the food delivery service API to get menu recommendations that take into account past ordering history and user preferences.

[0801] 5. The recommended menu is converted into voice using a speech synthesis library and provided to the user through the speaker.

[0802] 6. The user confirms the order details by voice and finally confirms the order.

[0803] Examples

[0804] For example, when a user makes a voice request from a landline saying, "What's the recommendation today?", the server captures the voice and converts it into text using a voice recognition library. Next, the AI ​​voiceprint authentication module authenticates the user and obtains menu recommendations using the food delivery service API. The obtained menu is converted into speech using a speech synthesis library and provided to the user through the speaker. When the user confirms by voice, "I'd like to order a sushi set," the server confirms the order.

[0805] Examples of prompt statements

[0806] User: Hello, Food Delivery Assistant, what's your name?

[0807] System: This is Tanaka.

[0808] User: Hello Tanaka-san. Please place your order.

[0809] System: What's your recommendation today?

[0810] User: Today's recommendation is the sushi set. Would you like to order the sushi set?

[0811] System: Yes.

[0812] User: Your order has been received, thank you.

[0813] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0814] Step 1:

[0815] A user requests information by voice from a landline phone. The server captures the user's voice through a microphone. The input is the user's voice and the output is voice data.

[0816] Step 2:

[0817] The server converts the acquired voice data into text using a voice recognition library (e.g., speech_recognition). The input is voice data, and the output is text data.

[0818] Step 3:

[0819] The server uses an AI voiceprint authentication module (e.g. voice_authentication) to authenticate the user's voiceprint from the text data. The input is the text data, and the output is the authentication result. If the authentication is successful, proceed to the next step.

[0820] Step 4:

[0821] The server uses a food delivery service API (e.g., food_delivery_service) to obtain recommended menu items that take into account the user's past order history and preferences. The input is the user authentication result and past order history data, and the output is the recommended menu items.

[0822] Step 5:

[0823] The server converts the text data of the recommended menu into speech using a speech synthesis library (e.g., pyttsx3). The input is the text data of the recommended menu, and the output is speech data.

[0824] Step 6:

[0825] The server provides the user with a recommended menu by voice through a speaker. The input is voice data, and the output is voice output to the user.

[0826] Step 7:

[0827] The user confirms the order details by voice on the landline phone. The server again picks up the user's voice through the microphone. The input is the user's voice, and the output is voice data.

[0828] Step 8:

[0829] The server converts the acquired voice data into text using a voice recognition library. The input is voice data and the output is text data.

[0830] Step 9:

[0831] The server analyzes the text data and confirms the order details. The input is the text data, and the output is the confirmation result of the order details.

[0832] Step 10:

[0833] The server confirms the order details and confirms the order using the food delivery service API. The input is the confirmation result of the order details, and the output is a notification that the order has been confirmed.

[0834] Step 11:

[0835] The server converts the order confirmation notification into speech using a speech synthesis library. The input is the order confirmation notification, and the output is speech data.

[0836] Step 12:

[0837] The server notifies the user of the order confirmation by voice through the speaker. The input is voice data, and the output is voice output to the user.

[0838] In addition, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0839] "Example 1"

[0840] In one embodiment of the present invention, an emotion engine that recognizes the user's emotions is incorporated. This emotion engine analyzes the user's emotions from text, voice, and images sent by the user through the messenger app. For example, if a user sends the text "Today is the worst," the emotion engine recognizes from this text that the user is feeling negative emotions. Also, if a user sends an image of a smiling face, the emotion engine recognizes from this image that the user is feeling positive emotions. In this way, the emotion engine can accurately grasp the user's emotions and provide information and services according to the emotions.

[0841] "Example 2"

[0842] In another embodiment of the present invention, an emotion engine is incorporated as a means of supporting users who are not digital natives. This emotion engine analyzes emotions from the user's voice. For example, if an elderly person makes a voice request from a landline, "What's the news today?", the emotion engine analyzes the user's emotions from the tone and strength of the voice. In this way, the emotion engine can accurately grasp the user's emotions and provide information and services accordingly.

[0843] "Example 5"

[0844] In a further embodiment of the present invention, an emotion engine is incorporated as a means that supports AI voiceprint authentication and can be used by elderly people even on landlines. This emotion engine analyzes emotions from the user's voice. For example, if an elderly person makes a voice request on a landline such as "What's the weather today?", the emotion engine analyzes the user's emotions from the tone and strength of the voice. In this way, the emotion engine can accurately grasp the user's emotions and provide information and services accordingly.

[0845] The process flow of each embodiment will be described below.

[0846] "Example 1"

[0847] Step 1: User sends a text, voice or image via a messenger app.

[0848] Step 2: The emotion engine analyzes the user's emotions from the submitted text, voice, and images.

[0849] Step 3: The system provides information and services based on the emotions analyzed by the emotion engine.

[0850] "Example 2"

[0851] Step 1: A non-digital native user makes a request via voice.

[0852] Step 2: The emotion engine analyzes the emotion from the user's voice.

[0853] Step 3: The system provides information and services based on the emotions analyzed by the emotion engine.

[0854] "Example 5"

[0855] Step 1: The senior makes a voice request from a landline phone.

[0856] Step 2: The emotion engine analyzes the emotion from the user's voice.

[0857] Step 3: The system provides information and services based on the emotions analyzed by the emotion engine.

[0858] Example 1

[0859] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0860] Conventional messenger apps lack the ability to analyze text, voice, and image messages sent by users and provide the necessary information. They also lack the ability to recognize users' emotions and respond appropriately, limiting the user experience. Furthermore, these systems are difficult to use for non-digital native users and the elderly.

[0861] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0862] In this invention, the server includes a means for providing a messenger app Internet search tool that can be used from text, voice, and image, a means for covering information on weather, disasters, news, electronic commerce, finance, etc., a means for enabling use from a messenger app on a mobile information terminal, a means for analyzing a user's text, voice, and image, a means for acquiring necessary information from the Internet based on the analysis result, a means including an emotion engine for recognizing the user's emotion, and a means for providing appropriate information to the user based on the analysis result and the acquired information. This makes it possible to not only analyze data sent by the user and provide necessary information, but also recognize the user's emotion and take appropriate action.

[0863] "Text" refers to textual information that a user sends through a messenger app.

[0864] "Audio" refers to sound information that a user sends through a messenger app.

[0865] An "image" is visual information that a user sends through a messenger app.

[0866] A "messenger app internet search tool" is a tool that analyzes data sent through a messenger app and retrieves and provides necessary information from the internet.

[0867] "Weather" is information about the weather.

[0868] "Disasters" refers to information about natural and man-made disasters.

[0869] "News" is reporting on current events or information.

[0870] "Electronic commerce" means commercial transactions conducted over the Internet.

[0871] "Finance" refers to information regarding the operation and management of funds.

[0872] A "portable information terminal" is a portable information processing device such as a smartphone or tablet.

[0873] "Analysis" is the process of analyzing the transmitted data to make sense of it.

[0874] "Obtaining from the Internet" refers to searching for and obtaining the necessary information via the Internet.

[0875] An "emotion engine" is software for analyzing emotions from data sent by users.

[0876] "Providing appropriate information" refers to responding with information that is useful to the user based on the analysis results and the information obtained.

[0877] This invention is a system that analyzes text, voice, and images sent by users through messenger apps and provides necessary information. It also incorporates an emotion engine that recognizes the user's emotions.

[0878] Hardware and Software Used

[0879] Hardware: Mobile information terminals (smartphones, tablets), servers

[0880] Software: Messenger app, Google Cloud Natural Language API, Amazon Rekognition, OpenWeatherMap API, IBM Watson Tone Analyzer

[0881] Data processing and calculation

[0882] 1. User Input

[0883] Users use messenger apps on their mobile devices to send text, voice, and images.

[0884] Example: A user enters the text "What's the weather like tomorrow?" and presses the submit button.

[0885] 2. Data transfer

[0886] The terminal transfers the data sent by the user to a server via the Internet.

[0887] What happens: The messenger app sends data to the server as an HTTP request.

[0888] 3. Data Analysis

[0889] The server analyzes the received text, audio, and images.

[0890] The analysis uses Google Cloud Natural Language API to analyze text and Amazon Rekognition to analyze images.

[0891] Example: Parsing the text "What's the weather like tomorrow?" to recognize that it is a request for weather forecast information.

[0892] 4. Acquisition of information

[0893] The server retrieves the necessary information from the Internet based on the analysis results.

[0894] Specific software: Obtains weather forecast information using the OpenWeatherMap API.

[0895] Example: In response to the request "What's the weather going to be like tomorrow?", weather forecast information is retrieved from the OpenWeatherMap API.

[0896] 5. Emotion Recognition

[0897] The server uses an emotion engine to analyze emotions from the user's text, voice, and images.

[0898] Specific software: Analyzing the sentiment of text using IBM Watson Tone Analyzer.

[0899] Example: If a user sends the text "Today is the worst", the sentiment engine will recognize the negative sentiment.

[0900] 6. Provision of information

[0901] The server provides the user with appropriate information based on the analysis results and the acquired information.

[0902] Specific behavior: The server generates the text "Tomorrow's weather will be sunny" and returns it to the user.

[0903] Example: If the user asks "What's the weather going to be like tomorrow?", the server responds with "The weather will be sunny tomorrow."

[0904] Examples of prompt statements

[0905] "Please explain the system that analyzes the text, voice, and images sent by the user through the messenger app and provides the necessary information. Please also specify the names of the specific software and APIs."

[0906] In this way, the present invention is a system that not only analyzes data sent by a user and provides necessary information, but also recognizes the user's emotions and takes appropriate action.

[0907] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0908] Step 1:

[0909] Users send text, voice, and images through messenger apps

[0910] Input: A user inputs text, voice, or images into a messenger app.

[0911] Specific behavior: The user enters the text "What's the weather like tomorrow?" and presses the send button.

[0912] Output: The messenger app prepares to send the input data.

[0913] Step 2:

[0914] The device forwards the transmitted data to the server.

[0915] Input: The data sent when the user presses the submit button.

[0916] What happens: The messenger app sends text data to the server as an HTTP request.

[0917] Output: The server receives the data.

[0918] Step 3:

[0919] The server analyzes the data it receives

[0920] Input: Text, audio, and image data sent from the device.

[0921] What happens next: The server uses the Google Cloud Natural Language API to parse the text and Amazon Rekognition to analyze the images.

[0922] Output: The analysis results include the user's request and the image content.

[0923] Example: Parsing the text "What's the weather like tomorrow?" to recognize that it is a request for weather forecast information.

[0924] Step 4:

[0925] The server retrieves the necessary information from the Internet.

[0926] Input: The user's request obtained as a result of analysis.

[0927] Specific behavior: The server retrieves weather forecast information using the OpenWeatherMap API.

[0928] Output: The desired information (e.g. weather forecast information) is obtained.

[0929] Example: In response to the request "What's the weather going to be like tomorrow?", weather forecast information is retrieved from the OpenWeatherMap API.

[0930] Step 5:

[0931] The server uses an emotion engine to analyze the user's emotions.

[0932] Input: Text, audio, and image data submitted by the user.

[0933] What happens: The server uses IBM Watson Tone Analyzer to analyze the sentiment of the text.

[0934] Output: The user's emotional state (e.g. positive, negative).

[0935] Example: If a user sends the text "Today is the worst", the sentiment engine will recognize the negative sentiment.

[0936] Step 6:

[0937] The server provides the user with appropriate information based on the analysis results and the acquired information.

[0938] Input: Analysis results and retrieved information.

[0939] Specific behavior: The server generates the text "Tomorrow's weather will be sunny" and returns it to the user.

[0940] Output: The user is provided with the appropriate information.

[0941] Example: If the user asks "What's the weather going to be like tomorrow?", the server responds with "The weather will be sunny tomorrow."

[0942] (Application example 1)

[0943] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0944] Conventional messenger apps only had limited functionality for analyzing the content of text, voice, and images sent by users and providing appropriate information. In addition, they lacked functionality for analyzing user emotions and taking appropriate action in emergencies, making them difficult to use in situations that required the provision of security-related information or emergency responses. Furthermore, their response to users who are not digital natives and the elderly was insufficient.

[0945] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0946] In this invention, the server includes a means for providing a messenger app Internet search tool that can be used from text, voice, and image, a means for covering information on weather, disasters, news, EC, finance, etc., a means for enabling use from a smartphone messenger app, a means for analyzing user emotions and taking appropriate action in an emergency, and a means for providing information on security. This makes it possible to analyze the text, voice, and image sent by the user and provide appropriate information, and in particular to provide information on security and respond quickly to emergencies. In addition, support for users who are not digital natives and the elderly is also strengthened.

[0947] A "messenger app internet search tool that can be used with text, voice, and images" is a messenger app that has the function of analyzing the text, voice, and images sent by the user and searching for and providing information on the internet.

[0948] "Means of covering information on weather, disasters, news, e-commerce, finance, etc." refers to the function of comprehensively providing various types of information, such as weather forecasts, disaster information, the latest news, information on e-commerce, and financial information.

[0949] "Means enabling access from a smartphone's messenger app" refers to a function that enables users to access various information through a messenger app running on their smartphone.

[0950] "A means of analyzing user emotions and taking appropriate action in an emergency" refers to a function that analyzes emotions from the text, voice, and images sent by the user, and takes appropriate action based on those emotions in the event of an emergency.

[0951] "Means of providing security-related information" refers to the function of analyzing information sent by users and providing appropriate security-related information and countermeasures.

[0952] "Measures to accommodate non-digital native users" refers to functions and interfaces that make it easy to use even for users who are not familiar with digital technology.

[0953] "A means that supports AI voiceprint authentication and allows elderly people to use the system via landline telephone" refers to a function that uses voiceprint authentication technology using artificial intelligence to enable elderly people to use the system via landline telephone.

[0954] The system for implementing this invention has the following configuration: First, a user sends text, voice, and images through a messenger app on their smartphone. These data are sent to a server, which then analyzes each data.

[0955] The server uses natural language processing libraries (e.g. TextBlob), speech recognition libraries (e.g. Google Speech-to-Text API), and image analysis libraries (e.g. PIL) for text, voice, and image analysis to extract the necessary information from the data submitted by the user and generate an appropriate response.

[0956] In addition, the server uses an emotion analysis engine to analyze the user's emotions. This emotion analysis engine determines the user's emotions from text, voice, and images, and takes appropriate action in the event of an emergency. For example, if the user sends a voice message saying "help," the server will send a notification to the emergency contact.

[0957] In order to provide information on security, the server analyzes the information sent by the user and provides appropriate security information and countermeasures. For example, if a user sends an image of a suspicious person, the server analyzes the image and notifies the police.

[0958] The system also takes into consideration users who are not digital natives and the elderly. The server uses AI voiceprint authentication technology to allow elderly people to use the system through a landline, making it easy for users who are not familiar with digital technology to use the system.

[0959] For example, if a user sends a text saying "I forgot my house keys," the server will analyze the text, search for nearby locksmiths and provide their contact information. If a user sends a text saying "I'm scared," the emotion analysis engine will recognize the negative emotion and take emergency action.

[0960] Examples of prompts to input to a generative AI model include:

[0961] If a user sends a text saying "I forgot my house keys", the sentiment engine should parse the user sentiment from this text and generate a response to search for a nearby locksmith.

[0962] In this way, a system can be realized that analyzes the text, voice, and images sent by users and provides appropriate information and takes emergency measures.

[0963] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0964] Step 1:

[0965] A user sends text, voice, and images through a messenger app on a smartphone. The input is the text, voice, and image data sent by the user, and the output is the transmission of these data to a server. A specific operation is when a user enters the text "I forgot my house key" and presses the send button.

[0966] Step 2:

[0967] The server analyzes the text, voice, and image data it receives. The input is the data sent by the user, and the output is the analysis result. Specifically, the server analyzes the text using a natural language processing library (e.g. TextBlob), converts the voice to text using a speech recognition library (e.g. Google Speech-to-Text API), and analyzes the image using an image analysis library (e.g. PIL).

[0968] Step 3:

[0969] The server analyzes the user's emotions using the emotion analysis engine. The input is the text, voice, and image data analyzed in step 2, and the output is the user's emotional state. In concrete terms, the server uses the emotion analysis engine to determine the user's emotions from the text and voice, and recognizes neutral emotions from the text, for example, "I forgot my house keys."

[0970] Step 4:

[0971] The server provides appropriate information based on the analysis results. The input is the analysis results of steps 2 and 3, and the output is the information provided to the user. In concrete terms, the server searches for a nearby locksmith for the text "I forgot my house keys" and provides the user with their contact information.

[0972] Step 5:

[0973] The server provides security information. The input is the data sent by the user and the analysis results, and the output is security information and countermeasures. In concrete terms, when a user sends an image of a suspicious person, the server analyzes the image and notifies the police.

[0974] Step 6:

[0975] The server takes appropriate action in case of emergency. The input is the result of the emotion analysis engine, and the output is the execution of emergency response. Specifically, when the user voices "help," the server sends a notification to the emergency contact.

[0976] Step 7:

[0977] The server supports users who are not digital natives and the elderly. The input is the user's usage status and voice data, and the output is a simple operation interface and voice authentication results. In concrete terms, the server uses AI voiceprint authentication technology to enable the elderly to use the system through a landline phone.

[0978] In this way, a system is realized that analyzes the text, voice, and images sent by the user and provides appropriate information and responds to emergencies.

[0979] Example 2

[0980] Next, a description will be given of Example 2 of the embodiment 2. In the following description, the data processing device 12 is referred to as a "server" and the smart glasses 214 are referred to as a "terminal".

[0981] Modern information provision systems are often difficult to use for non-digital native users and the elderly. In addition, there is a lack of means to provide appropriate information through voice recognition and emotion analysis to users who use landlines. For this reason, there is a demand for information provision systems that improve user convenience and are easy to use, especially for non-digital native users and the elderly.

[0982] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0983] In this invention, the server includes a means for providing a messenger app internet search tool that can be used from text, voice, and image, a means for covering information such as weather, disasters, news, e-commerce, and finance, a means for enabling use from a messenger app of a communication terminal, a means for receiving voice data through a landline and authenticating a user using an AI voiceprint authentication system, a means for analyzing the contents of a user's request and acquiring necessary information if authentication is successful, a means for converting the acquired information into voice data using a voice synthesis system and providing it to the user through a landline, and a means for analyzing emotions from the user's voice and providing information and services accordingly. This allows even users who are not digital natives or the elderly to easily acquire information through a landline and receive appropriate services.

[0984] A "text, voice and image messaging app internet search tool" is an application that allows users to search for information and send and receive messages using text, voice and images.

[0985] "Information on weather, disasters, news, e-commerce, finance, etc." refers to meteorological information, information on natural disasters, the latest news, information on online shopping, and information on financial markets and transactions.

[0986] A "communication terminal" is a device for communicating via the Internet or telephone lines, including smartphones and landline phones.

[0987] A "fixed line" is a wired communications line for carrying analog or digital telephone communications.

[0988] "Voice Data" means data that has been converted into digital form from a User's voice.

[0989] The "AI voiceprint authentication system" is an artificial intelligence system that analyzes the user's voice and authenticates individuals based on their voiceprints.

[0990] "User authentication" is the process by which a system verifies a user's identity and grants access rights.

[0991] "Request analysis" is the process of understanding the user's intent and the information they are looking for from the voice or text they provide.

[0992] "Obtaining information" means collecting the necessary data or information from external databases or APIs based on user requests.

[0993] A "voice synthesis system" is a technology that converts text data into voice data.

[0994] "Emotion analysis" is the process of determining the user's emotional state from their voice and taking appropriate action based on that information.

[0995] "Provision of information and services" refers to providing users with necessary information or specific services.

[0996] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0997] This invention provides an information provision system that can be used by non-digital native users and elderly people. Specifically, this system uses AI voiceprint authentication and a voice synthesis system to provide information to users who request information by voice using a landline phone.

[0998] Server Roles

[0999] The server receives voice requests from users through landline telephone lines. The received voice data is converted into digital format using a voice modem. The server then sends this voice data to an AI voiceprint authentication system to authenticate the user. This AI voiceprint authentication system uses common voice recognition software (e.g., voice recognition API). If authentication is successful, the server analyzes the content of the user's request and obtains the required information. For example, to obtain news information, the server uses a news API to obtain the latest news data. The server then converts the obtained information into voice data using a voice synthesis system (e.g., voice synthesis API) and provides it to the user through the landline telephone line.

[1000] Terminal Roles

[1001] The terminal receives the user's voice request via a landline telephone line. The received voice data is sent to a voice recognition module in the terminal and converted into text data. This voice recognition module uses general voice recognition software (e.g., voice recognition API). The converted text data is sent to a server, where the user is authenticated by an AI voiceprint authentication system. If authentication is successful, the terminal converts the information obtained from the server into voice data using a voice synthesis system (e.g., voice synthesis API) and provides it to the user via the landline telephone line.

[1002] User Roles

[1003] A user requests information by voice using a landline. For example, the user may make a request such as, "What's in the news today?" This voice request is sent to the server via the landline. The server authenticates the user using an AI voiceprint authentication system, and if authentication is successful, retrieves the information requested by the user. The retrieved information is converted into voice data using a voice synthesis system and provided to the user via the landline. The user can obtain the required information by listening to the provided voice information.

[1004] Examples

[1005] Example 1: Requesting news information

[1006] The user makes a voice request from a landline saying, "What's in the news today?" The server receives this voice data and authenticates the user using an AI voiceprint authentication system (e.g., voice recognition API). If authentication is successful, the server uses a news API to obtain the latest news data. The obtained news data is converted into voice data using a voice synthesis system (e.g., voice synthesis API) and provided to the user via the landline.

[1007] Example 2: Requesting weather information

[1008] A user makes a voice request from a landline saying, "What's the weather today?" The server receives this voice data and authenticates the user using an AI voiceprint authentication system (e.g., voice recognition API). If authentication is successful, the server uses a weather API to obtain the latest weather data. The obtained weather data is converted into voice data using a voice synthesis system (e.g., voice synthesis API) and provided to the user via the landline.

[1009] Examples of prompt statements

[1010] "Please explain in detail from the perspectives of the server, the terminal, and the user the process of a system that uses AI voiceprint authentication and a voice synthesis system to provide information to users who request information by voice from a landline phone. Please also add specific actions for each processing step."

[1011] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1012] Step 1:

[1013] The server receives voice requests from users over landline telephone lines.

[1014] Input: Voice data from a landline

[1015] Specific operation: The server uses a voice modem to convert analog voice signals into digital voice data.

[1016] Output: Digital audio data

[1017] Step 2:

[1018] The server sends the received voice data to the AI ​​voiceprint authentication system to authenticate the user.

[1019] Input: Digital audio data

[1020] Specific operation: The server calls the voice recognition API and converts the voice data into text data, which is then sent to the AI ​​voiceprint authentication system.

[1021] Output: Authentication result

[1022] Step 3:

[1023] The server receives the authentication results from the AI ​​voiceprint authentication system and authenticates the user.

[1024] Input: Authentication result

[1025] Specific operation: The server analyzes the authentication results returned by the AI ​​voiceprint authentication system and determines whether the authentication was successful.

[1026] Output: Authentication success or failure flag.

[1027] Step 4:

[1028] If the authentication is successful, the server analyzes the user's request and obtains the necessary information.

[1029] Input: Authentication success flag, voice data

[1030] Specific operation: The server uses a speech recognition API to convert the voice data into text data and analyzes the user's request. Based on the analysis results, the server retrieves the required information from external databases such as news APIs and weather APIs.

[1031] Output: Acquired information data

[1032] Step 5:

[1033] The server converts the acquired information into voice data using a voice synthesis system.

[1034] Input: Acquired information data

[1035] Specific operation: The server uses a speech synthesis API to convert the text data into speech data.

[1036] Output: Audio data

[1037] Step 6:

[1038] The server provides the generated voice data to the user via a fixed telephone line.

[1039] Input: Audio data

[1040] What it does: The server uses a voice modem to convert the digital voice data into an analog voice signal and transmit it to the user over a landline.

[1041] Output: Audio information provided to the user

[1042] Step 7:

[1043] A user requests information by voice using a landline telephone.

[1044] Input: User's voice request

[1045] Specific actions: A user picks up a landline phone receiver and speaks a request such as "What's in the news today?"

[1046] Output: Voice data sent over a landline

[1047] Step 8:

[1048] The user obtains the required information by listening to the provided audio information.

[1049] Input: Voice information provided via a landline telephone line

[1050] Specific operation: The user listens to news and weather information provided through a landline telephone receiver and obtains the required information.

[1051] Output: Information the user gets

[1052] (Application example 2)

[1053] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the smart glasses 214 will be referred to as a "terminal".

[1054] When non-digital native users and the elderly use landlines to obtain information, there are problems such as complex operations and difficulty in authentication. In addition, there is an issue that the user experience is not improved because appropriate information is not provided according to the user's emotions.

[1055] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1056] In this invention, the server includes a means for providing a messenger app internet search tool that can be used from text, voice, and image, a means for covering information on weather, disasters, news, EC, finance, etc., a means for enabling use from a messenger app on a smartphone, a means for requesting information by voice from a landline, a means for performing AI voiceprint authentication, a means including an emotion engine that analyzes emotions from the user's voice, and a means for providing information according to emotions. This allows even users who are not digital natives or the elderly to easily obtain information using a landline and provide appropriate information according to emotions.

[1057] A "text, voice and image enabled messenger app internet search tool" is a tool that allows users to search for information using text, voice and images and is accessible through a messenger app.

[1058] "Means that cover a wide range of information, including weather, disasters, news, e-commerce, and finance" refers to means that comprehensively provide a wide range of information, such as weather forecasts, disaster information, the latest news, e-commerce information, and financial information.

[1059] "Means enabling use from a messenger app on a smartphone" refers to means that enable information to be obtained or services to be used via a messenger app installed on a smartphone.

[1060] A "means for requesting information by voice from a landline telephone" is a means by which a user can request information by voice using a landline telephone.

[1061] "Means for performing AI voiceprint authentication" refers to a means for using artificial intelligence to authenticate a user's voiceprint and confirm the user's identity.

[1062] "Means including an emotion engine that analyzes emotions from a user's voice" refers to means including an emotion engine that can analyze user's voice data and read emotions from the voice.

[1063] A "means for providing information according to emotions" is a means for providing appropriate information according to the user's emotional state.

[1064] A system for carrying out this invention has the following configuration: The server includes a means for providing a messenger app internet search tool that can be used from text, voice, and image, a means for covering information on weather, disasters, news, EC, finance, etc., a means for enabling use from a messenger app on a smartphone, a means for requesting information by voice from a landline phone, a means for performing AI voiceprint authentication, a means including an emotion engine that analyzes emotions from the user's voice, and a means for providing information according to emotions.

[1065] Hardware and Software Used

[1066] Hardware: Smartphones, landlines

[1067] Software: Python, speech_recognition library, pyttsx3 library, custom emotion_recognition module, voice_authentication module

[1068] Data processing and calculation

[1069] The server first receives voice input from a landline phone. This voice data is converted to text using the speech_recognition library. Then, it authenticates the user's voiceprint from the voice data using the voice_authentication module. If authentication is successful, it analyzes the user's emotions from the voice data using the emotion_recognition module.

[1070] Based on the analyzed emotions, the server selects appropriate information and provides it by voice using the pyttsx3 library. This series of processes enables even non-digital native users and elderly people to easily obtain information using a landline phone and receive appropriate information according to their emotions.

[1071] Examples

[1072] For example, if a user makes a voice request from a landline saying, "What is the security situation at home?", the server will perform voice authentication and emotion analysis and respond by voice with, "The current security situation is normal." In this way, the user can obtain information with peace of mind.

[1073] Examples of prompt statements

[1074] When a user makes a voice request from a landline phone saying, "What is the security status at home?", the system should perform voice recognition and emotion analysis and provide emotion-based security information via voice.

[1075] This invention makes it possible for even non-digital native users and elderly people to easily obtain information using a landline phone and to be provided with appropriate information according to their emotions.

[1076] The flow of the specific process in the application example 2 will be described with reference to FIG.

[1077] Step 1:

[1078] A user requests information by voice from a landline phone. The user uses a landline phone to make a request by voice, for example, "What is the security situation at my home?" This voice data is input into the system.

[1079] Step 2:

[1080] The server receives the voice data and converts the voice to text using the speech_recognition library. The input is voice data and the output is text data. This conversion converts the voice data into analyzable text data.

[1081] Step 3:

[1082] The server uses the voice_authentication module to authenticate the user's voiceprint from the voice data corresponding to the text data. The input is the voice data, and the output is the authentication result. This authentication confirms the user's identity.

[1083] Step 4:

[1084] If the authentication is successful, the server uses the emotion_recognition module to analyze the user's emotions from the voice data. The input is the voice data, and the output is the emotion data. This analysis allows the user's emotional state to be understood.

[1085] Step 5:

[1086] The server selects appropriate information based on the analyzed emotion data. For example, if the user expresses anxiety, the server selects reassuring information such as "The current security situation is normal." The input is emotion data, and the output is the selected information.

[1087] Step 6:

[1088] The server converts the selected information into audio using the pyttsx3 library and provides it to the user through a landline phone. The input is the selected information and the output is audio data. This conversion allows the user to receive the information by audio.

[1089] Step 7:

[1090] The user receives the voice information provided via a landline phone and obtains the information he or she needs, allowing the user to obtain information with peace of mind.

[1091] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1092] The data generation model 58 is a so-called generative AI (Artificial Intelligence).

[1093] One example of a data generation model58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1094] Another example of generative AI is Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) are mentioned.

[1095] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1096] [Third embodiment]

[1097] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1098] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1099] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[1100] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1101] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[1102] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[1103] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[1104] Fig. 6 shows an example of main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1105] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1106] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1107] In the headset type terminal 314, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1108] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1109] "Example 1"

[1110] The present invention provides a messenger app internet search tool that can use text, voice, and images. Specifically, when a user sends text, voice, or image through a messenger app, the system analyzes it and provides the required information. For example, when a user sends a text such as "What's the weather like tomorrow?", the system retrieves weather forecast information and provides it to the user.

[1111] "Example 2"

[1112] Furthermore, the present invention supports AI voiceprint authentication to enable use by non-digital native users and elderly people. Specifically, when a user requests information by voice from a landline phone, the system performs AI voiceprint authentication to authenticate the user. If authentication is successful, the information requested by the user is provided by voice. For example, when an elderly person requests by voice from a landline phone, "What's in the news today?", the system obtains the news information and provides it by voice.

[1113] The process flow of each embodiment will be described below.

[1114] "Example 1"

[1115] Step 1: User sends a text, voice or image via a messenger app.

[1116] Step 2: The system analyzes the transmitted text, audio and images.

[1117] Step 3: Based on the analysis result, the system obtains the required information. For example, if the user sends a text "What's the weather like tomorrow?", the system will obtain weather forecast information.

[1118] Step 4: Provide the retrieved information to the user.

[1119] "Example 2"

[1120] Step 1: User requests information by voice over a landline phone.

[1121] Step 2: The system analyzes the voice and performs AI voiceprint authentication.

[1122] Step 3: If the authentication is successful, the system retrieves the information requested by the user. For example, if an elderly person requests "What's the news today?" by voice, the system retrieves the news information.

[1123] Step 4: Provide the retrieved information to the user via voice.

[1124] Example 1

[1125] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1126] Conventional messenger apps are limited to sending text, voice, and images, making it difficult to provide users with the information they need quickly and accurately. In addition, these apps are difficult to use for non-digital native users and elderly people. Furthermore, voice recognition and information provision functions are lacking for elderly people who use landlines. To solve these issues, a system that analyzes data sent by users and provides the necessary information is required.

[1127] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1128] In this invention, the server includes a means for receiving data sent by a user through a messenger app, a means for using a natural language processing engine to analyze the received data, and a means for acquiring necessary information based on the analysis result, which makes it possible to analyze the text, voice, and image sent by the user and provide the necessary information quickly and accurately.

[1129] "Text" refers to textual information that a user sends through a messenger app.

[1130] "Audio" refers to the sound information that a user sends through a messenger app.

[1131] An "image" is any visual information that a user sends through a messenger app.

[1132] "Messenger App Internet Search Tool" is an application that allows users to send text, voice, and images, which are then analyzed to provide the necessary information.

[1133] "Information on weather, disasters, news, e-commerce, finance, etc." are a wide range of information categories that users desire.

[1134] A "smartphone" is a portable information terminal that can connect to the Internet and use applications.

[1135] A "server" is a computer system that receives data sent by a user, analyzes it, and provides the necessary information.

[1136] A "natural language processing engine" is software that analyzes text data and understands its meaning and intent.

[1137] The "analysis results" are information obtained by the natural language processing engine analyzing text data.

[1138] "Means of obtaining information" refers to the process of obtaining the necessary information from external information sources based on the analysis results.

[1139] "Means of returning information" is the process by which retrieved information is provided to the user.

[1140] "Non-digital native users" are users who are not familiar with digital technology.

[1141] "Artificial intelligence voiceprint authentication" is a technology that analyzes voice to recognize the speaker.

[1142] "Elderly people" are older users who may have difficulty using digital technology.

[1143] A "fixed-line telephone" is a telephone that is fixed and cannot be moved.

[1144] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[1145] The present invention is a system that analyzes text, voice, and images sent by a user through a messenger app and provides necessary information. A specific embodiment of the system will be described below.

[1146] 1. System Configuration

[1147] The system consists of the following main components:

[1148] User device: The device through which a user uses the messenger app, such as a smartphone or landline phone.

[1149] Server: A computer system that receives data sent by users, analyzes it, and provides the required information.

[1150] Natural language processing engine: Software for analyzing text data and understanding its meaning and intent. Specifically, it uses natural language processing APIs.

[1151] Speech recognition software: Software for converting voice data into text, specifically using a speech recognition API.

[1152] Image recognition software: Software for analyzing image data. Specifically, it uses an image recognition API.

[1153] Information Acquisition API: An API for acquiring necessary information from external information sources based on the analysis results. Specifically, a weather forecast API is used.

[1154] 2. Program Processing

[1155] 1. User Input

[1156] User opens the Messenger app and sends a text, voice, or image.

[1157] Example: A user enters the text "What's the weather like tomorrow?" and presses the submit button.

[1158] 2. Receiving Data

[1159] The server receives data sent from the messenger app in real time and temporarily stores the received data in a database.

[1160] 3. Data Analysis

[1161] The server calls a natural language processing engine to analyze the data it receives. The API analyzes the text and understands the intent.

[1162] In the case of voice data, the server uses voice recognition software to convert the voice to text.

[1163] In the case of image data, the server uses image recognition software to analyze the image.

[1164] Example: The server sends the text "What's the weather like tomorrow?" to a natural language processing engine, and obtains the intent "I'm looking for a weather forecast" as the analysis result.

[1165] 4. Acquisition of information

[1166] Based on the analysis results, the server calls an information retrieval API and retrieves the specified information.

[1167] Example: Based on the analysis result of "What's the weather going to be tomorrow?", the server calls a weather forecast API and obtains the information "Tomorrow's weather will be sunny."

[1168] 5. Provision of information

[1169] The server returns the acquired information to the user. When returning the information, the notification function of the messenger app is used to display the information to the user.

[1170] Example: The server notifies the user that "tomorrow's weather will be sunny" via a messenger app and displays the information on the user's screen.

[1171] 3. Examples and prompts

[1172] Example 1: Text questions

[1173] User: "What's the weather like tomorrow?"

[1174] Server: Analyzes the text, retrieves weather forecast information, and replies to the user, "Tomorrow's weather will be sunny."

[1175] Example 2: Audio questions

[1176] User: Asks out loud, "What's the weather like tomorrow?"

[1177] Server: Converts the speech to text, analyzes it to obtain weather forecast information, and responds to the user, "Tomorrow's weather will be sunny."

[1178] Example 3: Image Question

[1179] User: Send weather related images.

[1180] Server: Analyzes the image, retrieves relevant weather forecast information, and responds to the user, "The weather in this area tomorrow will be sunny."

[1181] Example prompts for generative AI models

[1182] "If a user sends a text through your messenger app saying, 'What's the weather going to be like tomorrow?', please explain how you would get the weather forecast information and respond to the user."

[1183] In this way, a system can be created that analyzes the text, voice, and images sent by users through messenger apps and provides the necessary information.

[1184] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1185] Step 1:

[1186] User Input

[1187] How it works: A user opens a messaging app on their smartphone and sends a text, voice, or image.

[1188] Input: The user enters the text "What's the weather going to be like tomorrow?" and presses the submit button.

[1189] Output: The messenger app sends text data to the server.

[1190] Step 2:

[1191] Receiving data

[1192] How it works: The server receives data sent from the messenger app in real time, and temporarily stores it in a database.

[1193] Input: Text data sent from a messenger app: "What's the weather going to be like tomorrow?"

[1194] Output: The server prepares the received text data for analysis.

[1195] Step 3:

[1196] Analyzing data

[1197] How it works: The server calls a natural language processing engine to analyze the data it receives. The API analyzes the text and understands the intent. If it's voice data, the server uses voice recognition software to convert the speech to text. If it's image data, the server uses image recognition software to analyze the image.

[1198] Input: Text data received by the server: "What's the weather going to be like tomorrow?"

[1199] Output: The natural language processing engine returns the analysis result "You are looking for a weather forecast."

[1200] Step 4:

[1201] Obtaining information

[1202] Operation: Based on the analysis results, the server calls the information retrieval API and retrieves the specified information.

[1203] Input: Parsing result from the natural language processing engine: "I'm looking for a weather forecast."

[1204] Output: The information retrieval API returns the information "Tomorrow's weather will be sunny."

[1205] Step 5:

[1206] Provision of information

[1207] Behavior: The server returns the retrieved information to the user, using the notification feature of the messenger app to display the information to the user.

[1208] Input: Information from the information retrieval API: "Tomorrow's weather will be sunny."

[1209] Output: The server notifies the user that "tomorrow's weather will be sunny" through the messenger app and displays it on the user's screen.

[1210] In this way, a system can be created that analyzes the text, voice, and images sent by users through messenger apps and provides the necessary information.

[1211] (Application example 1)

[1212] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1213] Conventional messenger applications have the ability to provide information by analyzing text, voice, and images, but in autonomous vehicles, it is difficult to obtain information about the vehicle's status and surroundings in real time. In addition, there was a lack of functionality to provide information in response to user questions using voice or image recognition, which limited the user experience.

[1214] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means. In this invention, the server includes a means for providing a messenger app Internet search tool that can be used from text, voice, and image, a means for covering information on weather, disasters, news, EC, finance, etc., a means for enabling use from a smartphone messenger app, a means for acquiring the vehicle status and surrounding information in real time as an application installed in the autonomous driving vehicle, and a means for providing information in response to a user's question using voice recognition and image recognition. This makes it possible for a user to acquire necessary information in real time through text, voice, and image even in an autonomous driving vehicle.

[1215] "Text" is a means of expressing information using letters and symbols.

[1216] "Voice" is the means of recording and reproducing the human voice and other sounds as digital data.

[1217] An "image" is a means of recording and displaying visual information as digital data.

[1218] A "messenger app" is a software application that enables users to send and receive text, voice, and images.

[1219] An "Internet search tool" is software that searches for information on the Internet and provides it to users.

[1220] "Weather" is information that refers to the weather conditions in a particular area.

[1221] "Disaster" refers to information about damage caused by natural phenomena or man-made factors.

[1222] "News" is content intended to report current events or information.

[1223] "EC" is an abbreviation for electronic commerce, the act of buying and selling goods and services via the Internet.

[1224] "Finance" refers to information relating to the management, operation, and trading of funds.

[1225] A "smartphone" is a mobile phone that has Internet connectivity and a variety of applications.

[1226] An "autonomous vehicle" is a vehicle that drives autonomously using artificial intelligence and sensor technology.

[1227] "Real time" refers to a state in which processing and information provision are carried out immediately without delay.

[1228] "Speech recognition" is a technology that analyzes voice as digital data and converts it into text.

[1229] "Image recognition" is a technology that analyzes images and extracts specific information.

[1230] "User" refers to a person who uses a system or application.

[1231] A system for implementing this invention includes a means for providing a messenger app Internet search tool that can be used from text, voice, and images, a means for covering information on weather, disasters, news, e-commerce, finance, etc., a means for enabling use from a smartphone messenger app, a means for acquiring vehicle status and surrounding information in real time as an application to be installed in an autonomous vehicle, and a means for providing information in response to a user's questions using voice recognition and image recognition.

[1232] The server analyzes the text, voice, and images sent by the user and provides the necessary information. Specifically, when a user asks, "Where is the next gas station?", the server uses voice recognition technology to convert the voice into text, analyzes the text, and obtains the location information of the nearest gas station, which it then provides to the user.

[1233] To realize this system, the following hardware and software are used.

[1234] Hardware: Smartphones, microphones, cameras, self-driving vehicles

[1235] Software: Python, Requests library, SpeechRecognition library, Pillow library, Pytesseract library

[1236] The server uses the SpeechRecognition library for voice recognition and the Pytesseract library for image recognition, so that even if the user inputs an audio file or an image file, it can properly analyze it and provide information.

[1237] As a concrete example, consider the case where a user inputs a voice file saying "What's the weather tomorrow?" The server analyzes the voice file and converts it into text. It then uses WeatherAPI to get weather information for the specified location and provides it to the user.

[1238] Examples of prompt sentences to be input to a generative AI model include the following:

[1239] "The user has input an audio file 'weather_query.wav'. Write a program that parses this audio file and provides weather information."

[1240] In this way, users can obtain the information they need in real time through text, audio and images.

[1241] The flow of the specific process in the application example 1 will be described with reference to FIG.

[1242] Step 1:

[1243] The user inputs text, an audio file, or an image file. The input data is sent to the server. Inputs include text messages, audio files (e.g., .wav format), and image files (e.g., .jpg format).

[1244] Step 2:

[1245] The server determines the format of the input data. Specifically, it analyzes the file extension and header information of the data to determine whether the input is text, audio, or image. The input is the data sent by the user, and the output is the data format information.

[1246] Step 3:

[1247] The server selects the appropriate analysis method depending on the format of the input data. If it is text, it proceeds with the analysis. If it is an audio file, it uses the SpeechRecognition library to convert the audio to text. If it is an image file, it uses the Pytesseract library to extract text from the image. The input is the format information and the original data, and the output is the analyzed text data.

[1248] Step 4:

[1249] The server performs natural language processing based on the analyzed text data to understand the user's question. Specifically, it tokenizes the text data and extracts keywords. The input is the analyzed text data, and the output is the extracted keywords.

[1250] Step 5:

[1251] The server retrieves the required information from the appropriate source based on the extracted keywords. For example, if the keyword "weather" is included, the Weather API is used to retrieve the weather information for the specified location. The input is the extracted keywords, and the output is the retrieved information.

[1252] Step 6:

[1253] The server formats the acquired information into a format for replying to the user. Specifically, it summarizes the information in text format and provides it to the user in an easy-to-understand format. The input is the acquired information, and the output is a formatted reply message.

[1254] Step 7:

[1255] The server sends the formatted reply message to the user, and the user receives the reply message through a messenger app. The input is the formatted reply message, and the output is the message sent to the user.

[1256] Through the above processing steps, the user can obtain the required information in real time through text, voice, and images.

[1257] Example 2

[1258] Next, a second embodiment of the second form will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1259] Conventional information systems were often difficult to use for non-digital natives and the elderly. In addition, there were limited means of providing information by voice to users who used landlines, making it difficult for these users to easily obtain the latest information.

[1260] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for providing a messenger app Internet search tool that can be used from text, voice, and image, a means for covering information such as weather, disasters, news, electronic commerce, and finance, a means for enabling use from a messenger app of a mobile information terminal, a means for receiving voice data through a fixed telephone line, a means for analyzing the voice data and authenticating the user's voiceprint, a means for acquiring requested information when authentication is successful, a means for converting the acquired information into voice data, and a means for transmitting the voice data to the user through a fixed telephone line. This makes it possible for even users who are not digital natives or elderly people to easily acquire the latest information by voice using a fixed telephone line.

[1261] A "text, voice and image messaging app internet search tool" is an application that allows users to search for information and send and receive messages using text, voice and images.

[1262] "Information on weather, disasters, news, e-commerce, finance, etc." refers to a wide range of information that users need on a daily basis, such as meteorological information, information on natural disasters, the latest news, information on online shopping, and information on financial markets and banking.

[1263] A "mobile information terminal" is an electronic device, such as a smartphone or tablet, that is portable and can connect to the Internet.

[1264] A "fixed line" is a communication line for conducting voice communications over a traditional wired telephone network.

[1265] "Voice Data" means a digital representation of a User's voice.

[1266] An "AI voiceprint authentication system" is a system that uses artificial intelligence technology to identify individuals from voice data.

[1267] "Authentication" is the process by which a system verifies a user's identity and grants access.

[1268] A "data source" is an external source from which information is obtained.

[1269] "Speech synthesis software" refers to software for converting text data into voice data.

[1270] "Playing audio data" refers to the process of playing audio data as sound through an output device such as a speaker.

[1271] This invention provides an information provision system that can be used by non-digital native users and elderly people. In particular, we provide a system that allows users to request information by voice using a landline phone, authenticates the user using AI voiceprint authentication, and provides the requested information by voice.

[1272] Server Roles

[1273] The server receives voice data from the user via a landline. Specifically, when a user makes a voice request from a landline such as "What's in the news today?", the voice data is sent to the server. The server then sends the received voice data to an AI voiceprint authentication system, which analyzes the user's voiceprint. This authentication uses voice recognition software such as "Google Cloud Speech-to-Text." If authentication is successful, the server accesses data sources such as "News APIs" and "RSS feeds" to obtain the information requested by the user. The obtained information is converted into voice data using voice synthesis software such as "Amazon Polly" and provided to the user via the landline.

[1274] Terminal Roles

[1275] The terminal functions as a landline phone and is the interface when the user requests information by voice. When the user requests "What's the news today?" by voice from the landline phone, the terminal transmits the voice data to the server. When the terminal receives a response from the server, it plays the voice data and provides the information to the user.

[1276] User Roles

[1277] A user requests information by voice using a landline phone. For example, when an elderly person requests "What's in the news today?" from a landline phone, the voice data is sent to the server. The server authenticates the user's voiceprint, retrieves the requested news information, and provides it by voice.

[1278] Examples

[1279] Example 1: Requesting News Information

[1280] The user makes a voice request from a landline saying, "What's the news today?" The server receives this voice data and authenticates the user using an AI voiceprint authentication system. If authentication is successful, the server retrieves the latest news information from the news API, converts it into voice data, and provides it to the user.

[1281] Example 2: Requesting weather information

[1282] The user makes a voice request from a landline saying, "What's the weather today?" The server receives this voice data and authenticates the user using an AI voiceprint authentication system. If authentication is successful, the server retrieves the latest weather information from the weather API, converts it into voice data, and provides it to the user.

[1283] Examples of prompt statements

[1284] "Please explain the process for a system that authenticates the voiceprint of a user requesting information via voice over a landline phone and then provides the requested information via voice."

[1285] By inputting this prompt sentence into a generative AI model, it is possible to generate a sentence that explains the system's processing in natural language.

[1286] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1287] Step 1:

[1288] The server receives voice data from the user via a fixed telephone line.

[1289] Input: Voice data from users calling from landlines

[1290] Output: Audio data stored on the server

[1291] Specific operation: When a user voice requests "What's the news today?" from a landline phone, the voice data is sent to the server.

[1292] Step 2:

[1293] The server sends the received voice data to the AI ​​voiceprint authentication system and analyzes the user's voiceprint.

[1294] Input: Received audio data

[1295] Output: Analyzed voiceprint data

[1296] Specific operations: Voice data is converted into text using voice recognition software such as "Google Cloud Speech-to-Text," and voiceprint authentication is then performed based on the text data.

[1297] Step 3:

[1298] The server authenticates the user based on the analyzed voiceprint data.

[1299] Input: Analyzed voiceprint data

[1300] Output: Authentication result (success or failure)

[1301] Specific operation: The voiceprint data is compared with pre-registered voiceprint data, and if there is a match, the authentication is successful.

[1302] Step 4:

[1303] If authentication is successful, the server contacts the appropriate data source to obtain the information requested by the user.

[1304] Input: Authentication result (success) and user request

[1305] Output: Retrieved information data

[1306] Specific operation: Obtain the latest news information from "News API", "RSS feed", etc.

[1307] Step 5:

[1308] The server converts the acquired information into audio data.

[1309] Input: Acquired information data

[1310] Output: Audio data

[1311] Specific operation: Converts text data into speech data using speech synthesis software such as "Amazon Polly".

[1312] Step 6:

[1313] The server transmits the generated voice data to the user via a fixed telephone line.

[1314] Input: Generated audio data

[1315] Output: Voice data sent to the user's landline phone

[1316] What it does: It converts voice data into a format suitable for landline telephone lines and sends it to the user's landline.

[1317] Step 7:

[1318] The terminal receives the audio data transmitted from the server and plays it back.

[1319] Input: Audio data sent from the server

[1320] Output: Played audio information

[1321] Specific operation: Play audio data through the landline phone speaker to provide information to the user.

[1322] (Application example 2)

[1323] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1324] Modern digital services are designed primarily for young people and digital natives, and are often difficult to use for the elderly and users who are not familiar with digital technology. Food delivery services also require the use of smartphones and the Internet, which makes them difficult for the elderly to use. Furthermore, although the number of services using voice recognition technology is increasing, the accuracy and security of user authentication may not be sufficient.

[1325] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1326] In this invention, the server includes a means for providing a messenger app internet search tool that can be used from text, voice, and image, a means for covering information on weather, disasters, news, EC, finance, etc., a means for enabling use from a smartphone messenger app, and a means for accepting voice orders from a landline in a food delivery service, a means for authenticating a user by AI voiceprint authentication and providing recommended menu items taking into consideration past order history and preferences, and a means for confirming the order details by voice and finally confirming the order. This enables even elderly people and users who are not familiar with digital to use food delivery services safely and easily using a landline.

[1327] A "text, voice and image messaging app internet search tool" is an application that allows users to search for information and send and receive messages using text, voice and images.

[1328] "Means of covering information on weather, disasters, news, e-commerce, finance, etc." refers to a function that comprehensively provides various types of information, such as weather forecasts, disaster information, the latest news, e-commerce information, and financial information.

[1329] "Means enabling use from a messenger app on a smartphone" refers to a function that enables users to use the service through a messenger app running on their smartphone.

[1330] "A means for accepting voice orders from a landline telephone in a food delivery service" refers to a function that enables a user to place a food delivery order by voice using a landline telephone.

[1331] "A means of authenticating users through AI voiceprint recognition and providing recommended menu items taking into consideration their past order history and preferences" refers to a function that identifies users through voiceprint recognition technology using artificial intelligence, and presents recommended menu items based on the user's past order history and preferences.

[1332] "A means for confirming order details by voice and finally confirming the order" is a function that supports the process in which a user confirms the order details by voice and finally confirms the order.

[1333] A system for carrying out the present invention has the following configuration: The server includes means for providing a messenger app internet search tool that can be used from text, voice, and image, means for covering information on weather, disasters, news, EC, finance, etc., means for enabling use from a smartphone messenger app, means for accepting voice orders from a landline phone in a food delivery service, means for authenticating a user by AI voiceprint authentication and providing recommended menu items taking into consideration past order history and preferences, and means for confirming the order details by voice and finally confirming the order.

[1334] Hardware and Software Configuration

[1335] The server uses a voice recognition library (e.g., speech_recognition), a voice synthesis library (e.g., pyttsx3), an AI voiceprint authentication module (e.g., voice_authentication), and a food delivery service API (e.g., food_delivery_service). The user uses a landline phone to place an order by voice. The server captures the voice using a microphone and outputs the voice using a speaker.

[1336] Data processing and calculation

[1337] The server processes the data in the following steps:

[1338] 1. When a user requests information by voice from a landline phone, the server picks up the voice through a microphone.

[1339] 2. Use a speech recognition library to convert the captured audio into text.

[1340] 3. Use the AI ​​voiceprint recognition module to authenticate the user's voiceprint.

[1341] 4. If authentication is successful, use the food delivery service API to get menu recommendations that take into account past ordering history and user preferences.

[1342] 5. The recommended menu is converted into voice using a speech synthesis library and provided to the user through the speaker.

[1343] 6. The user confirms the order details by voice and finally confirms the order.

[1344] Examples

[1345] For example, when a user makes a voice request from a landline saying, "What's the recommendation today?", the server captures the voice and converts it into text using a voice recognition library. Next, the AI ​​voiceprint authentication module authenticates the user and obtains menu recommendations using the food delivery service API. The obtained menu is converted into speech using a speech synthesis library and provided to the user through the speaker. When the user confirms by voice, "I'd like to order a sushi set," the server confirms the order.

[1346] Examples of prompt statements

[1347] User: Hello, Food Delivery Assistant, what's your name?

[1348] System: This is Tanaka.

[1349] User: Hello Tanaka-san. Please place your order.

[1350] System: What's your recommendation today?

[1351] User: Today's recommendation is the sushi set. Would you like to order the sushi set?

[1352] System: Yes.

[1353] User: Your order has been received, thank you.

[1354] The flow of the specific process in the application example 2 will be described with reference to FIG.

[1355] Step 1:

[1356] A user requests information by voice from a landline phone. The server captures the user's voice through a microphone. The input is the user's voice and the output is voice data.

[1357] Step 2:

[1358] The server converts the acquired voice data into text using a voice recognition library (e.g., speech_recognition). The input is voice data, and the output is text data.

[1359] Step 3:

[1360] The server uses an AI voiceprint authentication module (e.g. voice_authentication) to authenticate the user's voiceprint from the text data. The input is the text data, and the output is the authentication result. If the authentication is successful, proceed to the next step.

[1361] Step 4:

[1362] The server uses a food delivery service API (e.g., food_delivery_service) to obtain recommended menu items that take into account the user's past order history and preferences. The input is the user authentication result and past order history data, and the output is the recommended menu items.

[1363] Step 5:

[1364] The server converts the text data of the recommended menu into speech using a speech synthesis library (e.g., pyttsx3). The input is the text data of the recommended menu, and the output is speech data.

[1365] Step 6:

[1366] The server provides the user with a recommended menu by voice through a speaker. The input is voice data, and the output is voice output to the user.

[1367] Step 7:

[1368] The user confirms the order details by voice on the landline phone. The server again picks up the user's voice through the microphone. The input is the user's voice, and the output is voice data.

[1369] Step 8:

[1370] The server converts the acquired voice data into text using a voice recognition library. The input is voice data and the output is text data.

[1371] Step 9:

[1372] The server analyzes the text data and confirms the order details. The input is the text data, and the output is the confirmation result of the order details.

[1373] Step 10:

[1374] The server confirms the order details and confirms the order using the food delivery service API. The input is the confirmation result of the order details, and the output is a notification that the order has been confirmed.

[1375] Step 11:

[1376] The server converts the order confirmation notification into speech using a speech synthesis library. The input is the order confirmation notification, and the output is speech data.

[1377] Step 12:

[1378] The server notifies the user of the order confirmation by voice through the speaker. The input is voice data, and the output is voice output to the user.

[1379] In addition, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1380] "Example 1"

[1381] In one embodiment of the present invention, an emotion engine that recognizes the user's emotions is incorporated. This emotion engine analyzes the user's emotions from text, voice, and images sent by the user through the messenger app. For example, if a user sends the text "Today is the worst," the emotion engine recognizes from this text that the user is feeling negative emotions. Also, if a user sends an image of a smiling face, the emotion engine recognizes from this image that the user is feeling positive emotions. In this way, the emotion engine can accurately grasp the user's emotions and provide information and services according to the emotions.

[1382] "Example 2"

[1383] In another embodiment of the present invention, an emotion engine is incorporated as a means of supporting users who are not digital natives. This emotion engine analyzes emotions from the user's voice. For example, if an elderly person makes a voice request from a landline, "What's the news today?", the emotion engine analyzes the user's emotions from the tone and strength of the voice. In this way, the emotion engine can accurately grasp the user's emotions and provide information and services accordingly.

[1384] "Example 5"

[1385] In a further embodiment of the present invention, an emotion engine is incorporated as a means that supports AI voiceprint authentication and can be used by elderly people even on landlines. This emotion engine analyzes emotions from the user's voice. For example, if an elderly person makes a voice request on a landline such as "What's the weather today?", the emotion engine analyzes the user's emotions from the tone and strength of the voice. In this way, the emotion engine can accurately grasp the user's emotions and provide information and services accordingly.

[1386] The process flow of each embodiment will be described below.

[1387] "Example 1"

[1388] Step 1: User sends a text, voice or image via a messenger app.

[1389] Step 2: The emotion engine analyzes the user's emotions from the submitted text, voice, and images.

[1390] Step 3: The system provides information and services based on the emotions analyzed by the emotion engine.

[1391] "Example 2"

[1392] Step 1: A non-digital native user makes a request via voice.

[1393] Step 2: The emotion engine analyzes the emotion from the user's voice.

[1394] Step 3: The system provides information and services based on the emotions analyzed by the emotion engine.

[1395] "Example 5"

[1396] Step 1: The senior makes a voice request from a landline phone.

[1397] Step 2: The emotion engine analyzes the emotion from the user's voice.

[1398] Step 3: The system provides information and services based on the emotions analyzed by the emotion engine.

[1399] Example 1

[1400] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1401] Conventional messenger apps lack the ability to analyze text, voice, and image messages sent by users and provide the necessary information. They also lack the ability to recognize users' emotions and respond appropriately, limiting the user experience. Furthermore, these systems are difficult to use for non-digital native users and the elderly.

[1402] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1403] In this invention, the server includes a means for providing a messenger app Internet search tool that can be used from text, voice, and image, a means for covering information on weather, disasters, news, electronic commerce, finance, etc., a means for enabling use from a messenger app on a mobile information terminal, a means for analyzing a user's text, voice, and image, a means for acquiring necessary information from the Internet based on the analysis result, a means including an emotion engine for recognizing the user's emotion, and a means for providing appropriate information to the user based on the analysis result and the acquired information. This makes it possible to not only analyze data sent by the user and provide necessary information, but also recognize the user's emotion and take appropriate action.

[1404] "Text" refers to textual information that a user sends through a messenger app.

[1405] "Audio" refers to sound information that a user sends through a messenger app.

[1406] An "image" is visual information that a user sends through a messenger app.

[1407] A "messenger app internet search tool" is a tool that analyzes data sent through a messenger app and retrieves and provides necessary information from the internet.

[1408] "Weather" is information about the weather.

[1409] "Disasters" refers to information about natural and man-made disasters.

[1410] "News" is reporting on current events or information.

[1411] "Electronic commerce" means commercial transactions conducted over the Internet.

[1412] "Finance" refers to information regarding the operation and management of funds.

[1413] A "portable information terminal" is a portable information processing device such as a smartphone or tablet.

[1414] "Analysis" is the process of analyzing the transmitted data to make sense of it.

[1415] "Obtaining from the Internet" refers to searching for and obtaining the necessary information via the Internet.

[1416] An "emotion engine" is software for analyzing emotions from data sent by users.

[1417] "Providing appropriate information" refers to responding with information that is useful to the user based on the analysis results and the information obtained.

[1418] This invention is a system that analyzes text, voice, and images sent by users through messenger apps and provides necessary information. It also incorporates an emotion engine that recognizes the user's emotions.

[1419] Hardware and Software Used

[1420] Hardware: Mobile information terminals (smartphones, tablets), servers

[1421] Software: Messenger app, Google Cloud Natural Language API, Amazon Rekognition, OpenWeatherMap API, IBM Watson Tone Analyzer

[1422] Data processing and calculation

[1423] 1. User Input

[1424] Users use messenger apps on their mobile devices to send text, voice, and images.

[1425] Example: A user enters the text "What's the weather like tomorrow?" and presses the submit button.

[1426] 2. Data transfer

[1427] The terminal transfers the data sent by the user to a server via the Internet.

[1428] What happens: The messenger app sends data to the server as an HTTP request.

[1429] 3. Data Analysis

[1430] The server analyzes the received text, audio, and images.

[1431] The analysis uses Google Cloud Natural Language API to analyze text and Amazon Rekognition to analyze images.

[1432] Example: Parsing the text "What's the weather like tomorrow?" to recognize that it is a request for weather forecast information.

[1433] 4. Acquisition of information

[1434] The server retrieves the necessary information from the Internet based on the analysis results.

[1435] Specific software: Obtains weather forecast information using the OpenWeatherMap API.

[1436] Example: In response to the request "What's the weather going to be like tomorrow?", weather forecast information is retrieved from the OpenWeatherMap API.

[1437] 5. Emotion Recognition

[1438] The server uses an emotion engine to analyze emotions from the user's text, voice, and images.

[1439] Specific software: Analyzing the sentiment of text using IBM Watson Tone Analyzer.

[1440] Example: If a user sends the text "Today is the worst", the sentiment engine will recognize the negative sentiment.

[1441] 6. Provision of information

[1442] The server provides the user with appropriate information based on the analysis results and the acquired information.

[1443] Specific behavior: The server generates the text "Tomorrow's weather will be sunny" and returns it to the user.

[1444] Example: If the user asks "What's the weather going to be like tomorrow?", the server responds with "The weather will be sunny tomorrow."

[1445] Examples of prompt statements

[1446] "Please explain the system that analyzes the text, voice, and images sent by the user through the messenger app and provides the necessary information. Please also specify the names of the specific software and APIs."

[1447] In this way, the present invention is a system that not only analyzes data sent by a user and provides necessary information, but also recognizes the user's emotions and takes appropriate action.

[1448] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1449] Step 1:

[1450] Users send text, voice, and images through messenger apps

[1451] Input: A user inputs text, voice, or images into a messenger app.

[1452] Specific behavior: The user enters the text "What's the weather like tomorrow?" and presses the send button.

[1453] Output: The messenger app prepares to send the input data.

[1454] Step 2:

[1455] The device forwards the transmitted data to the server.

[1456] Input: The data sent when the user presses the submit button.

[1457] What happens: The messenger app sends text data to the server as an HTTP request.

[1458] Output: The server receives the data.

[1459] Step 3:

[1460] The server analyzes the data it receives

[1461] Input: Text, audio, and image data sent from the device.

[1462] What happens next: The server uses the Google Cloud Natural Language API to parse the text and Amazon Rekognition to analyze the images.

[1463] Output: The analysis results include the user's request and the image content.

[1464] Example: Parsing the text "What's the weather like tomorrow?" to recognize that it is a request for weather forecast information.

[1465] Step 4:

[1466] The server retrieves the necessary information from the Internet.

[1467] Input: The user's request obtained as a result of analysis.

[1468] Specific behavior: The server retrieves weather forecast information using the OpenWeatherMap API.

[1469] Output: The desired information (e.g. weather forecast information) is obtained.

[1470] Example: In response to the request "What's the weather going to be like tomorrow?", weather forecast information is retrieved from the OpenWeatherMap API.

[1471] Step 5:

[1472] The server uses an emotion engine to analyze the user's emotions.

[1473] Input: Text, audio, and image data submitted by the user.

[1474] What happens: The server uses IBM Watson Tone Analyzer to analyze the sentiment of the text.

[1475] Output: The user's emotional state (e.g. positive, negative).

[1476] Example: If a user sends the text "Today is the worst", the sentiment engine will recognize the negative sentiment.

[1477] Step 6:

[1478] The server provides the user with appropriate information based on the analysis results and the acquired information.

[1479] Input: Analysis results and retrieved information.

[1480] Specific behavior: The server generates the text "Tomorrow's weather will be sunny" and returns it to the user.

[1481] Output: The user is provided with the appropriate information.

[1482] Example: If the user asks "What's the weather going to be like tomorrow?", the server responds with "The weather will be sunny tomorrow."

[1483] (Application example 1)

[1484] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1485] Conventional messenger apps only had limited functionality for analyzing the content of text, voice, and images sent by users and providing appropriate information. In addition, they lacked functionality for analyzing user emotions and taking appropriate action in emergencies, making them difficult to use in situations that required the provision of security-related information or emergency responses. Furthermore, their response to users who are not digital natives and the elderly was insufficient.

[1486] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1487] In this invention, the server includes a means for providing a messenger app Internet search tool that can be used from text, voice, and image, a means for covering information on weather, disasters, news, EC, finance, etc., a means for enabling use from a smartphone messenger app, a means for analyzing user emotions and taking appropriate action in an emergency, and a means for providing information on security. This makes it possible to analyze the text, voice, and image sent by the user and provide appropriate information, and in particular to provide information on security and respond quickly to emergencies. In addition, support for users who are not digital natives and the elderly is also strengthened.

[1488] A "messenger app internet search tool that can be used with text, voice, and images" is a messenger app that has the function of analyzing the text, voice, and images sent by the user and searching for and providing information on the internet.

[1489] "Means of covering information on weather, disasters, news, e-commerce, finance, etc." refers to the function of comprehensively providing various types of information, such as weather forecasts, disaster information, the latest news, information on e-commerce, and financial information.

[1490] "Means enabling access from a smartphone's messenger app" refers to a function that enables users to access various information through a messenger app running on their smartphone.

[1491] "A means of analyzing user emotions and taking appropriate action in an emergency" refers to a function that analyzes emotions from the text, voice, and images sent by the user, and takes appropriate action based on those emotions in the event of an emergency.

[1492] "Means of providing security-related information" refers to the function of analyzing information sent by users and providing appropriate security-related information and countermeasures.

[1493] "Measures to accommodate non-digital native users" refers to functions and interfaces that make it easy to use even for users who are not familiar with digital technology.

[1494] "A means that supports AI voiceprint authentication and allows elderly people to use the system via landline telephone" refers to a function that uses voiceprint authentication technology using artificial intelligence to enable elderly people to use the system via landline telephone.

[1495] The system for implementing this invention has the following configuration: First, a user sends text, voice, and images through a messenger app on their smartphone. These data are sent to a server, which then analyzes each data.

[1496] The server uses natural language processing libraries (e.g. TextBlob), speech recognition libraries (e.g. Google Speech-to-Text API), and image analysis libraries (e.g. PIL) for text, voice, and image analysis to extract the necessary information from the data submitted by the user and generate an appropriate response.

[1497] In addition, the server uses an emotion analysis engine to analyze the user's emotions. This emotion analysis engine determines the user's emotions from text, voice, and images, and takes appropriate action in the event of an emergency. For example, if the user sends a voice message saying "help," the server will send a notification to the emergency contact.

[1498] In order to provide information on security, the server analyzes the information sent by the user and provides appropriate security information and countermeasures. For example, if a user sends an image of a suspicious person, the server analyzes the image and notifies the police.

[1499] The system also takes into consideration users who are not digital natives and the elderly. The server uses AI voiceprint authentication technology to allow elderly people to use the system through a landline, making it easy for users who are not familiar with digital technology to use the system.

[1500] For example, if a user sends a text saying "I forgot my house keys," the server will analyze the text, search for nearby locksmiths and provide their contact information. If a user sends a text saying "I'm scared," the emotion analysis engine will recognize the negative emotion and take emergency action.

[1501] Examples of prompts to input to a generative AI model include:

[1502] If a user sends a text saying "I forgot my house keys", the sentiment engine should parse the user sentiment from this text and generate a response to search for a nearby locksmith.

[1503] In this way, a system can be realized that analyzes the text, voice, and images sent by users and provides appropriate information and takes emergency measures.

[1504] The flow of the specific process in the application example 1 will be described with reference to FIG.

[1505] Step 1:

[1506] A user sends text, voice, and images through a messenger app on a smartphone. The input is the text, voice, and image data sent by the user, and the output is the transmission of these data to a server. A specific operation is when a user enters the text "I forgot my house key" and presses the send button.

[1507] Step 2:

[1508] The server analyzes the text, voice, and image data it receives. The input is the data sent by the user, and the output is the analysis result. Specifically, the server analyzes the text using a natural language processing library (e.g. TextBlob), converts the voice to text using a speech recognition library (e.g. Google Speech-to-Text API), and analyzes the image using an image analysis library (e.g. PIL).

[1509] Step 3:

[1510] The server analyzes the user's emotions using the emotion analysis engine. The input is the text, voice, and image data analyzed in step 2, and the output is the user's emotional state. In concrete terms, the server uses the emotion analysis engine to determine the user's emotions from the text and voice, and recognizes neutral emotions from the text, for example, "I forgot my house keys."

[1511] Step 4:

[1512] The server provides appropriate information based on the analysis results. The input is the analysis results of steps 2 and 3, and the output is the information provided to the user. In concrete terms, the server searches for a nearby locksmith for the text "I forgot my house keys" and provides the user with their contact information.

[1513] Step 5:

[1514] The server provides security information. The input is the data sent by the user and the analysis results, and the output is security information and countermeasures. In concrete terms, when a user sends an image of a suspicious person, the server analyzes the image and notifies the police.

[1515] Step 6:

[1516] The server takes appropriate action in case of emergency. The input is the result of the emotion analysis engine, and the output is the execution of emergency response. Specifically, when the user voices "help," the server sends a notification to the emergency contact.

[1517] Step 7:

[1518] The server supports users who are not digital natives and the elderly. The input is the user's usage status and voice data, and the output is a simple operation interface and voice authentication results. In concrete terms, the server uses AI voiceprint authentication technology to enable the elderly to use the system through a landline phone.

[1519] In this way, a system is realized that analyzes the text, voice, and images sent by the user and provides appropriate information and responds to emergencies.

[1520] Example 2

[1521] Next, a second embodiment of the second form will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1522] Modern information provision systems are often difficult to use for non-digital native users and the elderly. In addition, there is a lack of means to provide appropriate information through voice recognition and emotion analysis to users who use landlines. For this reason, there is a demand for information provision systems that improve user convenience and are easy to use, especially for non-digital native users and the elderly.

[1523] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1524] In this invention, the server includes a means for providing a messenger app internet search tool that can be used from text, voice, and image, a means for covering information such as weather, disasters, news, e-commerce, and finance, a means for enabling use from a messenger app of a communication terminal, a means for receiving voice data through a landline and authenticating a user using an AI voiceprint authentication system, a means for analyzing the contents of a user's request and acquiring necessary information if authentication is successful, a means for converting the acquired information into voice data using a voice synthesis system and providing it to the user through a landline, and a means for analyzing emotions from the user's voice and providing information and services accordingly. This allows even users who are not digital natives or the elderly to easily acquire information through a landline and receive appropriate services.

[1525] A "text, voice and image messaging app internet search tool" is an application that allows users to search for information and send and receive messages using text, voice and images.

[1526] "Information on weather, disasters, news, e-commerce, finance, etc." refers to meteorological information, information on natural disasters, the latest news, information on online shopping, and information on financial markets and transactions.

[1527] A "communication terminal" is a device for communicating via the Internet or telephone lines, including smartphones and landline phones.

[1528] A "fixed line" is a wired communications line for carrying analog or digital telephone communications.

[1529] "Voice Data" means data that has been converted into digital form from a User's voice.

[1530] The "AI voiceprint authentication system" is an artificial intelligence system that analyzes the user's voice and authenticates individuals based on their voiceprints.

[1531] "User authentication" is the process by which a system verifies a user's identity and grants access rights.

[1532] "Request analysis" is the process of understanding the user's intent and the information they are looking for from the voice or text they provide.

[1533] "Obtaining information" means collecting the necessary data or information from external databases or APIs based on user requests.

[1534] A "voice synthesis system" is a technology that converts text data into voice data.

[1535] "Emotion analysis" is the process of determining the user's emotional state from their voice and taking appropriate action based on that information.

[1536] "Provision of information and services" refers to providing users with necessary information or specific services.

[1537] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[1538] This invention provides an information provision system that can be used by non-digital native users and elderly people. Specifically, this system uses AI voiceprint authentication and a voice synthesis system to provide information to users who request information by voice using a landline phone.

[1539] Server Roles

[1540] The server receives voice requests from users through landline telephone lines. The received voice data is converted into digital format using a voice modem. The server then sends this voice data to an AI voiceprint authentication system to authenticate the user. This AI voiceprint authentication system uses common voice recognition software (e.g., voice recognition API). If authentication is successful, the server analyzes the content of the user's request and obtains the required information. For example, to obtain news information, the server uses a news API to obtain the latest news data. The server then converts the obtained information into voice data using a voice synthesis system (e.g., voice synthesis API) and provides it to the user through the landline telephone line.

[1541] Terminal Roles

[1542] The terminal receives the user's voice request via a landline telephone line. The received voice data is sent to a voice recognition module in the terminal and converted into text data. This voice recognition module uses general voice recognition software (e.g., voice recognition API). The converted text data is sent to a server, where the user is authenticated by an AI voiceprint authentication system. If authentication is successful, the terminal converts the information obtained from the server into voice data using a voice synthesis system (e.g., voice synthesis API) and provides it to the user via the landline telephone line.

[1543] User Roles

[1544] A user requests information by voice using a landline. For example, the user may make a request such as, "What's in the news today?" This voice request is sent to the server via the landline. The server authenticates the user using an AI voiceprint authentication system, and if authentication is successful, retrieves the information requested by the user. The retrieved information is converted into voice data using a voice synthesis system and provided to the user via the landline. The user can obtain the required information by listening to the provided voice information.

[1545] Examples

[1546] Example 1: Requesting news information

[1547] The user makes a voice request from a landline saying, "What's in the news today?" The server receives this voice data and authenticates the user using an AI voiceprint authentication system (e.g., voice recognition API). If authentication is successful, the server uses a news API to obtain the latest news data. The obtained news data is converted into voice data using a voice synthesis system (e.g., voice synthesis API) and provided to the user via the landline.

[1548] Example 2: Requesting weather information

[1549] A user makes a voice request from a landline saying, "What's the weather today?" The server receives this voice data and authenticates the user using an AI voiceprint authentication system (e.g., voice recognition API). If authentication is successful, the server uses a weather API to obtain the latest weather data. The obtained weather data is converted into voice data using a voice synthesis system (e.g., voice synthesis API) and provided to the user via the landline.

[1550] Examples of prompt statements

[1551] "Please explain in detail from the perspectives of the server, the terminal, and the user the process of a system that uses AI voiceprint authentication and a voice synthesis system to provide information to users who request information by voice from a landline phone. Please also add specific actions for each processing step."

[1552] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1553] Step 1:

[1554] The server receives voice requests from users over landline telephone lines.

[1555] Input: Voice data from a landline

[1556] Specific operation: The server uses a voice modem to convert analog voice signals into digital voice data.

[1557] Output: Digital audio data

[1558] Step 2:

[1559] The server sends the received voice data to the AI ​​voiceprint authentication system to authenticate the user.

[1560] Input: Digital audio data

[1561] Specific operation: The server calls the voice recognition API and converts the voice data into text data, which is then sent to the AI ​​voiceprint authentication system.

[1562] Output: Authentication result

[1563] Step 3:

[1564] The server receives the authentication results from the AI ​​voiceprint authentication system and authenticates the user.

[1565] Input: Authentication result

[1566] Specific operation: The server analyzes the authentication results returned by the AI ​​voiceprint authentication system and determines whether the authentication was successful.

[1567] Output: Authentication success or failure flag.

[1568] Step 4:

[1569] If the authentication is successful, the server analyzes the user's request and obtains the necessary information.

[1570] Input: Authentication success flag, voice data

[1571] Specific operation: The server uses a speech recognition API to convert the voice data into text data and analyzes the user's request. Based on the analysis results, the server retrieves the required information from external databases such as news APIs and weather APIs.

[1572] Output: Acquired information data

[1573] Step 5:

[1574] The server converts the acquired information into voice data using a voice synthesis system.

[1575] Input: Acquired information data

[1576] Specific operation: The server uses a speech synthesis API to convert the text data into speech data.

[1577] Output: Audio data

[1578] Step 6:

[1579] The server provides the generated voice data to the user via a fixed telephone line.

[1580] Input: Audio data

[1581] What it does: The server uses a voice modem to convert the digital voice data into an analog voice signal and transmit it to the user over a landline.

[1582] Output: Audio information provided to the user

[1583] Step 7:

[1584] A user requests information by voice using a landline telephone.

[1585] Input: User's voice request

[1586] Specific actions: A user picks up a landline phone receiver and speaks a request such as "What's in the news today?"

[1587] Output: Voice data sent over a landline

[1588] Step 8:

[1589] The user obtains the required information by listening to the provided audio information.

[1590] Input: Voice information provided via a landline telephone line

[1591] Specific operation: The user listens to news and weather information provided through a landline telephone receiver and obtains the required information.

[1592] Output: Information the user gets

[1593] (Application example 2)

[1594] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1595] When non-digital native users and the elderly use landlines to obtain information, there are problems such as complex operations and difficulty in authentication. In addition, there is an issue that the user experience is not improved because appropriate information is not provided according to the user's emotions.

[1596] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1597] In this invention, the server includes a means for providing a messenger app internet search tool that can be used from text, voice, and image, a means for covering information on weather, disasters, news, EC, finance, etc., a means for enabling use from a messenger app on a smartphone, a means for requesting information by voice from a landline, a means for performing AI voiceprint authentication, a means including an emotion engine that analyzes emotions from the user's voice, and a means for providing information according to emotions. This allows even users who are not digital natives or the elderly to easily obtain information using a landline and provide appropriate information according to emotions.

[1598] A "text, voice and image enabled messenger app internet search tool" is a tool that allows users to search for information using text, voice and images and is accessible through a messenger app.

[1599] "Means that cover a wide range of information, including weather, disasters, news, e-commerce, and finance" refers to means that comprehensively provide a wide range of information, such as weather forecasts, disaster information, the latest news, e-commerce information, and financial information.

[1600] "Means enabling use from a messenger app on a smartphone" refers to means that enable information to be obtained or services to be used via a messenger app installed on a smartphone.

[1601] A "means for requesting information by voice from a landline telephone" is a means by which a user can request information by voice using a landline telephone.

[1602] "Means for performing AI voiceprint authentication" refers to a means for using artificial intelligence to authenticate a user's voiceprint and confirm the user's identity.

[1603] "Means including an emotion engine that analyzes emotions from a user's voice" refers to means including an emotion engine that can analyze user's voice data and read emotions from the voice.

[1604] A "means for providing information according to emotions" is a means for providing appropriate information according to the user's emotional state.

[1605] A system for carrying out this invention has the following configuration: The server includes a means for providing a messenger app internet search tool that can be used from text, voice, and image, a means for covering information on weather, disasters, news, EC, finance, etc., a means for enabling use from a messenger app on a smartphone, a means for requesting information by voice from a landline phone, a means for performing AI voiceprint authentication, a means including an emotion engine that analyzes emotions from the user's voice, and a means for providing information according to emotions.

[1606] Hardware and Software Used

[1607] Hardware: Smartphones, landlines

[1608] Software: Python, speech_recognition library, pyttsx3 library, custom emotion_recognition module, voice_authentication module

[1609] Data processing and calculation

[1610] The server first receives voice input from a landline phone. This voice data is converted to text using the speech_recognition library. Then, it authenticates the user's voiceprint from the voice data using the voice_authentication module. If authentication is successful, it analyzes the user's emotions from the voice data using the emotion_recognition module.

[1611] Based on the analyzed emotions, the server selects appropriate information and provides it by voice using the pyttsx3 library. This series of processes enables even non-digital native users and elderly people to easily obtain information using a landline phone and receive appropriate information according to their emotions.

[1612] Examples

[1613] For example, if a user makes a voice request from a landline saying, "What is the security situation at home?", the server will perform voice authentication and emotion analysis and respond by voice with, "The current security situation is normal." In this way, the user can obtain information with peace of mind.

[1614] Examples of prompt statements

[1615] When a user makes a voice request from a landline phone saying, "What is the security status at home?", the system should perform voice recognition and emotion analysis and provide emotion-based security information via voice.

[1616] This invention makes it possible for even non-digital native users and elderly people to easily obtain information using a landline phone and to be provided with appropriate information according to their emotions.

[1617] The flow of the specific process in the application example 2 will be described with reference to FIG.

[1618] Step 1:

[1619] A user requests information by voice from a landline phone. The user uses a landline phone to make a request by voice, for example, "What is the security situation at my home?" This voice data is input into the system.

[1620] Step 2:

[1621] The server receives the voice data and converts the voice to text using the speech_recognition library. The input is voice data and the output is text data. This conversion converts the voice data into analyzable text data.

[1622] Step 3:

[1623] The server uses the voice_authentication module to authenticate the user's voiceprint from the voice data corresponding to the text data. The input is the voice data, and the output is the authentication result. This authentication confirms the user's identity.

[1624] Step 4:

[1625] If the authentication is successful, the server uses the emotion_recognition module to analyze the user's emotions from the voice data. The input is the voice data, and the output is the emotion data. This analysis allows the user's emotional state to be understood.

[1626] Step 5:

[1627] The server selects appropriate information based on the analyzed emotion data. For example, if the user expresses anxiety, the server selects reassuring information such as "The current security situation is normal." The input is emotion data, and the output is the selected information.

[1628] Step 6:

[1629] The server converts the selected information into audio using the pyttsx3 library and provides it to the user through a landline phone. The input is the selected information and the output is audio data. This conversion allows the user to receive the information by audio.

[1630] Step 7:

[1631] The user receives the voice information provided via a landline phone and obtains the information he or she needs, allowing the user to obtain information with peace of mind.

[1632] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input for the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1633] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1634] Another example of generative AI is Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) are mentioned.

[1635] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1636] [Fourth embodiment]

[1637] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1638] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1639] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[1640] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. In addition, the microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1641] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[1642] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[1643] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[1644] The control target 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, legs, etc. The posture and behavior of the robot 414 are controlled by controlling the motors of the arms, hands, legs, etc. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1645] Fig. 8 shows an example of main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1646] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1647] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1648] In the robot 414, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1649] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1650] "Example 1"

[1651] The present invention provides a messenger app internet search tool that can use text, voice, and images. Specifically, when a user sends text, voice, or image through a messenger app, the system analyzes it and provides the required information. For example, when a user sends a text such as "What's the weather like tomorrow?", the system retrieves weather forecast information and provides it to the user.

[1652] "Example 2"

[1653] Furthermore, the present invention supports AI voiceprint authentication to enable use by non-digital native users and elderly people. Specifically, when a user requests information by voice from a landline phone, the system performs AI voiceprint authentication to authenticate the user. If authentication is successful, the information requested by the user is provided by voice. For example, when an elderly person requests by voice from a landline phone, "What's in the news today?", the system obtains the news information and provides it by voice.

[1654] The process flow of each embodiment will be described below.

[1655] "Example 1"

[1656] Step 1: User sends a text, voice or image via a messenger app.

[1657] Step 2: The system analyzes the transmitted text, audio and images.

[1658] Step 3: Based on the analysis result, the system obtains the required information. For example, if the user sends a text "What's the weather like tomorrow?", the system will obtain weather forecast information.

[1659] Step 4: Provide the retrieved information to the user.

[1660] "Example 2"

[1661] Step 1: User requests information by voice over a landline phone.

[1662] Step 2: The system analyzes the voice and performs AI voiceprint authentication.

[1663] Step 3: If the authentication is successful, the system retrieves the information requested by the user. For example, if an elderly person requests "What's the news today?" by voice, the system retrieves the news information.

[1664] Step 4: Provide the retrieved information to the user via voice.

[1665] Example 1

[1666] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1667] Conventional messenger apps are limited to sending text, voice, and images, making it difficult to provide users with the information they need quickly and accurately. In addition, these apps are difficult to use for non-digital native users and elderly people. Furthermore, voice recognition and information provision functions are lacking for elderly people who use landlines. To solve these issues, a system that analyzes data sent by users and provides the necessary information is required.

[1668] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1669] In this invention, the server includes a means for receiving data sent by a user through a messenger app, a means for using a natural language processing engine to analyze the received data, and a means for acquiring necessary information based on the analysis result, which makes it possible to analyze the text, voice, and image sent by the user and provide the necessary information quickly and accurately.

[1670] "Text" refers to textual information that a user sends through a messenger app.

[1671] "Audio" refers to the sound information that a user sends through a messenger app.

[1672] An "image" is any visual information that a user sends through a messenger app.

[1673] "Messenger App Internet Search Tool" is an application that allows users to send text, voice, and images, which are then analyzed to provide the necessary information.

[1674] "Information on weather, disasters, news, e-commerce, finance, etc." are a wide range of information categories that users desire.

[1675] A "smartphone" is a portable information terminal that can connect to the Internet and use applications.

[1676] A "server" is a computer system that receives data sent by a user, analyzes it, and provides the necessary information.

[1677] A "natural language processing engine" is software that analyzes text data and understands its meaning and intent.

[1678] The "analysis results" are information obtained by the natural language processing engine analyzing text data.

[1679] "Means of obtaining information" refers to the process of obtaining the necessary information from external information sources based on the analysis results.

[1680] "Means of returning information" is the process by which retrieved information is provided to the user.

[1681] "Non-digital native users" are users who are not familiar with digital technology.

[1682] "Artificial intelligence voiceprint authentication" is a technology that analyzes voice to recognize the speaker.

[1683] "Elderly people" are older users who may have difficulty using digital technology.

[1684] A "fixed-line telephone" is a telephone that is fixed and cannot be moved.

[1685] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[1686] The present invention is a system that analyzes text, voice, and images sent by a user through a messenger app and provides necessary information. A specific embodiment of the system will be described below.

[1687] 1. System Configuration

[1688] The system consists of the following main components:

[1689] User device: The device through which a user uses the messenger app, such as a smartphone or landline phone.

[1690] Server: A computer system that receives data sent by users, analyzes it, and provides the required information.

[1691] Natural language processing engine: Software for analyzing text data and understanding its meaning and intent. Specifically, it uses natural language processing APIs.

[1692] Speech recognition software: Software for converting voice data into text, specifically using a speech recognition API.

[1693] Image recognition software: Software for analyzing image data. Specifically, it uses an image recognition API.

[1694] Information Acquisition API: An API for acquiring necessary information from external information sources based on the analysis results. Specifically, a weather forecast API is used.

[1695] 2. Program Processing

[1696] 1. User Input

[1697] User opens the Messenger app and sends a text, voice, or image.

[1698] Example: A user enters the text "What's the weather like tomorrow?" and presses the submit button.

[1699] 2. Receiving Data

[1700] The server receives data sent from the messenger app in real time and temporarily stores the received data in a database.

[1701] 3. Data Analysis

[1702] The server calls a natural language processing engine to analyze the data it receives. The API analyzes the text and understands the intent.

[1703] In the case of voice data, the server uses voice recognition software to convert the voice to text.

[1704] In the case of image data, the server uses image recognition software to analyze the image.

[1705] Example: The server sends the text "What's the weather like tomorrow?" to a natural language processing engine, and obtains the intent "I'm looking for a weather forecast" as the analysis result.

[1706] 4. Acquisition of information

[1707] Based on the analysis results, the server calls an information retrieval API and retrieves the specified information.

[1708] Example: Based on the analysis result of "What's the weather going to be tomorrow?", the server calls a weather forecast API and obtains the information "Tomorrow's weather will be sunny."

[1709] 5. Provision of information

[1710] The server returns the acquired information to the user. When returning the information, the notification function of the messenger app is used to display the information to the user.

[1711] Example: The server notifies the user that "tomorrow's weather will be sunny" via a messenger app and displays the information on the user's screen.

[1712] 3. Examples and prompts

[1713] Example 1: Text questions

[1714] User: "What's the weather like tomorrow?"

[1715] Server: Analyzes the text, retrieves weather forecast information, and replies to the user, "Tomorrow's weather will be sunny."

[1716] Example 2: Audio questions

[1717] User: Asks out loud, "What's the weather like tomorrow?"

[1718] Server: Converts the speech to text, analyzes it to obtain weather forecast information, and responds to the user, "Tomorrow's weather will be sunny."

[1719] Example 3: Image Question

[1720] User: Send weather related images.

[1721] Server: Analyzes the image, retrieves relevant weather forecast information, and responds to the user, "The weather in this area tomorrow will be sunny."

[1722] Example prompts for generative AI models

[1723] "If a user sends a text through your messenger app saying, 'What's the weather going to be like tomorrow?', please explain how you would get the weather forecast information and respond to the user."

[1724] In this way, a system can be created that analyzes the text, voice, and images sent by users through messenger apps and provides the necessary information.

[1725] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1726] Step 1:

[1727] User Input

[1728] How it works: A user opens a messaging app on their smartphone and sends a text, voice, or image.

[1729] Input: The user enters the text "What's the weather going to be like tomorrow?" and presses the submit button.

[1730] Output: The messenger app sends text data to the server.

[1731] Step 2:

[1732] Receiving data

[1733] How it works: The server receives data sent from the messenger app in real time, and temporarily stores it in a database.

[1734] Input: Text data sent from a messenger app: "What's the weather going to be like tomorrow?"

[1735] Output: The server prepares the received text data for analysis.

[1736] Step 3:

[1737] Analyzing data

[1738] How it works: The server calls a natural language processing engine to analyze the data it receives. The API analyzes the text and understands the intent. If it's voice data, the server uses voice recognition software to convert the speech to text. If it's image data, the server uses image recognition software to analyze the image.

[1739] Input: Text data received by the server: "What's the weather going to be like tomorrow?"

[1740] Output: The natural language processing engine returns the analysis result "You are looking for a weather forecast."

[1741] Step 4:

[1742] Obtaining information

[1743] Operation: Based on the analysis results, the server calls the information retrieval API and retrieves the specified information.

[1744] Input: Parsing result from the natural language processing engine: "I'm looking for a weather forecast."

[1745] Output: The information retrieval API returns the information "Tomorrow's weather will be sunny."

[1746] Step 5:

[1747] Provision of information

[1748] Behavior: The server returns the retrieved information to the user, using the notification feature of the messenger app to display the information to the user.

[1749] Input: Information from the information retrieval API: "Tomorrow's weather will be sunny."

[1750] Output: The server notifies the user that "tomorrow's weather will be sunny" through the messenger app and displays it on the user's screen.

[1751] In this way, a system can be created that analyzes the text, voice, and images sent by users through messenger apps and provides the necessary information.

[1752] (Application example 1)

[1753] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1754] Conventional messenger applications have the ability to provide information by analyzing text, voice, and images, but in autonomous vehicles, it is difficult to obtain information about the vehicle's status and surroundings in real time. In addition, there was a lack of functionality to provide information in response to user questions using voice or image recognition, which limited the user experience.

[1755] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means. In this invention, the server includes a means for providing a messenger app Internet search tool that can be used from text, voice, and image, a means for covering information on weather, disasters, news, EC, finance, etc., a means for enabling use from a smartphone messenger app, a means for acquiring the vehicle status and surrounding information in real time as an application installed in the autonomous driving vehicle, and a means for providing information in response to a user's question using voice recognition and image recognition. This makes it possible for a user to acquire necessary information in real time through text, voice, and image even in an autonomous driving vehicle.

[1756] "Text" is a means of expressing information using letters and symbols.

[1757] "Voice" is the means of recording and reproducing the human voice and other sounds as digital data.

[1758] An "image" is a means of recording and displaying visual information as digital data.

[1759] A "messenger app" is a software application that enables users to send and receive text, voice, and images.

[1760] An "Internet search tool" is software that searches for information on the Internet and provides it to users.

[1761] "Weather" is information that refers to the weather conditions in a particular area.

[1762] "Disaster" refers to information about damage caused by natural phenomena or man-made factors.

[1763] "News" is content intended to report current events or information.

[1764] "EC" is an abbreviation for electronic commerce, the act of buying and selling goods and services via the Internet.

[1765] "Finance" refers to information relating to the management, operation, and trading of funds.

[1766] A "smartphone" is a mobile phone that has Internet connectivity and a variety of applications.

[1767] An "autonomous vehicle" is a vehicle that drives autonomously using artificial intelligence and sensor technology.

[1768] "Real time" refers to a state in which processing and information provision are carried out immediately without delay.

[1769] "Speech recognition" is a technology that analyzes voice as digital data and converts it into text.

[1770] "Image recognition" is a technology that analyzes images and extracts specific information.

[1771] "User" refers to a person who uses a system or application.

[1772] A system for implementing this invention includes a means for providing a messenger app Internet search tool that can be used from text, voice, and images, a means for covering information on weather, disasters, news, e-commerce, finance, etc., a means for enabling use from a smartphone messenger app, a means for acquiring vehicle status and surrounding information in real time as an application to be installed in an autonomous vehicle, and a means for providing information in response to a user's questions using voice recognition and image recognition.

[1773] The server analyzes the text, voice, and images sent by the user and provides the necessary information. Specifically, when a user asks, "Where is the next gas station?", the server uses voice recognition technology to convert the voice into text, analyzes the text, and obtains the location information of the nearest gas station, which it then provides to the user.

[1774] To realize this system, the following hardware and software are used.

[1775] Hardware: Smartphones, microphones, cameras, self-driving vehicles

[1776] Software: Python, Requests library, SpeechRecognition library, Pillow library, Pytesseract library

[1777] The server uses the SpeechRecognition library for voice recognition and the Pytesseract library for image recognition, so that even if the user inputs an audio file or an image file, it can properly analyze it and provide information.

[1778] As a concrete example, consider the case where a user inputs a voice file saying "What's the weather tomorrow?" The server analyzes the voice file and converts it into text. It then uses WeatherAPI to get weather information for the specified location and provides it to the user.

[1779] Examples of prompt sentences to be input to a generative AI model include the following:

[1780] "The user has input an audio file 'weather_query.wav'. Write a program that parses this audio file and provides weather information."

[1781] In this way, users can obtain the information they need in real time through text, audio and images.

[1782] The flow of the specific process in the application example 1 will be described with reference to FIG.

[1783] Step 1:

[1784] The user inputs text, an audio file, or an image file. The input data is sent to the server. Inputs include text messages, audio files (e.g., .wav format), and image files (e.g., .jpg format).

[1785] Step 2:

[1786] The server determines the format of the input data. Specifically, it analyzes the file extension and header information of the data to determine whether the input is text, audio, or image. The input is the data sent by the user, and the output is the data format information.

[1787] Step 3:

[1788] The server selects the appropriate analysis method depending on the format of the input data. If it is text, it proceeds with the analysis. If it is an audio file, it uses the SpeechRecognition library to convert the audio to text. If it is an image file, it uses the Pytesseract library to extract text from the image. The input is the format information and the original data, and the output is the analyzed text data.

[1789] Step 4:

[1790] The server performs natural language processing based on the analyzed text data to understand the user's question. Specifically, it tokenizes the text data and extracts keywords. The input is the analyzed text data, and the output is the extracted keywords.

[1791] Step 5:

[1792] The server retrieves the required information from the appropriate source based on the extracted keywords. For example, if the keyword "weather" is included, the Weather API is used to retrieve the weather information for the specified location. The input is the extracted keywords, and the output is the retrieved information.

[1793] Step 6:

[1794] The server formats the acquired information into a format for replying to the user. Specifically, it summarizes the information in text format and provides it to the user in an easy-to-understand format. The input is the acquired information, and the output is a formatted reply message.

[1795] Step 7:

[1796] The server sends the formatted reply message to the user, and the user receives the reply message through a messenger app. The input is the formatted reply message, and the output is the message sent to the user.

[1797] Through the above processing steps, the user can obtain the required information in real time through text, voice, and images.

[1798] Example 2

[1799] Next, a second embodiment of the second embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1800] Conventional information systems were often difficult to use for non-digital natives and the elderly. In addition, there were limited means of providing information by voice to users who used landlines, making it difficult for these users to easily obtain the latest information.

[1801] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for providing a messenger app Internet search tool that can be used from text, voice, and image, a means for covering information such as weather, disasters, news, electronic commerce, and finance, a means for enabling use from a messenger app of a mobile information terminal, a means for receiving voice data through a fixed telephone line, a means for analyzing the voice data and authenticating the user's voiceprint, a means for acquiring requested information when authentication is successful, a means for converting the acquired information into voice data, and a means for transmitting the voice data to the user through a fixed telephone line. This makes it possible for even users who are not digital natives or elderly people to easily acquire the latest information by voice using a fixed telephone line.

[1802] A "text, voice and image messaging app internet search tool" is an application that allows users to search for information and send and receive messages using text, voice and images.

[1803] "Information on weather, disasters, news, e-commerce, finance, etc." refers to a wide range of information that users need on a daily basis, such as meteorological information, information on natural disasters, the latest news, information on online shopping, and information on financial markets and banking.

[1804] A "mobile information terminal" is an electronic device, such as a smartphone or tablet, that is portable and can connect to the Internet.

[1805] A "fixed line" is a communication line for conducting voice communications over a traditional wired telephone network.

[1806] "Voice Data" means a digital representation of a User's voice.

[1807] An "AI voiceprint authentication system" is a system that uses artificial intelligence technology to identify individuals from voice data.

[1808] "Authentication" is the process by which a system verifies a user's identity and grants access.

[1809] A "data source" is an external source from which information is obtained.

[1810] "Speech synthesis software" refers to software for converting text data into voice data.

[1811] "Playing audio data" refers to the process of playing audio data as sound through an output device such as a speaker.

[1812] This invention provides an information provision system that can be used by non-digital native users and elderly people. In particular, we provide a system that allows users to request information by voice using a landline phone, authenticates the user using AI voiceprint authentication, and provides the requested information by voice.

[1813] Server Roles

[1814] The server receives voice data from the user via a landline. Specifically, when a user makes a voice request from a landline such as "What's in the news today?", the voice data is sent to the server. The server then sends the received voice data to an AI voiceprint authentication system, which analyzes the user's voiceprint. This authentication uses voice recognition software such as "Google Cloud Speech-to-Text." If authentication is successful, the server accesses data sources such as "News APIs" and "RSS feeds" to obtain the information requested by the user. The obtained information is converted into voice data using voice synthesis software such as "Amazon Polly" and provided to the user via the landline.

[1815] Terminal Roles

[1816] The terminal functions as a landline phone and is the interface when the user requests information by voice. When the user requests "What's the news today?" by voice from the landline phone, the terminal transmits the voice data to the server. When the terminal receives a response from the server, it plays the voice data and provides the information to the user.

[1817] User Roles

[1818] A user requests information by voice using a landline phone. For example, when an elderly person requests "What's in the news today?" from a landline phone, the voice data is sent to the server. The server authenticates the user's voiceprint, retrieves the requested news information, and provides it by voice.

[1819] Examples

[1820] Example 1: Requesting News Information

[1821] The user makes a voice request from a landline saying, "What's the news today?" The server receives this voice data and authenticates the user using an AI voiceprint authentication system. If authentication is successful, the server retrieves the latest news information from the news API, converts it into voice data, and provides it to the user.

[1822] Example 2: Requesting weather information

[1823] The user makes a voice request from a landline saying, "What's the weather today?" The server receives this voice data and authenticates the user using an AI voiceprint authentication system. If authentication is successful, the server retrieves the latest weather information from the weather API, converts it into voice data, and provides it to the user.

[1824] Examples of prompt statements

[1825] "Please explain the process for a system that authenticates the voiceprint of a user requesting information via voice over a landline phone and then provides the requested information via voice."

[1826] By inputting this prompt sentence into a generative AI model, it is possible to generate a sentence that explains the system's processing in natural language.

[1827] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1828] Step 1:

[1829] The server receives voice data from the user via a fixed telephone line.

[1830] Input: Voice data from users calling from landlines

[1831] Output: Audio data stored on the server

[1832] Specific operation: When a user voice requests "What's the news today?" from a landline phone, the voice data is sent to the server.

[1833] Step 2:

[1834] The server sends the received voice data to the AI ​​voiceprint authentication system and analyzes the user's voiceprint.

[1835] Input: Received audio data

[1836] Output: Analyzed voiceprint data

[1837] Specific operations: Voice data is converted into text using voice recognition software such as "Google Cloud Speech-to-Text," and voiceprint authentication is then performed based on the text data.

[1838] Step 3:

[1839] The server authenticates the user based on the analyzed voiceprint data.

[1840] Input: Analyzed voiceprint data

[1841] Output: Authentication result (success or failure)

[1842] Specific operation: The voiceprint data is compared with pre-registered voiceprint data, and if there is a match, the authentication is successful.

[1843] Step 4:

[1844] If authentication is successful, the server contacts the appropriate data source to obtain the information requested by the user.

[1845] Input: Authentication result (success) and user request

[1846] Output: Retrieved information data

[1847] Specific operation: Obtain the latest news information from "News API", "RSS feed", etc.

[1848] Step 5:

[1849] The server converts the acquired information into audio data.

[1850] Input: Acquired information data

[1851] Output: Audio data

[1852] Specific operation: Converts text data into speech data using speech synthesis software such as "Amazon Polly".

[1853] Step 6:

[1854] The server transmits the generated voice data to the user via a fixed telephone line.

[1855] Input: Generated audio data

[1856] Output: Voice data sent to the user's landline phone

[1857] What it does: It converts voice data into a format suitable for landline telephone lines and sends it to the user's landline.

[1858] Step 7:

[1859] The terminal receives the audio data transmitted from the server and plays it back.

[1860] Input: Audio data sent from the server

[1861] Output: Played audio information

[1862] Specific operation: Play audio data through the landline phone speaker to provide information to the user.

[1863] (Application example 2)

[1864] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1865] Modern digital services are designed primarily for young people and digital natives, and are often difficult to use for the elderly and users who are not familiar with digital technology. Food delivery services also require the use of smartphones and the Internet, which makes them difficult for the elderly to use. Furthermore, although the number of services using voice recognition technology is increasing, the accuracy and security of user authentication may not be sufficient.

[1866] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1867] In this invention, the server includes a means for providing a messenger app internet search tool that can be used from text, voice, and image, a means for covering information on weather, disasters, news, EC, finance, etc., a means for enabling use from a smartphone messenger app, and a means for accepting voice orders from a landline in a food delivery service, a means for authenticating a user by AI voiceprint authentication and providing recommended menu items taking into consideration past order history and preferences, and a means for confirming the order details by voice and finally confirming the order. This enables even elderly people and users who are not familiar with digital to use food delivery services safely and easily using a landline.

[1868] A "text, voice and image messaging app internet search tool" is an application that allows users to search for information and send and receive messages using text, voice and images.

[1869] "Means of covering information on weather, disasters, news, e-commerce, finance, etc." refers to a function that comprehensively provides various types of information, such as weather forecasts, disaster information, the latest news, e-commerce information, and financial information.

[1870] "Means enabling use from a messenger app on a smartphone" refers to a function that enables users to use the service through a messenger app running on their smartphone.

[1871] "A means for accepting voice orders from a landline telephone in a food delivery service" refers to a function that enables a user to place a food delivery order by voice using a landline telephone.

[1872] "A means of authenticating users through AI voiceprint recognition and providing recommended menu items taking into consideration their past order history and preferences" refers to a function that identifies users through voiceprint recognition technology using artificial intelligence, and presents recommended menu items based on the user's past order history and preferences.

[1873] "A means for confirming order details by voice and finally confirming the order" is a function that supports the process in which a user confirms the order details by voice and finally confirms the order.

[1874] A system for carrying out the present invention has the following configuration: The server includes means for providing a messenger app internet search tool that can be used from text, voice, and image, means for covering information on weather, disasters, news, EC, finance, etc., means for enabling use from a smartphone messenger app, means for accepting voice orders from a landline phone in a food delivery service, means for authenticating a user by AI voiceprint authentication and providing recommended menu items taking into consideration past order history and preferences, and means for confirming the order details by voice and finally confirming the order.

[1875] Hardware and Software Configuration

[1876] The server uses a voice recognition library (e.g., speech_recognition), a voice synthesis library (e.g., pyttsx3), an AI voiceprint authentication module (e.g., voice_authentication), and a food delivery service API (e.g., food_delivery_service). The user uses a landline phone to place an order by voice. The server captures the voice using a microphone and outputs the voice using a speaker.

[1877] Data processing and calculation

[1878] The server processes the data in the following steps:

[1879] 1. When a user requests information by voice from a landline phone, the server picks up the voice through a microphone.

[1880] 2. Use a speech recognition library to convert the captured audio into text.

[1881] 3. Use the AI ​​voiceprint recognition module to authenticate the user's voiceprint.

[1882] 4. If authentication is successful, use the food delivery service API to get menu recommendations that take into account past ordering history and user preferences.

[1883] 5. The recommended menu is converted into voice using a speech synthesis library and provided to the user through the speaker.

[1884] 6. The user confirms the order details by voice and finally confirms the order.

[1885] Examples

[1886] For example, when a user makes a voice request from a landline saying, "What's the recommendation today?", the server captures the voice and converts it into text using a voice recognition library. Next, the AI ​​voiceprint authentication module authenticates the user and obtains menu recommendations using the food delivery service API. The obtained menu is converted into speech using a speech synthesis library and provided to the user through the speaker. When the user confirms by voice, "I'd like to order a sushi set," the server confirms the order.

[1887] Examples of prompt statements

[1888] User: Hello, Food Delivery Assistant, what's your name?

[1889] System: This is Tanaka.

[1890] User: Hello Tanaka-san. Please place your order.

[1891] System: What's your recommendation today?

[1892] User: Today's recommendation is the sushi set. Would you like to order the sushi set?

[1893] System: Yes.

[1894] User: Your order has been received, thank you.

[1895] The flow of the specific process in the application example 2 will be described with reference to FIG.

[1896] Step 1:

[1897] A user requests information by voice from a landline phone. The server captures the user's voice through a microphone. The input is the user's voice and the output is voice data.

[1898] Step 2:

[1899] The server converts the acquired voice data into text using a voice recognition library (e.g., speech_recognition). The input is voice data, and the output is text data.

[1900] Step 3:

[1901] The server uses an AI voiceprint authentication module (e.g. voice_authentication) to authenticate the user's voiceprint from the text data. The input is the text data, and the output is the authentication result. If the authentication is successful, proceed to the next step.

[1902] Step 4:

[1903] The server uses a food delivery service API (e.g., food_delivery_service) to obtain recommended menu items that take into account the user's past order history and preferences. The input is the user authentication result and past order history data, and the output is the recommended menu items.

[1904] Step 5:

[1905] The server converts the text data of the recommended menu into speech using a speech synthesis library (e.g., pyttsx3). The input is the text data of the recommended menu, and the output is speech data.

[1906] Step 6:

[1907] The server provides the user with a recommended menu by voice through a speaker. The input is voice data, and the output is voice output to the user.

[1908] Step 7:

[1909] The user confirms the order detai...

Claims

1. A means for acquiring inquiry data including text, voice or image sent by a user through a messenger application; A means for analyzing the inquiry data and deriving information desired by the user as an analysis result; A means for calling an information acquisition API and acquiring information requested by the user; means for presenting the obtained information using a notification function of the messenger application; A system including:

2. A means for acquiring inquiry data including text, voice or image sent by a user; means for analyzing the inquiry data according to a format thereof and converting the inquiry data into text data; means for extracting keywords from the text data by performing natural language processing on the text data; A means for acquiring information desired by the user from an information source based on the extracted keywords; means for sending to the user a reply message summarizing the acquired information in a text format; A system including:

3. A means for receiving a user's voice transmitted through a fixed telephone line; A means for performing user authentication using a voiceprint authentication system based on the received voice; means for, if authentication is successful, accessing a data source to obtain information desired by said user as indicated by the received voice; A means for converting the acquired information into audio data; means for transmitting said voice data to said user over a fixed telephone line; A system including:

Images