System

Abstract

To provide a system.SOLUTION: A system includes: means for making a call to a person designated by a user based on information input by the user; means for generating a response from the person by converting the person's voice into text; means for detecting an unexpected question included in the person's response by analyzing the person's voice; reply request means for requesting the user to input an answer to the unexpected question when the unexpected question is detected during the conversation of the user with the person; means for analyzing order information input by the user as the answer, and inputting a specific prompt sentence into a generative AI model, to generate an instruction for the person to place the order by telephone; an emotion engine for recognizing the emotions of the user when making the answer; and means for adjusting the tone of the user's voice in the conversation with the person based on the recognized emotion.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Many users find it difficult to make reservations or inquiries over the phone due to time and location constraints, language barriers, etc. In particular, it is difficult to respond to unexpected questions from the person on the other end of the phone. [Means for solving the problem]

[0005] As a means for solving this problem, the system of the present invention includes: means for making a call to a party designated by the user based on information input by the user; means for generating a response from the party by converting the voice of the party to text; means for detecting an unexpected question included in the party's response by analyzing the voice of the party; answer request means for, when the unexpected question is detected while the user is conversing with the party, prompting the user to input an answer to the unexpected question; means for analyzing order information input by the user as the answer and inputting a specific prompt sentence into a generative AI model to generate instructions for the party to place the order via telephone; an emotion engine for recognizing the emotions of the user when giving the answer; and means for adjusting the tone of the user's voice in the conversation with the party based on the recognized emotion. [Brief explanation of the drawings]

[0006] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 2 is a sequence diagram showing a flow of processing in the data processing system according to the first embodiment of the first form example. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Embodiment 1. [Figure 13] FIG. 10 is a sequence diagram showing a processing flow of a data processing system in a second embodiment of the second form example. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Embodiment Example 2. [Figure 15] FIG. 10 is a sequence diagram showing the flow of processing in a data processing system according to a third embodiment of the third embodiment. [Figure 16] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Embodiment 3. [Figure 17] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the first embodiment of the first form example when an emotion engine is combined. [Figure 18] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Form Example 1 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0007] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0008] First, the terms used in the following description will be explained.

[0009] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, the processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)).

[0010] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0011] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0012] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0014] [First embodiment]

[0015] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0016] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0017] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0018] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0019] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0020] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0021] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0022] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0023] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0024] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0025] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0026] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0027] "Example 1"

[0028] In one embodiment of the present invention, a user inputs information such as reservation date and time, number of people, and names into a smartphone application. This information is analyzed by generative AI, which generates instructions to call the specified number. Once the call is connected, the system uses AI voice recognition technology to converse with the person on the other end of the line. If an irregular question is asked during the conversation, the system responds with "Please wait."

[0029] " and then asks the user to enter a response on the app. Once the user has entered a response, the app will resume the conversation with the person on the other end of the phone based on that response.

[0030] "Example 2"

[0031] As a specific example, when a user wants to make a restaurant reservation, they input the reservation date and time, number of people, and name on the application. This information is analyzed by generative AI, which generates instructions to call the specified restaurant. Once the call is connected, AI voice recognition technology is used to converse with the restaurant staff and complete the reservation. If the restaurant asks an unexpected question, the app puts the call on hold by saying "Please wait" and asks the user to enter a response on the app. Once the user has entered a response, the conversation with the restaurant staff continues based on that response, and the reservation is confirmed.

[0032] The processing flow of each embodiment will be described below.

[0033] "Example 1"

[0034] Step 1: The user enters information such as reservation date and time, number of people, and names on the smartphone application.

[0035] Step 2: This information is analyzed by a generative AI to generate instructions to call the specified number.

[0036] Step 3: Once the call is connected, use AI voice recognition technology to talk to the person on the other end.

[0037] Step 4: If an irregular question comes up during the conversation, put it on hold with "Please wait" and ask the user to enter the answer on the app.

[0038] Step 5: Once the user has entered their answer, the conversation with the person on the other end of the phone continues based on that answer.

[0039] "Example 2"

[0040] Step 1: When a user wants to make a restaurant reservation, they input the reservation date and time, number of people, and name on the application.

[0041] Step 2: This information is analyzed by a generative AI, which generates instructions to call the specified restaurant.

[0042] Step 3: Once the call is connected, use AI voice recognition technology to speak to the restaurant staff and complete the reservation.

[0043] Step 4: If the restaurant asks an unexpected question, the app will put the question on hold and ask the user to enter the answer on the app.

[0044] Step 5: Once the user has entered their answer, they can continue the conversation with the restaurant staff based on their answer and confirm the reservation.

[0045] Example 1

[0046] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0047] Conventional reservation systems require users to manually make phone calls and make reservations, which is time-consuming and makes it difficult to respond appropriately when an irregular question comes up during a conversation with the person on the other end of the phone. Furthermore, there is a lack of a way to efficiently analyze the information entered by the user and generate an appropriate response.

[0048] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0049] In this invention, the server includes means for making a call to a specified telephone number based on information input by the user, speech recognition technology means for conversing with the person at the telephone number, means for prompting the user to input a response when an irregular question is asked during the conversation, means for using a generative AI model for analyzing the information input by the user, means for generating a prompt sentence based on the analysis result, and means for analyzing the speech of the person at the telephone number in real time and generating an appropriate response. This allows the user to make a reservation without hassle and makes it possible to appropriately respond to irregular questions.

[0050] "User" means an individual or group that uses the system to make reservations or enter information.

[0051] "Information" refers to data such as reservation date and time, number of people, and names entered by the user.

[0052] "Telephone number" refers to the telephone number at which a user makes a reservation or inquiry.

[0053] "Means for making phone calls" refers to the ability to make calls to designated telephone numbers using VoIP technology.

[0054] "Voice recognition technology means" refers to technology that converts the speech of the person on the other end of the phone into text in real time.

[0055] An "irregular question" refers to an unexpected question or confirmation from the person on the other end of the phone.

[0056] "Means for requesting answer input" refers to an interface that allows the user to input an answer when an irregular question is asked.

[0057] A "generative AI model" refers to an artificial intelligence model that analyzes information entered by a user and generates appropriate responses or instructions.

[0058] A "prompt" refers to an instruction or question that is input into a generative AI model.

[0059] "Means of analysis" refers to the function of analyzing information entered by the user using a generative AI model.

[0060] "Means for generating a response" refers to the function of analyzing what the person on the other end of the phone says and generating an appropriate reply.

[0061] This invention is a system that efficiently and automatically makes calls and provides necessary information when a user makes a reservation or makes an inquiry using a smartphone application. Specific embodiments of this system are described below.

[0062] Hardware and software used

[0063] Hardware

[0064] Smartphone (iOS or ANDROID (registered trademark))

[0065] Server (Cloud server or on-premise server)

[0066] software

[0067] Smartphone application

[0068] Generative AI models (e.g., OpenAI's GPT-4)

[0069] Voice recognition technology (e.g., Google® Cloud Speech-to-Text)

[0070] VoIP technology (e.g., Twilio API)

[0071] Data processing and calculation

[0072] User Input

[0073] The user opens the smartphone application and enters information such as the reservation date and time, number of people, and name into the reservation form. Once the information is complete, the user presses the send button.

[0074] Sending data

[0075] The device converts the information entered by the user into JSON format and sends it to a server using the HTTPS protocol, with the destination URL set in advance.

[0076] Data analysis

[0077] The server analyzes the received JSON data and inputs it as a prompt to the generative AI model. An example of a prompt might be, "The user would like to make a restaurant reservation. Please call the restaurant and make the reservation using the following information: Reservation date and time: December 24, 2023, 7:00 PM, Number of people: 4, Name: Yamada Taro."

[0078] Making a call

[0079] The server uses VoIP technology to make a call to the specified phone number, for example, by using the Twilio API. Once the call is connected, it uses AI voice recognition technology to start a conversation with the person on the other end.

[0080] Conversation progression

[0081] The server uses AI speech recognition technology to convert the caller's speech into text in real time, and then uses a generative AI model to generate an appropriate response. For example, if the caller says, "Your reservation has been accepted," the server will respond with, "Thank you."

[0082] Prompt the user

[0083] If an irregular question comes up during a conversation, the server will tell the other party "Please wait" and put the conversation on hold. The device will display the irregular question to the user on the application and ask them to enter an answer. For example, if the other party asks "Do you have any allergies?", the device will display the question to the user and ask them to enter an answer.

[0084] Restarting the conversation

[0085] When the user enters "no allergies," that information is sent to the server. The server then continues the conversation with the person on the other end of the phone based on the user's response. For example, it might respond, "Sorry to keep you waiting. I don't have any allergies." This completes the reservation process.

[0086] Specific examples

[0087] For example, when a user makes a restaurant reservation, they input the following prompt into the generative AI model:

[0088] Example prompt sentence:

[0089] A user would like to make a restaurant reservation. Please call the restaurant using the following information to make the reservation:

[0090] Reservation date and time: December 24, 2023 7:00 PM

[0091] Number of people: 4 people

[0092] Name: Taro Yamada

[0093] Based on this prompt, the generative AI model generates the following instructions:

[0094] "We would like to make a reservation for 4 people on December 24, 2023 at 7:00 PM. Can you accept our reservation?"

[0095] After the call is connected, the app uses AI voice recognition technology to continue the conversation, prompting the user for confirmation as needed. For example, if the restaurant asks if the user has any allergies, the app displays the question to the user and allows them to enter their answer. If the user enters "no allergies," the app resumes the conversation based on that information.

[0096] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0097] Step 1:

[0098] The user opens the smartphone application and enters information such as the reservation date and time, number of people, and name into the reservation form. Once the information is complete, the user presses the send button.

[0099] Input: Information such as reservation date and time, number of people, name, etc.

[0100] Output: When the submit button is pressed, the input information is sent to the next step.

[0101] Step 2:

[0102] The device converts the information entered by the user into JSON format and sends it to a server using the HTTPS protocol, with the destination URL set in advance.

[0103] Input: Information entered by the user

[0104] Output: JSON format data is sent to the server

[0105] Step 3:

[0106] The server analyzes the received JSON data and inputs it as a prompt to the generative AI model. An example of a prompt might be, "The user would like to make a restaurant reservation. Please call the restaurant and make the reservation using the following information: Reservation date and time: December 24, 2023, 7:00 PM, Number of people: 4, Name: Yamada Taro."

[0107] Input: JSON format data

[0108] Output: The prompt sentence that is input to the generative AI model

[0109] Step 4:

[0110] The server uses a generative AI model to generate instructions for making a call to a specified phone number based on the prompt text.

[0111] Input: prompt statement

[0112] Output: Instructions to make a call

[0113] Step 5:

[0114] The server places the call to the specified destination using VoIP technology, for example, by using the Twilio API to make the call.

[0115] Input: Instructions to make a call

[0116] Output: Phone is connected

[0117] Step 6:

[0118] Once the call is connected, the server uses AI voice recognition technology to begin a conversation with the person on the other end of the line.

[0119] Input: Phone connected

[0120] Output: A conversation with the person on the other end of the phone is initiated

[0121] Step 7:

[0122] The server uses AI speech recognition technology to convert the caller's speech into text in real time, and then uses a generative AI model to generate an appropriate response. For example, if the caller says, "Your reservation has been accepted," the server will respond with, "Thank you."

[0123] Input: What the person on the other end of the phone says

[0124] Output: Translated utterances and generated responses

[0125] Step 8:

[0126] If an irregular question comes up during a conversation, the server will tell the other party to wait and put the conversation on hold. The device will then display the irregular question to the user on the application and ask them to enter an answer.

[0127] Input: Irregular Question

[0128] Output: The question and answer prompt that are displayed to the user

[0129] Step 9:

[0130] Once the user enters their answer, the information is sent to the server. For example, if someone asks if they have any allergies, the server displays the question and prompts the user to enter their answer.

[0131] Input: User's answer

[0132] Output: The answer sent to the server

[0133] Step 10:

[0134] The server continues the conversation with the person on the other end of the phone based on the user's response. For example, it might respond, "Sorry to keep you waiting. I don't have any allergies." This completes the reservation process.

[0135] Input: User's answer

[0136] Output: A response that is conveyed to the person on the other end of the phone and the completion of the reservation

[0137] (Application example 1)

[0138] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0139] Conventional food delivery services often require users to make reservations or orders over the phone when ordering from a restaurant. However, this poses problems, such as the time and effort required for users to make phone calls and the need to respond to irregular questions that arise during conversations with restaurant staff. Furthermore, depending on the time of day when a user calls, the restaurant may be busy and the response may be delayed. This reduces user convenience and impairs the quality of service, creating issues.

[0140] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0141] In this invention, the server includes: means for making a call to a specified telephone number based on information input by a user; AI speech recognition technology means for conversing with the person at the telephone number; means for prompting the user to input an answer when an irregular question is asked during the conversation; means for continuing the conversation with the person at the telephone number based on the answer input by the user; means for analyzing the information input by the user using generative AI and generating the telephone call content; means for reading out the generated telephone call content using speech synthesis technology; and means for recognizing the question from the person at the telephone number using speech recognition technology. This allows users to easily make reservations and orders through a smartphone app and can quickly respond to irregular questions, thereby improving user convenience and improving the quality of service.

[0142] "User" means an individual or corporation that makes reservations or orders using the system.

[0143] "Input information" refers to data such as reservation date and time, number of people, names, order details, etc. that the user provides to the system.

[0144] The "designated telephone number" is the telephone number of the restaurant or service provider for which the user wishes to make a reservation or order.

[0145] "Means for making a call" is a function that allows the system to automatically make a call to a specified telephone number.

[0146] "AI voice recognition technology means" is a technology that uses artificial intelligence to recognize the voice of the person on the other end of the phone and convert it into text data.

[0147] An "irregular question" is a situation in which an unexpected question or request arises from the person on the other end of the phone.

[0148] The "means for prompting the user to input an answer" is a function that prompts the user to input an appropriate answer when an irregular question occurs.

[0149] The "means for continuing the conversation again" is a function for continuing the conversation with the person on the other end of the phone call based on the answer entered by the user.

[0150] "Generative AI" is an artificial intelligence technology that analyzes information entered by the user and generates appropriate telephone content.

[0151] "Means for generating telephone content" is a function that uses generative AI to create the content to be conveyed over the phone based on the information entered by the user.

[0152] "Speech synthesis technology" is a technology that converts text data into speech and reads it out in a natural voice.

[0153] "A means for reading out loud" is a function that uses voice synthesis technology to convey the generated telephone call contents aloud.

[0154] "Voice recognition technology" is a technology that recognizes the voice of the person on the other end of the phone and converts it into text data.

[0155] The "means for recognizing questions" is a function that uses voice recognition technology to understand the question asked by the person on the other end of the phone and acquires it as text data.

[0156] To implement this invention, the following hardware and software are required. The hardware requires a smartphone and microphone. The software uses Python, OpenAI API, the SpeechRecognition library, and the pyttsx3 library.

[0157] The server provides a means to call a specified number based on the information entered by the user. When a user enters reservation or order information through a smartphone app, the information is sent to the server. The server uses generative AI to analyze the information entered by the user and generate the phone call content. This generated phone call content is read aloud using speech synthesis technology.

[0158] The terminal provides AI speech recognition technology for conversation with the person on the other end of the phone. When the call is connected, the AI ​​speech recognition technology is used to recognize the voice of the person on the other end of the phone and convert it into text data. If an irregular question is asked during the conversation, the terminal puts the call on hold and asks the user to enter a response. Once the user enters a response, the terminal continues the conversation with the person on the other end based on that response.

[0159] As a concrete example, if a user inputs "I would like to make a reservation for two people at 7pm tomorrow," the server analyzes this information and calls the specified restaurant. The generative AI generates the phone call message "Please make a reservation for two people at 7pm tomorrow," and reads it out loud using speech synthesis technology. If a restaurant staff member asks, "Do you have any allergies?", the device recognizes this question and asks the user for a response. If the user inputs "no," the AI ​​responds again with "I don't have any allergies," and the conversation continues.

[0160] An example of a prompt is as follows:

[0161] User input: I would like to make a reservation for 2 people tomorrow at 7pm.

[0162] Call the restaurant and tell them the following:

[0163] In this way, users can easily make reservations and orders through a smartphone app, and irregular inquiries can be quickly responded to, improving user convenience and the quality of service.

[0164] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0165] Step 1:

[0166] Users enter reservation and order information through a smartphone app.

[0167] Input: Information such as reservation date and time, number of people, name, order details, etc.

[0168] Output: The entered information is sent to the server

[0169] Specific behavior: The user enters the required information into the app's input form and presses the submit button.

[0170] Step 2:

[0171] The server uses generative AI to analyze the information entered by the user and generate the content of the call.

[0172] Input: Information entered by the user

[0173] Output: Generated call contents (text format)

[0174] Specific operation: The server calls the OpenAI API, passes the input information as a prompt, and generates the appropriate call content.

[0175] Step 3:

[0176] The server uses speech synthesis technology to read out the generated phone call content aloud.

[0177] Input: Generated call content (text format)

[0178] Output: Audio file

[0179] Specific operation: The server uses the pyttsx3 library to convert the generated text into speech and create an audio file.

[0180] Step 4:

[0181] The server makes a call to the specified telephone number.

[0182] Input: specified phone number, audio file

[0183] Output: Establishing a telephone connection

[0184] Specific operation: The server uses the telephone API to call the specified phone number and play the audio file.

[0185] Step 5:

[0186] The device uses AI voice recognition technology to recognize the voice of the person on the other end of the phone and convert it into text data.

[0187] Input: The voice of the person you are calling

[0188] Output: Recognized text data

[0189] What it does: The device uses the SpeechRecognition library to convert the other person's voice into text in real time.

[0190] Step 6:

[0191] If an irregular question comes up during a conversation, the terminal will put the question on hold and ask the user to enter an answer.

[0192] Input: Recognized text data (irregular questions)

[0193] Output: prompts the user for input

[0194] Specific behavior: The device detects an irregular question and notifies the user to enter an answer in the app.

[0195] Step 7:

[0196] The user enters the answer in the app.

[0197] Input: User's answer

[0198] Output: The entered answer is sent to the server

[0199] Specific behavior: The user enters answers into the app's input form and presses the submit button.

[0200] Step 8:

[0201] The server continues the conversation with the person on the other end of the phone based on the user's response.

[0202] Input: User's answer

[0203] Output: Re-conversation content (text format)

[0204] Specific operation: The server calls the generative AI again, passes the user's answer as a prompt, and generates appropriate re-conversation content.

[0205] Step 9:

[0206] The server uses speech synthesis technology to read out the contents of the re-conversation aloud.

[0207] Input: Re-conversation content (text format)

[0208] Output: Audio file

[0209] Specific operation: The server uses the pyttsx3 library to convert the replay content into audio and create an audio file.

[0210] Step 10:

[0211] The server then plays the audio file again to the person on the other end of the line, and the conversation continues.

[0212] Input: Audio file

[0213] Output: Audio playback to the other party

[0214] Specific operation: The server uses the telephone API to play the audio file of the conversation and continue the conversation.

[0215] Example 2

[0216] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0217] Conventional restaurant reservation systems require users to manually make phone calls, making the reservation process cumbersome. Furthermore, if an unexpected question arose during the call, the user was often unable to respond immediately, resulting in a delay in confirming the reservation. Furthermore, there was a lack of a system that could efficiently analyze the information entered by the user and provide an appropriate response.

[0218] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0219] In this invention, the server includes means for making a call to a specified telephone number based on information input by the user, artificial intelligence speech recognition technology means for conversing with the person at the telephone number, means for prompting the user to input a response when an irregular question is asked during the conversation, artificial intelligence model generation means for analyzing the information input by the user, communication API means for making the call, and means for notifying the user of the confirmation of the reservation. This enables users to easily make restaurant reservations and quickly respond to unexpected questions.

[0220] "User" refers to an individual or organization that uses the system to make restaurant reservations.

[0221] "Phone number" refers to the restaurant or other facility where the user wishes to make a reservation.

[0222] "Artificial intelligence speech recognition technology" refers to technology that converts speech into text and understands the content of conversations.

[0223] "Irregular questions" refer to unexpected questions or confirmations that arise during the reservation process.

[0224] A "generative artificial intelligence model" refers to an artificial intelligence model that analyzes information entered by a user and generates appropriate instructions and responses.

[0225] "Communications API" refers to the application programming interface used by the server to place a call.

[0226] A "reservation confirmation notice" refers to a notice to inform the user that the reservation has been confirmed.

[0227] This invention is a system that allows users to easily make restaurant reservations. The system uses artificial intelligence speech recognition technology to call a specified number based on information entered by the user and to converse with the person on the phone. The system also includes a means for prompting the user to input a response if an irregular question is asked during the conversation.

[0228] Hardware and software used

[0229] 1. Server

[0230] The server uses a generative artificial intelligence model (e.g., GPT-4) to analyze the information entered by the user and generate appropriate instructions.

[0231] The server uses a communications API (e.g., Twilio) to make the call.

[0232] The server uses artificial intelligence speech recognition technology (e.g., Google Speech-to-Text) to conduct a conversation with the person on the other end of the phone.

[0233] 2. Terminal

[0234] The terminal transmits the information entered by the user to the server.

[0235] The terminal receives instructions from the server and displays them to the user.

[0236] 3. Users

[0237] The user uses the application to enter reservation information.

[0238] The user inputs answers to the irregular questions.

[0239] Specific examples

[0240] The user opens the application and enters the following information:

[0241] Reservation date and time: December 24, 2023 7:00 PM

[0242] Number of people: 4 people

[0243] Name: Taro Yamada

[0244] This information is analyzed by a generative AI model (GPT-4), which generates instructions to call the specified restaurant using Twilio. Once the call is connected, a conversation with the restaurant staff takes place using Google Speech-to-Text. For example, if the staff asks, "Do you have any allergies?", the server puts the call on hold with a "Please wait" message and displays "Do you have any allergies?" on the terminal. If the user enters "No," the conversation with the staff continues based on that response, and the reservation is confirmed.

[0245] Prompt Sentence Examples

[0246] An example of a prompt to be input to the generative AI model is as follows:

[0247] The user entered the following information:

[0248] Reservation date and time: December 24, 2023 7:00 PM

[0249] Number of people: 4 people

[0250] Name: Taro Yamada

[0251] Use this information to generate instructions to call the designated restaurant and complete the reservation.

[0252] In this way, a system is realized that allows users to easily make reservations at restaurants.

[0253] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0254] Step 1:

[0255] The user enters the reservation information.

[0256] Input: reservation date and time, number of people, name

[0257] Specific operation: The user opens the application and enters the reservation date and time "December 24, 2023 19:00", the number of people "4", and the name "Yamada Taro".

[0258] Output: The entered reservation information is saved on the device.

[0259] Step 2:

[0260] The terminal sends the input information to the server.

[0261] Input: Booking information entered by the user

[0262] Specific operation: The terminal sends the reservation information entered by the user to the server.

[0263] Output: The server receives the reservation information.

[0264] Step 3:

[0265] The server analyzes the input information using a generative AI model.

[0266] Input: Received reservation information

[0267] How it works: The server uses a generative AI model (e.g., GPT-4) to analyze the reservation information entered by the user.

[0268] Output: The parsing results in instructions to call the restaurant.

[0269] Step 4:

[0270] The server uses the phone API to call the restaurant.

[0271] Input: Instructions generated as a result of the analysis

[0272] Specific operation: The server uses a telephone API such as Twilio to call the specified restaurant.

[0273] Output: A telephone connection is established with the restaurant.

[0274] Step 5:

[0275] The server uses AI voice recognition technology to converse with restaurant staff.

[0276] Input: Conversation with restaurant staff

[0277] Specific operation: The server uses AI voice recognition technology such as Google Speech-to-Text to converse with restaurant staff and proceed with the reservation.

[0278] Output: The conversation is parsed into text.

[0279] Step 6:

[0280] The server prompts the user for an answer to an unexpected question.

[0281] Input: Unexpected question from restaurant staff

[0282] Specific operation: When a restaurant staff member asks, "Do you have any allergies?", the server puts the customer on hold by saying, "Please wait," and displays "Do you have any allergies?" on the terminal.

[0283] Output: The question is displayed to the user.

[0284] Step 7:

[0285] The user enters the answer.

[0286] Input: User's answer to the question displayed on the terminal

[0287] Specific action: The user enters "No" in response to the question displayed on the terminal.

[0288] Output: The entered answer is saved on the device.

[0289] Step 8:

[0290] The terminal sends the user's answer to the server.

[0291] Input: The answer entered by the user

[0292] Specific operation: The device sends the answer entered by the user to the server.

[0293] Output: The server receives the answer.

[0294] Step 9:

[0295] The server continues the conversation with the restaurant staff.

[0296] Input: User's answer

[0297] Specific operation: Based on the user's answer, the server continues the conversation with the restaurant staff and confirms the reservation.

[0298] Output: The reservation is confirmed.

[0299] Step 10:

[0300] The server confirms the reservation and notifies the user.

[0301] Input: Confirmed reservation information

[0302] Specific behavior: The server notifies the user that the reservation has been confirmed and displays the reservation details.

[0303] Output: A confirmation of the booking is sent to the user.

[0304] (Application example 2)

[0305] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0306] With conventional food delivery systems, users had to manually call to place an order, which was time-consuming and could lead to miscommunication and other communication issues. Another issue was that it was difficult to respond immediately to unexpected questions from the restaurant, which meant it took a long time to confirm the order.

[0307] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0308] In this invention, the server includes: a means for making a call to a specified telephone number based on information entered by the user; an AI speech recognition technology means for conversing with the person at the telephone number; a means for prompting the user to input a response if an irregular question is asked during the conversation; a means for analyzing the order information entered by the user and generating instructions for the telephone order using a generative AI model; and a means for communicating the generated instructions to the person at the telephone number using speech synthesis technology. This eliminates the need for the user to make manual calls, reduces miscommunication of order details, and enables quick response to unexpected questions.

[0309] "User" means an individual or corporation that uses the system to input information and receive services.

[0310] "Input information" refers to data provided by the user to the system, and includes order details, reservation information, and the like.

[0311] A "designated telephone number" is a telephone number that the system will call based on the information entered by the user.

[0312] "Means for making a call" is a function that allows the system to automatically make a call to a specified telephone number.

[0313] "AI speech recognition technology means" is a technology that uses artificial intelligence to convert speech into text and understand the content of the conversation.

[0314] An "irregular question" is a question or request that the system does not anticipate and that requires user input.

[0315] The "means for prompting the user to input an answer" is a function that prompts the user to input an answer when an irregular question occurs.

[0316] "Order information" refers to the specific order details provided by the user to the system, including the name of the dish, the quantity, the delivery address, the desired delivery time, and the like.

[0317] A "generative AI model" is an artificial intelligence model that analyzes information entered by the user and generates appropriate instructions and answers.

[0318] "Telephone order instructions" are the specific order details and instructions generated by the generative AI model that should be conveyed to the person on the other end of the phone.

[0319] "Speech synthesis technology" is a technology that converts text data into speech and communicates it to the person on the other end of the phone in a natural voice.

[0320] A system for implementing this invention includes means for making a call to a specified telephone number based on information entered by a user, AI voice recognition technology means for conducting a conversation with the person at the telephone, means for prompting the user to input a response when an irregular question arises during the conversation, means for analyzing the order information entered by the user and generating instructions for a telephone order using a generation AI model, and means for conveying the generated instructions to the person at the telephone using voice synthesis technology.

[0321] The server receives order information entered by the user on the food delivery application, including the name of the dish, the quantity, the delivery address, the desired delivery time, etc. The server analyzes this information and generates phone order instructions using a generative AI model (e.g., OpenAI's API).

[0322] The generated instructions are used to automatically call the specified phone number using the Twilio API. After the call is connected, the generated instructions are conveyed to the caller using speech synthesis technology (e.g., pyttsx3). If the caller asks an unexpected question, the server puts the call on hold and asks the user to enter an answer in the application. Once the user enters an answer, the server uses the generative AI model again to generate new instructions and convey them to the caller.

[0323] As a specific example, if a user inputs "one pizza, two salads, delivery destination Shibuya-ku, Tokyo, desired delivery time 6:00 p.m.", the following prompt sentence will be input to the generative AI model.

[0324] Example prompt sentence:

[0325] "User would like to place the following order: 1 pizza, 2 salads, delivery location: Shibuya-ku, Tokyo, desired delivery time: 6 PM. Please call the restaurant to complete the order."

[0326] Based on this prompt, the generative AI model generates instructions to call the restaurant, and then uses the Twilio API to actually make the call. The order is then conveyed using speech synthesis technology, and the order is completed.

[0327] This system saves users the trouble of manually making phone calls, reduces miscommunication of orders, and allows for quick response to unexpected questions.

[0328] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0329] Step 1:

[0330] A user enters order information on a food delivery application.

[0331] Input: Order information such as dish name, quantity, delivery address, desired delivery time, etc.

[0332] Output: Order information sent to the server

[0333] Specific operation: The user enters the required information into the application's order form and presses the "Order" button.

[0334] Step 2:

[0335] The server analyzes the order information received from the user.

[0336] Input: Order information submitted by the user

[0337] Output: Parsed order information

[0338] Specific operation: The server receives the order information and extracts each item (food name, quantity, delivery address, desired delivery time).

[0339] Step 3:

[0340] The server uses the generative AI model to generate phone order instructions.

[0341] Input: Parsed order information

[0342] Output: Generated phone order instructions

[0343] Specific operation: The server converts the parsed order information into prompt sentences and inputs them into a generative AI model (e.g., OpenAI's API) to generate telephone order instructions.

[0344] Step 4:

[0345] The server automatically makes a call to the specified phone number using the Twilio API.

[0346] Input: Generated phone order instructions, designated phone number

[0347] Output: Establishing a telephone connection

[0348] Specific operation: The server calls the Twilio API and makes a call to the specified phone number.

[0349] Step 5:

[0350] The server uses voice synthesis technology to generate instructions and transmit them to the person on the other end of the phone.

[0351] Input: Generated phone order instructions

[0352] Output: Voice instructions that are conveyed to the person on the other end of the call

[0353] Specific operation: The server uses a speech synthesis engine (e.g., pyttsx3) to convert the generated instructions into speech and convey them to the person on the other end of the phone.

[0354] Step 6:

[0355] If the person on the other end of the phone asks an unexpected question, the server puts the call on hold and asks the user to enter an answer on the application.

[0356] Input: An unexpected question from the person on the other end of the phone

[0357] Output: Prompt the user to enter an answer

[0358] Specific operation: The server uses voice recognition technology to detect unexpected questions, puts the call on hold with a "Please wait" message, and sends a notification to the user requesting them to enter an answer in the application.

[0359] Step 7:

[0360] The user enters an answer into the application, and the server again uses the generative AI model to generate new instructions and convey them to the person on the other end of the phone.

[0361] Input: User response

[0362] Output: Newly generated instructions, voice instructions to be conveyed to the person on the other end of the call

[0363] Specific operation: The user enters an answer on the application, and the server receives the answer and uses the generative AI model to generate new instructions, which are then conveyed to the person on the other end of the phone using voice synthesis technology.

[0364] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0365] "Example 1"

[0366] One embodiment of the present invention is a system that incorporates an emotion engine. This system includes a means for making a call to a specified telephone number based on information entered by a user, an AI voice recognition technology means for conducting a conversation with the person at the telephone number, a means for prompting the user to input a response when an irregular question is asked during the conversation, and an emotion engine that recognizes the user's emotions. The emotion engine recognizes emotions from the tone of the user's voice and adjusts the tone of the conversation based on that emotion. For example, if the user feels angry, the emotion engine conveys that information to the AI ​​voice recognition technology means, and the AI ​​adjusts the tone of the conversation based on that information. This enables a more natural conversation that reflects the user's emotions.

[0367] "Example 2"

[0368] Another embodiment of the present invention is a system that incorporates an emotion engine. This system includes a means for making a call to a specified telephone number based on information entered by a user, an AI voice recognition technology means for conducting a conversation with the person at the telephone number, a means for prompting the user to input a response when an irregular question is asked during the conversation, and an emotion engine that recognizes the user's emotions. The emotion engine recognizes emotions from the tone of the user's voice and adjusts the tone of the conversation based on that emotion. For example, if the user feels happy, the emotion engine conveys that information to the AI ​​voice recognition technology means, and the AI ​​adjusts the tone of the conversation based on that information. This enables a more natural conversation that reflects the user's emotions.

[0369] The processing flow of each embodiment will be described below.

[0370] "Example 1"

[0371] Step 1: The user enters reservation details into the app.

[0372] Step 2: The system will call the specified number.

[0373] Step 3: AI voice recognition technology means the conversation with the person on the other end of the phone.

[0374] Step 4: If an irregular question comes up during the conversation, the system prompts the user to enter an answer.

[0375] Step 5: The emotion engine recognizes the emotion from the user's tone of voice.

[0376] Step 6: The emotion engine adjusts the tone of the conversation based on that emotion.

[0377] "Example 2"

[0378] Step 1: The user enters reservation details into the app.

[0379] Step 2: The system will call the specified number.

[0380] Step 3: AI voice recognition technology means the conversation with the person on the other end of the phone.

[0381] Step 4: If an irregular question comes up during the conversation, the system prompts the user to enter an answer.

[0382] Step 5: The emotion engine recognizes the emotion from the user's tone of voice.

[0383] Step 6: The emotion engine adjusts the tone of the conversation based on that emotion.

[0384] Example 1

[0385] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0386] Conventional telephone reservation systems require users to make a direct call, which makes it difficult to respond to irregular inquiries. Furthermore, they are unable to respond to the user's emotions, which can lead to a poor user experience. There is a need to solve these problems and provide a more natural and efficient telephone reservation system.

[0387] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0388] In this invention, the server includes means for making a call to a specified telephone number based on information input by the user, a voice recognition technology means for conversing with the person at the telephone number, means for prompting the user to input a response if an irregular question is asked during the conversation, and means for recognizing the user's emotions and adjusting the tone of the conversation based on the emotions. This allows the user to make reservations efficiently without having to make a direct call, and irregular questions can be quickly responded to. Furthermore, it is possible to respond according to the user's emotions, providing a more natural and satisfying user experience.

[0389] "User" means an individual or group that uses the system to make reservations or enter information.

[0390] "Information" refers to data that users input into the system, such as reservation date and time, number of people, and names.

[0391] "Telephone number" refers to a location or organization that has a telephone number where a user can make a reservation or inquiry.

[0392] The "means for making a call" refers to the function or technology that enables the server to make a call to a specified telephone number.

[0393] "Speech recognition technology means" refers to technology that converts the speech of the person on the other end of the phone into text in real time and generates an appropriate response.

[0394] An "irregular question" is an unexpected question or request that is not part of the normal booking process or conversation flow.

[0395] The "means for requesting input of an answer" is a function that prompts the user to input an answer to an irregular question when the question is asked.

[0396] The "means for recognizing emotions" is a technology that analyzes and recognizes emotions from the tone of a user's voice and other voice characteristics.

[0397] The "means for adjusting the tone of the conversation" is a function that appropriately changes the tone and content of the voice response generated by the system based on the recognized user's emotions.

[0398] This invention is a system that allows a user to input reservation information using a smartphone application and then makes a call to a specified telephone number based on that information. The system analyzes the information entered by the user and uses voice recognition technology to converse with the person on the other end of the phone. The system also includes a means for prompting the user to input a response if an irregular question is asked during the conversation. It also includes a means for recognizing the user's emotions and adjusting the tone of the conversation based on those emotions.

[0399] Hardware and software used

[0400] Hardware

[0401] Smartphone: A device used by users to enter reservation information.

[0402] Server: A central processing unit for analyzing data, making calls, performing voice recognition, and emotion recognition.

[0403] software

[0404] Application: Runs on a smartphone and provides an interface for users to enter reservation information.

[0405] Generative AI models: For example, OpenAI's GPT-4 is used to analyze user-entered information and generate appropriate instructions.

[0406] Speech recognition technology: For example, Google Cloud Speech-to-Text is used to convert what the person on the other end of the phone says into text in real time.

[0407] Emotion Recognition Technology: For example, Affectiva's emotion recognition technology is used to recognize emotions from the user's tone of voice.

[0408] Cloud communications platforms: For example, using Twilio to make calls to specified destinations.

[0409] Specific examples

[0410] Example 1: Restaurant reservations

[0411] A user opens a smartphone application and enters "October 15, 2023, 7 PM, 4 people, Taro Tanaka." The device sends this information to the server. The server uses a generative AI model to generate the instruction "Please make a reservation for Taro Tanaka on October 15, 2023 at 7 PM for 4 people," and uses Twilio to call the restaurant. When the call connects, the server says, "Hello, my name is Taro Tanaka. I'd like to make a reservation for 4 people on October 15 at 7 PM." If the restaurant asks, "Do you have any special requests?" the server replies, "Please wait," and sends a notification to the device. If the user enters, "I'd like a window seat," the device sends this information to the server, and the server again tells the person on the phone, "I'd like a window seat."

[0412] Example 2: Using the Emotion Engine

[0413] The user enters "October 15, 2023, 7:00 PM, 4 people, Taro Tanaka" into the app, and the call is connected. If the user begins to speak in an angry tone during the conversation, the emotion engine recognizes that emotion and passes it on to the server. The server passes that information on to the AI, which responds calmly with, "I'm sorry, I'll check right away."

[0414] Prompt Sentence Examples

[0415] "Please explain a system in which a user inputs reservation information into a smartphone application, which then uses that information to call a specified number. Once the call is connected, the system uses AI voice recognition technology to conduct the conversation, and if an irregular question is asked, please include a mechanism to prompt the user for an answer. Also, please explain how an emotion engine is used to adjust the tone of the conversation according to the user's emotions."

[0416] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0417] Step 1:

[0418] The user opens the application on their smartphone and enters reservation information (reservation date and time, number of people, name, etc.).

[0419] Input: Information such as reservation date and time, number of people, name, etc.

[0420] Output: Reservation information entered

[0421] Specific operation: The user enters "October 15, 2023, 7:00 PM, 4 people, Taro Tanaka" into the application's input form.

[0422] Step 2:

[0423] The terminal transmits the reservation information input by the user to the server.

[0424] Input: Reservation information entered by the user

[0425] Output: Reservation information sent to the server

[0426] Specific operation: The terminal uses an HTTP request to send the entered reservation information to the server in JSON format.

[0427] Step 3:

[0428] The server analyzes the received reservation information using a generative AI model and generates instructions to call the specified phone number.

[0429] Input: Reservation information sent to the server

[0430] Output: Instructions to make a call

[0431] Specific operation: The server uses a generative AI model (e.g., GPT-4) to generate the instruction, "Please make a reservation for Taro Tanaka on October 15, 2023 at 7 p.m. for four people."

[0432] Step 4:

[0433] The server places a call to the specified telephone number based on the generated instruction.

[0434] Input: Instructions to make a call

[0435] Output: Make a phone call

[0436] Specific operation: The server uses Twilio's API to make a call to the specified restaurant.

[0437] Step 5:

[0438] Once the call is connected, the server uses voice recognition technology to converse with the person on the other end of the line.

[0439] Input: What the person on the other end of the phone says

[0440] Output: Speech converted to text

[0441] How it works: The server uses Google Cloud Speech-to-Text to convert what the person on the other end of the phone says into text in real time.

[0442] Step 6:

[0443] The server uses a generative AI model to generate an appropriate response and deliver it to the person on the other end of the call.

[0444] Input: Speech converted to text

[0445] Output: The generated response

[0446] Specific operation: The server uses the generative AI model to generate a response that says, "Hello, my name is Taro Tanaka. I would like to make a reservation for four people at 7 p.m. on October 15th," and conveys this to the person on the other end of the line.

[0447] Step 7:

[0448] If an irregular question comes up during a conversation, the server will tell the other party to wait and put the call on hold.

[0449] Input: Irregular Question

[0450] Output: "Please wait" response

[0451] Specific behavior: When the server receives the question "Do you have any special requests?", it tells the caller "Please wait" and puts the call on hold.

[0452] Step 8:

[0453] The server sends a notification to the user at the terminal, requesting input of an answer.

[0454] Input: Irregular Question

[0455] Output: User notification

[0456] Specific operation: The server sends a push notification to the device, asking the user, "Do you have any special requests?"

[0457] Step 9:

[0458] Users enter their answers in the app.

[0459] Input: User notification

[0460] Output: User's answer

[0461] Specific operation: The user enters "I would like a window seat" in the app.

[0462] Step 10:

[0463] The terminal transmits the user's answer to the server.

[0464] Input: User's answer

[0465] Output: The answer sent to the server

[0466] Specific operation: The terminal uses an HTTP request to send the user's answer to the server.

[0467] Step 11:

[0468] The server continues the conversation with the person on the other end of the phone based on the user's response.

[0469] Input: The answer sent to the server

[0470] Output: Response to the person on the phone

[0471] Specific operation: The server generates a response saying "I would like a window seat" and conveys this to the person on the other end of the phone.

[0472] Step 12:

[0473] The server uses emotion recognition technology to recognize emotions from the tone of the user's voice and transmits that information to AI voice recognition technology.

[0474] Input: User's voice tone

[0475] Output: Recognized emotion information

[0476] Specific operation: The server uses emotion recognition technology to determine that the user is angry and conveys that information to the AI ​​voice recognition technology.

[0477] Step 13:

[0478] The server adjusts the tone of the conversation based on the recognized emotional information.

[0479] Input: Recognized emotion information

[0480] Output: Modulated conversational tone

[0481] What it does: The server uses AI speech recognition technology to respond in a calm tone, saying, "Sorry, we'll check it right away."

[0482] (Application example 1)

[0483] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0484] Conventional food delivery systems require users to place orders over the phone, which can lead to cumbersome communication and difficulty in responding to irregular questions. Furthermore, they often fail to consider the user's feelings, resulting in a slow ordering process. This can lead to poor user experience, cancellations of orders, and dissatisfaction.

[0485] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0486] In this invention, the server includes: means for making a call to a specified telephone number based on information input by the user; artificial intelligence speech recognition technology means for conversing with the person at the telephone number; means for prompting the user to input a response when an irregular question is asked during the conversation; emotion engine means for recognizing the user's emotion and adjusting the tone of the conversation based on the emotion; and generative artificial intelligence model means for analyzing the information input by the user and generating instructions to call the telephone number. This not only enables the user to place an order smoothly and quickly respond to irregular questions, but also enables natural conversation that reflects the user's emotion.

[0487] "Information entered by the user" refers to information such as reservation date and time, number of people, names, menu, etc., entered by the user via a smartphone or other device.

[0488] "Means for calling a specified telephone number" refers to technology for automatically calling a specific telephone number based on information entered by the user.

[0489] "Artificial intelligence speech recognition technology means" refers to artificial intelligence technology that converts the voice of the person on the other end of the phone into text and generates an appropriate response.

[0490] The "means for prompting the user to input an answer when an irregular question is asked" is an interface that allows the user to input an answer to a question when an unexpected question is asked during a conversation.

[0491] The "emotion engine means" is a technology for recognizing emotions from the tone of a user's voice and input content, and adjusting the tone of the conversation based on those emotions.

[0492] The "generative artificial intelligence model means" is an artificial intelligence model that analyzes information input by the user and generates instructions to be called and conversation content.

[0493] In the system for implementing this invention, a user inputs order information using a smartphone or other device, and an AI automatically makes a phone call based on that information to confirm the order. Specific embodiments of this system are described below.

[0494] System configuration

[0495] 1. User Interface:

[0496] Use a smartphone application (iOS / Android).

[0497] Provide a form for users to enter order information (date and time, number of people, menu, name, etc.).

[0498] 2. Data analysis and generative AI models:

[0499] The information entered by the user is sent to a server in the cloud.

[0500] On the server side, a generative AI model (e.g., OpenAI's GPT-4) is used to generate instructions to call the destination.

[0501] 3. Telephone connection and AI voice recognition:

[0502] Use APIs such as Twilio to automatically call specified numbers.

[0503] Once the call is connected, the Google Cloud Speech-to-Text API is used to convert the caller's voice into text.

[0504] Use generative AI models to generate appropriate answers to the other person's questions.

[0505] 4. Handling irregular questions:

[0506] If an irregular question is received, the app will put it on hold with a "Please wait" message and ask the user to enter the answer on the app.

[0507] The conversation continues based on the answers entered by the user.

[0508] 5. Emotion Engine:

[0509] Use an emotion recognition API (e.g., IBM Watson® Tone Analyzer) to analyze the user's tone of voice.

[0510] The emotion engine recognizes the user's emotions and conveys that information to the generative AI model.

[0511] A generative AI model adjusts the tone of the conversation to create a natural conversation that responds to the user's emotions.

[0512] Specific examples

[0513] A user types "I'd like to order two pizzas tomorrow night at 7pm" into a smartphone app. The server receives this information and uses a generative AI model to generate instructions to call the restaurant. The Twilio API is used to automatically make the call, and the Google Cloud Speech-to-Text API is used to convert the caller's voice into text. If the restaurant asks, "Do you have specific toppings?", the app prompts the user for a response. If the user types "pepperoni and mushrooms," the AI ​​continues the conversation based on that information. If the user is angry, the emotion engine recognizes this information and the generative AI model adjusts the tone of the conversation.

[0514] Prompt Sentence Examples

[0515] A user types, "I'd like to order two pizzas tomorrow night at 7pm." Call the restaurant to confirm the order. If they ask about specific toppings, ask them in-app and continue the conversation based on their answer. Also, recognize the user's emotions and adjust the tone of the conversation accordingly.

[0516] In this way, users can enjoy a smoother and more natural ordering experience in food delivery applications.

[0517] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0518] Step 1:

[0519] The user launches the smartphone application and enters order information (date and time, number of people, menu, name, etc.). The entered information is sent from the device to a server on the cloud. The input data is sent to the server in the form of reservation date and time, number of people, menu, name, etc.

[0520] Step 2:

[0521] The server analyzes the received order information and uses a generative AI model (e.g., OpenAI's GPT-4) to generate instructions to call the destination. The generated instructions are output as a specific prompt for making the call. For example, a prompt such as "I'd like to order two pizzas tomorrow night at 7 p.m." is generated.

[0522] Step 3:

[0523] The server automatically calls the specified phone number using an API such as Twilio. Once the call is connected, the server uses the Google Cloud Speech-to-Text API to convert the caller's voice into text in real time. The converted text data is then sent to the server.

[0524] Step 4:

[0525] The server inputs the converted text data into a generative AI model to generate an appropriate answer to the caller's question. The generated answer is then converted back into speech and transmitted to the caller. For example, in response to the question "Do you have any specific toppings?", the answer might be "pepperoni and mushrooms."

[0526] Step 5:

[0527] If an irregular question comes up during a conversation, the server puts the question on hold and asks the user to enter the answer in the app. The answer entered by the user is sent from the device to the server. For example, in response to the question "Do you have any specific toppings?", the user might enter "pepperoni and mushrooms."

[0528] Step 6:

[0529] The server continues the conversation based on the answers entered by the user. It uses a generative AI model to generate an appropriate answer to the caller's question, converts it into speech, and delivers it to the caller. For example, it might respond with "pepperoni and mushrooms."

[0530] Step 7:

[0531] The server uses an emotion recognition API (e.g., IBM Watson Tone Analyzer) to analyze the user's tone of voice and recognize their emotions. The recognized emotion data is input into the generative AI model. For example, if the user is angry, that information is conveyed to the generative AI model.

[0532] Step 8:

[0533] The generative AI model adjusts the tone of the conversation based on the perceived emotion. For example, if the user is angry, the generative AI model will continue the conversation in a calmer tone. The adjusted tone of the conversation is converted into audio and transmitted to the other party on the call.

[0534] In this way, the user can smoothly place an order, irregular questions can be quickly responded to, and natural conversations can be held according to the user's emotions.

[0535] Example 2

[0536] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0537] Conventional reservation systems have the problem of being unable to respond appropriately when unexpected questions arise when making a phone call based on the information entered by the user. Furthermore, the system responds mechanically without considering the user's feelings, which can result in a poor user experience. There is a need to solve these problems and provide a more natural and user-friendly reservation system.

[0538] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0539] In this invention, the server includes means for making a call to a specified telephone number based on information input by the user, voice recognition technology means for conversing with the person at the telephone number, means for prompting the user to input a response when an unexpected question is asked during the conversation, and emotion recognition means for recognizing the user's emotion and adjusting the tone of the conversation based on that emotion. This allows for appropriate responses to unexpected questions and enables natural conversation that reflects the user's emotion.

[0540] "User" means an individual or group that uses the system to make reservations or enter information.

[0541] A "means for making a call" is a device or software that has the function of automatically making a call to a specified telephone number based on information entered by the user.

[0542] "Voice recognition technology means" refers to a device or software that has the technology to analyze the voice of the person on the other end of the phone and convert it into text data.

[0543] An "unexpected question" is a question or request that the system does not anticipate in advance, and may require input from the user.

[0544] The "means for prompting the user to input an answer" is a device or software that has the function of prompting the user to input an answer when an unexpected question is presented.

[0545] An "emotion recognition means" is a device or software that has the technology to analyze emotions from the user's tone of voice and vocabulary, and adjust the system's response based on that information.

[0546] The "reservation date and time" refers to the specific date and time that the user desires to make a reservation.

[0547] "Number of Participants" means the total number of Participants in a Booking.

[0548] "Name" refers to the name of the user making the reservation.

[0549] The "holding means" is a device or software that has the function of temporarily interrupting a conversation when an unexpected question is asked and asking the user for confirmation.

[0550] This invention is a system for allowing users to make restaurant reservations, and uses voice recognition technology to call a designated number based on information entered by the user and to converse with the person at the call. It also includes means for prompting the user to input a response when an unexpected question is asked during the conversation, and emotion recognition means for recognizing the user's emotion and adjusting the tone of the conversation based on that emotion.

[0551] Hardware and software used

[0552] 1. Server:

[0553] Generative AI model: OpenAI's GPT-4 is used to analyze user-entered information.

[0554] Telephone API: Uses Twilio to make a call to a specified phone number.

[0555] Speech recognition technology: Google Cloud Speech-to-Text is used to convert the voice of the person on the other end of the call into text data.

[0556] Text-to-speech technology: Amazon Polly is used to provide voice-over reservation information.

[0557] Emotion Recognition Technology: Affectiva is used to analyze emotions from the user's tone of voice.

[0558] 2. Terminal:

[0559] User interface: Provides applications that run on devices such as smartphones and tablets.

[0560] Program processing

[0561] 1. User enters reservation information:

[0562] The user launches the application and accesses the reservation form.

[0563] The user inputs the reservation date and time, number of people, and name.

[0564] Example: A user enters the following on their smartphone screen: "I would like to make a reservation for four people on October 15, 2023 at 7:00 PM. My name is Taro Tanaka."

[0565] 2. The server receives and analyzes the information:

[0566] The server receives the information entered by the user.

[0567] The server uses a generative AI model (e.g., OpenAI's GPT-4) to analyze the input information.

[0568] Example: The server generates the analysis result "Taro Tanaka would like to make a reservation for 4 people on October 15, 2023 at 7:00 PM."

[0569] 3. The server makes the call:

[0570] The server uses a telephone API such as Twilio to make a call to the specified restaurant.

[0571] Example: Your server calls the Twilio API to call a restaurant's phone number.

[0572] 4. Servers converse with restaurant staff:

[0573] The server uses AI speech synthesis technology (e.g., Amazon Polly) to communicate the reservation information to restaurant staff.

[0574] Example: The server says, "Hello, my name is Taro Tanaka. I'd like to make a reservation for 4 people at 7:00 PM on October 15, 2023."

[0575] 5. The server responds to unexpected questions:

[0576] If an unexpected question comes from restaurant staff, the server will put the call on hold and ask the user to enter the answer on the app.

[0577] Example: The server receives the question "Would you like a window seat?" and responds "Please wait. I'll check with the user."

[0578] 6. User answers the question:

[0579] Users enter answers to questions on the app.

[0580] Example: A user types "Yes, I'd like a window seat" into an input field in an app.

[0581] 7. The server resumes the conversation and confirms the reservation:

[0582] The server continues the conversation with the restaurant staff based on the user's response and confirms the reservation.

[0583] Example: A server tells restaurant staff, "I'd like a window seat," and confirms that the reservation is confirmed.

[0584] Prompt Sentence Examples

[0585] "When a user wants to make a restaurant reservation, they enter the reservation date and time, number of people, and their name. Analyze this information and generate instructions to call the specified restaurant. Once the call is connected, use AI voice recognition technology to speak with the restaurant staff and complete the reservation. If an unexpected question comes up, ask the user to enter an answer and continue the conversation based on that answer."

[0586] The above is an embodiment of the invention.

[0587] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0588] Step 1:

[0589] The user enters the reservation information.

[0590] Input: The user accesses the application's reservation form and enters the reservation date and time, number of people, and name.

[0591] Specific operation: The user enters on the smartphone screen, "I would like to make a reservation for four people on October 15, 2023 at 7:00 PM. My name is Tanaka Taro."

[0592] Output: The entered reservation information is sent to the server.

[0593] Step 2:

[0594] The server receives and analyzes the information.

[0595] Input: Booking information entered by the user.

[0596] What it does: The server uses a generative AI model (e.g., OpenAI's GPT-4) to analyze the input information.

[0597] Data processing: Analyze the reservation date and time, number of people, and names, and format the reservation details in text format.

[0598] Output: Analysis result: "Taro Tanaka would like to make a reservation for 4 people on October 15, 2023 at 7:00 PM."

[0599] Step 3:

[0600] The server makes the call.

[0601] Input: Parsed reservation information.

[0602] Specific operation: The server uses a telephone API such as Twilio to make a call to the specified restaurant.

[0603] Data processing: Convert reservation information into a format to be passed to the telephone API.

[0604] Output: A call is made to the restaurant's phone number.

[0605] Step 4:

[0606] The server talks to the restaurant staff.

[0607] Input: Reservation information and phone connection.

[0608] Specific operation: The server uses AI speech synthesis technology (e.g., Amazon Polly) to communicate the reservation information to restaurant staff.

[0609] Data processing: Convert reservation information in text format into audio data.

[0610] Output: A voice message saying, "Hello, my name is Taro Tanaka. I'd like to make a reservation for 4 people on October 15, 2023 at 7:00 PM."

[0611] Step 5:

[0612] The server responds to unexpected questions.

[0613] Input: Question from restaurant staff.

[0614] What happens: The server uses speech recognition technology (e.g., Google Cloud Speech-to-Text) to convert the question into text data.

[0615] Data processing: Convert the voice data into text data and analyze the question content.

[0616] Output: Text data: "Would you like a window seat?"

[0617] Step 6:

[0618] The user answers the question.

[0619] Input: The question from the server.

[0620] What happens: The user enters an answer into an input field in the app.

[0621] Data processing: The user's answers are sent to the server as text data.

[0622] Output: "Yes, I'd like a window seat."

[0623] Step 7:

[0624] The server resumes the conversation and confirms the reservation.

[0625] Input: The user's answer.

[0626] Specific operation: The server again uses AI speech synthesis technology to convey the user's response to the restaurant staff.

[0627] Data processing: Convert the user's answers into audio data.

[0628] Output: A voice message saying "I would like a window seat."

[0629] The above is the flow of processing of the program of this system.

[0630] (Application example 2)

[0631] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0632] With conventional food delivery services, when a user places an order, they must manually enter information and confirm the order via phone or app. However, this is time-consuming and labor-intensive, and the user must respond again, especially if unexpected questions or changes arise. Furthermore, the system is unable to provide natural conversations that reflect the user's emotions, resulting in a poor user experience. To solve these issues, a system is needed that can automatically place orders based on the user's input information and enable natural conversations that reflect the user's emotions.

[0633] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0634] In this invention, the server includes means for making a call to a specified telephone number based on information input by the user, voice recognition technology means for conversing with the person at the telephone number, means for prompting the user to input a response when an irregular question is asked during the conversation, an emotion engine for recognizing the user's emotions, and means for adjusting the tone of the conversation based on the emotions recognized by the emotion engine. This enables a user to automatically communicate the order details, delivery time, and special requests to a restaurant over the phone when using a food delivery service, and also enables a natural conversation that reflects the user's emotions.

[0635] "User" means an individual or organization that uses the system to input information and receive services.

[0636] "Information" refers to data that a user enters into the system, including order details, delivery times, special requests, etc.

[0637] "Telephone number" refers to a contact point designated by the user, typically a restaurant or store that provides food delivery services.

[0638] "Means for making a call" is a function that allows the system to automatically make a call to a specified telephone number.

[0639] "Speech recognition technology means" refers to technology that enables the system to understand the conversation with the person on the other end of the phone and respond appropriately.

[0640] An "irregular question" is when the person on the other end of the phone asks an unexpected question or makes a request.

[0641] The "means for requesting input of an answer" is a function that prompts the user to input an answer when an irregular question is asked.

[0642] The "emotion engine" is a technology that recognizes emotions from the user's tone of voice and input content.

[0643] The "means for adjusting the tone of a conversation" is a function by which the system adjusts the tone of a conversation based on the emotion recognized by the emotion engine.

[0644] "System" means a device or program that includes a series of technical means for automatically making calls based on information entered by the user and for realizing natural conversations using voice recognition technology and an emotion engine.

[0645] A system for carrying out this invention includes means for making a call to a specified telephone number based on information input by a user, voice recognition technology means for conducting a conversation with the person at the telephone number, means for prompting the user to input a response when an irregular question is asked during the conversation, an emotion engine for recognizing the user's emotions, and means for adjusting the tone of the conversation based on the emotions recognized by the emotion engine.

[0646] Hardware and software used

[0647] Hardware: Smartphone

[0648] Software: Google Cloud Dialogflow, IBM Watson Tone Analyzer

[0649] Data processing and calculation

[0650] Receiving user input

[0651] Using a smartphone application, users enter information such as order details, delivery time, and special requests, which is then sent to a server.

[0652] sentiment analysis

[0653] The server uses IBM Watson Tone Analyzer to analyze the emotions in the user's text input. The emotion engine recognizes emotions from the user's tone of voice and input, and adjusts the tone of the conversation based on the results.

[0654] Intent detection

[0655] The server uses Google Cloud Dialogflow to detect the user's intent from the text they input, which then generates an appropriate response and instructions to call the specified phone number.

[0656] Making a phone reservation

[0657] Based on the generated response, the server uses AI to call the restaurant and place the order. Once the call is connected, the customer uses voice recognition technology to converse with the restaurant staff and complete the order.

[0658] Responding to irregular questions

[0659] If an unexpected question comes from the restaurant, the server puts the request on hold and asks the user to enter an answer on the application. Once the user enters the answer, the conversation with the restaurant staff continues based on the answer and the order is confirmed.

[0660] Specific examples

[0661] If a user types "Two pizzas, please deliver them at 7 PM," the emotion engine analyzes the user's emotions and Dialogflow generates an appropriate response. For example, if the user is happy, the AI ​​will proceed with the conversation in a tone such as "Thank you! Two pizzas will be delivered at 7 PM."

[0662] Prompt Sentence Examples

[0663] If a user types "Two pizzas, please deliver them at 7 PM," the emotion engine analyzes the user's emotions and Dialogflow generates an appropriate response. For example, if the user is happy, the AI ​​will proceed with the conversation in a tone such as "Thank you! Two pizzas will be delivered at 7 PM."

[0664] In this way, a food delivery assistant application that can have natural conversations according to the user's emotions is completed.

[0665] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0666] Step 1:

[0667] A user uses a smartphone application to enter information such as order details, delivery time, and special requests.

[0668] Input: Order details, delivery time, special requests

[0669] Output: The entered information is sent to the server

[0670] Specific behavior: When a user enters information into an application's input form and presses the "Submit" button, the information is sent to the server.

[0671] Step 2:

[0672] The server uses IBM Watson Tone Analyzer to analyze the sentiment from the user's input text.

[0673] Input: Information entered by the user

[0674] Output: Sentiment analysis results

[0675] Specific operation: The server sends the input text data to IBM Watson Tone Analyzer and receives the sentiment analysis results.

[0676] Step 3:

[0677] The server uses Google Cloud Dialogflow to detect the user's intent from the input text.

[0678] Input: Information entered by the user

[0679] Output: Intent detection result

[0680] Specific operation: The server sends the entered text data to Google Cloud Dialogflow and receives the intent detection result.

[0681] Step 4:

[0682] Based on the generated response, the server generates an instruction to call the specified telephone number.

[0683] Input: Intention detection results, sentiment analysis results

[0684] Output: Instructions to make a call

[0685] Specific operation: The server generates an appropriate response based on the intent detection and sentiment analysis results, and generates instructions to make a call.

[0686] Step 5:

[0687] The server calls the designated number and uses voice recognition technology to talk to restaurant staff.

[0688] Input: Instructions to make a call

[0689] Output: Conversation

[0690] What it does: The server automatically makes a call and uses voice recognition technology to speak to restaurant staff.

[0691] Step 6:

[0692] If an unexpected question comes from the restaurant, the server will put the request on hold and ask the user to enter an answer on the application.

[0693] Input: Unexpected question

[0694] Output: Prompt the user for an answer

[0695] What happens: The server detects the unexpected question and sends a message to the application prompting the user to enter an answer.

[0696] Step 7:

[0697] The user inputs an answer, and the server continues the conversation with the restaurant staff based on the answer and confirms the order.

[0698] Input: User's answer

[0699] Output: Confirmed order details

[0700] Specific behavior: The user enters an answer into the application, and the server uses the answer to continue the conversation with the restaurant staff and confirm the order.

[0701] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0702] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0703] Another example of generative AI is Gemini (registered trademark) (Internet search engine). <url: https: gemini.google.com ?hl="ja">) are listed.

[0704] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0705] [Second embodiment]

[0706] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0707] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0708] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0709] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0710] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0711] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0712] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0713] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0714] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0715] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0716] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0717] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0718] "Example 1"

[0719] In one embodiment of the present invention, a user inputs information such as reservation date and time, number of people, and names into a smartphone application. This information is analyzed by generative AI, which generates instructions to call the specified number. Once the call is connected, the conversation with the person on the other end is carried out using AI voice recognition technology. If an irregular question is asked during the conversation, the system puts the call on hold with a "Please wait" message and prompts the user to enter a response on the app. Once the user has entered a response, the conversation with the person on the other end is continued based on that response.

[0720] "Example 2"

[0721] As a specific example, when a user wants to make a restaurant reservation, they input the reservation date and time, number of people, and name on the application. This information is analyzed by generative AI, which generates instructions to call the specified restaurant. Once the call is connected, AI voice recognition technology is used to converse with the restaurant staff and complete the reservation. If the restaurant asks an unexpected question, the app puts the call on hold by saying "Please wait" and asks the user to enter a response on the app. Once the user has entered a response, the conversation with the restaurant staff continues based on that response, and the reservation is confirmed.

[0722] The processing flow of each embodiment will be described below.

[0723] "Example 1"

[0724] Step 1: The user enters information such as reservation date and time, number of people, and name on the smartphone application.

[0725] Step 2: This information is analyzed by a generative AI to generate instructions to call the specified number.

[0726] Step 3: Once the call is connected, use AI voice recognition technology to talk to the person on the other end.

[0727] Step 4: If an irregular question comes up during the conversation, put it on hold with "Please wait" and ask the user to enter the answer on the app.

[0728] Step 5: Once the user has entered their answer, the conversation with the person on the other end of the phone continues based on that answer.

[0729] "Example 2"

[0730] Step 1: When a user wants to make a restaurant reservation, they input the reservation date and time, number of people, and name on the application.

[0731] Step 2: This information is analyzed by a generative AI, which generates instructions to call the specified restaurant.

[0732] Step 3: Once the call is connected, AI voice recognition technology is used to speak to the restaurant staff and complete the reservation.

[0733] Step 4: If the restaurant asks an unexpected question, the app will put the question on hold and ask the user to enter the answer on the app.

[0734] Step 5: Once the user has entered their answer, they can continue the conversation with the restaurant staff based on their answer and confirm the reservation.

[0735] Example 1

[0736] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0737] Conventional reservation systems require users to manually make phone calls and make reservations, which is time-consuming and makes it difficult to respond appropriately when an irregular question comes up during a conversation with the person on the other end of the phone. Furthermore, there is a lack of a way to efficiently analyze the information entered by the user and generate an appropriate response.

[0738] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0739] In this invention, the server includes means for making a call to a specified telephone number based on information input by the user, speech recognition technology means for conversing with the person at the telephone number, means for prompting the user to input a response when an irregular question is asked during the conversation, means for using a generative AI model for analyzing the information input by the user, means for generating a prompt sentence based on the analysis result, and means for analyzing the speech of the person at the telephone number in real time and generating an appropriate response. This allows the user to make a reservation without hassle and makes it possible to appropriately respond to irregular questions.

[0740] "User" means an individual or group that uses the system to make reservations or enter information.

[0741] "Information" refers to data such as reservation date and time, number of people, and names entered by the user.

[0742] "Telephone number" refers to the telephone number at which a user makes a reservation or inquiry.

[0743] "Means for making phone calls" refers to the ability to make calls to designated telephone numbers using VoIP technology.

[0744] "Voice recognition technology means" refers to technology that converts the speech of the person on the other end of the phone into text in real time.

[0745] An "irregular question" refers to an unexpected question or confirmation from the person on the other end of the phone.

[0746] "Means for requesting answer input" refers to an interface that allows the user to input an answer when an irregular question is asked.

[0747] A "generative AI model" refers to an artificial intelligence model that analyzes information entered by a user and generates appropriate responses or instructions.

[0748] A "prompt" refers to an instruction or question that is input into a generative AI model.

[0749] "Means of analysis" refers to the function of analyzing information entered by the user using a generative AI model.

[0750] "Means for generating a response" refers to the function of analyzing what the person on the other end of the phone says and generating an appropriate reply.

[0751] This invention is a system that efficiently and automatically makes calls and provides necessary information when a user makes a reservation or makes an inquiry using a smartphone application. Specific embodiments of this system are described below.

[0752] Hardware and software used

[0753] Hardware

[0754] Smartphone (iOS or Android)

[0755] Server (Cloud server or on-premise server)

[0756] software

[0757] Smartphone application

[0758] Generative AI models (e.g., OpenAI's GPT-4)

[0759] Speech recognition technology (e.g., Google Cloud Speech-to-Text)

[0760] VoIP technology (e.g., Twilio API)

[0761] Data processing and calculation

[0762] User Input

[0763] The user opens the smartphone application and enters information such as the reservation date and time, number of people, and name into the reservation form. Once the information is complete, the user presses the send button.

[0764] Sending data

[0765] The device converts the information entered by the user into JSON format and sends it to a server using the HTTPS protocol, with the destination URL set in advance.

[0766] Data analysis

[0767] The server analyzes the received JSON data and inputs it as a prompt to the generative AI model. An example of a prompt might be, "The user would like to make a restaurant reservation. Please call the restaurant and make the reservation using the following information: Reservation date and time: December 24, 2023, 7:00 PM, Number of people: 4, Name: Yamada Taro."

[0768] Making a call

[0769] The server uses VoIP technology to make a call to the specified phone number, for example, by using the Twilio API. Once the call is connected, it uses AI voice recognition technology to start a conversation with the person on the other end.

[0770] Conversation progression

[0771] The server uses AI speech recognition technology to convert the caller's speech into text in real time, and then uses a generative AI model to generate an appropriate response. For example, if the caller says, "Your reservation has been accepted," the server will respond with, "Thank you."

[0772] Prompt the user

[0773] If an irregular question comes up during a conversation, the server will tell the other party "Please wait" and put the conversation on hold. The device will display the irregular question to the user on the application and ask them to enter an answer. For example, if the other party asks "Do you have any allergies?", the device will display the question to the user and ask them to enter an answer.

[0774] Restarting the conversation

[0775] When the user enters "no allergies," that information is sent to the server. The server then continues the conversation with the person on the other end of the phone based on the user's response. For example, it might respond, "Sorry to keep you waiting. I don't have any allergies." This completes the reservation process.

[0776] Specific examples

[0777] For example, when a user makes a restaurant reservation, they input the following prompt into the generative AI model:

[0778] Example prompt sentence:

[0779] A user would like to make a restaurant reservation. Please call the restaurant using the following information to make the reservation:

[0780] Reservation date and time: December 24, 2023 7:00 PM

[0781] Number of people: 4 people

[0782] Name: Taro Yamada

[0783] Based on this prompt, the generative AI model generates the following instructions:

[0784] "We would like to make a reservation for 4 people on December 24, 2023 at 7:00 PM. Can you accept our reservation?"

[0785] After the call is connected, the app uses AI voice recognition technology to continue the conversation, prompting the user for confirmation as needed. For example, if the restaurant asks if the user has any allergies, the app displays the question to the user and allows them to enter their answer. If the user enters "no allergies," the app resumes the conversation based on that information.

[0786] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0787] Step 1:

[0788] The user opens the smartphone application and enters information such as the reservation date and time, number of people, and name into the reservation form. Once the information is complete, the user presses the send button.

[0789] Input: Information such as reservation date and time, number of people, name, etc.

[0790] Output: When the submit button is pressed, the input information is sent to the next step.

[0791] Step 2:

[0792] The device converts the information entered by the user into JSON format and sends it to a server using the HTTPS protocol, with the destination URL set in advance.

[0793] Input: Information entered by the user

[0794] Output: JSON format data is sent to the server

[0795] Step 3:

[0796] The server analyzes the received JSON data and inputs it as a prompt to the generative AI model. An example of a prompt might be, "The user would like to make a restaurant reservation. Please call the restaurant and make the reservation using the following information: Reservation date and time: December 24, 2023, 7:00 PM, Number of people: 4, Name: Yamada Taro."

[0797] Input: JSON format data

[0798] Output: The prompt sentence that is input to the generative AI model

[0799] Step 4:

[0800] The server uses the generative AI model to generate instructions for calling the specified phone number based on the prompt text.

[0801] Input: prompt statement

[0802] Output: Instructions to make a call

[0803] Step 5:

[0804] The server places the call to the specified destination using VoIP technology, for example, by using the Twilio API to make the call.

[0805] Input: Instructions to make a call

[0806] Output: Phone is connected

[0807] Step 6:

[0808] Once the call is connected, the server uses AI voice recognition technology to begin a conversation with the person on the other end of the line.

[0809] Input: Phone connected

[0810] Output: A conversation with the person on the other end of the phone is initiated

[0811] Step 7:

[0812] The server uses AI speech recognition technology to convert the caller's speech into text in real time, and then uses a generative AI model to generate an appropriate response. For example, if the caller says, "Your reservation has been accepted," the server will respond with, "Thank you."

[0813] Input: What the person on the other end of the phone says

[0814] Output: Translated utterances and generated responses

[0815] Step 8:

[0816] If an irregular question comes up during a conversation, the server will tell the other party to wait and put the conversation on hold. The device will then display the irregular question to the user on the application and ask them to enter an answer.

[0817] Input: Irregular Question

[0818] Output: The question and answer prompt that are displayed to the user

[0819] Step 9:

[0820] Once the user enters their answer, the information is sent to the server. For example, if someone asks if they have any allergies, the server displays the question and prompts the user to enter their answer.

[0821] Input: User's answer

[0822] Output: The answer sent to the server

[0823] Step 10:

[0824] The server continues the conversation with the person on the other end of the phone based on the user's response. For example, it might respond, "Sorry to keep you waiting. I don't have any allergies." This completes the reservation process.

[0825] Input: User's answer

[0826] Output: A response that is conveyed to the person on the other end of the phone and the completion of the reservation

[0827] (Application example 1)

[0828] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0829] Conventional food delivery services often require users to make reservations or orders over the phone when ordering from a restaurant. However, this poses problems, such as the time and effort required for users to make phone calls and the need to respond to irregular questions that arise during conversations with restaurant staff. Furthermore, depending on the time of day when a user calls, the restaurant may be busy and the response may be delayed. This reduces user convenience and impairs the quality of service, creating issues.

[0830] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0831] In this invention, the server includes: means for making a call to a specified telephone number based on information input by a user; AI speech recognition technology means for conversing with the person at the telephone number; means for prompting the user to input an answer when an irregular question is asked during the conversation; means for continuing the conversation with the person at the telephone number based on the answer input by the user; means for analyzing the information input by the user using generative AI and generating the telephone call content; means for reading out the generated telephone call content using speech synthesis technology; and means for recognizing the question from the person at the telephone number using speech recognition technology. This allows users to easily make reservations and orders through a smartphone app and can quickly respond to irregular questions, thereby improving user convenience and improving the quality of service.

[0832] "User" means an individual or corporation that makes reservations or orders using the system.

[0833] "Input information" refers to data such as reservation date and time, number of people, names, order details, etc. that the user provides to the system.

[0834] The "designated telephone number" is the telephone number of the restaurant or service provider for which the user wishes to make a reservation or order.

[0835] "Means for making a call" is a function that allows the system to automatically make a call to a specified telephone number.

[0836] "AI voice recognition technology means" is a technology that uses artificial intelligence to recognize the voice of the person on the other end of the phone and convert it into text data.

[0837] An "irregular question" is a situation in which an unexpected question or request arises from the person on the other end of the phone.

[0838] The "means for prompting the user to input an answer" is a function that prompts the user to input an appropriate answer when an irregular question occurs.

[0839] The "means for continuing the conversation again" is a function for continuing the conversation with the person on the other end of the phone call based on the answer entered by the user.

[0840] "Generative AI" is an artificial intelligence technology that analyzes information entered by the user and generates appropriate telephone content.

[0841] "Means for generating telephone content" is a function that uses generative AI to create the content to be conveyed over the phone based on the information entered by the user.

[0842] "Speech synthesis technology" is a technology that converts text data into speech and reads it out in a natural voice.

[0843] "A means for reading out loud" is a function that uses voice synthesis technology to convey the generated telephone call contents aloud.

[0844] "Voice recognition technology" is a technology that recognizes the voice of the person on the other end of the phone and converts it into text data.

[0845] The "means for recognizing questions" is a function that uses voice recognition technology to understand the question asked by the person on the other end of the phone and acquires it as text data.

[0846] To implement this invention, the following hardware and software are required. The hardware requires a smartphone and a microphone. The software uses Python, OpenAI API, the SpeechRecognition library, and the pyttsx3 library.

[0847] The server provides a means to call a specified number based on the information entered by the user. When a user enters reservation or order information through a smartphone app, the information is sent to the server. The server uses generative AI to analyze the information entered by the user and generate the phone call content. This generated phone call content is read aloud using speech synthesis technology.

[0848] The terminal provides AI speech recognition technology for conversation with the person on the other end of the phone. When the call is connected, the AI ​​speech recognition technology is used to recognize the voice of the person on the other end of the phone and convert it into text data. If an irregular question is asked during the conversation, the terminal puts the call on hold and asks the user to enter a response. Once the user enters a response, the terminal continues the conversation with the person on the other end based on that response.

[0849] As a specific example, if a user inputs "I would like to make a reservation for two people at 7pm tomorrow," the server analyzes this information and calls the specified restaurant. The generative AI generates the phone call message "Please make a reservation for two people at 7pm tomorrow," and reads it out loud using speech synthesis technology. If a restaurant staff member asks, "Do you have any allergies?", the device recognizes this question and asks the user for a response. If the user inputs "no," the AI ​​again replies, "I don't have any allergies," and the conversation continues.

[0850] An example of a prompt is as follows:

[0851] User input: I would like to make a reservation for 2 people tomorrow at 7pm.

[0852] Call the restaurant and tell them the following:

[0853] In this way, users can easily make reservations and orders through a smartphone app, and irregular inquiries can be quickly responded to, improving user convenience and the quality of service.

[0854] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0855] Step 1:

[0856] Users enter reservation and order information through a smartphone app.

[0857] Input: Information such as reservation date and time, number of people, name, order details, etc.

[0858] Output: The entered information is sent to the server

[0859] Specific behavior: The user enters the required information into the app's input form and presses the submit button.

[0860] Step 2:

[0861] The server uses generative AI to analyze the information entered by the user and generate the content of the call.

[0862] Input: Information entered by the user

[0863] Output: Generated call contents (text format)

[0864] Specific operation: The server calls the OpenAI API, passes the input information as a prompt, and generates the appropriate call content.

[0865] Step 3:

[0866] The server uses speech synthesis technology to read out the generated phone call content aloud.

[0867] Input: Generated call content (text format)

[0868] Output: Audio file

[0869] Specific operation: The server uses the pyttsx3 library to convert the generated text into speech and create an audio file.

[0870] Step 4:

[0871] The server makes a call to the specified telephone number.

[0872] Input: specified phone number, audio file

[0873] Output: Establishing a telephone connection

[0874] Specific operation: The server uses the telephone API to call the specified phone number and play the audio file.

[0875] Step 5:

[0876] The device uses AI voice recognition technology to recognize the voice of the person on the other end of the phone and convert it into text data.

[0877] Input: The voice of the person you are calling

[0878] Output: Recognized text data

[0879] What it does: The device uses the SpeechRecognition library to convert the other person's voice into text in real time.

[0880] Step 6:

[0881] If an irregular question comes up during a conversation, the terminal will put the question on hold and ask the user to enter an answer.

[0882] Input: Recognized text data (irregular questions)

[0883] Output: prompts the user for input

[0884] Specific behavior: The device detects an irregular question and notifies the user to enter an answer in the app.

[0885] Step 7:

[0886] The user enters the answer in the app.

[0887] Input: User's answer

[0888] Output: The entered answer is sent to the server

[0889] Specific behavior: The user enters answers into the app's input form and presses the submit button.

[0890] Step 8:

[0891] The server continues the conversation with the person on the other end of the phone based on the user's response.

[0892] Input: User's answer

[0893] Output: Re-conversation content (text format)

[0894] Specific operation: The server calls the generative AI again, passes the user's answer as a prompt, and generates appropriate re-conversation content.

[0895] Step 9:

[0896] The server uses speech synthesis technology to read out the contents of the re-conversation aloud.

[0897] Input: Re-conversation content (text format)

[0898] Output: Audio file

[0899] Specific operation: The server uses the pyttsx3 library to convert the replay content into audio and create an audio file.

[0900] Step 10:

[0901] The server then plays the audio file again to the person on the other end of the line, and the conversation continues.

[0902] Input: Audio file

[0903] Output: Audio playback to the person you are calling

[0904] Specific operation: The server uses the telephone API to play the audio file of the conversation and continue the conversation.

[0905] Example 2

[0906] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0907] Conventional restaurant reservation systems require users to manually make phone calls, making the reservation process cumbersome. Furthermore, if an unexpected question arose during the call, the user was often unable to respond immediately, resulting in a delay in confirming the reservation. Furthermore, there was a lack of a system that could efficiently analyze the information entered by the user and provide an appropriate response.

[0908] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0909] In this invention, the server includes means for making a call to a specified telephone number based on information input by the user, artificial intelligence speech recognition technology means for conversing with the person at the telephone number, means for prompting the user to input a response when an irregular question is asked during the conversation, artificial intelligence model generation means for analyzing the information input by the user, communication API means for making the call, and means for notifying the user of the confirmation of the reservation. This enables users to easily make restaurant reservations and quickly respond to unexpected questions.

[0910] "User" refers to an individual or organization that uses the system to make restaurant reservations.

[0911] "Telephone number" refers to the restaurant or other facility where the user wishes to make a reservation.

[0912] "Artificial intelligence speech recognition technology" refers to technology that converts speech into text and understands the content of conversations.

[0913] "Irregular questions" refer to unexpected questions or confirmations that arise during the reservation process.

[0914] A "generative artificial intelligence model" refers to an artificial intelligence model that analyzes information entered by a user and generates appropriate instructions and responses.

[0915] "Communications API" refers to the application programming interface used by the server to place a call.

[0916] A "reservation confirmation notice" refers to a notice to inform the user that the reservation has been confirmed.

[0917] This invention is a system that allows users to easily make restaurant reservations. The system uses artificial intelligence speech recognition technology to call a specified telephone number based on information entered by the user and to converse with the person at the telephone number. The system also includes a means for prompting the user to input a response if an irregular question is asked during the conversation.

[0918] Hardware and software used

[0919] 1. Server

[0920] The server uses a generative artificial intelligence model (e.g., GPT-4) to analyze the information entered by the user and generate appropriate instructions.

[0921] The server uses a communications API (e.g., Twilio) to make the call.

[0922] The server uses artificial intelligence speech recognition technology (e.g., Google Speech-to-Text) to conduct a conversation with the person on the other end of the phone.

[0923] 2. Terminal

[0924] The terminal transmits the information entered by the user to the server.

[0925] The terminal receives instructions from the server and displays them to the user.

[0926] 3. Users

[0927] The user uses the application to enter reservation information.

[0928] The user inputs answers to the irregular questions.

[0929] Specific examples

[0930] The user opens the application and enters the following information:

[0931] Reservation date and time: December 24, 2023 7:00 PM

[0932] Number of people: 4 people

[0933] Name: Taro Yamada

[0934] This information is analyzed by a generative AI model (GPT-4), which generates instructions to call the specified restaurant using Twilio. Once the call is connected, a conversation with the restaurant staff takes place using Google Speech-to-Text. For example, if the staff asks, "Do you have any allergies?", the server puts the call on hold with a "Please wait" message and displays "Do you have any allergies?" on the terminal. If the user enters "No," the conversation with the staff continues based on that response, and the reservation is confirmed.

[0935] Prompt Sentence Examples

[0936] An example of a prompt to be input to the generative AI model is as follows:

[0937] The user entered the following information:

[0938] Reservation date and time: December 24, 2023 7:00 PM

[0939] Number of people: 4 people

[0940] Name: Taro Yamada

[0941] Use this information to generate instructions to call the designated restaurant and complete the reservation.

[0942] In this way, a system is realized that allows users to easily make reservations at restaurants.

[0943] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0944] Step 1:

[0945] The user enters the reservation information.

[0946] Input: reservation date and time, number of people, name

[0947] Specific operation: The user opens the application and enters the reservation date and time "December 24, 2023 19:00", the number of people "4", and the name "Yamada Taro".

[0948] Output: The entered reservation information is saved on the device.

[0949] Step 2:

[0950] The terminal sends the input information to the server.

[0951] Input: Booking information entered by the user

[0952] Specific operation: The terminal sends the reservation information entered by the user to the server.

[0953] Output: The server receives the reservation information.

[0954] Step 3:

[0955] The server analyzes the input information using a generative AI model.

[0956] Input: Received reservation information

[0957] How it works: The server uses a generative AI model (e.g., GPT-4) to analyze the reservation information entered by the user.

[0958] Output: The parsing results in instructions to call the restaurant.

[0959] Step 4:

[0960] The server uses the phone API to call the restaurant.

[0961] Input: Instructions generated as a result of the analysis

[0962] Specific operation: The server uses a telephone API such as Twilio to call the specified restaurant.

[0963] Output: A telephone connection is established with the restaurant.

[0964] Step 5:

[0965] The server uses AI voice recognition technology to converse with restaurant staff.

[0966] Input: Conversation with restaurant staff

[0967] Specific operation: The server uses AI voice recognition technology such as Google Speech-to-Text to converse with restaurant staff and proceed with the reservation.

[0968] Output: The conversation is parsed into text.

[0969] Step 6:

[0970] The server prompts the user for an answer to an unexpected question.

[0971] Input: Unexpected question from restaurant staff

[0972] Specific operation: When a restaurant staff member asks, "Do you have any allergies?", the server puts the customer on hold by saying, "Please wait," and displays "Do you have any allergies?" on the terminal.

[0973] Output: The question is displayed to the user.

[0974] Step 7:

[0975] The user enters the answer.

[0976] Input: User's answer to the question displayed on the terminal

[0977] Specific action: The user enters "No" in response to the question displayed on the terminal.

[0978] Output: The entered answer is saved on the device.

[0979] Step 8:

[0980] The terminal sends the user's answer to the server.

[0981] Input: The answer entered by the user

[0982] Specific operation: The device sends the answer entered by the user to the server.

[0983] Output: The server receives the answer.

[0984] Step 9:

[0985] The server continues the conversation with the restaurant staff.

[0986] Input: User's answer

[0987] Specific operation: Based on the user's answer, the server continues the conversation with the restaurant staff and confirms the reservation.

[0988] Output: The reservation is confirmed.

[0989] Step 10:

[0990] The server confirms the reservation and notifies the user.

[0991] Input: Confirmed reservation information

[0992] Specific behavior: The server notifies the user that the reservation has been confirmed and displays the reservation details.

[0993] Output: A confirmation of the booking is sent to the user.

[0994] (Application example 2)

[0995] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0996] With conventional food delivery systems, users had to manually call to place an order, which was time-consuming and could lead to miscommunication and other communication issues. Another issue was that it was difficult to respond immediately to unexpected questions from the restaurant, which meant it took a long time to confirm the order.

[0997] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0998] In this invention, the server includes: a means for making a call to a specified telephone number based on information entered by the user; an AI speech recognition technology means for conversing with the person at the telephone number; a means for prompting the user to input a response if an irregular question is asked during the conversation; a means for analyzing the order information entered by the user and generating instructions for the telephone order using a generative AI model; and a means for communicating the generated instructions to the person at the telephone number using speech synthesis technology. This eliminates the need for the user to make manual calls, reduces miscommunication of order details, and enables quick response to unexpected questions.

[0999] "User" means an individual or corporation that uses the system to input information and receive services.

[1000] "Input information" refers to data provided by the user to the system, and includes order details, reservation information, and the like.

[1001] A "designated telephone number" is a telephone number that the system will call based on the information entered by the user.

[1002] "Means for making a call" is a function that allows the system to automatically make a call to a specified telephone number.

[1003] "AI speech recognition technology means" is a technology that uses artificial intelligence to convert speech into text and understand the content of the conversation.

[1004] An "irregular question" is a question or request that the system does not anticipate and that requires user input.

[1005] The "means for prompting the user to input an answer" is a function that prompts the user to input an answer when an irregular question occurs.

[1006] "Order information" refers to the specific order details provided by the user to the system, including the name of the dish, the quantity, the delivery address, the desired delivery time, and the like.

[1007] A "generative AI model" is an artificial intelligence model that analyzes information entered by the user and generates appropriate instructions and answers.

[1008] "Telephone order instructions" are the specific order details and instructions generated by the generative AI model that should be conveyed to the person on the other end of the phone.

[1009] "Speech synthesis technology" is a technology that converts text data into speech and communicates it to the person on the other end of the phone in a natural voice.

[1010] A system for implementing this invention includes means for making a call to a specified telephone number based on information entered by a user, AI voice recognition technology means for conducting a conversation with the person at the telephone, means for prompting the user to input a response when an irregular question arises during the conversation, means for analyzing the order information entered by the user and generating instructions for a telephone order using a generation AI model, and means for conveying the generated instructions to the person at the telephone using voice synthesis technology.

[1011] The server receives order information entered by the user on the food delivery application, including the name of the dish, the quantity, the delivery address, the desired delivery time, etc. The server analyzes this information and generates phone order instructions using a generative AI model (e.g., OpenAI's API).

[1012] The generated instructions are used to automatically call the specified phone number using the Twilio API. After the call is connected, the generated instructions are conveyed to the caller using speech synthesis technology (e.g., pyttsx3). If the caller asks an unexpected question, the server puts the call on hold and asks the user to enter an answer in the application. Once the user enters an answer, the server uses the generative AI model again to generate new instructions and convey them to the caller.

[1013] As a specific example, if a user inputs "one pizza, two salads, delivery destination Shibuya-ku, Tokyo, desired delivery time 6:00 p.m.", the following prompt sentence will be input to the generative AI model.

[1014] Example prompt sentence:

[1015] "User would like to place the following order: 1 pizza, 2 salads, delivery location: Shibuya-ku, Tokyo, desired delivery time: 6 PM. Please call the restaurant to complete the order."

[1016] Based on this prompt, the generative AI model generates instructions to call the restaurant, and then uses the Twilio API to actually make the call. The order is then conveyed using speech synthesis technology, and the order is completed.

[1017] This system saves users the trouble of manually making phone calls, reduces miscommunication of orders, and allows for quick response to unexpected questions.

[1018] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1019] Step 1:

[1020] A user enters order information on a food delivery application.

[1021] Input: Order information such as dish name, quantity, delivery address, desired delivery time, etc.

[1022] Output: Order information sent to the server

[1023] Specific operation: The user enters the required information into the application's order form and presses the "Order" button.

[1024] Step 2:

[1025] The server analyzes the order information received from the user.

[1026] Input: Order information submitted by the user

[1027] Output: Parsed order information

[1028] Specific operation: The server receives the order information and extracts each item (food name, quantity, delivery address, desired delivery time).

[1029] Step 3:

[1030] The server uses the generative AI model to generate phone order instructions.

[1031] Input: Parsed order information

[1032] Output: Generated phone order instructions

[1033] Specific operation: The server converts the parsed order information into prompt sentences and inputs them into a generative AI model (e.g., OpenAI's API) to generate telephone order instructions.

[1034] Step 4:

[1035] The server automatically makes a call to the specified phone number using the Twilio API.

[1036] Input: Generated phone order instructions, designated phone number

[1037] Output: Establishing a telephone connection

[1038] Specific operation: The server calls the Twilio API and makes a call to the specified phone number.

[1039] Step 5:

[1040] The server uses voice synthesis technology to generate instructions and transmit them to the person on the other end of the phone.

[1041] Input: Generated phone order instructions

[1042] Output: Voice instructions that are conveyed to the person on the other end of the call

[1043] Specific operation: The server uses a speech synthesis engine (e.g., pyttsx3) to convert the generated instructions into speech and convey them to the person on the other end of the phone.

[1044] Step 6:

[1045] If the person on the other end of the phone asks an unexpected question, the server puts the call on hold and asks the user to enter an answer on the application.

[1046] Input: An unexpected question from the person on the other end of the phone

[1047] Output: Prompt the user to enter an answer

[1048] Specific operation: The server uses voice recognition technology to detect unexpected questions, puts the call on hold with a "Please wait" message, and sends a notification to the user requesting them to enter an answer in the application.

[1049] Step 7:

[1050] The user enters an answer into the application, and the server again uses the generative AI model to generate new instructions and convey them to the person on the other end of the phone.

[1051] Input: User response

[1052] Output: Newly generated instructions, voice instructions to be conveyed to the person on the other end of the call

[1053] Specific operation: The user enters an answer on the application, and the server receives the answer and uses the generative AI model to generate new instructions, which are then conveyed to the person on the other end of the phone using voice synthesis technology.

[1054] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1055] "Example 1"

[1056] One embodiment of the present invention is a system that incorporates an emotion engine. This system includes a means for making a call to a specified telephone number based on information entered by a user, an AI voice recognition technology means for conducting a conversation with the person at the telephone number, a means for prompting the user to input a response when an irregular question is asked during the conversation, and an emotion engine that recognizes the user's emotions. The emotion engine recognizes emotions from the tone of the user's voice and adjusts the tone of the conversation based on that emotion. For example, if the user feels angry, the emotion engine conveys that information to the AI ​​voice recognition technology means, and the AI ​​adjusts the tone of the conversation based on that information. This enables a more natural conversation that reflects the user's emotions.

[1057] "Example 2"

[1058] Another embodiment of the present invention is a system that incorporates an emotion engine. This system includes a means for making a call to a specified telephone number based on information entered by a user, an AI voice recognition technology means for conducting a conversation with the person at the telephone number, a means for prompting the user to input a response when an irregular question is asked during the conversation, and an emotion engine that recognizes the user's emotions. The emotion engine recognizes emotions from the tone of the user's voice and adjusts the tone of the conversation based on that emotion. For example, if the user feels happy, the emotion engine conveys that information to the AI ​​voice recognition technology means, and the AI ​​adjusts the tone of the conversation based on that information. This enables a more natural conversation that reflects the user's emotions.

[1059] The processing flow of each embodiment will be described below.

[1060] "Example 1"

[1061] Step 1: The user enters reservation details into the app.

[1062] Step 2: The system will call the specified number.

[1063] Step 3: AI voice recognition technology means the conversation with the person on the other end of the phone.

[1064] Step 4: If an irregular question comes up during the conversation, the system prompts the user to enter an answer.

[1065] Step 5: The emotion engine recognizes the emotion from the user's tone of voice.

[1066] Step 6: The emotion engine adjusts the tone of the conversation based on that emotion.

[1067] "Example 2"

[1068] Step 1: The user enters reservation details into the app.

[1069] Step 2: The system will call the specified number.

[1070] Step 3: AI voice recognition technology means the conversation with the person on the other end of the phone.

[1071] Step 4: If an irregular question comes up during the conversation, the system prompts the user to enter an answer.

[1072] Step 5: The emotion engine recognizes the emotion from the user's tone of voice.

[1073] Step 6: The emotion engine adjusts the tone of the conversation based on that emotion.

[1074] Example 1

[1075] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1076] Conventional telephone reservation systems require users to make a direct call, which makes it difficult to respond to irregular inquiries. Furthermore, they are unable to respond to the user's emotions, which can lead to a poor user experience. There is a need to solve these problems and provide a more natural and efficient telephone reservation system.

[1077] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1078] In this invention, the server includes means for making a call to a specified telephone number based on information input by the user, a voice recognition technology means for conversing with the person at the telephone number, means for prompting the user to input a response if an irregular question is asked during the conversation, and means for recognizing the user's emotions and adjusting the tone of the conversation based on the emotions. This allows the user to make reservations efficiently without having to make a direct call, and irregular questions can be quickly responded to. Furthermore, it is possible to respond according to the user's emotions, providing a more natural and satisfying user experience.

[1079] "User" means an individual or group that uses the system to make reservations or enter information.

[1080] "Information" refers to data that users input into the system, such as reservation date and time, number of people, and names.

[1081] "Telephone number" refers to a location or organization that has a telephone number where a user can make a reservation or inquiry.

[1082] The "means for making a call" refers to the function or technology that enables the server to make a call to a specified telephone number.

[1083] "Speech recognition technology means" refers to technology that converts the speech of the person on the other end of the phone into text in real time and generates an appropriate response.

[1084] An "irregular question" is an unexpected question or request that is not part of the normal booking process or conversation flow.

[1085] The "means for requesting input of an answer" is a function that prompts the user to input an answer to an irregular question when the question is asked.

[1086] The "means for recognizing emotions" is a technology that analyzes and recognizes emotions from the tone of a user's voice and other voice characteristics.

[1087] The "means for adjusting the tone of the conversation" is a function that appropriately changes the tone and content of the voice response generated by the system based on the recognized user's emotions.

[1088] This invention is a system that allows a user to input reservation information using a smartphone application and then makes a call to a specified telephone number based on that information. The system analyzes the information entered by the user and uses voice recognition technology to converse with the person on the other end of the phone. The system also includes a means for prompting the user to input a response if an irregular question is asked during the conversation. It also includes a means for recognizing the user's emotions and adjusting the tone of the conversation based on those emotions.

[1089] Hardware and software used

[1090] Hardware

[1091] Smartphone: A device used by users to enter reservation information.

[1092] Server: A central processing unit for analyzing data, making calls, performing voice recognition, and emotion recognition.

[1093] software

[1094] Application: Runs on a smartphone and provides an interface for users to enter reservation information.

[1095] Generative AI models: For example, OpenAI's GPT-4 is used to analyze user-entered information and generate appropriate instructions.

[1096] Speech recognition technology: For example, Google Cloud Speech-to-Text is used to convert what the person on the other end of the phone says into text in real time.

[1097] Emotion Recognition Technology: For example, Affectiva's emotion recognition technology is used to recognize emotions from the user's tone of voice.

[1098] Cloud communications platforms: For example, using Twilio to make calls to specified destinations.

[1099] Specific examples

[1100] Example 1: Restaurant reservations

[1101] A user opens a smartphone application and enters "October 15, 2023, 7 PM, 4 people, Taro Tanaka." The device sends this information to the server. The server uses a generative AI model to generate the instruction "Please make a reservation for Taro Tanaka on October 15, 2023 at 7 PM for 4 people," and uses Twilio to call the restaurant. When the call connects, the server says, "Hello, my name is Taro Tanaka. I'd like to make a reservation for 4 people on October 15 at 7 PM." If the restaurant asks, "Do you have any special requests?" the server replies, "Please wait," and sends a notification to the device. If the user enters, "I'd like a window seat," the device sends this information to the server, and the server again tells the person on the phone, "I'd like a window seat."

[1102] Example 2: Using the Emotion Engine

[1103] The user enters "October 15, 2023, 7:00 PM, 4 people, Taro Tanaka" into the app, and the call is connected. If the user begins to speak in an angry tone during the conversation, the emotion engine recognizes that emotion and passes it on to the server. The server passes that information on to the AI, which responds calmly with, "I'm sorry, I'll check right away."

[1104] Prompt Sentence Examples

[1105] "Please explain a system in which a user inputs reservation information into a smartphone application, which then uses that information to call a specified number. Once the call is connected, the system uses AI voice recognition technology to conduct the conversation, and if an irregular question is asked, please include a mechanism to prompt the user for an answer. Also, please explain how an emotion engine is used to adjust the tone of the conversation according to the user's emotions."

[1106] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1107] Step 1:

[1108] The user opens the application on their smartphone and enters reservation information (reservation date and time, number of people, name, etc.).

[1109] Input: Information such as reservation date and time, number of people, name, etc.

[1110] Output: Reservation information entered

[1111] Specific operation: The user enters "October 15, 2023, 7:00 PM, 4 people, Taro Tanaka" into the application's input form.

[1112] Step 2:

[1113] The terminal transmits the reservation information input by the user to the server.

[1114] Input: Reservation information entered by the user

[1115] Output: Reservation information sent to the server

[1116] Specific operation: The terminal uses an HTTP request to send the entered reservation information to the server in JSON format.

[1117] Step 3:

[1118] The server analyzes the received reservation information using a generative AI model and generates instructions to call the specified phone number.

[1119] Input: Reservation information sent to the server

[1120] Output: Instructions to make a call

[1121] Specific operation: The server uses a generative AI model (e.g., GPT-4) to generate the instruction, "Please make a reservation for Taro Tanaka on October 15, 2023 at 7 p.m. for four people."

[1122] Step 4:

[1123] The server places a call to the specified telephone number based on the generated instruction.

[1124] Input: Instructions to make a call

[1125] Output: Make a phone call

[1126] Specific operation: The server uses Twilio's API to make a call to the specified restaurant.

[1127] Step 5:

[1128] Once the call is connected, the server uses voice recognition technology to converse with the person on the other end of the line.

[1129] Input: What the person on the other end of the phone says

[1130] Output: Speech converted to text

[1131] How it works: The server uses Google Cloud Speech-to-Text to convert what the person on the other end of the phone says into text in real time.

[1132] Step 6:

[1133] The server uses a generative AI model to generate an appropriate response and deliver it to the person on the other end of the call.

[1134] Input: Speech converted to text

[1135] Output: The generated response

[1136] Specific operation: The server uses the generative AI model to generate a response that says, "Hello, my name is Taro Tanaka. I would like to make a reservation for four people at 7 p.m. on October 15th," and conveys this to the person on the other end of the line.

[1137] Step 7:

[1138] If an irregular question comes up during a conversation, the server will tell the other party to wait and put the call on hold.

[1139] Input: Irregular Question

[1140] Output: "Please wait" response

[1141] Specific behavior: When the server receives the question "Do you have any special requests?", it tells the caller "Please wait" and puts the call on hold.

[1142] Step 8:

[1143] The server sends a notification to the user at the terminal, requesting input of an answer.

[1144] Input: Irregular Question

[1145] Output: User notification

[1146] Specific operation: The server sends a push notification to the device, asking the user, "Do you have any special requests?"

[1147] Step 9:

[1148] Users enter their answers in the app.

[1149] Input: User notification

[1150] Output: User's answer

[1151] Specific operation: The user enters "I would like a window seat" in the app.

[1152] Step 10:

[1153] The terminal transmits the user's answer to the server.

[1154] Input: User's answer

[1155] Output: The answer sent to the server

[1156] Specific operation: The terminal uses an HTTP request to send the user's answer to the server.

[1157] Step 11:

[1158] The server continues the conversation with the person on the other end of the phone based on the user's response.

[1159] Input: The answer sent to the server

[1160] Output: Response to the person on the phone

[1161] Specific operation: The server generates a response saying "I would like a window seat" and conveys this to the person on the other end of the phone.

[1162] Step 12:

[1163] The server uses emotion recognition technology to recognize emotions from the tone of the user's voice and transmits that information to AI voice recognition technology.

[1164] Input: User's voice tone

[1165] Output: Recognized emotion information

[1166] Specific operation: The server uses emotion recognition technology to determine that the user is angry and conveys that information to the AI ​​voice recognition technology.

[1167] Step 13:

[1168] The server adjusts the tone of the conversation based on the recognized emotional information.

[1169] Input: Recognized emotion information

[1170] Output: Modulated conversational tone

[1171] What it does: The server uses AI speech recognition technology to respond in a calm tone, saying, "Sorry, we'll check it right away."

[1172] (Application example 1)

[1173] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1174] Conventional food delivery systems require users to place orders over the phone, which can lead to cumbersome communication and difficulty in responding to irregular questions. Furthermore, they often fail to consider the user's feelings, resulting in a slow ordering process. This can lead to poor user experience, cancellations of orders, and dissatisfaction.

[1175] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1176] In this invention, the server includes: means for making a call to a specified telephone number based on information input by the user; artificial intelligence speech recognition technology means for conversing with the person at the telephone number; means for prompting the user to input a response when an irregular question is asked during the conversation; emotion engine means for recognizing the user's emotion and adjusting the tone of the conversation based on the emotion; and generative artificial intelligence model means for analyzing the information input by the user and generating instructions to call the telephone number. This not only enables the user to place an order smoothly and quickly respond to irregular questions, but also enables natural conversation that reflects the user's emotion.

[1177] "Information entered by the user" refers to information such as reservation date and time, number of people, names, menu, etc., entered by the user via a smartphone or other device.

[1178] "Means for calling a specified telephone number" refers to technology for automatically calling a specific telephone number based on information entered by the user.

[1179] "Artificial intelligence speech recognition technology means" refers to artificial intelligence technology that converts the voice of the person on the other end of the phone into text and generates an appropriate response.

[1180] The "means for prompting the user to input an answer when an irregular question is asked" is an interface that allows the user to input an answer to a question when an unexpected question is asked during a conversation.

[1181] The "emotion engine means" is a technology for recognizing emotions from the tone of a user's voice and input content, and adjusting the tone of the conversation based on those emotions.

[1182] The "generative artificial intelligence model means" is an artificial intelligence model that analyzes information input by the user and generates instructions to be called and conversation content.

[1183] In the system for implementing this invention, a user inputs order information using a smartphone or other device, and an AI automatically makes a phone call based on that information to confirm the order. Specific embodiments of this system are described below.

[1184] System configuration

[1185] 1. User Interface:

[1186] Use a smartphone application (iOS / Android).

[1187] Provide a form for users to enter order information (date and time, number of people, menu, name, etc.).

[1188] 2. Data analysis and generative AI models:

[1189] The information entered by the user is sent to a server in the cloud.

[1190] On the server side, a generative AI model (e.g., OpenAI's GPT-4) is used to generate instructions to call the destination.

[1191] 3. Telephone connection and AI voice recognition:

[1192] Use APIs such as Twilio to automatically call specified numbers.

[1193] Once the call is connected, the Google Cloud Speech-to-Text API is used to convert the caller's voice into text.

[1194] Use generative AI models to generate appropriate answers to the other person's questions.

[1195] 4. Handling irregular questions:

[1196] If an irregular question is received, the app will put it on hold with a "Please wait" message and ask the user to enter the answer on the app.

[1197] The conversation continues based on the answers entered by the user.

[1198] 5. Emotion Engine:

[1199] Use an emotion recognition API (e.g. IBM Watson Tone Analyzer) to analyze the tone of the user's voice.

[1200] The emotion engine recognizes the user's emotions and conveys that information to the generative AI model.

[1201] A generative AI model adjusts the tone of the conversation to create a natural conversation that responds to the user's emotions.

[1202] Specific examples

[1203] A user types "I'd like to order two pizzas tomorrow night at 7pm" into a smartphone app. The server receives this information and uses a generative AI model to generate instructions to call the restaurant. The Twilio API is used to automatically make the call, and the Google Cloud Speech-to-Text API is used to convert the caller's voice into text. If the restaurant asks, "Do you have specific toppings?", the app prompts the user for a response. If the user types "pepperoni and mushrooms," the AI ​​continues the conversation based on that information. If the user is angry, the emotion engine recognizes this information and the generative AI model adjusts the tone of the conversation.

[1204] Prompt Sentence Examples

[1205] A user types, "I'd like to order two pizzas tomorrow night at 7pm." Call the restaurant to confirm the order. If they ask about specific toppings, ask them in-app and continue the conversation based on their answer. Also, recognize the user's emotions and adjust the tone of the conversation accordingly.

[1206] In this way, users can enjoy a smoother and more natural ordering experience in food delivery applications.

[1207] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1208] Step 1:

[1209] The user launches the smartphone application and enters order information (date and time, number of people, menu, name, etc.). The entered information is sent from the device to a server on the cloud. The input data is sent to the server in the form of reservation date and time, number of people, menu, name, etc.

[1210] Step 2:

[1211] The server analyzes the received order information and uses a generative AI model (e.g., OpenAI's GPT-4) to generate instructions to call the destination. The generated instructions are output as a specific prompt for making the call. For example, a prompt such as "I'd like to order two pizzas tomorrow night at 7 p.m." is generated.

[1212] Step 3:

[1213] The server automatically calls the specified phone number using an API such as Twilio. Once the call is connected, the server uses the Google Cloud Speech-to-Text API to convert the caller's voice into text in real time. The converted text data is then sent to the server.

[1214] Step 4:

[1215] The server inputs the converted text data into a generative AI model to generate an appropriate answer to the caller's question. The generated answer is then converted back into speech and transmitted to the caller. For example, in response to the question "Do you have any specific toppings?", the answer might be "pepperoni and mushrooms."

[1216] Step 5:

[1217] If an irregular question comes up during a conversation, the server puts the question on hold and asks the user to enter the answer in the app. The answer entered by the user is sent from the device to the server. For example, in response to the question "Do you have any specific toppings?", the user might enter "pepperoni and mushrooms."

[1218] Step 6:

[1219] The server continues the conversation based on the answers entered by the user. It uses a generative AI model to generate an appropriate answer to the caller's question, converts it into speech, and delivers it to the caller. For example, it might respond with "pepperoni and mushrooms."

[1220] Step 7:

[1221] The server uses an emotion recognition API (e.g., IBM Watson Tone Analyzer) to analyze the user's tone of voice and recognize their emotions. The recognized emotion data is input into the generative AI model. For example, if the user is angry, that information is conveyed to the generative AI model.

[1222] Step 8:

[1223] The generative AI model adjusts the tone of the conversation based on the perceived emotion. For example, if the user is angry, the generative AI model will continue the conversation in a calmer tone. The adjusted tone of the conversation is converted into audio and transmitted to the other party on the call.

[1224] In this way, the user can smoothly place an order, irregular questions can be quickly responded to, and natural conversations can be held according to the user's emotions.

[1225] Example 2

[1226] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1227] Conventional reservation systems have the problem of being unable to respond appropriately when unexpected questions arise when making a phone call based on the information entered by the user. Furthermore, the system responds mechanically without considering the user's feelings, which can result in a poor user experience. There is a need to solve these problems and provide a more natural and user-friendly reservation system.

[1228] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1229] In this invention, the server includes means for making a call to a specified telephone number based on information input by the user, voice recognition technology means for conversing with the person at the telephone number, means for prompting the user to input a response when an unexpected question is asked during the conversation, and emotion recognition means for recognizing the user's emotion and adjusting the tone of the conversation based on that emotion. This allows for appropriate responses to unexpected questions and enables natural conversation that reflects the user's emotion.

[1230] "User" means an individual or group that uses the system to make reservations or enter information.

[1231] A "means for making a call" is a device or software that has the function of automatically making a call to a specified telephone number based on information entered by the user.

[1232] "Voice recognition technology means" refers to a device or software that has the technology to analyze the voice of the person on the other end of the phone and convert it into text data.

[1233] An "unexpected question" is a question or request that the system does not anticipate in advance, and may require input from the user.

[1234] The "means for prompting the user to input an answer" is a device or software that has the function of prompting the user to input an answer when an unexpected question is presented.

[1235] An "emotion recognition means" is a device or software that has the technology to analyze emotions from the user's tone of voice and vocabulary, and adjust the system's response based on that information.

[1236] The "reservation date and time" refers to the specific date and time that the user desires to make a reservation.

[1237] "Number of Participants" means the total number of Participants in a Booking.

[1238] "Name" refers to the name of the user making the reservation.

[1239] The "holding means" is a device or software that has the function of temporarily interrupting a conversation when an unexpected question is asked and asking the user for confirmation.

[1240] This invention is a system for allowing users to make restaurant reservations, and uses voice recognition technology to call a designated number based on information entered by the user and to converse with the person at the call. It also includes means for prompting the user to input a response when an unexpected question is asked during the conversation, and emotion recognition means for recognizing the user's emotion and adjusting the tone of the conversation based on that emotion.

[1241] Hardware and software used

[1242] 1. Server:

[1243] Generative AI model: OpenAI's GPT-4 is used to analyze user-entered information.

[1244] Telephone API: Uses Twilio to make a call to a specified phone number.

[1245] Speech recognition technology: Google Cloud Speech-to-Text is used to convert the voice of the person on the other end of the call into text data.

[1246] Text-to-speech technology: Amazon Polly is used to provide voice-over reservation information.

[1247] Emotion Recognition Technology: Affectiva is used to analyze emotions from the user's tone of voice.

[1248] 2. Terminal:

[1249] User interface: Provides applications that run on devices such as smartphones and tablets.

[1250] Program processing

[1251] 1. User enters reservation information:

[1252] The user launches the application and accesses the reservation form.

[1253] The user inputs the reservation date and time, number of people, and name.

[1254] Example: A user enters the following on their smartphone screen: "I would like to make a reservation for four people on October 15, 2023 at 7:00 PM. My name is Taro Tanaka."

[1255] 2. The server receives and analyzes the information:

[1256] The server receives the information entered by the user.

[1257] The server uses a generative AI model (e.g., OpenAI's GPT-4) to analyze the input information.

[1258] Example: The server generates the analysis result "Taro Tanaka would like to make a reservation for 4 people on October 15, 2023 at 7:00 PM."

[1259] 3. The server makes the call:

[1260] The server uses a telephone API such as Twilio to make a call to the specified restaurant.

[1261] Example: Your server calls the Twilio API to call a restaurant's phone number.

[1262] 4. Servers converse with restaurant staff:

[1263] The server uses AI speech synthesis technology (e.g., Amazon Polly) to communicate the reservation information to restaurant staff.

[1264] Example: The server says, "Hello, my name is Taro Tanaka. I'd like to make a reservation for 4 people at 7:00 PM on October 15, 2023."

[1265] 5. The server responds to unexpected questions:

[1266] If an unexpected question comes from restaurant staff, the server will put the call on hold and ask the user to enter the answer on the app.

[1267] Example: The server receives the question "Would you like a window seat?" and responds "Please wait. I'll check with the user."

[1268] 6. User answers the question:

[1269] Users enter answers to questions on the app.

[1270] Example: A user types "Yes, I'd like a window seat" into an input field in an app.

[1271] 7. The server resumes the conversation and confirms the reservation:

[1272] The server continues the conversation with the restaurant staff based on the user's response and confirms the reservation.

[1273] Example: A server tells restaurant staff, "I'd like a window seat," and confirms that the reservation is confirmed.

[1274] Prompt Sentence Examples

[1275] "When a user wants to make a restaurant reservation, they enter the reservation date and time, number of people, and their name. Analyze this information and generate instructions to call the specified restaurant. Once the call is connected, use AI voice recognition technology to speak with the restaurant staff and complete the reservation. If an unexpected question comes up, ask the user to enter an answer and continue the conversation based on that answer."

[1276] The above is an embodiment of the invention.

[1277] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1278] Step 1:

[1279] The user enters the reservation information.

[1280] Input: The user accesses the application's reservation form and enters the reservation date and time, number of people, and name.

[1281] Specific operation: The user enters on the smartphone screen, "I would like to make a reservation for four people on October 15, 2023 at 7:00 PM. My name is Tanaka Taro."

[1282] Output: The entered reservation information is sent to the server.

[1283] Step 2:

[1284] The server receives and analyzes the information.

[1285] Input: Booking information entered by the user.

[1286] What it does: The server uses a generative AI model (e.g., OpenAI's GPT-4) to analyze the input information.

[1287] Data processing: Analyze the reservation date and time, number of people, and names, and format the reservation details in text format.

[1288] Output: Analysis result: "Taro Tanaka would like to make a reservation for 4 people on October 15, 2023 at 7:00 PM."

[1289] Step 3:

[1290] The server makes the call.

[1291] Input: Parsed reservation information.

[1292] Specific operation: The server uses a telephone API such as Twilio to make a call to the specified restaurant.

[1293] Data processing: Convert reservation information into a format to be passed to the telephone API.

[1294] Output: A call is made to the restaurant's phone number.

[1295] Step 4:

[1296] The server talks to the restaurant staff.

[1297] Input: Reservation information and phone connection.

[1298] Specific operation: The server uses AI speech synthesis technology (e.g., Amazon Polly) to communicate the reservation information to restaurant staff.

[1299] Data processing: Convert reservation information in text format into audio data.

[1300] Output: A voice message saying, "Hello, my name is Taro Tanaka. I'd like to make a reservation for 4 people on October 15, 2023 at 7:00 PM."

[1301] Step 5:

[1302] The server responds to unexpected questions.

[1303] Input: Question from restaurant staff.

[1304] What happens: The server uses speech recognition technology (e.g., Google Cloud Speech-to-Text) to convert the question into text data.

[1305] Data processing: Convert the voice data into text data and analyze the question content.

[1306] Output: Text data: "Would you like a window seat?"

[1307] Step 6:

[1308] The user answers the question.

[1309] Input: The question from the server.

[1310] What happens: The user enters an answer into an input field in the app.

[1311] Data processing: The user's answers are sent to the server as text data.

[1312] Output: "Yes, I'd like a window seat."

[1313] Step 7:

[1314] The server resumes the conversation and confirms the reservation.

[1315] Input: The user's answer.

[1316] Specific operation: The server again uses AI speech synthesis technology to convey the user's response to the restaurant staff.

[1317] Data processing: Convert the user's answers into audio data.

[1318] Output: A voice message saying "I would like a window seat."

[1319] The above is the flow of processing of the program of this system.

[1320] (Application example 2)

[1321] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1322] With conventional food delivery services, when a user places an order, they must manually enter information and confirm the order via phone or app. However, this is time-consuming and labor-intensive, and the user must respond again, especially if unexpected questions or changes arise. Furthermore, the system is unable to provide natural conversations that reflect the user's emotions, resulting in a poor user experience. To solve these issues, a system is needed that can automatically place orders based on the user's input information and enable natural conversations that reflect the user's emotions.

[1323] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1324] In this invention, the server includes means for making a call to a specified telephone number based on information input by the user, voice recognition technology means for conversing with the person at the telephone number, means for prompting the user to input a response when an irregular question is asked during the conversation, an emotion engine for recognizing the user's emotions, and means for adjusting the tone of the conversation based on the emotions recognized by the emotion engine. This enables a user to automatically communicate the order details, delivery time, and special requests to a restaurant over the phone when using a food delivery service, and also enables a natural conversation that reflects the user's emotions.

[1325] "User" means an individual or organization that uses the system to input information and receive services.

[1326] "Information" refers to data that a user enters into the system, including order details, delivery times, special requests, etc.

[1327] "Telephone number" refers to a contact point designated by the user, typically a restaurant or store that provides food delivery services.

[1328] "Means for making a call" is a function that allows the system to automatically make a call to a specified telephone number.

[1329] "Speech recognition technology means" refers to technology that enables the system to understand the conversation with the person on the other end of the phone and respond appropriately.

[1330] An "irregular question" is when the person on the other end of the phone asks an unexpected question or makes a request.

[1331] The "means for requesting input of an answer" is a function that prompts the user to input an answer when an irregular question is asked.

[1332] The "emotion engine" is a technology that recognizes emotions from the user's tone of voice and input content.

[1333] The "means for adjusting the tone of a conversation" is a function by which the system adjusts the tone of a conversation based on the emotion recognized by the emotion engine.

[1334] "System" means a device or program that includes a series of technical means for automatically making calls based on information entered by the user and for realizing natural conversations using voice recognition technology and an emotion engine.

[1335] A system for carrying out this invention includes means for making a call to a specified telephone number based on information input by a user, voice recognition technology means for conducting a conversation with the person at the telephone number, means for prompting the user to input a response when an irregular question is asked during the conversation, an emotion engine for recognizing the user's emotions, and means for adjusting the tone of the conversation based on the emotions recognized by the emotion engine.

[1336] Hardware and software used

[1337] Hardware: Smartphone

[1338] Software: Google Cloud Dialogflow, IBM Watson Tone Analyzer

[1339] Data processing and calculation

[1340] Receiving user input

[1341] Using a smartphone application, users enter information such as order details, delivery time, and special requests, which is then sent to a server.

[1342] sentiment analysis

[1343] The server uses IBM Watson Tone Analyzer to analyze the emotions in the user's text input. The emotion engine recognizes emotions from the user's tone of voice and input, and adjusts the tone of the conversation based on the results.

[1344] Intent detection

[1345] The server uses Google Cloud Dialogflow to detect the user's intent from the text they input, which then generates an appropriate response and instructions to call the specified phone number.

[1346] Making a phone reservation

[1347] Based on the generated response, the server uses AI to call the restaurant and place the order. Once the call is connected, the customer uses voice recognition technology to converse with the restaurant staff and complete the order.

[1348] Responding to irregular questions

[1349] If an unexpected question comes from the restaurant, the server puts the request on hold and asks the user to enter an answer on the application. Once the user enters the answer, the conversation with the restaurant staff continues based on the answer and the order is confirmed.

[1350] Specific examples

[1351] If a user types "Two pizzas, please deliver them at 7 PM," the emotion engine analyzes the user's emotions and Dialogflow generates an appropriate response. For example, if the user is happy, the AI ​​will proceed with the conversation in a tone such as "Thank you! Two pizzas will be delivered at 7 PM."

[1352] Prompt Sentence Examples

[1353] If a user types "Two pizzas, please deliver them at 7 PM," the emotion engine analyzes the user's emotions and Dialogflow generates an appropriate response. For example, if the user is happy, the AI ​​will proceed with the conversation in a tone such as "Thank you! Two pizzas will be delivered at 7 PM."

[1354] In this way, a food delivery assistant application that can have natural conversations according to the user's emotions is completed.

[1355] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1356] Step 1:

[1357] A user uses a smartphone application to enter information such as order details, delivery time, and special requests.

[1358] Input: Order details, delivery time, special requests

[1359] Output: The entered information is sent to the server

[1360] Specific behavior: When a user enters information into an application's input form and presses the "Submit" button, the information is sent to the server.

[1361] Step 2:

[1362] The server uses IBM Watson Tone Analyzer to analyze the sentiment from the user's input text.

[1363] Input: Information entered by the user

[1364] Output: Sentiment analysis results

[1365] Specific operation: The server sends the input text data to IBM Watson Tone Analyzer and receives the sentiment analysis results.

[1366] Step 3:

[1367] The server uses Google Cloud Dialogflow to detect the user's intent from the input text.

[1368] Input: Information entered by the user

[1369] Output: Intent detection result

[1370] Specific operation: The server sends the entered text data to Google Cloud Dialogflow and receives the intent detection result.

[1371] Step 4:

[1372] Based on the generated response, the server generates an instruction to call the specified telephone number.

[1373] Input: Intention detection results, sentiment analysis results

[1374] Output: Instructions to make a call

[1375] Specific operation: The server generates an appropriate response based on the intent detection and sentiment analysis results, and generates instructions to make a call.

[1376] Step 5:

[1377] The server calls the designated number and uses voice recognition technology to talk to restaurant staff.

[1378] Input: Instructions to make a call

[1379] Output: Conversation

[1380] What it does: The server automatically makes a call and uses voice recognition technology to speak to restaurant staff.

[1381] Step 6:

[1382] If an unexpected question comes from the restaurant, the server will put the request on hold and ask the user to enter an answer on the application.

[1383] Input: Unexpected question

[1384] Output: Prompt the user for an answer

[1385] What happens: The server detects the unexpected question and sends a message to the application prompting the user to enter an answer.

[1386] Step 7:

[1387] The user inputs an answer, and the server continues the conversation with the restaurant staff based on the answer and confirms the order.

[1388] Input: User's answer

[1389] Output: Confirmed order details

[1390] Specific behavior: The user enters an answer into the application, and the server uses the answer to continue the conversation with the restaurant staff and confirm the order.

[1391] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1392] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1393] Another example of generative AI is Gemini (internet search engine). <url: https: gemini.google.com ?hl="ja">) are listed.

[1394] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1395] [Third embodiment]

[1396] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1397] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1398] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1399] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1400] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1401] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1402] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1403] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1404] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1405] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1406] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1407] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1408] "Example 1"

[1409] In one embodiment of the present invention, a user inputs information such as reservation date and time, number of people, and names into a smartphone application. This information is analyzed by generative AI, which generates instructions to call the specified number. Once the call is connected, the conversation with the person on the other end is carried out using AI voice recognition technology. If an irregular question is asked during the conversation, the system puts the call on hold with a "Please wait" message and prompts the user to enter a response on the app. Once the user has entered a response, the conversation with the person on the other end is continued based on that response.

[1410] "Example 2"

[1411] As a specific example, when a user wants to make a restaurant reservation, they input the reservation date and time, number of people, and name on the application. This information is analyzed by generative AI, which generates instructions to call the specified restaurant. Once the call is connected, AI voice recognition technology is used to converse with the restaurant staff and complete the reservation. If the restaurant asks an unexpected question, the app puts the call on hold by saying "Please wait" and asks the user to enter a response on the app. Once the user has entered a response, the conversation with the restaurant staff continues based on that response, and the reservation is confirmed.

[1412] The processing flow of each embodiment will be described below.

[1413] "Example 1"

[1414] Step 1: The user enters information such as reservation date and time, number of people, and names on the smartphone application.

[1415] Step 2: This information is analyzed by a generative AI to generate instructions to call the specified number.

[1416] Step 3: Once the call is connected, use AI voice recognition technology to talk to the person on the other end.

[1417] Step 4: If an irregular question comes up during the conversation, put it on hold with "Please wait" and ask the user to enter the answer on the app.

[1418] Step 5: Once the user has entered their answer, the conversation with the person on the other end of the phone continues based on that answer.

[1419] "Example 2"

[1420] Step 1: When a user wants to make a restaurant reservation, they input the reservation date and time, number of people, and name on the application.

[1421] Step 2: This information is analyzed by a generative AI, which generates instructions to call the specified restaurant.

[1422] Step 3: Once the call is connected, use AI voice recognition technology to speak to the restaurant staff and complete the reservation.

[1423] Step 4: If the restaurant asks an unexpected question, the app will put the question on hold and ask the user to enter the answer on the app.

[1424] Step 5: Once the user has entered their answer, they can continue the conversation with the restaurant staff based on their answer and confirm the reservation.

[1425] Example 1

[1426] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1427] Conventional reservation systems require users to manually make phone calls and make reservations, which is time-consuming and makes it difficult to respond appropriately when an irregular question comes up during a conversation with the person on the other end of the phone. Furthermore, there is a lack of a way to efficiently analyze the information entered by the user and generate an appropriate response.

[1428] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1429] In this invention, the server includes means for making a call to a specified telephone number based on information input by the user, speech recognition technology means for conversing with the person at the telephone number, means for prompting the user to input a response when an irregular question is asked during the conversation, means for using a generative AI model for analyzing the information input by the user, means for generating a prompt sentence based on the analysis result, and means for analyzing the speech of the person at the telephone number in real time and generating an appropriate response. This allows the user to make a reservation without hassle and makes it possible to appropriately respond to irregular questions.

[1430] "User" means an individual or group that uses the system to make reservations or enter information.

[1431] "Information" refers to data such as reservation date and time, number of people, and names entered by the user.

[1432] "Telephone number" refers to the telephone number at which a user makes a reservation or inquiry.

[1433] "Means for making phone calls" refers to the ability to make calls to designated telephone numbers using VoIP technology.

[1434] "Voice recognition technology means" refers to technology that converts the speech of the person on the other end of the phone into text in real time.

[1435] An "irregular question" refers to an unexpected question or confirmation from the person on the other end of the phone.

[1436] "Means for requesting answer input" refers to an interface that allows the user to input an answer when an irregular question is asked.

[1437] A "generative AI model" refers to an artificial intelligence model that analyzes information entered by a user and generates appropriate responses or instructions.

[1438] A "prompt" refers to an instruction or question that is input into a generative AI model.

[1439] "Means of analysis" refers to the function of analyzing information entered by the user using a generative AI model.

[1440] "Means for generating a response" refers to the function of analyzing what the person on the other end of the phone says and generating an appropriate reply.

[1441] This invention is a system that efficiently and automatically makes calls and provides necessary information when a user makes a reservation or inquiry using a smartphone application. Specific embodiments of this system are described below.

[1442] Hardware and software used

[1443] Hardware

[1444] Smartphone (iOS or Android)

[1445] Server (Cloud server or on-premise server)

[1446] software

[1447] Smartphone application

[1448] Generative AI models (e.g., OpenAI's GPT-4)

[1449] Speech recognition technology (e.g., Google Cloud Speech-to-Text)

[1450] VoIP technology (e.g., Twilio API)

[1451] Data processing and calculation

[1452] User Input

[1453] The user opens the smartphone application and enters information such as the reservation date and time, number of people, and name into the reservation form. Once the information is complete, the user presses the send button.

[1454] Sending data

[1455] The device converts the information entered by the user into JSON format and sends it to a server using the HTTPS protocol, with the destination URL set in advance.

[1456] Data analysis

[1457] The server analyzes the received JSON data and inputs it as a prompt to the generative AI model. An example of a prompt might be, "The user would like to make a restaurant reservation. Please call the restaurant and make the reservation using the following information: Reservation date and time: December 24, 2023, 7:00 PM, Number of people: 4, Name: Yamada Taro."

[1458] Making a call

[1459] The server uses VoIP technology to make a call to the specified phone number, for example, by using the Twilio API. Once the call is connected, it uses AI voice recognition technology to start a conversation with the person on the other end.

[1460] Conversation progression

[1461] The server uses AI speech recognition technology to convert the caller's speech into text in real time, and then uses a generative AI model to generate an appropriate response. For example, if the caller says, "Your reservation has been accepted," the server will respond with, "Thank you."

[1462] Prompt the user

[1463] If an irregular question comes up during a conversation, the server will tell the other party "Please wait" and put the conversation on hold. The device will display the irregular question to the user on the application and ask them to enter an answer. For example, if the other party asks "Do you have any allergies?", the device will display the question to the user and ask them to enter an answer.

[1464] Restarting the conversation

[1465] When the user enters "no allergies," that information is sent to the server. The server then continues the conversation with the person on the other end of the phone based on the user's response. For example, it might respond, "Sorry to keep you waiting. I don't have any allergies." This completes the reservation process.

[1466] Specific examples

[1467] For example, when a user makes a restaurant reservation, they input the following prompt into the generative AI model:

[1468] Example prompt sentence:

[1469] A user would like to make a restaurant reservation. Please call the restaurant using the following information to make the reservation:

[1470] Reservation date and time: December 24, 2023 7:00 PM

[1471] Number of people: 4 people

[1472] Name: Taro Yamada

[1473] Based on this prompt, the generative AI model generates the following instructions:

[1474] "We would like to make a reservation for 4 people on December 24, 2023 at 7:00 PM. Can you accept our reservation?"

[1475] After the call is connected, the app uses AI voice recognition technology to continue the conversation, prompting the user for confirmation as needed. For example, if the restaurant asks if the user has any allergies, the app displays the question to the user and allows them to enter their answer. If the user enters "no allergies," the app resumes the conversation based on that information.

[1476] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1477] Step 1:

[1478] The user opens the smartphone application and enters information such as the reservation date and time, number of people, and name into the reservation form. Once the information is complete, the user presses the send button.

[1479] Input: Information such as reservation date and time, number of people, name, etc.

[1480] Output: When the submit button is pressed, the input information is sent to the next step.

[1481] Step 2:

[1482] The device converts the information entered by the user into JSON format and sends it to a server using the HTTPS protocol, with the destination URL set in advance.

[1483] Input: Information entered by the user

[1484] Output: JSON format data is sent to the server

[1485] Step 3:

[1486] The server analyzes the received JSON data and inputs it as a prompt to the generative AI model. An example of a prompt might be, "The user would like to make a restaurant reservation. Please call the restaurant and make the reservation using the following information: Reservation date and time: December 24, 2023, 7:00 PM, Number of people: 4, Name: Yamada Taro."

[1487] Input: JSON format data

[1488] Output: The prompt sentence that is input to the generative AI model

[1489] Step 4:

[1490] The server uses a generative AI model to generate instructions for making a call to a specified phone number based on the prompt text.

[1491] Input: prompt statement

[1492] Output: Instructions to make a call

[1493] Step 5:

[1494] The server places the call to the specified destination using VoIP technology, for example, by using the Twilio API to make the call.

[1495] Input: Instructions to make a call

[1496] Output: Phone is connected

[1497] Step 6:

[1498] Once the call is connected, the server uses AI voice recognition technology to begin a conversation with the person on the other end of the line.

[1499] Input: Phone connected

[1500] Output: A conversation with the person on the other end of the phone is initiated

[1501] Step 7:

[1502] The server uses AI speech recognition technology to convert the caller's speech into text in real time, and then uses a generative AI model to generate an appropriate response. For example, if the caller says, "Your reservation has been accepted," the server will respond with, "Thank you."

[1503] Input: What the person on the other end of the phone says

[1504] Output: Translated utterances and generated responses

[1505] Step 8:

[1506] If an irregular question comes up during a conversation, the server will tell the other party to wait and put the conversation on hold. The device will then display the irregular question to the user on the application and ask them to enter an answer.

[1507] Input: Irregular Question

[1508] Output: The question and answer prompt that are displayed to the user

[1509] Step 9:

[1510] Once the user enters their answer, the information is sent to the server. For example, if someone asks if they have any allergies, the server displays the question and prompts the user to enter their answer.

[1511] Input: User's answer

[1512] Output: The answer sent to the server

[1513] Step 10:

[1514] The server continues the conversation with the person on the other end of the phone based on the user's response. For example, it might respond, "Sorry to keep you waiting. I don't have any allergies." This completes the reservation process.

[1515] Input: User's answer

[1516] Output: A response that is conveyed to the person on the other end of the phone and the completion of the reservation

[1517] (Application example 1)

[1518] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1519] Conventional food delivery services often require users to make reservations or orders over the phone when ordering from a restaurant. However, this poses problems, such as the time and effort required for users to make phone calls and the need to respond to irregular questions that arise during conversations with restaurant staff. Furthermore, depending on the time of day when a user calls, the restaurant may be busy and the response may be delayed. This reduces user convenience and impairs the quality of service, creating issues.

[1520] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1521] In this invention, the server includes: means for making a call to a specified telephone number based on information input by a user; AI speech recognition technology means for conversing with the person at the telephone number; means for prompting the user to input an answer when an irregular question is asked during the conversation; means for continuing the conversation with the person at the telephone number based on the answer input by the user; means for analyzing the information input by the user using generative AI and generating the telephone call content; means for reading out the generated telephone call content using speech synthesis technology; and means for recognizing the question from the person at the telephone number using speech recognition technology. This allows users to easily make reservations and orders through a smartphone app and can quickly respond to irregular questions, thereby improving user convenience and improving the quality of service.

[1522] "User" means an individual or corporation that makes reservations or orders using the system.

[1523] "Input information" refers to data such as reservation date and time, number of people, names, order details, etc. that the user provides to the system.

[1524] The "designated telephone number" is the telephone number of the restaurant or service provider for which the user wishes to make a reservation or order.

[1525] "Means for making a call" is a function that allows the system to automatically make a call to a specified telephone number.

[1526] "AI voice recognition technology means" is a technology that uses artificial intelligence to recognize the voice of the person on the other end of the phone and convert it into text data.

[1527] An "irregular question" is a situation in which an unexpected question or request arises from the person on the other end of the phone.

[1528] The "means for prompting the user to input an answer" is a function that prompts the user to input an appropriate answer when an irregular question occurs.

[1529] The "means for continuing the conversation again" is a function for continuing the conversation with the person on the other end of the phone call based on the answer entered by the user.

[1530] "Generative AI" is an artificial intelligence technology that analyzes information entered by the user and generates appropriate telephone content.

[1531] "Means for generating telephone content" is a function that uses generative AI to create the content to be conveyed over the phone based on the information entered by the user.

[1532] "Speech synthesis technology" is a technology that converts text data into speech and reads it out in a natural voice.

[1533] "A means for reading out loud" is a function that uses voice synthesis technology to convey the generated telephone call contents aloud.

[1534] "Voice recognition technology" is a technology that recognizes the voice of the person on the other end of the phone and converts it into text data.

[1535] The "means for recognizing questions" is a function that uses voice recognition technology to understand the question asked by the person on the other end of the phone and acquires it as text data.

[1536] To implement this invention, the following hardware and software are required. The hardware requires a smartphone and a microphone. The software uses Python, OpenAI API, the SpeechRecognition library, and the pyttsx3 library.

[1537] The server provides a means to call a specified number based on the information entered by the user. When a user enters reservation or order information through a smartphone app, the information is sent to the server. The server uses generative AI to analyze the information entered by the user and generate the phone call content. This generated phone call content is read aloud using speech synthesis technology.

[1538] The terminal provides AI speech recognition technology for conversation with the person on the other end of the phone. When the call is connected, the AI ​​speech recognition technology is used to recognize the voice of the person on the other end of the phone and convert it into text data. If an irregular question is asked during the conversation, the terminal puts the call on hold and asks the user to enter a response. Once the user enters a response, the terminal continues the conversation with the person on the other end based on that response.

[1539] As a specific example, if a user inputs "I would like to make a reservation for two people at 7pm tomorrow," the server analyzes this information and calls the specified restaurant. The generative AI generates the phone call message "Please make a reservation for two people at 7pm tomorrow," and reads it out loud using speech synthesis technology. If a restaurant staff member asks, "Do you have any allergies?", the device recognizes this question and asks the user for a response. If the user inputs "no," the AI ​​again replies, "I don't have any allergies," and the conversation continues.

[1540] An example of a prompt is as follows:

[1541] User input: I would like to make a reservation for 2 people tomorrow at 7pm.

[1542] Call the restaurant and tell them the following:

[1543] In this way, users can easily make reservations and orders through a smartphone app, and irregular inquiries can be quickly responded to, improving user convenience and the quality of service.

[1544] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1545] Step 1:

[1546] Users enter reservation and order information through a smartphone app.

[1547] Input: Information such as reservation date and time, number of people, name, order details, etc.

[1548] Output: The entered information is sent to the server

[1549] Specific behavior: The user enters the required information into the app's input form and presses the submit button.

[1550] Step 2:

[1551] The server uses generative AI to analyze the information entered by the user and generate the content of the call.

[1552] Input: Information entered by the user

[1553] Output: Generated call contents (text format)

[1554] Specific operation: The server calls the OpenAI API, passes the input information as a prompt, and generates the appropriate call content.

[1555] Step 3:

[1556] The server uses speech synthesis technology to read out the generated phone call content aloud.

[1557] Input: Generated call content (text format)

[1558] Output: Audio file

[1559] Specific operation: The server uses the pyttsx3 library to convert the generated text into speech and create an audio file.

[1560] Step 4:

[1561] The server makes a call to the specified telephone number.

[1562] Input: specified phone number, audio file

[1563] Output: Establishing a telephone connection

[1564] Specific operation: The server uses the telephone API to call the specified phone number and play the audio file.

[1565] Step 5:

[1566] The device uses AI voice recognition technology to recognize the voice of the person on the other end of the phone and convert it into text data.

[1567] Input: The voice of the person you are calling

[1568] Output: Recognized text data

[1569] What it does: The device uses the SpeechRecognition library to convert the other person's voice into text in real time.

[1570] Step 6:

[1571] If an irregular question comes up during a conversation, the terminal will put the question on hold and ask the user to enter an answer.

[1572] Input: Recognized text data (irregular questions)

[1573] Output: prompts the user for input

[1574] Specific behavior: The device detects an irregular question and notifies the user to enter an answer in the app.

[1575] Step 7:

[1576] The user enters the answer in the app.

[1577] Input: User's answer

[1578] Output: The entered answer is sent to the server

[1579] Specific behavior: The user enters answers into the app's input form and presses the submit button.

[1580] Step 8:

[1581] The server continues the conversation with the person on the other end of the phone based on the user's response.

[1582] Input: User's answer

[1583] Output: Re-conversation content (text format)

[1584] Specific operation: The server calls the generative AI again, passes the user's answer as a prompt, and generates appropriate re-conversation content.

[1585] Step 9:

[1586] The server uses speech synthesis technology to read out the contents of the re-conversation aloud.

[1587] Input: Re-conversation content (text format)

[1588] Output: Audio file

[1589] Specific operation: The server uses the pyttsx3 library to convert the replay content into audio and create an audio file.

[1590] Step 10:

[1591] The server then plays the audio file again to the person on the other end of the line, and the conversation continues.

[1592] Input: Audio file

[1593] Output: Audio playback to the person you are calling

[1594] Specific operation: The server uses the telephone API to play the audio file of the conversation and continue the conversation.

[1595] Example 2

[1596] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1597] Conventional restaurant reservation systems require users to manually make phone calls, making the reservation process cumbersome. Furthermore, if an unexpected question arose during the call, the user was often unable to respond immediately, resulting in a delay in confirming the reservation. Furthermore, there was a lack of a system that could efficiently analyze the information entered by the user and provide an appropriate response.

[1598] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1599] In this invention, the server includes means for making a call to a specified telephone number based on information input by the user, artificial intelligence speech recognition technology means for conversing with the person at the telephone number, means for prompting the user to input a response when an irregular question is asked during the conversation, artificial intelligence model generation means for analyzing the information input by the user, communication API means for making the call, and means for notifying the user of the confirmation of the reservation. This enables users to easily make restaurant reservations and quickly respond to unexpected questions.

[1600] "User" refers to an individual or organization that uses the system to make restaurant reservations.

[1601] "Telephone number" refers to the restaurant or other facility where the user wishes to make a reservation.

[1602] "Artificial intelligence speech recognition technology" refers to technology that converts speech into text and understands the content of conversations.

[1603] "Irregular questions" refer to unexpected questions or confirmations that arise during the reservation process.

[1604] A "generative artificial intelligence model" refers to an artificial intelligence model that analyzes information entered by a user and generates appropriate instructions and responses.

[1605] "Communications API" refers to the application programming interface used by the server to place a call.

[1606] A "reservation confirmation notice" refers to a notice to inform the user that the reservation has been confirmed.

[1607] This invention is a system that allows users to easily make restaurant reservations. The system uses artificial intelligence speech recognition technology to call a specified telephone number based on information entered by the user and to converse with the person at the telephone number. The system also includes a means for prompting the user to input a response if an irregular question is asked during the conversation.

[1608] Hardware and software used

[1609] 1. Server

[1610] The server uses a generative artificial intelligence model (e.g., GPT-4) to analyze the information entered by the user and generate appropriate instructions.

[1611] The server uses a communications API (e.g., Twilio) to make the call.

[1612] The server uses artificial intelligence speech recognition technology (e.g., Google Speech-to-Text) to conduct a conversation with the person on the other end of the phone.

[1613] 2. Terminal

[1614] The terminal transmits the information entered by the user to the server.

[1615] The terminal receives instructions from the server and displays them to the user.

[1616] 3. Users

[1617] The user uses the application to enter reservation information.

[1618] The user inputs answers to the irregular questions.

[1619] Specific examples

[1620] The user opens the application and enters the following information:

[1621] Reservation date and time: December 24, 2023 7:00 PM

[1622] Number of people: 4 people

[1623] Name: Taro Yamada

[1624] This information is analyzed by a generative AI model (GPT-4), which generates instructions to call the specified restaurant using Twilio. Once the call is connected, a conversation with the restaurant staff takes place using Google Speech-to-Text. For example, if the staff asks, "Do you have any allergies?", the server puts the call on hold with a "Please wait" message and displays "Do you have any allergies?" on the terminal. If the user enters "No," the conversation with the staff continues based on that response, and the reservation is confirmed.

[1625] Prompt Sentence Examples

[1626] An example of a prompt to be input to the generative AI model is as follows:

[1627] The user entered the following information:

[1628] Reservation date and time: December 24, 2023 7:00 PM

[1629] Number of people: 4 people

[1630] Name: Taro Yamada

[1631] Use this information to generate instructions to call the designated restaurant and complete the reservation.

[1632] In this way, a system is realized that allows users to easily make reservations at restaurants.

[1633] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1634] Step 1:

[1635] The user enters the reservation information.

[1636] Input: reservation date and time, number of people, name

[1637] Specific operation: The user opens the application and enters the reservation date and time "December 24, 2023 19:00", the number of people "4", and the name "Yamada Taro".

[1638] Output: The entered reservation information is saved on the device.

[1639] Step 2:

[1640] The terminal sends the input information to the server.

[1641] Input: Booking information entered by the user

[1642] Specific operation: The terminal sends the reservation information entered by the user to the server.

[1643] Output: The server receives the reservation information.

[1644] Step 3:

[1645] The server analyzes the input information using a generative AI model.

[1646] Input: Received reservation information

[1647] How it works: The server uses a generative AI model (e.g., GPT-4) to analyze the reservation information entered by the user.

[1648] Output: The parsing results in instructions to call the restaurant.

[1649] Step 4:

[1650] The server uses the phone API to call the restaurant.

[1651] Input: Instructions generated as a result of the analysis

[1652] Specific operation: The server uses a telephone API such as Twilio to call the specified restaurant.

[1653] Output: A telephone connection is established with the restaurant.

[1654] Step 5:

[1655] The server uses AI voice recognition technology to converse with restaurant staff.

[1656] Input: Conversation with restaurant staff

[1657] Specific operation: The server uses AI voice recognition technology such as Google Speech-to-Text to converse with restaurant staff and proceed with the reservation.

[1658] Output: The conversation is parsed into text.

[1659] Step 6:

[1660] The server prompts the user for an answer to an unexpected question.

[1661] Input: Unexpected question from restaurant staff

[1662] Specific operation: When a restaurant staff member asks, "Do you have any allergies?", the server puts the customer on hold by saying, "Please wait," and displays "Do you have any allergies?" on the terminal.

[1663] Output: The question is displayed to the user.

[1664] Step 7:

[1665] The user enters the answer.

[1666] Input: User's answer to the question displayed on the terminal

[1667] Specific action: The user enters "No" in response to the question displayed on the terminal.

[1668] Output: The entered answer is saved on the device.

[1669] Step 8:

[1670] The terminal sends the user's answer to the server.

[1671] Input: The answer entered by the user

[1672] Specific operation: The device sends the answer entered by the user to the server.

[1673] Output: The server receives the answer.

[1674] Step 9:

[1675] The server continues the conversation with the restaurant staff.

[1676] Input: User's answer

[1677] Specific operation: Based on the user's answer, the server continues the conversation with the restaurant staff and confirms the reservation.

[1678] Output: The reservation is confirmed.

[1679] Step 10:

[1680] The server confirms the reservation and notifies the user.

[1681] Input: Confirmed reservation information

[1682] Specific behavior: The server notifies the user that the reservation has been confirmed and displays the reservation details.

[1683] Output: A confirmation of the booking is sent to the user.

[1684] (Application example 2)

[1685] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1686] With conventional food delivery systems, users had to manually call to place an order, which was time-consuming and could lead to miscommunication and other communication issues. Another issue was that it was difficult to respond immediately to unexpected questions from the restaurant, which meant it took a long time to confirm the order.

[1687] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1688] In this invention, the server includes: a means for making a call to a specified telephone number based on information entered by the user; an AI speech recognition technology means for conversing with the person at the telephone number; a means for prompting the user to input a response if an irregular question is asked during the conversation; a means for analyzing the order information entered by the user and generating instructions for the telephone order using a generative AI model; and a means for communicating the generated instructions to the person at the telephone number using speech synthesis technology. This eliminates the need for the user to make manual calls, reduces miscommunication of order details, and enables quick response to unexpected questions.

[1689] "User" means an individual or corporation that uses the system to input information and receive services.

[1690] "Input information" refers to data provided by the user to the system, including order details, reservation information, and the like.

[1691] A "designated telephone number" is a telephone number that the system will call based on the information entered by the user.

[1692] "Means for making a call" is a function that allows the system to automatically make a call to a specified telephone number.

[1693] "AI speech recognition technology means" is a technology that uses artificial intelligence to convert speech into text and understand the content of the conversation.

[1694] An "irregular question" is a question or request that the system does not anticipate and that requires user input.

[1695] The "means for prompting the user to input an answer" is a function that prompts the user to input an answer when an irregular question occurs.

[1696] "Order information" refers to the specific order details provided by the user to the system, and includes the name of the dish, quantity, delivery address, desired delivery time, etc.

[1697] A "generative AI model" is an artificial intelligence model that analyzes information entered by the user and generates appropriate instructions and answers.

[1698] "Telephone order instructions" are the specific order details and instructions generated by the generative AI model that should be conveyed to the person on the other end of the phone.

[1699] "Speech synthesis technology" is a technology that converts text data into speech and conveys it to the person on the other end of the phone in a natural voice.

[1700] A system for implementing this invention includes means for making a call to a specified telephone number based on information entered by a user, AI voice recognition technology means for conducting a conversation with the person at the telephone, means for prompting the user to input a response when an irregular question arises during the conversation, means for analyzing the order information entered by the user and generating instructions for a telephone order using a generation AI model, and means for conveying the generated instructions to the person at the telephone using voice synthesis technology.

[1701] The server receives order information entered by the user on the food delivery application, including the name of the dish, the quantity, the delivery address, the desired delivery time, etc. The server analyzes this information and generates phone order instructions using a generative AI model (e.g., OpenAI's API).

[1702] The generated instructions are used to automatically call the specified phone number using the Twilio API. After the call is connected, the generated instructions are conveyed to the caller using speech synthesis technology (for example, pyttsx3). If the caller asks an unexpected question, the server puts the call on hold and asks the user to enter an answer in the application. Once the user enters an answer, the server uses the generative AI model again to generate new instructions and convey them to the caller.

[1703] As a specific example, if a user inputs "one pizza, two salads, delivery destination Shibuya-ku, Tokyo, desired delivery time 6:00 p.m.", the following prompt sentence will be input to the generative AI model.

[1704] Example prompt sentence:

[1705] "User would like to place the following order: 1 pizza, 2 salads, delivery location: Shibuya-ku, Tokyo, desired delivery time: 6 PM. Please call the restaurant to complete the order."

[1706] Based on this prompt, the generative AI model generates instructions to call the restaurant, and then uses the Twilio API to actually make the call. The order is then conveyed using speech synthesis technology, and the order is completed.

[1707] This system saves users the trouble of manually making phone calls, reduces miscommunication of orders, and allows for quick response to unexpected questions.

[1708] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1709] Step 1:

[1710] A user enters order information on a food delivery application.

[1711] Input: Order information such as dish name, quantity, delivery address, desired delivery time, etc.

[1712] Output: Order information sent to the server

[1713] Specific operation: The user enters the required information into the application's order form and presses the "Order" button.

[1714] Step 2:

[1715] The server analyzes the order information received from the user.

[1716] Input: Order information submitted by the user

[1717] Output: Parsed order information

[1718] Specific operation: The server receives the order information and extracts each item (food name, quantity, delivery address, desired delivery time).

[1719] Step 3:

[1720] The server uses the generative AI model to generate phone order instructions.

[1721] Input: Parsed order information

[1722] Output: Generated phone order instructions

[1723] Specific operation: The server converts the parsed order information into prompt sentences and inputs them into a generative AI model (e.g., OpenAI's API) to generate telephone order instructions.

[1724] Step 4:

[1725] The server automatically makes a call to the specified phone number using the Twilio API.

[1726] Input: Generated phone order instructions, designated phone number

[1727] Output: Establishing a telephone connection

[1728] Specific operation: The server calls the Twilio API and makes a call to the specified phone number.

[1729] Step 5:

[1730] The server uses voice synthesis technology to generate instructions and transmit them to the person on the other end of the phone.

[1731] Input: Generated phone order instructions

[1732] Output: Voice instructions that are conveyed to the person on the other end of the call

[1733] Specific operation: The server uses a speech synthesis engine (e.g., pyttsx3) to convert the generated instructions into speech and convey them to the person on the other end of the phone.

[1734] Step 6:

[1735] If the person on the other end of the phone asks an unexpected question, the server puts the call on hold and asks the user to enter an answer on the application.

[1736] Input: An unexpected question from the person on the other end of the phone

[1737] Output: Prompt the user to enter an answer

[1738] Specific operation: The server uses voice recognition technology to detect unexpected questions, puts the call on hold with a "Please wait" message, and sends a notification to the user requesting them to enter an answer in the application.

[1739] Step 7:

[1740] The user enters an answer into the application, and the server again uses the generative AI model to generate new instructions and convey them to the person on the other end of the phone.

[1741] Input: User response

[1742] Output: Newly generated instructions, voice instructions to be conveyed to the person on the other end of the call

[1743] Specific operation: The user enters an answer on the application, and the server receives the answer and uses the generative AI model to generate new instructions, which are then conveyed to the person on the other end of the phone using voice synthesis technology.

[1744] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1745] "Example 1"

[1746] One embodiment of the present invention is a system that incorporates an emotion engine. This system includes a means for making a call to a specified telephone number based on information entered by a user, an AI voice recognition technology means for conducting a conversation with the person at the telephone number, a means for prompting the user to input a response when an irregular question is asked during the conversation, and an emotion engine that recognizes the user's emotions. The emotion engine recognizes emotions from the tone of the user's voice and adjusts the tone of the conversation based on that emotion. For example, if the user feels angry, the emotion engine conveys that information to the AI ​​voice recognition technology means, and the AI ​​adjusts the tone of the conversation based on that information. This enables a more natural conversation that reflects the user's emotions.

[1747] "Example 2"

[1748] Another embodiment of the present invention is a system that incorporates an emotion engine. This system includes a means for making a call to a specified telephone number based on information entered by a user, an AI voice recognition technology means for conducting a conversation with the person at the telephone number, a means for prompting the user to input a response when an irregular question is asked during the conversation, and an emotion engine that recognizes the user's emotions. The emotion engine recognizes emotions from the tone of the user's voice and adjusts the tone of the conversation based on that emotion. For example, if the user feels happy, the emotion engine conveys that information to the AI ​​voice recognition technology means, and the AI ​​adjusts the tone of the conversation based on that information. This enables a more natural conversation that reflects the user's emotions.

[1749] The processing flow of each embodiment will be described below.

[1750] "Example 1"

[1751] Step 1: The user enters reservation details into the app.

[1752] Step 2: The system will call the specified number.

[1753] Step 3: AI voice recognition technology means the conversation with the person on the other end of the phone.

[1754] Step 4: If an irregular question comes up during the conversation, the system prompts the user to enter an answer.

[1755] Step 5: The emotion engine recognizes the emotion from the user's tone of voice.

[1756] Step 6: The emotion engine adjusts the tone of the conversation based on that emotion.

[1757] "Example 2"

[1758] Step 1: The user enters reservation details into the app.

[1759] Step 2: The system will call the specified number.

[1760] Step 3: AI voice recognition technology means the conversation with the person on the other end of the phone.

[1761] Step 4: If an irregular question comes up during the conversation, the system prompts the user to enter an answer.

[1762] Step 5: The emotion engine recognizes the emotion from the user's tone of voice.

[1763] Step 6: The emotion engine adjusts the tone of the conversation based on that emotion.

[1764] Example 1

[1765] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1766] Conventional telephone reservation systems require users to make a direct call, which makes it difficult to respond to irregular inquiries. Furthermore, they are unable to respond to the user's emotions, which can lead to a poor user experience. There is a need to solve these problems and provide a more natural and efficient telephone reservation system.

[1767] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1768] In this invention, the server includes means for making a call to a specified telephone number based on information input by the user, a voice recognition technology means for conversing with the person at the telephone number, means for prompting the user to input a response if an irregular question is asked during the conversation, and means for recognizing the user's emotions and adjusting the tone of the conversation based on the emotions. This allows the user to make reservations efficiently without having to make a direct call, and irregular questions can be quickly responded to. Furthermore, it is possible to respond according to the user's emotions, providing a more natural and satisfying user experience.

[1769] "User" means an individual or group that uses the system to make reservations or enter information.

[1770] "Information" refers to data that users input into the system, such as reservation date and time, number of people, and names.

[1771] "Telephone number" refers to a location or organization that has a telephone number where a user can make a reservation or inquiry.

[1772] The "means for making a call" refers to the function or technology that enables the server to make a call to a specified telephone number.

[1773] "Speech recognition technology means" refers to technology that converts what the person on the other end of the phone says into text in real time and generates an appropriate response.

[1774] An "irregular question" is an unexpected question or request that is not part of the normal booking process or conversation flow.

[1775] The "means for requesting input of an answer" is a function that prompts the user to input an answer to an irregular question when the question is asked.

[1776] The "means for recognizing emotions" is a technology that analyzes and recognizes emotions from the tone of a user's voice and other voice characteristics.

[1777] The "means for adjusting the tone of the conversation" is a function that appropriately changes the tone and content of the voice response generated by the system based on the recognized user's emotions.

[1778] This invention is a system that allows a user to input reservation information using a smartphone application and then makes a call to a specified telephone number based on that information. The system analyzes the information entered by the user and uses voice recognition technology to converse with the person on the other end of the line. The system also includes a means for prompting the user to input a response if an irregular question is asked during the conversation. The system also includes a means for recognizing the user's emotions and adjusting the tone of the conversation based on those emotions.

[1779] Hardware and software used

[1780] Hardware

[1781] Smartphone: A device used by users to enter reservation information.

[1782] Server: A central processing unit for analyzing data, making calls, performing voice recognition, and emotion recognition.

[1783] software

[1784] Application: Runs on a smartphone and provides an interface for users to enter reservation information.

[1785] Generative AI models: For example, OpenAI's GPT-4 is used to analyze user-entered information and generate appropriate instructions.

[1786] Speech recognition technology: For example, Google Cloud Speech-to-Text is used to convert what the person on the other end of the phone says into text in real time.

[1787] Emotion Recognition Technology: For example, Affectiva's emotion recognition technology is used to recognize emotions from the user's tone of voice.

[1788] Cloud communications platforms: For example, using Twilio to make calls to specified destinations.

[1789] Specific examples

[1790] Example 1: Restaurant reservations

[1791] A user opens a smartphone application and enters "October 15, 2023, 7 PM, 4 people, Taro Tanaka." The device sends this information to the server. The server uses a generative AI model to generate the instruction "Please make a reservation for Taro Tanaka on October 15, 2023 at 7 PM for 4 people," and uses Twilio to call the restaurant. When the call connects, the server says, "Hello, my name is Taro Tanaka. I'd like to make a reservation for 4 people on October 15 at 7 PM." If the restaurant asks, "Do you have any special requests?" the server replies, "Please wait," and sends a notification to the device. If the user enters, "I'd like a window seat," the device sends this information to the server, and the server again tells the person on the phone, "I'd like a window seat."

[1792] Example 2: Using the Emotion Engine

[1793] The user enters "October 15, 2023, 7:00 PM, 4 people, Taro Tanaka" into the app, and the call is connected. If the user begins to speak in an angry tone during the conversation, the emotion engine recognizes that emotion and passes it on to the server. The server passes that information on to the AI, which responds calmly with, "I'm sorry, I'll check right away."

[1794] Prompt Sentence Examples

[1795] "Please explain a system in which a user inputs reservation information into a smartphone application, which then uses that information to call a specified number. Once the call is connected, the system uses AI voice recognition technology to conduct the conversation, and if an irregular question is asked, please include a mechanism to prompt the user for an answer. Also, please explain how an emotion engine is used to adjust the tone of the conversation according to the user's emotions."

[1796] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1797] Step 1:

[1798] The user opens the application on their smartphone and enters reservation information (reservation date and time, number of people, name, etc.).

[1799] Input: Information such as reservation date and time, number of people, name, etc.

[1800] Output: Reservation information entered

[1801] Specific operation: The user enters "October 15, 2023, 7:00 PM, 4 people, Taro Tanaka" into the application's input form.

[1802] Step 2:

[1803] The terminal transmits the reservation information input by the user to the server.

[1804] Input: Reservation information entered by the user

[1805] Output: Reservation information sent to the server

[1806] Specific operation: The terminal uses an HTTP request to send the entered reservation information to the server in JSON format.

[1807] Step 3:

[1808] The server analyzes the received reservation information using a generative AI model and generates instructions to call the specified phone number.

[1809] Input: Reservation information sent to the server

[1810] Output: Instructions to make a call

[1811] Specific operation: The server uses a generative AI model (e.g., GPT-4) to generate the instruction, "Please make a reservation for Taro Tanaka on October 15, 2023 at 7 p.m. for four people."

[1812] Step 4:

[1813] The server places a call to the specified telephone number based on the generated instruction.

[1814] Input: Instructions to make a call

[1815] Output: Make a phone call

[1816] Specific operation: The server uses Twilio's API to make a call to the specified restaurant.

[1817] Step 5:

[1818] Once the call is connected, the server uses voice recognition technology to converse with the person on the other end of the line.

[1819] Input: What the person on the other end of the phone says

[1820] Output: Speech converted to text

[1821] How it works: The server uses Google Cloud Speech-to-Text to convert what the person on the other end of the phone says into text in real time.

[1822] Step 6:

[1823] The server uses a generative AI model to generate an appropriate response and deliver it to the person on the other end of the call.

[1824] Input: Speech converted to text

[1825] Output: The generated response

[1826] Specific operation: The server uses the generative AI model to generate a response that says, "Hello, my name is Taro Tanaka. I would like to make a reservation for four people at 7 p.m. on October 15th," and conveys this to the person on the other end of the phone.

[1827] Step 7:

[1828] If an irregular question comes up during a conversation, the server will tell the other party to wait and put the call on hold.

[1829] Input: Irregular Question

[1830] Output: "Please wait" response

[1831] Specific behavior: When the server receives the question "Do you have any special requests?", it tells the caller "Please wait" and puts the call on hold.

[1832] Step 8:

[1833] The server sends a notification to the user at the terminal, requesting input of an answer.

[1834] Input: Irregular Question

[1835] Output: User notification

[1836] Specific operation: The server sends a push notification to the device, asking the user, "Do you have any special requests?"

[1837] Step 9:

[1838] Users enter their answers in the app.

[1839] Input: User notification

[1840] Output: User's answer

[1841] Specific operation: The user enters "I would like a window seat" in the app.

[1842] Step 10:

[1843] The terminal transmits the user's answer to the server.

[1844] Input: User's answer

[1845] Output: The answer sent to the server

[1846] Specific operation: The terminal uses an HTTP request to send the user's answer to the server.

[1847] Step 11:

[1848] The server continues the conversation with the person on the other end of the phone based on the user's response.

[1849] Input: The answer sent to the server

[1850] Output: Response to the person on the phone

[1851] Specific operation: The server generates a response saying "I would like a window seat" and conveys this to the person on the other end of the phone.

[1852] Step 12:

[1853] The server uses emotion recognition technology to recognize emotions from the tone of the user's voice and transmits that information to AI voice recognition technology.

[1854] Input: User's voice tone

[1855] Output: Recognized emotion information

[1856] Specific operation: The server uses emotion recognition technology to determine that the user is angry and conveys that information to the AI ​​voice recognition technology.

[1857] Step 13:

[1858] The server adjusts the tone of the conversation based on the recognized emotional information.

[1859] Input: Recognized emotion information

[1860] Output: Modulated conversational tone

[1861] What it does: The server uses AI voice recognition technology to respond in a calm tone, saying, "Sorry, we'll check it right away."

[1862] (Application example 1)

[1863] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1864] Conventional food delivery systems require users to place orders over the phone, which can lead to cumbersome communication and difficulty in responding to irregular questions. Furthermore, they often fail to consider the user's feelings, resulting in a slow ordering process. This can lead to poor user experience, cancellations of orders, and dissatisfaction.

[1865] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1866] In this invention, the server includes: means for making a call to a specified telephone number based on information input by the user; artificial intelligence speech recognition technology means for conversing with the person at the telephone number; means for prompting the user to input a response when an irregular question is asked during the conversation; emotion engine means for recognizing the user's emotion and adjusting the tone of the conversation based on the emotion; and generative artificial intelligence model means for analyzing the information input by the user and generating instructions to call the telephone number. This not only enables the user to place an order smoothly and quickly respond to irregular questions, but also enables natural conversation that reflects the user's emotion.

[1867] "Information entered by the user" refers to information such as reservation date and time, number of people, names, menu, etc., entered by the user via a smartphone or other device.

[1868] "Means for calling a specified telephone number" refers to technology for automatically calling a specific telephone number based on information entered by the user.

[1869] "Artificial intelligence speech recognition technology means" refers to artificial intelligence technology that converts the voice of the person on the other end of the phone into text and generates an appropriate response.

[1870] The "means for prompting the user to input an answer when an irregular question is asked" is an interface that allows the user to input an answer to a question when an unexpected question is asked during a conversation.

[1871] The "emotion engine means" is a technology for recognizing emotions from the tone of a user's voice and input content, and adjusting the tone of the conversation based on those emotions.

[1872] The "generative artificial intelligence model means" is an artificial intelligence model that analyzes information input by the user and generates instructions to be called and conversation content.

[1873] In the system for implementing this invention, a user inputs order information using a smartphone or other device, and an AI automatically makes a phone call based on that information to confirm the order. Specific embodiments of this system are described below.

[1874] System configuration

[1875] 1. User Interface:

[1876] Use a smartphone application (iOS / Android).

[1877] Provide a form for users to enter order information (date and time, number of people, menu, name, etc.).

[1878] 2. Data analysis and generative AI models:

[1879] The information entered by the user is sent to a server in the cloud.

[1880] On the server side, a generative AI model (e.g., OpenAI's GPT-4) is used to generate instructions to call the destination.

[1881] 3. Telephone connection and AI voice recognition:

[1882] Use APIs such as Twilio to automatically call specified numbers.

[1883] Once the call is connected, the Google Cloud Speech-to-Text API is used to convert the caller's voice into text.

[1884] Use generative AI models to generate appropriate answers to the other person's questions.

[1885] 4. Handling irregular questions:

[1886] If an irregular question is received, the app will put it on hold with a "Please wait" message and ask the user to enter the answer on the app.

[1887] The conversation continues based on the answers entered by the user.

[1888] 5. Emotion Engine:

[1889] Use an emotion recognition API (e.g. IBM Watson Tone Analyzer) to analyze the tone of the user's voice.

[1890] The emotion engine recognizes the user's emotions and conveys that information to the generative AI model.

[1891] A generative AI model adjusts the tone of the conversation to create a natural conversation that responds to the user's emotions.

[1892] Specific examples

[1893] A user types "I'd like to order two pizzas tomorrow night at 7pm" into a smartphone app. The server receives this information and uses a generative AI model to generate instructions to call the restaurant. The Twilio API is used to automatically make the call, and the Google Cloud Speech-to-Text API is used to convert the caller's voice into text. If the restaurant asks, "Do you have specific toppings?", the app prompts the user for a response. If the user types "pepperoni and mushrooms," the AI ​​continues the conversation based on that information. If the user is angry, the emotion engine recognizes this information and the generative AI model adjusts the tone of the conversation.

[1894] Prompt Sentence Examples

[1895] A user types, "I'd like to order two pizzas tomorrow night at 7pm." Call the restaurant to confirm the order. If they ask about specific toppings, ask them in-app and continue the conversation based on their answer. Also, recognize the user's emotions and adjust the tone of the conversation accordingly.

[1896] In this way, users can enjoy a smoother and more natural ordering experience in food delivery applications.

[1897] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1898] Step 1:

[1899] The user launches the smartphone application and enters order information (date and time, number of people, menu, name, etc.). The entered information is sent from the device to a server on the cloud. The input data is sent to the server in the form of reservation date and time, number of people, menu, name, etc.

[1900] Step 2:

[1901] The server analyzes the received order information and uses a generative AI model (e.g., OpenAI's GPT-4) to generate instructions to call the destination. The generated instructions are output as a specific prompt for making the call. For example, a prompt such as "I'd like to order two pizzas tomorrow night at 7 p.m." is generated.

[1902] Step 3:

[1903] The server automatically calls the specified phone number using an API such as Twilio. Once the call is connected, the server uses the Google Cloud Speech-to-Text API to convert the caller's voice into text in real time. The converted text data is then sent to the server.

[1904] Step 4:

[1905] The server inputs the converted text data into a generative AI model to generate an appropriate answer to the caller's question. The generated answer is then converted back into speech and transmitted to the caller. For example, in response to the question "Do you have any specific toppings?", the answer might be "pepperoni and mushrooms."

[1906] Step 5:

[1907] If an irregular question comes up during a conversation, the server puts the question on hold and asks the user to enter the answer in the app. The answer entered by the user is sent from the device to the server. For example, in response to the question "Do you have any specific toppings?", the user might enter "pepperoni and mushrooms."

[1908] Step 6:

[1909] The server continues the conversation based on the answers entered by the user. It uses a generative AI model to generate an appropriate answer to the caller's question, converts it into speech, and delivers it to the caller. For example, it might respond with "pepperoni and mushrooms."

[1910] Step 7:

[1911] The server uses an emotion recognition API (e.g., IBM Watson Tone Analyzer) to analyze the user's tone of voice and recognize their emotions. The recognized emotion data is input into the generative AI model. For example, if the user is angry, that information is conveyed to the generative AI model.

[1912] Step 8:

[1913] The generative AI model adjusts the tone of the conversation based on the perceived emotion. For example, if the user is angry, the generative AI model will continue the conversation in a calmer tone. The adjusted tone of the conversation is converted into audio and transmitted to the other party on the call.

[1914] In this way, the user can smoothly place an order, irregular questions can be quickly responded to, and natural conversations can be held according to the user's emotions.

[1915] Example 2

[1916] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1917] Conventional reservation systems have the problem of being unable to respond appropriately when unexpected questions arise when making a phone call based on the information entered by the user. Furthermore, the system responds mechanically without considering the user's feelings, which can result in a poor user experience. There is a need to solve these problems and provide a more natural and user-friendly reservation system.

[1918] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1919] In this invention, the server includes means for making a call to a specified telephone number based on information input by the user, voice recognition technology means for conversing with the person at the telephone number, means for prompting the user to input a response when an unexpected question is asked during the conversation, and emotion recognition means for recognizing the user's emotion and adjusting the tone of the conversation based on that emotion. This allows for appropriate responses to unexpected questions and enables natural conversation that reflects the user's emotion.

[1920] "User" means an individual or group that uses the system to make reservations or enter information.

[1921] A "means for making a call" is a device or software that has the function of automatically making a call to a specified telephone number based on information entered by the user.

[1922] "Voice recognition technology means" refers to a device or software that has the technology to analyze the voice of the person on the other end of the phone and convert it into text data.

[1923] An "unexpected question" is a question or request that the system does not anticipate in advance, and may require input from the user.

[1924] The "means for prompting the user to input an answer" is a device or software that has the function of prompting the user to input an answer when an unexpected question is presented.

[1925] An "emotion recognition means" is a device or software that has the technology to analyze emotions from the user's tone of voice and vocabulary, and adjust the system's response based on that information.

[1926] The "reservation date and time" refers to the specific date and time that the user desires to make a reservation.

[1927] "Number of Participants" means the total number of Participants in a Booking.

[1928] "Name" refers to the name of the user making the reservation.

[1929] The "holding means" is a device or software that has the function of temporarily interrupting a conversation when an unexpected question is asked and asking the user for confirmation.

[1930] This invention is a system for allowing users to make restaurant reservations, and uses voice recognition technology to call a designated number based on information entered by the user and to converse with the person at the call. It also includes means for prompting the user to input a response when an unexpected question is asked during the conversation, and emotion recognition means for recognizing the user's emotion and adjusting the tone of the conversation based on that emotion.

[1931] Hardware and software used

[1932] 1. Server:

[1933] Generative AI model: OpenAI's GPT-4 is used to analyze user-entered information.

[1934] Telephone API: Uses Twilio to make a call to a specified phone number.

[1935] Speech recognition technology: Google Cloud Speech-to-Text is used to convert the voice of the person on the other end of the call into text data.

[1936] Text-to-speech technology: Amazon Polly is used to provide voice-over reservation information.

[1937] Emotion Recognition Technology: Affectiva is used to analyze emotions from the user's tone of voice.

[1938] 2. Terminal:

[1939] User interface: Provides applications that run on devices such as smartphones and tablets.

[1940] Program processing

[1941] 1. User enters reservation information:

[1942] The user launches the application and accesses the reservation form.

[1943] The user inputs the reservation date and time, number of people, and name.

[1944] Example: A user enters on their smartphone screen, "I would like to make a reservation for four people on October 15, 2023 at 7:00 PM. My name is Taro Tanaka."

[1945] 2. The server receives and analyzes the information:

[1946] The server receives the information entered by the user.

[1947] The server uses a generative AI model (e.g., OpenAI's GPT-4) to analyze the input information.

[1948] Example: The server generates the analysis result "Taro Tanaka would like to make a reservation for 4 people on October 15, 2023 at 7:00 PM."

[1949] 3. The server makes the call:

[1950] The server uses a telephone API such as Twilio to make a call to the specified restaurant.

[1951] Example: Your server calls the Twilio API to place a call to a restaurant's phone number.

[1952] 4. Servers converse with restaurant staff:

[1953] The server uses AI speech synthesis technology (e.g., Amazon Polly) to communicate the reservation information to restaurant staff.

[1954] Example: The server says, "Hello, my name is Taro Tanaka. I'd like to make a reservation for 4 people at 7:00 PM on October 15, 2023."

[1955] 5. The server responds to unexpected questions:

[1956] If an unexpected question comes from restaurant staff, the server will put the call on hold and ask the user to enter the answer on the app.

[1957] Example: The server receives the question "Would you like a window seat?" and responds "Please wait. I'll check with the user."

[1958] 6. User answers the question:

[1959] The user enters answers to questions on the app.

[1960] Example: A user types "Yes, I'd like a window seat" into an input field in an app.

[1961] 7. The server resumes the conversation and confirms the reservation:

[1962] The server continues the conversation with the restaurant staff based on the user's response and confirms the reservation.

[1963] Example: A server tells restaurant staff, "I'd like a table by the window," and confirms that the reservation is confirmed.

[1964] Prompt Sentence Examples

[1965] "When a user wants to make a restaurant reservation, they enter the reservation date and time, number of people, and their name. Analyze this information and generate instructions to call the specified restaurant. Once the call is connected, use AI voice recognition technology to speak with the restaurant staff and complete the reservation. If an unexpected question comes up, ask the user to enter an answer and continue the conversation based on that answer."

[1966] The above is an embodiment of the invention.

[1967] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1968] Step 1:

[1969] The user enters the reservation information.

[1970] Input: The user accesses the application's reservation form and enters the reservation date and time, number of people, and name.

[1971] Specific operation: The user enters on the smartphone screen, "I would like to make a reservation for four people on October 15, 2023 at 7:00 PM. My name is Tanaka Taro."

[1972] Output: The entered reservation information is sent to the server.

[1973] Step 2:

[1974] The server receives and analyzes the information.

[1975] Input: Booking information entered by the user.

[1976] What it does: The server uses a generative AI model (e.g., OpenAI's GPT-4) to analyze the input information.

[1977] Data processing: Analyze the reservation date and time, number of people, and names, and format the reservation details in text format.

[1978] Output: Analysis result: "Taro Tanaka would like to make a reservation for 4 people on October 15, 2023 at 7:00 PM."

[1979] Step 3:

[1980] The server makes the call.

[1981] Input: Parsed reservation information.

[1982] Specific operation: The server uses a telephone API such as Twilio to make a call to the specified restaurant.

[1983] Data processing: Convert reservation information into a format to be passed to the telephone API.

[1984] Output: A call is made to the restaurant's phone number.

[1985] Step 4:

[1986] The server talks to the restaurant staff.

[1987] Input: Reservation information and phone connection.

[1988] Specific operation: The server uses AI speech synthesis technology (e.g., Amazon Polly) to convey the reservation information to restaurant staff.

[1989] Data processing: Convert reservation information in text format into audio data.

[1990] Output: A voice message saying, "Hello, my name is Taro Tanaka. I'd like to make a reservation for 4 people on October 15, 2023 at 7:00 PM."

[1991] Step 5:

[1992] The server responds to unexpected questions.

[1993] Input: Question from restaurant staff.

[1994] What happens: The server uses speech recognition technology (e.g., Google Cloud Speech-to-Text) to convert the question into text data.

[1995] Data processing: Convert the voice data into text data and analyze the question content.

[1996] Output: Text data: "Would you like a window seat?"

[1997] Step 6:

[1998] The user answers the question.

[1999] Input: The question from the server.

[2000] What happens: The user enters an answer into an input field in the app.

[2001] Data processing: The user's answers are sent to the server as text data.

[2002] Output: Text data: "Yes, I'd like a window seat."

[2003] Step 7:

[2004] The server resumes the conversation and confirms the reservation.

[2005] Input: The user's answer.

[2006] Specific operation: The server again uses AI speech synthesis technology to convey the user's response to the restaurant staff.

[2007] Data processing: Convert the user's answers into audio data.

[2008] Output: A voice message saying "I would like a window seat."

[2009] The above is the flow of processing of the program of this system.

[2010] (Application example 2)

[2011] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[2012] With conventional food delivery services, when a user places an order, they must manually enter information and confirm the order via phone or app. However, this is time-consuming and labor-intensive, and the user must respond again, especially if unexpected questions or changes arise. Furthermore, the system is unable to provide natural conversations that reflect the user's emotions, resulting in a poor user experience. To solve these issues, a system is needed that can automatically place orders based on the user's input information and enable natural conversations that reflect the user's emotions.

[2013] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2014] In this invention, the server includes means for making a call to a specified telephone number based on information input by the user, voice recognition technology means for conversing with the person at the telephone number, means for prompting the user to input a response when an irregular question is asked during the conversation, an emotion engine for recognizing the user's emotions, and means for adjusting the tone of the conversation based on the emotions recognized by the emotion engine. This enables a user to automatically communicate the order details, delivery time, and special requests to a restaurant over the phone when using a food delivery service, and also enables a natural conversation that reflects the user's emotions.

[2015] "User" means an individual or organization that uses the system to input information and receive services.

[2016] "Information" refers to data that a user enters into the system, including order details, delivery times, special requests, etc.

[2017] "Telephone number" refers to a contact point designated by the user, typically a restaurant or store that provides food delivery services.

[2018] "Means for making a call" is a function that allows the system to automatically make a call to a specified telephone number.

[2019] "Speech recognition technology means" refers to technology that enables the system to understand the conversation with the person on the other end of the phone and respond appropriately.

[2020] An "irregular question" is when the person on the other end of the phone asks an unexpected question or makes a request.

[2021] The "means for requesting input of an answer" is a function that prompts the user to input an answer when an irregular question is asked.

[2022] The "emotion engine" is a technology that recognizes emotions from the user's tone of voice and input content.

[2023] The "means for adjusting the tone of a conversation" is a function by which the system adjusts the tone of a conversation based on the emotion recognized by the emotion engine.

[2024] "System" means a device or program that includes a series of technical means for automatically making calls based on information entered by the user and for realizing natural conversations using voice recognition technology and an emotion engine.

[2025] A system for carrying out this invention includes means for making a call to a specified telephone number based on information input by a user, voice recognition technology means for conducting a conversation with the person at the telephone number, means for prompting the user to input a response when an irregular question is asked during the conversation, an emotion engine for recognizing the user's emotions, and means for adjusting the tone of the conversation based on the emotions recognized by the emotion engine.

[2026] Hardware and software used

[2027] Hardware: Smartphone

[2028] Software: Google Cloud Dialogflow, IBM Watson Tone Analyzer

[2029] Data processing and calculation

[2030] Receiving user input

[2031] Using a smartphone application, users enter information such as order details, delivery time, and special requests, which is then sent to a server.

[2032] sentiment analysis

[2033] The server uses IBM Watson Tone Analyzer to analyze the emotions in the user's text input. The emotion engine recognizes emotions from the user's tone of voice and input, and adjusts the tone of the conversation based on the results.

[2034] Intent detection

[2035] The server uses Google Cloud Dialogflow to detect the user's intent from the text they input, which then generates an appropriate response and instructions to call the specified phone number.

[2036] Making a phone reservation

[2037] Based on the generated response, the server uses AI to call the restaurant and place the order. Once the call is connected, the customer uses voice recognition technology to converse with the restaurant staff and complete the order.

[2038] Responding to irregular questions

[2039] If an unexpected question comes from the restaurant, the server puts the request on hold and asks the user to enter an answer on the application. Once the user enters the answer, the conversation with the restaurant staff continues based on the answer and the order is confirmed.

[2040] Specific examples

[2041] If a user types "Two pizzas, please deliver them at 7 PM," the emotion engine analyzes the user's emotions and Dialogflow generates an appropriate response. For example, if the user is happy, the AI ​​will proceed with the conversation in a tone such as "Thank you! Two pizzas will be delivered at 7 PM."

[2042] Prompt Sentence Examples

[2043] If a user types "Two pizzas, please deliver them at 7 PM," the emotion engine analyzes the user's emotions and Dialogflow generates an appropriate response. For example, if the user is happy, the AI ​​will proceed with the conversation in a tone such as "Thank you! Two pizzas will be delivered at 7 PM."

[2044] In this way, a food delivery assistant application that can have natural conversations according to the user's emotions is completed.

[2045] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2046] Step 1:

[2047] A user uses a smartphone application to enter information such as order details, delivery time, and special requests.

[2048] Input: Order details, delivery time, special requests

[2049] Output: The entered information is sent to the server

[2050] Specific behavior: When a user enters information into an application's input form and presses the "Submit" button, the information is sent to the server.

[2051] Step 2:

[2052] The server uses IBM Watson Tone Analyzer to analyze the sentiment from the user's input text.

[2053] Input: Information entered by the user

[2054] Output: Sentiment analysis results

[2055] Specific operation: The server sends the input text data to IBM Watson Tone Analyzer and receives the sentiment analysis results.

[2056] Step 3:

[2057] The server uses Google Cloud Dialogflow to detect the user's intent from the input text.

[2058] Input: Information entered by the user

[2059] Output: Intent detection result

[2060] Specific operation: The server sends the entered text data to Google Cloud Dialogflow and receives the intent detection result.

[2061] Step 4:

[2062] The server generates an instruction to call the specified telephone number based on the generated response.

[2063] Input: Intention detection results, sentiment analysis results

[2064] Output: Instructions to make a call

[2065] Specific operation: The server generates an appropriate response based on the intent detection and sentiment analysis results, and generates instructions to make a call.

[2066] Step 5:

[2067] The server calls the designated number and uses voice recognition technology to talk to restaurant staff.

[2068] Input: Instructions to make a call

[2069] Output: Conversation

[2070] What it does: The server automatically makes a call and uses voice recognition technology to speak to restaurant staff.

[2071] Step 6:

[2072] If an unexpected question comes from the restaurant, the server will put the request on hold and ask the user to enter the answer on the application.

[2073] Input: Unexpected question

[2074] Output: Prompt the user for an answer

[2075] What happens: The server detects the unexpected question and sends a message to the application prompting the user to enter an answer.

[2076] Step 7:

[2077] The user inputs an answer, and the server continues the conversation with the restaurant staff based on the answer and confirms the order.

[2078] Input: User's answer

[2079] Output: Confirmed order details

[2080] Specific behavior: The user enters an answer into the application, and the server continues the conversation with the restaurant staff based on the answer and confirms the order.

[2081] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[2082] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2083] Another example of generative AI is Gemini (internet search engine). <url: https: gemini.google.com ?hl="ja">) are listed.

[2084] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[2085] [Fourth embodiment]

[2086] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[2087] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[2088] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[2089] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[2090] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[2091] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[2092] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[2093] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[2094] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[2095] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[2096] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[2097] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[2098] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[2099] "Example 1"

[2100] In one embodiment of the present invention, a user inputs information such as reservation date and time, number of people, and names into a smartphone application. This information is analyzed by generative AI, which generates instructions to call the specified number. Once the call is connected, the conversation with the person on the other end is carried out using AI voice recognition technology. If an irregular question is asked during the conversation, the system puts the call on hold with a "Please wait" message and prompts the user to enter a response on the app. Once the user has entered a response, the conversation with the person on the other end is continued based on that response.

[2101] "Example 2"

[2102] As a specific example, when a user wants to make a restaurant reservation, they input the reservation date and time, number of people, and name on the application. This information is analyzed by generative AI, which generates instructions to call the specified restaurant. Once the call is connected, AI voice recognition technology is used to converse with the restaurant staff and complete the reservation. If the restaurant asks an unexpected question, the app puts the call on hold by saying "Please wait" and asks the user to enter a response on the app. Once the user has entered a response, the conversation with the restaurant staff continues based on that response, and the reservation is confirmed.

[2103] The processing flow of each embodiment will be described below.

[2104] "Example 1"

[2105] Step 1: The user enters information such as reservation date and time, number of people, and names on the smartphone application.

[2106] Step 2: This information is analyzed by a generative AI to generate instructions to call the specified number.

[2107] Step 3: Once the call is connected, use AI voice recognition technology to talk to the person on the other end.

[2108] Step 4: If an irregular question comes up during the conversation, put it on hold with "Please wait" and ask the user to enter the answer on the app.

[2109] Step 5: Once the user has entered their answer, the conversation with the person on the other end of the phone continues based on that answer.

[2110] "Example 2"

[2111] Step 1: When a user wants to make a restaurant reservation, they input the reservation date and time, number of people, and name on the application.

[2112] Step 2: This information is analyzed by a generative AI, which generates instructions to call the specified restaurant.

[2113] Step 3: Once the call is connected, use AI voice recognition technology to speak to the restaurant staff and complete the reservation.

[2114] Step 4: If the restaurant asks an unexpected question, the app will put the question on hold and ask the user to enter the answer on the app.

[2115] Step 5: Once the user has entered their answer, they can continue the conversation with the restaurant staff based on their answer and confirm the reservation.

[2116] Example 1

[2117] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2118] Conventional reservation systems require users to manually make phone calls and make reservations, which is time-consuming and makes it difficult to respond appropriately when an irregular question comes up during a conversation with the person on the other end of the phone. Furthermore, there is a lack of a way to efficiently analyze the information entered by the user and generate an appropriate response.

[2119] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[2120] In this invention, the server includes means for making a call to a specified telephone number based on information input by the user, speech recognition technology means for conversing with the person at the telephone number, means for prompting the user to input a response when an irregular question is asked during the conversation, means for using a generative AI model for analyzing the information input by the user, means for generating a prompt sentence based on the analysis result, and means for analyzing the speech of the person at the telephone number in real time and generating an appropriate response. This allows the user to make a reservation without hassle and makes it possible to appropriately respond to irregular questions.

[2121] "User" means an individual or group that uses the system to make reservations or enter information.

[2122] "Information" refers to data such as reservation date and time, number of people, and names entered by the user.

[2123] "Telephone number" refers to the telephone number at which a user makes a reservation or inquiry.

[2124] "Means for making phone calls" refers to the ability to make calls to designated telephone numbers using VoIP technology.

[2125] "Voice recognition technology means" refers to technology that converts the speech of the person on the other end of the phone into text in real time.

[2126] An "irregular question" refers to an unexpected question or confirmation from the person on the other end of the phone.

[2127] "Means for requesting answer input" refers to an interface that allows the user to input an answer when an irregular question is asked.

[2128] A "generative AI model" refers to an artificial intelligence model that analyzes information entered by a user and generates appropriate responses or instructions.

[2129] A "prompt" refers to an instruction or question that is input into a generative AI model.

[2130] "Means of analysis" refers to the function of analyzing information entered by the user using a generative AI model.

[2131] "Means for generating a response" refers to the function of analyzing what the person on the other end of the phone says and generating an appropriate reply.

[2132] This invention is a system that efficiently and automatically makes calls and provides necessary information when a user makes a reservation or inquiry using a smartphone application. Specific embodiments of this system are described below.

[2133] Hardware and software used

[2134] Hardware

[2135] Smartphone (iOS or Android)

[2136] Server (Cloud server or on-premise server)

[2137] software

[2138] Smartphone application

[2139] Generative AI models (e.g., OpenAI's GPT-4)

[2140] Speech recognition technology (e....

Claims

1. means for making a call to a party designated by a user based on information input by the user; means for converting the voice of the other party into text to generate a response of the other party; A means for detecting an unexpected question included in a response of the other party by analyzing the voice of the other party; an answer request means for requesting the user to input an answer to the unexpected question when the unexpected question is detected during a conversation between the user and the other party; A means for analyzing the order information input by the user as the answer and inputting a specific prompt sentence into a generating AI model to generate an instruction for the telephone order to the other party; an emotion engine that recognizes the emotion of the user when providing the answer; means for adjusting a tone of voice of the user in a conversation with the other person based on the recognized emotion; A system including:

2. The system according to claim 1 , wherein the input information includes the date and time of reservation at the restaurant, the number of people, and the name of the person.

3. 2. The system according to claim 1, wherein, when the unexpected question is detected, the answer requesting means requests input of an answer to the unexpected question and plays back a guidance to the other party to put the conversation on hold.

Images