System

Abstract

To provide a system.SOLUTION: A system includes: first means for receiving a question by voice from a user in an automatic driving vehicle; second means for generating a prompt sentence for generating an answer to the question on the basis of contents of the question from the user; third means for inputting the generated prompt sentence into a generative AI model to generate the answer; and fourth means for outputting the generated answer to the user using a speaker in the automatic driving vehicle.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including a description and related instruction sentence regarding the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-180282 A Summary of the Invention [Problem to be solved by the invention]

[0004] Currently, technical support for smartphones is mainly provided through text-based chat. However, this requires users to input text, and can result in waiting times. Furthermore, when follow-up by a human is required, it is difficult to provide efficient suggestions for a solution. [Means for solving the problem]

[0005] This invention converts voice input into text using a voice recognition model, generates a response based on the text input using generative AI, and converts the generated text response into voice using a text-to-speech AI. Furthermore, by recording a response log, even if follow-up by a human is required, it is possible to make a waste-free proposal for a solution. This realizes voice response without waiting time 24 hours a day, 365 days a year, reducing the user's effort and providing efficient support. [Brief description of the drawings]

[0006] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Diagram 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. FIG. [Diagram 3] FIG. 11 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Diagram 5] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 13 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 2 is a sequence diagram showing a process flow of the data processing system in the first embodiment of the first form example. [Figure 12]FIG. 11 is a sequence diagram showing a process flow of the data processing system in the application example 1 of the first embodiment. [Figure 13] FIG. 11 is a sequence diagram showing a process flow of the data processing system in the second embodiment of the second embodiment. [Figure 14] FIG. 13 is a sequence diagram showing a process flow of the data processing system in application example 2 of embodiment 2. [Figure 15] FIG. 13 is a sequence diagram showing a process flow of a data processing system in a third embodiment of the present invention. [Figure 16] FIG. 13 is a sequence diagram showing a process flow of the data processing system in the third application example of the third embodiment. [Figure 17] FIG. 11 is a sequence diagram showing the flow of processing in the data processing system in the first embodiment of the first form example when an emotion engine is combined. [Figure 18] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Form Example 1 when combined with an emotion engine. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0008] First, the terms used in the following description will be explained.

[0009] In the following embodiments, a signed processor (hereinafter simply referred to as a "processor") may be one arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be one type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)).

[0010] In the following embodiments, a signed RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by the processor.

[0011] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0012] In the following embodiments, a communication I / F (Interface) with a code is an interface including a communication processor and an antenna. The communication I / F controls communication between multiple computers. An example of a communication standard applied to the communication I / F is a wireless communication standard including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. In addition, in this specification, the same idea as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0014] [First embodiment]

[0015] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0016] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0017] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0018] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0019] The reception device 38 includes a touch panel 38A and a microphone 38B, and receives user input. The touch panel 38A detects contact with a pointer (e.g., a pen or a finger) to receive user input by the touch of the pointer. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0020] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (e.g., voice and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs voice according to instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a Complementary Metal-Oxide-Semiconductor (CMOS) image sensor or a Charge Coupled Device (CCD) image sensor.

[0021] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54.

[0022] FIG. 2 shows an example of main functions of the data processing device 12 and the smart device 14.

[0023] As shown in Fig. 2, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32. The specific process program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific process program 56 from the storage 32, and executes the read specific process program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific process program 56 executed on the RAM 30.

[0024] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0025] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores a reception output program 60. The reception output program 60 is used together with the specific processing program 56 by the data processing system 10. The processor 46 reads out the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0026] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0027] "Example 1"

[0028] The present invention is a system for providing technical support for smartphones. Voice input from a user is converted to text by a voice recognition model. For example, WhisperAPI is used as this voice recognition model. The converted text input is analyzed by a generative AI, and an appropriate response is generated. The generated text response is converted to voice by a text-to-speech AI and output to the user. For example, ReadSpeaker is used as this text-to-speech AI. In addition, the system records a response log, and when follow-up by personnel is required, a proposal for a solution is made based on this. This enables voice response 24 hours a day, 365 days a year without waiting time, reducing the burden on users and providing efficient support.

[0029] "Example 2"

[0030] The present invention is a system for providing technical support for smartphones. Voice input from a user is converted to text by a voice recognition model. For example, WhisperAPI is used as this voice recognition model. The converted text input is analyzed by a generative AI, and an appropriate response is generated. The generated text response is converted to voice by a text-to-speech AI and output to the user. For example, ReadSpeaker is used as this text-to-speech AI. In addition, the system records a response log, and when follow-up by personnel is required, a proposal for a solution is made based on this. This enables voice response 24 hours a day, 365 days a year without waiting time, reducing the burden on users and providing efficient support.

[0031] The process flow of each embodiment will be described below.

[0032] "Example 1"

[0033] Step 1: Receive voice input from the user. This voice input is done through the smartphone microphone.

[0034] Step 2: The received voice input is converted into text using a voice recognition model, such as WhisperAPI.

[0035] Step 3: The textual input is analyzed by a generative AI to generate an appropriate response. This generative AI has the ability to understand the user's problem and suggest appropriate solutions.

[0036] Step 4: The generated text response is converted to audio by a speech AI and output to the user. For example, ReadSpeaker is used as the speech AI. Step 5: The system records the response log. This allows the system to make suggestions for a solution based on the log, even if follow-up by a human is required.

[0037] Example 1

[0038] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal".

[0039] Conventional smartphone technical support systems often required users to wait a long time to resolve their issues, making it difficult to provide efficient support. In addition, the process of converting voice input into text, generating an appropriate response, and returning it by voice was sometimes inconsistent, which reduced the user experience.

[0040] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0041] In this invention, the server includes a means for converting voice input into text using a voice recognition model, a means for generating a response based on the converted text input using generative artificial intelligence, a means for converting the generated text response into voice using speech-to-speech artificial intelligence, a means for recording a response log, and a means for following up with personnel, thereby enabling efficient support 24 hours a day, 365 days a year without waiting times.

[0042] A "speech recognition model" is an algorithm or system for analyzing speech data and converting it into corresponding text data.

[0043] "Generative artificial intelligence" is an artificial intelligence technology that analyzes input data and generates appropriate responses or information.

[0044] "Read-to-speech artificial intelligence" is an artificial intelligence technology that converts text data into audio data and outputs it as audio.

[0045] A "response log" is a database or file that maintains a record of user interactions and the responses generated.

[0046] "Follow-up" is the process by which personnel respond when a user's problem is not resolved or additional support is required.

[0047] A "voice recognition API" is an application programming interface for converting voice data into text data.

[0048] A "speech synthesis API" is an application programming interface for converting text data into speech data.

[0049] This invention is a system for providing technical support for smartphones. Voice input from a user is converted to text by a voice recognition model. A voice recognition API is used as this voice recognition model. The text input is analyzed by a generative artificial intelligence, and an appropriate response is generated. The generated text response is converted to voice by a voice-to-speech artificial intelligence, and output to the user. A voice synthesis API is used as this voice-to-speech artificial intelligence. In addition, the system records a response log, and when follow-up by personnel is required, a proposal for a solution is made based on this. This enables voice response 24 hours a day, 365 days a year with no waiting time, reducing the burden on users and providing efficient support.

[0050] Specifically, a user speaks a question or problem into a smartphone. For example, they might say, "My smartphone's battery is running low." The device records this voice and sends the data to a server. The server uses a speech recognition API to convert the voice data into text. Specifically, it analyzes the voice signal and generates a corresponding string of characters.

[0051] The server then passes the converted user input to a generative AI, which uses natural language processing techniques, for example, to analyze the user's question or problem and generate an appropriate response, such as "To reduce battery drain, we recommend closing apps running in the background."

[0052] The generated text response is passed from the server to the speech-to-speech AI, which converts the text into speech and sends it to the device. The device then plays the speech to the user. Specifically, it generates an audio file and plays it through the smartphone speaker.

[0053] In addition, the server records a log of all responses. If the user's problem is not resolved or additional support is required, the server can use this log to send a follow-up call to a human. For example, if a user says, "Wi-Fi is not working," the server will check the log and escalate the call to the appropriate technician.

[0054] An example of a prompt sentence is, "What should you do if your smartphone battery runs out quickly?" By inputting this prompt sentence into a generative artificial intelligence system, an appropriate response is generated.

[0055] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0056] Step 1:

[0057] A user speaks a question or problem into a smartphone. For example, they might say, "My smartphone's battery is running low." The device records this voice and sends the data to a server. The input is the voice data, and the output is the voice data sent to the server.

[0058] Step 2:

[0059] The server uses a speech recognition API to convert this voice data into text: it analyzes the audio signal and generates a corresponding string of characters. The input is the voice data, and the output is the text data.

[0060] Step 3:

[0061] The server passes the textualized user input to the generative AI. The generative AI uses natural language processing techniques to analyze the user's question or problem and generate an appropriate response. For example, it generates a response such as "To reduce battery drain, we recommend closing apps running in the background." The input is text data, and the output is the response text.

[0062] Step 4:

[0063] The generated text response is passed from the server to the speech AI, which converts the text into speech and sends it to the terminal. Specifically, it generates an audio file and plays it through the smartphone speaker. The input is the response text, and the output is the audio data.

[0064] Step 5:

[0065] The server records all response logs. If the user's problem is not resolved or additional support is required, the server uses these logs to send personnel to follow up. For example, if a user says "Wi-Fi is not working," the server checks the logs and escalates the issue to the appropriate technician. The input is the response log, and the output is the information for follow-up.

[0066] (Application example 1)

[0067] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0068] As autonomous vehicles become more widespread, users need to receive support for vehicle operation and troubleshooting quickly and efficiently. However, conventional systems require users to spend a lot of time and effort to solve problems, making it difficult to respond quickly, especially in emergencies.

[0069] The identification process by the identification processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means. In this invention, the server includes a means for converting a voice input into text using a voice recognition model, a means for generating a response based on the input converted into text using a generative AI, a means for converting the generated text response into voice using a text-to-speech AI, a means for recording a response log, and a means for analyzing a voice input of a user in an autonomous vehicle and providing support regarding vehicle operation and troubleshooting. This allows the user to receive support quickly and efficiently using voice in an autonomous vehicle.

[0070] A "speech recognition model" is an algorithm or software for converting voice input into text data.

[0071] "Generative AI" is an artificial intelligence technology that analyzes input text data and generates appropriate responses.

[0072] "Reading AI" refers to algorithms and software for converting text data into speech.

[0073] A "response log" is a database or file that stores records of user interactions.

[0074] An "autonomous vehicle" is a vehicle that uses artificial intelligence and sensor technology to drive autonomously.

[0075] "User's voice input" refers to voice data generated by a user through a voice input device such as a microphone.

[0076] "Vehicle operation" refers to the act of controlling various vehicle functions (e.g., air conditioning, navigation, audio system, etc.).

[0077] "Troubleshooting" is the process of identifying and resolving problems with a system or device.

[0078] "Support" is a service that provides appropriate solutions and information to problems or questions faced by users.

[0079] A system for implementing the present invention includes a speech recognition model, a generative AI, a text-to-speech AI, and a means for recording a response log, as well as a means for analyzing a user's voice input in an autonomous vehicle and providing support for operating and troubleshooting the vehicle.

[0080] System Configuration

[0081] 1. Speech recognition model:

[0082] For the speech recognition model, we use a speech recognition API, which is an algorithm or software that converts a user's voice input into text data.

[0083] 2. Generative AI:

[0084] As a generative AI, we use a generative AI model, which is an artificial intelligence technology that analyzes textual input and generates an appropriate response.

[0085] 3. Text-to-speech AI:

[0086] For speech-to-speech AI, we use a text-to-speech engine, which is an algorithm or software that converts generated text responses into speech.

[0087] 4. Response log:

[0088] A response log is a database or file that stores a record of user interactions so that they can be viewed at a later time.

[0089] 5. Support measures in autonomous vehicles:

[0090] This includes a means for analyzing a user's voice input in an autonomous vehicle and providing assistance regarding vehicle operation and troubleshooting.

[0091] System Operation

[0092] When a user speaks to the server while in an autonomous vehicle, the server converts the speech into text data using a speech recognition model. Next, a generative AI analyzes the text data and generates an appropriate response. The generated response is converted into speech by a text-to-speech AI and provided to the user. Response logs are recorded, and follow-up is provided by personnel as necessary.

[0093] Examples

[0094] For example, consider a user asking how to operate the air conditioning in a self-driving vehicle.

[0095] User speaks: "How do I turn down the air conditioner?"

[0096] Example prompt: "How can I turn down the air conditioner?"

[0097] When this prompt sentence is input to a generative AI, the AI ​​generates the following response:

[0098] Generated response: "To turn down the air conditioning temperature, press the temperature control button located on the center dashboard. To turn down the temperature, press the minus button."

[0099] The response is then converted into audio by a speech-to-text AI and provided to the user, allowing them to receive support quickly and efficiently.

[0100] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0101] Step 1:

[0102] A user performs voice input in an autonomous vehicle. The user's voice input is transmitted to a server through a microphone in the vehicle. The input data is in the form of voice data.

[0103] Step 2:

[0104] The server converts the voice input into text data using a voice recognition model. Specifically, it uses a voice recognition API to analyze the voice data and generate corresponding text data. The input is voice data, and the output is text data.

[0105] Step 3:

[0106] The server uses generative AI to generate a response based on the textual input. It uses a generative AI model to analyze the text data and generate an appropriate response. The input is the text data and the output is the response text.

[0107] Step 4:

[0108] The server converts the generated text response using the speech AI into voice data. It uses a speech synthesis engine to convert the text data into voice data. The input is the response text, and the output is the voice data.

[0109] Step 5:

[0110] The server provides the generated voice data to the user, specifically by playing a voice response through the vehicle's speakers. The input is the voice data and the output is the speech heard by the user.

[0111] Step 6:

[0112] The server records the response log. It saves the user's voice input, the converted text data, the generated response text, and the voice data in a database. This makes it possible to refer to them later. The input is various data, and the output is the recorded log data.

[0113] Step 7:

[0114] If necessary, the server will send a human to follow up. If the problem is not resolved or additional support is required based on the response log, a specialist staff member will respond. The input is the response log, and the output is the follow-up support.

[0115] Example 2

[0116] Next, a second embodiment of the second embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0117] Traditional technical support systems often required users to wait for long periods of time to resolve their issues, and the problem was that they did not provide efficient support. In addition, there was a lack of a system that could consistently convert voice input into text, generate an appropriate response, and return it by voice. Furthermore, there was insufficient follow-up when the user's problem was not resolved.

[0118] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0119] In this invention, the server includes a means for converting voice input to text using a speech recognition model, a means for generating a response based on the converted text input using generative artificial intelligence, a means for converting the generated text response to voice using text-to-speech artificial intelligence, a means for recording a response log, and a means for performing follow-up by personnel, thereby enabling users to receive fast and efficient technical support 24 hours a day, 365 days a year without waiting times.

[0120] A "speech recognition model" is an algorithm or software for converting voice data into text data.

[0121] "Generative artificial intelligence" is an artificial intelligence technology that analyzes input data and generates appropriate responses or information.

[0122] "Read-to-speech artificial intelligence" refers to algorithms and software for converting text data into audio data.

[0123] A "response log" is a database or file that records user interactions and the responses generated.

[0124] "Follow-up" is the process for providing additional support if the user's issue is not resolved.

[0125] This invention is a technical support system using a smartphone, which converts voice input from a user into text, generates an appropriate response using generative artificial intelligence, and returns the response by voice using a text-to-speech artificial intelligence. A specific embodiment of this system will be described below.

[0126] First, the user speaks a question or problem into the smartphone. The device receives this voice input through the microphone and sends it to the server. The server uses a speech recognition model to convert the voice data into text data. For example, a speech recognition API is used as this speech recognition model.

[0127] The server then passes the converted user input to a generative AI, which analyzes the input text and understands the user's intent. For example, a model using natural language processing technology is used as the generative AI. Based on the analysis results, the generative AI generates an appropriate response.

[0128] The generated text response is passed from the server to a voice-to-speech AI, which converts the text into speech. For example, a voice synthesis API is used as the voice-to-speech AI. The converted voice data is sent to the terminal, which plays the voice data and provides the response to the user.

[0129] In addition, the server records all response logs. If the user's problem is not resolved or additional support is required, personnel will follow up based on the recorded logs. This allows users to receive fast and efficient technical support 24 hours a day, 365 days a year, without waiting.

[0130] As a concrete example, consider the case where a user says "Wi-Fi is not connecting" to a smartphone. The device sends this voice to the server, and the speech recognition API converts it into text "Wi-Fi is not connecting." The generative AI analyzes this text and generates a response saying "Please try restarting the router." The text-to-speech AI converts this response into speech, and the device tells the user aloud, "Please try restarting the router."

[0131] An example of a prompt sentence is, "If a user says to their smartphone, 'I can't connect to Wi-Fi,' how would you respond?"

[0132] This system allows users to receive technical support quickly and efficiently.

[0133] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0134] Step 1:

[0135] A user speaks a question or problem into a smartphone. The device receives this voice input through a microphone. The input is voice data, and the output is voice data as is. A specific example of how this works is when a user says, "I can't connect to Wi-Fi."

[0136] Step 2:

[0137] The device sends the received voice data to the server. The server uses a voice recognition model to convert the voice data into text data. The input is voice data, and the output is text data. In concrete terms, the voice recognition API converts the voice saying "Wi-Fi is not connecting" into the text "Wi-Fi is not connecting."

[0138] Step 3:

[0139] The server passes the converted user input to the generative AI. The generative AI analyzes the input text and understands the user's intent. The input is text data, and the output is the analysis result. In concrete terms, the generative AI analyzes the text "Wi-Fi is not connecting" and determines that the problem is with the network connection.

[0140] Step 4:

[0141] The server uses generative AI to generate an appropriate response based on the analysis results. The input is the analysis results, and the output is the response text. The generative AI generates the response "Try restarting your router."

[0142] Step 5:

[0143] The generated text response is passed from the server to the speech AI, which converts the text to speech. The input is the response text, and the output is the voice data. In concrete terms, the speech synthesis API converts the text "Please try restarting the router" into voice data.

[0144] Step 6:

[0145] The server sends voice data to the terminal. The terminal plays the voice data and provides a response to the user. The input is voice data, and the output is voice output. In concrete terms, the terminal tells the user by voice, "Please try restarting the router."

[0146] Step 7:

[0147] The server records all response logs. If the user's problem is not resolved or additional support is required, personnel will follow up based on the recorded logs. The input is the response log, and the output is follow-up suggestions. In concrete terms, if the user inquires again saying "still can't connect," the server will refer to the log and suggest the next step, such as "check your network settings."

[0148] (Application example 2)

[0149] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0150] There is a demand for prompt and appropriate support for technical problems and questions that arise inside autonomous vehicles. However, with conventional systems, when a user faces a technical problem inside the vehicle, there are limited ways to respond immediately, making it difficult to provide efficient support. This leads to issues such as reduced convenience for users and a loss of satisfaction.

[0151] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0152] In this invention, the server includes a means for converting voice input into text using a voice recognition model, a means for generating a response based on the converted text input using a generative AI, a means for converting the generated text response into voice using a text-to-speech AI, a means for recording a response log, and a means for receiving voice input in the autonomous vehicle and outputting a voice response using a speaker in the vehicle, thereby enabling a user in the autonomous vehicle to receive prompt and appropriate support for technical issues or questions.

[0153] A "speech recognition model" is an algorithm or software for converting voice input into text data.

[0154] "Generative AI" is an artificial intelligence technology that generates appropriate responses based on input text data.

[0155] "Reading AI" refers to algorithms or software for converting text data into audio data.

[0156] A "response log" is data that records user input and the system's response to it.

[0157] An "autonomous vehicle" is a vehicle that drives autonomously using artificial intelligence and sensor technology.

[0158] A "speaker" is a device for outputting audio data as sound.

[0159] As an embodiment of the present invention, a technical support system in an autonomous vehicle is shown below. The system includes a speech recognition model, a generative AI, a voice-to-speech AI, a response log recording means, and a voice input and output means in the autonomous vehicle.

[0160] System Configuration

[0161] 1. Speech Recognition Model

[0162] A voice recognition model is an algorithm or software that converts a user's voice input into text data. Specifically, it uses a voice recognition API. It receives voice input through a microphone in the vehicle and converts it into text.

[0163] 2. Generative AI

[0164] Generative AI is an artificial intelligence technique that generates appropriate responses based on textual input. Specifically, it uses generative AI models to analyze text data and generate appropriate responses to user questions.

[0165] 3. Text-to-speech AI

[0166] Text-to-speech AI is an algorithm or software that converts the generated text response into audio data, specifically using a text-to-speech API, which then converts the generated text response into audio and outputs it to the user through the vehicle's speakers.

[0167] 4. Response logging methods

[0168] The response log recorder is a database that records user input and the system's response to that input, so that later follow-up by personnel may be required.

[0169] 5. Voice input and output means in autonomous vehicles

[0170] A microphone is used as the voice input means in an autonomous vehicle, and a speaker is used as the voice output means, allowing users to receive voice support in the vehicle.

[0171] System Operation

[0172] The server receives voice input from the vehicle's microphone and converts it to text using a speech recognition model. It then uses generative AI to analyze the text and generate an appropriate response. The generated text response is converted to audio using speech AI and output to the user through the vehicle's speakers. Response logs are recorded and follow-ups are provided by personnel as needed.

[0173] Examples

[0174] For example, if a user asks, "How do I turn up the air conditioner temperature?" the system would do the following:

[0175] 1. A microphone in the vehicle captures the user's voice.

[0176] 2. The speech recognition model converts the speech to text.

[0177] 3. Generative AI analyzes the text and generates an appropriate response.

[0178] 4. Text-to-speech AI converts the generated text response into audio.

[0179] 5. Speakers in the vehicle output a voice response to the user.

[0180] Examples of prompt statements

[0181] User: How do I turn up the heat on my air conditioner?

[0182] In this way, quick and appropriate voice responses can be provided to user questions within an autonomous vehicle.

[0183] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0184] Step 1:

[0185] The user inputs voice into a microphone in the vehicle.

[0186] Input: User's voice

[0187] Output: Audio data

[0188] Specific action: The user says, "Tell me how to turn up the air conditioner temperature."

[0189] Step 2:

[0190] The server converts the voice data into text data using a voice recognition model.

[0191] Input: Audio data

[0192] Output: Text data

[0193] Specific operation: The speech recognition API analyzes the voice data and generates the text "Tell me how to turn up the air conditioner temperature."

[0194] Step 3:

[0195] The server uses generative AI to analyze the text data and generate an appropriate response.

[0196] Input: Text data

[0197] Output: Response text

[0198] Specific behavior: The generative AI model analyzes the text "Tell me how to turn up the temperature on the air conditioner" and generates the response text "To turn up the temperature on the air conditioner, select the 'Air Conditioning' menu on the central touchscreen and increase the temperature setting."

[0199] Step 4:

[0200] The server uses a text-to-speech AI to convert the response text into audio data.

[0201] Input: Response text

[0202] Output: Audio data

[0203] Specific operation: The speech synthesis API analyzes the response text and generates speech data saying, "To increase the temperature of the air conditioner, select the 'Air Conditioning' menu on the central touch screen and increase the temperature setting."

[0204] Step 5:

[0205] The server outputs the audio data to the user using a speaker in the vehicle.

[0206] Input: Audio data

[0207] Output: Voice response

[0208] Specific operation: The speaker in the vehicle plays audio data, informing the user, "To increase the air conditioning temperature, select the 'Air Conditioning' menu on the central touchscreen and increase the temperature setting."

[0209] Step 6:

[0210] The server logs the response.

[0211] Input: User voice input, generated response text

[0212] Output: Response log

[0213] What happens: The server stores the user's voice input and the generated response text in a database in case a later human follow-up is required.

[0214] Furthermore, an emotion engine that estimates the emotion of the user may be combined. That is, the identification processing unit 290 may estimate the emotion of the user using the emotion identification model 59, and perform identification processing using the emotion of the user.

[0215] "Example 1"

[0216] As one embodiment of the present invention, a system incorporating an emotion engine is provided. This system converts voice input into text using a voice recognition model, generates a response based on the text input using a generative AI, converts the generated text response into voice using a speech-to-speech AI, and records a response log. Furthermore, this system includes an emotion engine that recognizes the user's emotion from the voice input. Specifically, when a user expresses an emotion such as anger or joy by voice, the emotion engine recognizes the emotion, and the generative AI generates a response based on the emotion. For example, when a user shows an emotion of anger, the generative AI is adjusted to generate a more polite response. In addition, the emotion engine can adjust the priority of responses based on the user's emotion. For example, when a user shows a strong emotion of anger, the system is adjusted to prioritize processing of inquiries from that user. This makes it possible to provide more appropriate support according to the user's emotion.

[0217] "Example 2"

[0218] As one embodiment of the present invention, a system incorporating an emotion engine is provided. This system converts voice input into text using a voice recognition model, generates a response based on the text input using a generative AI, converts the generated text response into voice using a speech-to-speech AI, and records a response log. Furthermore, this system includes an emotion engine that recognizes the user's emotion from the voice input. Specifically, when a user expresses an emotion such as anger or joy by voice, the emotion engine recognizes the emotion, and the generative AI generates a response based on the emotion. For example, when a user shows an emotion of anger, the generative AI is adjusted to generate a more polite response. In addition, the emotion engine can adjust the priority of responses based on the user's emotion. For example, when a user shows a strong emotion of anger, the system is adjusted to prioritize processing of inquiries from that user. This makes it possible to provide more appropriate support according to the user's emotion.

[0219] The process flow of each embodiment will be described below.

[0220] "Example 1"

[0221] Step 1: Receive voice input from the user. This voice input is received through a smartphone microphone or similar.

[0222] Step 2: Convert the received voice input into text using WhisperAPI. WhisperAPI is a speech recognition model that has the ability to convert speech to text.

[0223] Step 3: Recognize user emotions from the textual input using the emotion engine. The emotion engine has the ability to recognize user emotions (e.g., anger, joy) from the text.

[0224] Step 4: Generative AI generates a response based on the recognized emotion. Generative AI has the ability to generate an appropriate response based on the textual input and the recognized emotion.

[0225] Step 5: The generated text response is converted into speech using ReadSpeaker, a speech-to-speech AI capable of converting text to speech.

[0226] Step 6: Provide a spoken response to the user and simultaneously record the response log, which can be used for later follow-up and system improvement.

[0227] Example 1

[0228] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal".

[0229] Conventional technical support systems provide uniform responses without considering the user's feelings, which can lead to low user satisfaction. In addition, there are often waiting times, and responses are delayed even in situations where a quick response is required. This increases user stress and reduces support efficiency.

[0230] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0231] In this invention, the server includes a means for converting a voice input into text using a voice recognition model, a means for generating a response based on the input converted into text using generative artificial intelligence, a means for converting the generated text response into voice using a text-to-speech artificial intelligence, a means for recording a response log, a means for recognizing a user's emotion from the voice input using an emotion engine and adjusting the response based on the emotion, and a means for adjusting the priority of responses based on the user's emotion, thereby making it possible to quickly provide an appropriate response according to the user's emotion.

[0232] A "speech recognition model" is an algorithm or software for analyzing voice data and converting it into text data.

[0233] "Generative artificial intelligence" is an artificial intelligence technology that analyzes input data and generates appropriate responses or results.

[0234] "Read-to-speech artificial intelligence" is an artificial intelligence technology that converts text data into audio data and outputs it as audio.

[0235] A "response log" is data that records the content of inquiries from users and the responses generated by the system.

[0236] An "emotion engine" is a technology that analyzes user emotions from voice input and text data and recognizes those emotions.

[0237] "Means for adjusting priority" is a function for changing the order and importance of responses based on the user's emotions and the content of the inquiry.

[0238] This invention is a system for providing technical support for smartphones. Voice input from a user is converted to text by a voice recognition model. A voice recognition API is used as this voice recognition model. The text input is analyzed by generative artificial intelligence, and an appropriate response is generated. The generated text response is converted to voice by a speech-to-speech artificial intelligence, and output to the user. A voice synthesis API is used as this speech-to-speech artificial intelligence. In addition, the system records a response log, and when follow-up by a human is required, a proposal for a solution is made based on this. This enables voice support 24 hours a day, 365 days a year with no waiting time, reducing the burden on users and providing efficient support.

[0239] Furthermore, this system combines an emotion engine. The emotion engine recognizes the user's emotion from the voice input, and the generative AI generates a response based on that emotion. For example, if the user expresses anger, the generative AI is adjusted to generate a more polite response. The emotion engine can also adjust the priority of responses based on the user's emotion. For example, if a user expresses strong anger, the system is adjusted to prioritize inquiries from that user. This makes it possible to provide more appropriate support according to the user's emotion.

[0240] As a concrete example, consider the case where a user makes a voice inquiry to their smartphone saying, "Wi-Fi won't connect." The device converts this voice into text using a speech recognition API, generating the text "Wi-Fi won't connect." The server passes this text to a generative artificial intelligence system, which generates an appropriate response. For example, a response might be generated saying, "Please try restarting your Wi-Fi router." The speech synthesis API converts this response into speech, and the device outputs the response to the user in voice.

[0241] An example of a prompt sentence is, "The user spoke to his smartphone saying, 'I can't connect to Wi-Fi.' Please convert this speech into text, generate an appropriate response, and reply by voice."

[0242] This system allows users to receive voice support 24 hours a day, 365 days a year, without waiting, providing efficient support.

[0243] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0244] Step 1:

[0245] A user makes a voice inquiry into a smartphone. For example, they say, "I can't connect to Wi-Fi." The input is the user's voice data, and the output is the voice data sent to the device.

[0246] Step 2:

[0247] The device receives the user's voice input and converts the voice to text using a voice recognition API. Specifically, the device sends voice data to the voice recognition API, which analyzes the voice data and generates the text "Wi-Fi is not connected." The input is voice data, and the output is text data.

[0248] Step 3:

[0249] The server receives the text sent from the terminal. The server passes this text to the generative AI for analysis. The generative AI understands the content of the user's inquiry and creates a prompt sentence to generate an appropriate response. The input is text data, and the output is a prompt sentence.

[0250] Step 4:

[0251] The generative AI generates an appropriate response based on the prompt sentence. For example, a response such as "Please try restarting your Wi-Fi router" is generated. The server stores this response in text format. The input is the prompt sentence, and the output is the response text.

[0252] Step 5:

[0253] The server sends the generated text response to the device. The device uses a speech synthesis API to convert the text response into speech. Specifically, the device passes the text data to the speech synthesis API, which generates speech data. The input is text data, and the output is speech data.

[0254] Step 6:

[0255] The device outputs the generated voice data to the user. The user can hear the voice response, "Please try restarting your Wi-Fi router." The input is the voice data, and the output is the voice response to the user.

[0256] Step 7:

[0257] The server records all response logs, including the user's voice input, the transcribed input, and the generated response. If necessary, a human follow-up is performed and a resolution suggestion is provided. The input is the data, and the output is the recorded response log.

[0258] Step 8:

[0259] The server uses an emotion engine to recognize user emotions from voice input. For example, if a user expresses an emotion of anger, the emotion engine recognizes the emotion and adjusts the generative artificial intelligence to generate a more polite response. The emotion engine can also adjust the priority of responses. User queries that express a strong emotion of anger are given priority. The input is voice data and text data, and the output is the adjusted response and priority.

[0260] (Application example 1)

[0261] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0262] In autonomous vehicles, there is a need to provide prompt and appropriate support when users ask questions about how to operate the vehicle or troubleshooting. However, conventional systems are unable to respond in a way that takes into account the user's emotions, which can lead to a decrease in user satisfaction. In addition, the accuracy and speed of generating responses based on voice input are also issues.

[0263] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0264] In this invention, the server includes a means for converting voice input into text using a voice recognition model, a means for generating a response based on the text input using a generative AI, a means for converting the generated text response into voice using a text-to-speech AI, a means for recording a response log, a means for recognizing a user's emotion from the voice input using an emotion engine, and a means for adjusting the tone and priority of the response based on the user's emotion. This enables prompt and appropriate support according to the user's emotion.

[0265] A "speech recognition model" is an algorithm or software for converting voice input into text.

[0266] "Generative AI" is an artificial intelligence technology that generates appropriate responses based on text input.

[0267] "Reading AI" refers to algorithms or software that converts text into speech and outputs it.

[0268] A "response log" is a database that records user inquiries and the system's responses to them.

[0269] An "emotion engine" is an algorithm or software that analyzes and recognizes a user's emotions from voice input.

[0270] "Tone" refers to the vocal cadence or nuance of the response produced.

[0271] "Priority" is a criterion for determining the order in which multiple inquiries or tasks are processed.

[0272] The system for implementing this invention is composed of a combination of a speech recognition model, a generative AI, a reading AI, and an emotion engine. Below, we will explain the details of each component and how they work together.

[0273] Speech Recognition Model

[0274] The server converts the user's voice input into text using a speech recognition model, which is provided by a speech recognition API, that analyzes the speech data and generates corresponding text data.

[0275] Generative AI

[0276] The server uses a generative AI to generate responses based on the textual input. The generative AI uses advanced natural language processing models such as GPT-3. The generative AI leverages pre-trained data to generate appropriate responses to user questions.

[0277] Reading AI

[0278] The server uses a voice AI to convert the generated text response into voice. A voice synthesis engine is used as the voice AI. The voice synthesis engine converts the text data into natural voice and outputs it to the user.

[0279] Emotion Engine

[0280] The server recognizes the user's emotions from the voice input using an emotion engine. The emotion engine analyzes the tone, pitch, and rate of the voice data to identify the user's emotional state. If the user is expressing an emotion such as anger or joy, a response is generated according to that emotion.

[0281] Adjusting the tone and priority of your response

[0282] The server adjusts the tone and priority of the responses generated by the generative AI based on the user's emotions recognized by the emotion engine. For example, if the user expresses anger, the generative AI is adjusted to generate more polite responses. In addition, inquiries from users who show strong emotions are given priority.

[0283] Examples

[0284] Consider a case where a user asks how to operate the air conditioner in an autonomous vehicle. When the user asks verbally, "How do I adjust the temperature of the air conditioner?", a speech recognition model converts this speech into text. A generative AI then generates a response based on this text: "To adjust the temperature of the air conditioner, tap the temperature setting on the center display." The emotion engine analyzes the user's emotions, and if there is no particular change in emotion, the reading AI vocalizes this response in a normal tone and provides it to the user.

[0285] Examples of prompt statements

[0286] User sentiment: Neutral

[0287] User Question: How do I adjust the temperature on my air conditioner?

[0288] Generate an appropriate response.

[0289] In this way, it is possible to provide prompt and appropriate support according to the user's emotions.

[0290] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0291] Step 1:

[0292] The user performs voice input. For example, the user may ask a question in an autonomous vehicle, such as "How do I adjust the temperature of the air conditioner?" This voice data is sent to the server.

[0293] Step 2:

[0294] The server converts the voice input to text using a voice recognition model. The voice recognition API analyzes the voice data and generates the corresponding text data: "How do I adjust the temperature of the air conditioner?" The input is voice data, and the output is text data.

[0295] Step 3:

[0296] The server uses an emotion engine to recognize the user's emotion from the voice input. The emotion engine analyzes the tone, pitch, and speed of the voice data and identifies the user's emotional state as "neutral." The input is the voice data, and the output is the emotional state.

[0297] Step 4:

[0298] The server uses a generative AI to generate a response based on the textual input. The generative AI (e.g., GPT-3) uses the prompt sentence to generate an appropriate response: "To adjust the temperature of the air conditioner, tap the temperature setting on the center display." The input is the text data and the emotional state, and the output is the response text.

[0299] Step 5:

[0300] The server adjusts the tone and priority of the response generated by the generative AI based on the user's emotion recognized by the emotion engine. For example, because the user's emotion is "normal," a response is generated in a normal tone. The input is the response text and the emotional state, and the output is the adjusted response text.

[0301] Step 6:

[0302] The server uses a voice-to-speech AI to convert the generated text response into voice. The voice synthesis engine converts the text data into natural speech and outputs it to the user. The input is the adjusted response text, and the output is the voice data.

[0303] Step 7:

[0304] The server records the response log. User queries and the system's responses to them are stored in a database. Inputs are voice data, text data, emotional state, and response text, and output is the recorded log data.

[0305] In this way, prompt and appropriate support can be provided according to the user's feelings.

[0306] Example 2

[0307] Next, a second embodiment of the second embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0308] Conventional smartphone technical support systems convert users' voice input into text, generate responses using generative AI, and convert them into voice using text-to-speech AI, but they have a problem in that they cannot adjust responses to take the user's emotions into account. This can lead to lower user satisfaction and make it difficult to provide efficient support.

[0309] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0310] In this invention, the server includes a means for converting a voice input into text using a voice recognition model, a means for generating a response based on the converted text input using generative artificial intelligence, a means for converting the generated text response into voice using a text-to-speech artificial intelligence, a means for recording a response log, a means for recognizing a user's emotion from the voice input using an emotion engine, and a means for adjusting a response based on the user's emotion, thereby enabling more appropriate support according to the user's emotion.

[0311] A "speech recognition model" is an algorithm or software for converting voice input into text data.

[0312] "Generative artificial intelligence" is an artificial intelligence technology that analyzes input text data and generates appropriate responses.

[0313] "Read-to-speech artificial intelligence" is a technology that converts text data into voice data and provides a voice response to the user.

[0314] A "response log" is data for recording inquiries from users and responses to those inquiries.

[0315] An "emotion engine" is a technology for recognizing a user's emotions from voice input and adjusting responses based on those emotions.

[0316] "Text conversion" is the process of converting audio data into text data.

[0317] "Analysis" is the process of analyzing input data and understanding its meaning and intent.

[0318] "Adjustment" is the process of changing responses or behavior in response to specific conditions or situations.

[0319] This invention is a system for providing technical support for smartphones, which converts voice input from the user into text, analyzes the text to generate an appropriate response, and returns it to the user as voice. It also has a function to recognize the user's emotions and provide a response according to those emotions.

[0320] System Configuration

[0321] 1. Voice to text conversion

[0322] Hardware / Software: The server converts voice input to text using a speech recognition model, which is provided by a speech recognition API.

[0323] Example: When a user says, "My smartphone battery is running out quickly, what should I do?" the server converts this speech into text.

[0324] 2. Text Analysis and Response Generation

[0325] Hardware / Software: The server uses generative artificial intelligence (e.g., a generative AI model) to parse the textual input and generate an appropriate response.

[0326] Example: A server parses the text "My phone's battery is draining quickly, what should I do?" and generates a response that reads "To extend battery life, you can close apps running in the background and adjust the screen brightness."

[0327] 3. Audio Responses

[0328] Hardware / Software: The server converts the generated text response into speech using a speech-to-text AI, which uses a text-to-speech API.

[0329] Example: The server converts the text "To extend battery life, you can close background apps and adjust your screen brightness." into speech, which the device then outputs to the user.

[0330] 4. Response logging

[0331] Hardware / Software: The server uses a database to log all responses.

[0332] Example: The server stores the user's inquiry and the generated response in a database. For example, the following log is recorded: "Inquiry: My smartphone's battery runs out quickly. What should I do?" "Response: To extend the battery life, it is effective to close apps running in the background and adjust the screen brightness."

[0333] 5. Emotion recognition and response adjustment

[0334] Hardware / Software: The server uses an emotion engine to recognize emotions from the user's voice.

[0335] Example: If a user expresses anger, the server uses the emotion engine to recognize the emotion. For example, if the user says in an angry tone, "Why is my battery running out so fast?", the generative AI generates a more polite response such as, "I'm sorry. We will give you a solution to your battery problem right away." The system is also adjusted so that inquiries from users who express strong emotions are given priority.

[0336] Examples of prompt statements

[0337] "My smartphone battery runs out quickly, what should I do?"

[0338] "The app keeps crashing, what should I do?"

[0339] "I can't connect to Wi-Fi, what should I do?"

[0340] This system allows users to receive voice support 24 hours a day, 365 days a year, without waiting, providing efficient support.

[0341] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0342] Step 1:

[0343] Receiving audio input

[0344] How it works: A user speaks a query into their smartphone.

[0345] Input: The user's voice input.

[0346] Output: The audio data.

[0347] Specific behavior: A user says, "My smartphone battery is running out quickly. What should I do?" The device receives this voice through the microphone.

[0348] Step 2:

[0349] Voice to text conversion

[0350] How it works: The server converts voice input into text using a speech recognition model.

[0351] Input: Audio data.

[0352] Output: Text data.

[0353] What happens: The server calls a speech recognition API to convert the voice data into text such as, "My smartphone's battery is running out quickly. What should I do?"

[0354] Step 3:

[0355] Text analysis and response generation

[0356] How it works: The server uses generative artificial intelligence to parse the textual input and generate an appropriate response.

[0357] Input: Text data.

[0358] Output: The response text.

[0359] How it works: The server inputs the text "My smartphone's battery is draining quickly, what should I do?" into the generative AI model, which generates a response saying "To extend battery life, it is effective to close apps running in the background and adjust the screen brightness."

[0360] Step 4:

[0361] Spoken responses

[0362] How it works: The server uses text-to-speech artificial intelligence to convert the generated text response into speech.

[0363] Input: The response text.

[0364] Output: The audio data.

[0365] Specific operation: The server calls the speech synthesis API to convert the text "To extend battery life, it is effective to close apps running in the background and adjust the screen brightness." into speech, and the device outputs the speech to the user.

[0366] Step 5:

[0367] Response logging

[0368] Behavior: The server logs all responses.

[0369] Input: The user's query and the generated response.

[0370] Output: The response log data.

[0371] Specific operation: The server records the following log in the database: "Query: My smartphone's battery is draining quickly. What should I do?" "Response: To extend battery life, it is effective to close apps running in the background and adjust the screen brightness."

[0372] Step 6:

[0373] Emotion recognition and response adjustment

[0374] How it works: The server uses an emotion engine to recognize emotions from the user's voice and tailor its response.

[0375] Input: The user's voice data.

[0376] Output: The adjusted response text.

[0377] Specific behavior: If a user expresses anger, the server uses the emotion engine to recognize the emotion. For example, if a user says in an angry tone, "Why is my battery running out so fast?", the generative AI generates a more polite response such as, "I'm sorry. We will give you a solution to your battery problem right away." The system is also adjusted so that inquiries from users who express strong emotions are given priority.

[0378] (Application example 2)

[0379] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0380] In autonomous vehicles, when a user asks a question about vehicle operation or navigation by voice while driving, it is necessary to provide a quick and appropriate response. However, conventional systems are unable to respond taking into account the user's emotions, which may reduce user satisfaction. In addition, it is difficult to respond quickly to questions with high urgency because it is not possible to adjust the priority of responses based on emotions. To solve these issues, a system is needed that recognizes the user's emotions, generates responses based on them, and provides responses with an appropriate tone and priority.

[0381] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0382] In this invention, the server includes a means for converting a voice input into text using a voice recognition model, a means for generating a response based on the input converted into text using a generative AI, a means for converting the generated text response into voice using a text-to-speech AI, a means for recording a response log, a means for recognizing a user's emotion from the voice input using an emotion engine, a means for adjusting the tone and priority of the response based on the user's emotion, and a means for being installed in the infotainment system of the autonomous vehicle. This enables a quick and appropriate response that takes the user's emotion into consideration, thereby improving user satisfaction.

[0383] A "speech recognition model" is an algorithm or software for converting voice input into text.

[0384] "Generative AI" is an artificial intelligence technology that generates appropriate responses based on input text data.

[0385] "Read-to-speech AI" is an artificial intelligence technology that converts text data into audio and outputs it.

[0386] A "response log" is a database that records user inquiries and the system's responses to them.

[0387] An "emotion engine" is a technology for recognizing a user's emotions from voice input and adjusting responses based on those emotions.

[0388] An "infotainment system" is a system that provides information and entertainment within a vehicle.

[0389] An "autonomous vehicle" is a vehicle that drives itself using artificial intelligence and sensor technology.

[0390] As an embodiment of the present invention, a program installed in the infotainment system of an autonomous vehicle will be described. The server uses a speech recognition model, generative AI, text-to-speech AI, and emotion engine to generate a response based on the user's voice input, and outputs it by voice.

[0391] First, the user speaks to a vehicle operation or navigation question. The voice input is converted to text using a speech recognition model, which uses a speech recognition API. The text input is then analyzed by an emotion engine to recognize the user's emotion. The emotion engine recognizes the user's emotion from the voice input and adjusts the tone and priority of the response based on that emotion.

[0392] A generative AI model is then used to generate an appropriate response based on the textual input and the recognized emotion. The response is then converted to speech by a text-to-speech AI, which uses a text-to-speech API. The generated voice response is then output to the user through the infotainment system.

[0393] In addition, response logs are recorded on the server and can be viewed later, making it possible to track user inquiry history and response content, and to have personnel follow up if necessary.

[0394] As a concrete example, consider a user asking, "Where is the next gas station?" In this case, the emotion engine recognizes the user's impatience, and the generative AI generates a quick and polite response, such as, "The next gas station is 3 kilometers away on the right. Don't worry."

[0395] Example prompt:

[0396] User Question: Where is the next gas station?

[0397] User Sentiment: Impatience

[0398] Appropriate response:

[0399] In this way, a quick and appropriate response that takes the user's feelings into consideration is possible, thereby improving user satisfaction.

[0400] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0401] Step 1:

[0402] The user asks questions about vehicle operation and navigation by voice.

[0403] Input: User's voice input

[0404] Output: Audio data

[0405] Specific Action: A user asks a question to the infotainment system of an autonomous vehicle.

[0406] Step 2:

[0407] The server converts the voice input into text using a speech recognition model.

[0408] Input: Audio data

[0409] Output: Text data

[0410] Specific operation: The speech recognition API analyzes the speech data and generates corresponding text data.

[0411] Step 3:

[0412] The server uses an emotion engine to recognize the user's emotion from the text data.

[0413] Input: Text data

[0414] Output: Emotion data

[0415] Specific behavior: The emotion engine analyzes text data and recognizes the user's emotions (e.g., impatience, anger, joy).

[0416] Step 4:

[0417] The server uses generative AI to generate a response based on text data and emotion data.

[0418] Input: Text data, emotion data

[0419] Output: Response text

[0420] Specific operation: The generative AI model generates an appropriate response text based on text data and emotion data.

[0421] Step 5:

[0422] The server uses text-to-speech AI to convert the response text into speech.

[0423] Input: Response text

[0424] Output: Audio data

[0425] Specific operation: The speech synthesis API analyzes the response text and generates corresponding speech data.

[0426] Step 6:

[0427] The server outputs the generated voice data to the user through the infotainment system.

[0428] Input: Audio data

[0429] Output: A voice response to the user

[0430] Specific Action: The infotainment system plays the audio data and provides a response to the user.

[0431] Step 7:

[0432] The server logs the response.

[0433] Input: Text data, emotion data, response text

[0434] Output: Response log

[0435] Specific behavior: The server records the user's question, the recognized emotion, and the generated response in a database.

[0436] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0437] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0438] Another example of generative AI is Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) are mentioned.

[0439] In the above embodiment, an example has been given in which the specific process is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0440] [Second embodiment]

[0441] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0442] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0443] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0444] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0445] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[0446] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[0447] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[0448] Fig. 4 shows an example of main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0449] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0450] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0451] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0452] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0453] "Example 1"

[0454] The present invention is a system for providing technical support for smartphones. Voice input from a user is converted to text by a voice recognition model. For example, WhisperAPI is used as this voice recognition model. The converted text input is analyzed by a generative AI, and an appropriate response is generated. The generated text response is converted to voice by a text-to-speech AI and output to the user. For example, ReadSpeaker is used as this text-to-speech AI. In addition, the system records a response log, and when follow-up by personnel is required, a proposal for a solution is made based on this. This enables voice response 24 hours a day, 365 days a year without waiting time, reducing the burden on users and providing efficient support.

[0455] "Example 2"

[0456] The present invention is a system for providing technical support for smartphones. Voice input from a user is converted to text by a voice recognition model. For example, WhisperAPI is used as this voice recognition model. The converted text input is analyzed by a generative AI, and an appropriate response is generated. The generated text response is converted to voice by a text-to-speech AI and output to the user. For example, ReadSpeaker is used as this text-to-speech AI. In addition, the system records a response log, and when follow-up by personnel is required, a proposal for a solution is made based on this. This enables voice response 24 hours a day, 365 days a year without waiting time, reducing the burden on users and providing efficient support.

[0457] The process flow of each embodiment will be described below.

[0458] "Example 1"

[0459] Step 1: Receive voice input from the user. This voice input is done through the smartphone microphone.

[0460] Step 2: The received voice input is converted into text using a voice recognition model, such as WhisperAPI.

[0461] Step 3: The textual input is analyzed by a generative AI to generate an appropriate response. This generative AI has the ability to understand the user's problem and suggest appropriate solutions.

[0462] Step 4: The generated text response is converted to audio by a speech AI and output to the user. For example, ReadSpeaker is used as the speech AI. Step 5: The system records the response log. This allows the system to make suggestions for a solution based on the log, even if follow-up by a human is required.

[0463] Example 1

[0464] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0465] Conventional smartphone technical support systems often required users to wait a long time to resolve their issues, making it difficult to provide efficient support. In addition, the process of converting voice input into text, generating an appropriate response, and returning it by voice was sometimes inconsistent, which reduced the user experience.

[0466] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0467] In this invention, the server includes a means for converting voice input into text using a voice recognition model, a means for generating a response based on the converted text input using generative artificial intelligence, a means for converting the generated text response into voice using speech-to-speech artificial intelligence, a means for recording a response log, and a means for following up with personnel, thereby enabling efficient support 24 hours a day, 365 days a year without waiting times.

[0468] A "speech recognition model" is an algorithm or system for analyzing speech data and converting it into corresponding text data.

[0469] "Generative artificial intelligence" is an artificial intelligence technology that analyzes input data and generates appropriate responses or information.

[0470] "Read-to-speech artificial intelligence" is an artificial intelligence technology that converts text data into audio data and outputs it as audio.

[0471] A "response log" is a database or file that maintains a record of user interactions and the responses generated.

[0472] "Follow-up" is the process by which personnel respond when a user's problem is not resolved or additional support is required.

[0473] A "voice recognition API" is an application programming interface for converting voice data into text data.

[0474] A "speech synthesis API" is an application programming interface for converting text data into speech data.

[0475] This invention is a system for providing technical support for smartphones. Voice input from a user is converted to text by a voice recognition model. A voice recognition API is used as this voice recognition model. The text input is analyzed by a generative artificial intelligence, and an appropriate response is generated. The generated text response is converted to voice by a voice-to-speech artificial intelligence, and output to the user. A voice synthesis API is used as this voice-to-speech artificial intelligence. In addition, the system records a response log, and when follow-up by personnel is required, a proposal for a solution is made based on this. This enables voice response 24 hours a day, 365 days a year with no waiting time, reducing the burden on users and providing efficient support.

[0476] Specifically, a user speaks a question or problem into a smartphone. For example, they might say, "My smartphone's battery is running low." The device records this voice and sends the data to a server. The server uses a speech recognition API to convert the voice data into text. Specifically, it analyzes the voice signal and generates a corresponding string of characters.

[0477] The server then passes the converted user input to a generative AI, which uses natural language processing techniques, for example, to analyze the user's question or problem and generate an appropriate response, such as "To reduce battery drain, we recommend closing apps running in the background."

[0478] The generated text response is passed from the server to the speech-to-speech AI, which converts the text into speech and sends it to the device. The device then plays the speech to the user. Specifically, it generates an audio file and plays it through the smartphone speaker.

[0479] In addition, the server records a log of all responses. If the user's problem is not resolved or additional support is required, the server can use this log to send a follow-up call to the appropriate technician. For example, if a user says, "I can't connect to Wi-Fi," the server will check the log and escalate the call to the appropriate technician.

[0480] An example of a prompt sentence is, "What should you do if your smartphone battery runs out quickly?" By inputting this prompt sentence into a generative artificial intelligence system, an appropriate response is generated.

[0481] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0482] Step 1:

[0483] A user speaks a question or problem into a smartphone. For example, they might say, "My smartphone's battery is running low." The device records this voice and sends the data to a server. The input is the voice data, and the output is the voice data sent to the server.

[0484] Step 2:

[0485] The server uses a speech recognition API to convert this voice data into text: it analyzes the audio signal and generates a corresponding string of characters. The input is the voice data, and the output is the text data.

[0486] Step 3:

[0487] The server passes the textualized user input to the generative AI. The generative AI uses natural language processing techniques to analyze the user's question or problem and generate an appropriate response. For example, it generates a response such as "To reduce battery drain, we recommend closing apps running in the background." The input is text data, and the output is the response text.

[0488] Step 4:

[0489] The generated text response is passed from the server to the speech AI, which converts the text into speech and sends it to the terminal. Specifically, it generates an audio file and plays it through the smartphone speaker. The input is the response text, and the output is the audio data.

[0490] Step 5:

[0491] The server records all response logs. If the user's problem is not resolved or additional support is required, the server uses these logs to send personnel to follow up. For example, if a user says "Wi-Fi is not working," the server checks the logs and escalates the issue to the appropriate technician. The input is the response log, and the output is the information for follow-up.

[0492] (Application example 1)

[0493] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0494] As autonomous vehicles become more widespread, users need to receive support for vehicle operation and troubleshooting quickly and efficiently. However, conventional systems require users to spend a lot of time and effort to solve problems, making it difficult to respond quickly, especially in emergencies.

[0495] The identification process by the identification processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means. In this invention, the server includes a means for converting a voice input into text using a voice recognition model, a means for generating a response based on the input converted into text using a generative AI, a means for converting the generated text response into voice using a text-to-speech AI, a means for recording a response log, and a means for analyzing a voice input of a user in an autonomous vehicle and providing support regarding vehicle operation and troubleshooting. This allows the user to receive support quickly and efficiently using voice in an autonomous vehicle.

[0496] A "speech recognition model" is an algorithm or software for converting voice input into text data.

[0497] "Generative AI" is an artificial intelligence technology that analyzes input text data and generates appropriate responses.

[0498] "Reading AI" refers to algorithms and software for converting text data into speech.

[0499] A "response log" is a database or file that stores records of user interactions.

[0500] An "autonomous vehicle" is a vehicle that uses artificial intelligence and sensor technology to drive autonomously.

[0501] "User's voice input" refers to voice data generated by a user through a voice input device such as a microphone.

[0502] "Vehicle operation" refers to the act of controlling various vehicle functions (e.g., air conditioning, navigation, audio system, etc.).

[0503] "Troubleshooting" is the process of identifying and resolving problems with a system or device.

[0504] "Support" is a service that provides appropriate solutions and information to problems or questions faced by users.

[0505] A system for implementing the invention includes a speech recognition model, a generative AI, a text-to-speech AI, and a means for recording a response log, as well as a means for analyzing a user's voice input in an autonomous vehicle and providing support for operating and troubleshooting the vehicle.

[0506] System Configuration

[0507] 1. Speech recognition model:

[0508] For the speech recognition model, we use a speech recognition API, which is an algorithm or software that converts a user's voice input into text data.

[0509] 2. Generative AI:

[0510] As a generative AI, we use a generative AI model, which is an artificial intelligence technology that analyzes textual input and generates an appropriate response.

[0511] 3. Text-to-speech AI:

[0512] For speech-to-speech AI, we use a text-to-speech engine, which is an algorithm or software that converts generated text responses into speech.

[0513] 4. Response log:

[0514] A response log is a database or file that stores a record of user interactions so that they can be viewed at a later time.

[0515] 5. Support measures in autonomous vehicles:

[0516] This includes a means for analyzing a user's voice input in an autonomous vehicle and providing assistance regarding vehicle operation and troubleshooting.

[0517] System Operation

[0518] When a user speaks to the server while in an autonomous vehicle, the server converts the speech into text data using a speech recognition model. Next, a generative AI analyzes the text data and generates an appropriate response. The generated response is converted into speech by a text-to-speech AI and provided to the user. Response logs are recorded, and follow-up is provided by personnel as necessary.

[0519] Examples

[0520] For example, consider a user asking how to operate the air conditioning in a self-driving vehicle.

[0521] User speaks: "How do I turn down the air conditioner?"

[0522] Example prompt: "How can I turn down the air conditioner?"

[0523] When this prompt sentence is input to a generative AI, the AI ​​generates the following response:

[0524] Generated response: "To turn down the air conditioning temperature, press the temperature control button located on the center dashboard. To turn down the temperature, press the minus button."

[0525] The response is then converted into audio by a speech-to-text AI and provided to the user, allowing them to receive support quickly and efficiently.

[0526] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0527] Step 1:

[0528] A user performs voice input in an autonomous vehicle. The user's voice input is transmitted to a server through a microphone in the vehicle. The input data is in the form of voice data.

[0529] Step 2:

[0530] The server converts the voice input into text data using a voice recognition model. Specifically, it uses a voice recognition API to analyze the voice data and generate corresponding text data. The input is voice data, and the output is text data.

[0531] Step 3:

[0532] The server uses generative AI to generate a response based on the textual input. It uses a generative AI model to analyze the text data and generate an appropriate response. The input is the text data and the output is the response text.

[0533] Step 4:

[0534] The server converts the generated text response using the speech AI into voice data. It uses a speech synthesis engine to convert the text data into voice data. The input is the response text, and the output is the voice data.

[0535] Step 5:

[0536] The server provides the generated voice data to the user, specifically by playing a voice response through the vehicle's speakers. The input is the voice data and the output is the speech heard by the user.

[0537] Step 6:

[0538] The server records the response log. It saves the user's voice input, the converted text data, the generated response text, and the voice data in a database. This makes it possible to refer to them later. The input is various data, and the output is the recorded log data.

[0539] Step 7:

[0540] If necessary, the server will send a human to follow up. If the problem is not resolved or additional support is required based on the response log, a specialist staff member will respond. The input is the response log, and the output is the follow-up support.

[0541] Example 2

[0542] Next, a description will be given of Example 2 of the embodiment 2. In the following description, the data processing device 12 is referred to as a "server" and the smart glasses 214 are referred to as a "terminal".

[0543] Traditional technical support systems often required users to wait for long periods of time to resolve their issues, and the problem was that they did not provide efficient support. In addition, there was a lack of a system that could consistently convert voice input into text, generate an appropriate response, and return it by voice. Furthermore, there was insufficient follow-up when the user's problem was not resolved.

[0544] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0545] In this invention, the server includes a means for converting voice input to text using a speech recognition model, a means for generating a response based on the converted text input using generative artificial intelligence, a means for converting the generated text response to voice using text-to-speech artificial intelligence, a means for recording a response log, and a means for performing follow-up by personnel, thereby enabling users to receive fast and efficient technical support 24 hours a day, 365 days a year without waiting times.

[0546] A "speech recognition model" is an algorithm or software for converting voice data into text data.

[0547] "Generative artificial intelligence" is an artificial intelligence technology that analyzes input data and generates appropriate responses or information.

[0548] "Read-to-speech artificial intelligence" refers to algorithms and software for converting text data into audio data.

[0549] A "response log" is a database or file that records user interactions and the responses generated.

[0550] "Follow-up" is the process for providing additional support if the user's issue is not resolved.

[0551] This invention is a technical support system using a smartphone, which converts voice input from a user into text, generates an appropriate response using generative artificial intelligence, and returns the response by voice using a text-to-speech artificial intelligence. A specific embodiment of this system will be described below.

[0552] First, the user speaks a question or problem into the smartphone. The device receives this voice input through the microphone and sends it to the server. The server uses a speech recognition model to convert the voice data into text data. For example, a speech recognition API is used as this speech recognition model.

[0553] The server then passes the converted user input to a generative AI, which analyzes the input text and understands the user's intent. For example, a model using natural language processing technology is used as the generative AI. Based on the analysis results, the generative AI generates an appropriate response.

[0554] The generated text response is passed from the server to a voice-to-speech AI, which converts the text into speech. For example, a voice synthesis API is used as the voice-to-speech AI. The converted voice data is sent to the terminal, which plays the voice data and provides the response to the user.

[0555] In addition, the server records all response logs. If the user's problem is not resolved or additional support is required, personnel will follow up based on the recorded logs. This allows users to receive fast and efficient technical support 24 hours a day, 365 days a year, without waiting.

[0556] As a concrete example, consider the case where a user says "Wi-Fi is not connecting" to a smartphone. The device sends this voice to the server, and the speech recognition API converts it into text "Wi-Fi is not connecting." The generative AI analyzes this text and generates a response saying "Please try restarting the router." The text-to-speech AI converts this response into speech, and the device tells the user aloud, "Please try restarting the router."

[0557] An example of a prompt sentence is, "If a user says to their smartphone, 'I can't connect to Wi-Fi,' how would you respond?"

[0558] This system allows users to receive technical support quickly and efficiently.

[0559] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0560] Step 1:

[0561] A user speaks a question or problem into a smartphone. The device receives this voice input through a microphone. The input is voice data, and the output is voice data as is. A specific example of how this works is when a user says, "I can't connect to Wi-Fi."

[0562] Step 2:

[0563] The device sends the received voice data to the server. The server uses a voice recognition model to convert the voice data into text data. The input is voice data, and the output is text data. In concrete terms, the voice recognition API converts the voice saying "Wi-Fi is not connecting" into the text "Wi-Fi is not connecting."

[0564] Step 3:

[0565] The server passes the converted user input to the generative AI. The generative AI analyzes the input text and understands the user's intent. The input is text data, and the output is the analysis result. In concrete terms, the generative AI analyzes the text "Wi-Fi is not connecting" and determines that the problem is with the network connection.

[0566] Step 4:

[0567] The server uses generative AI to generate an appropriate response based on the analysis results. The input is the analysis results, and the output is the response text. The generative AI generates the response "Try restarting your router."

[0568] Step 5:

[0569] The generated text response is passed from the server to the speech AI, which converts the text to speech. The input is the response text, and the output is the voice data. In concrete terms, the speech synthesis API converts the text "Please try restarting the router" into voice data.

[0570] Step 6:

[0571] The server sends voice data to the terminal. The terminal plays the voice data and provides a response to the user. The input is voice data, and the output is voice output. In concrete terms, the terminal tells the user by voice, "Please try restarting the router."

[0572] Step 7:

[0573] The server records all response logs. If the user's problem is not resolved or additional support is required, personnel will follow up based on the recorded logs. The input is the response log, and the output is follow-up suggestions. In concrete terms, if the user inquires again saying "still can't connect," the server will refer to the log and suggest the next step, such as "check your network settings."

[0574] (Application example 2)

[0575] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the smart glasses 214 will be referred to as a "terminal".

[0576] There is a demand for prompt and appropriate support for technical problems and questions that arise inside autonomous vehicles. However, with conventional systems, when a user faces a technical problem inside the vehicle, there are limited ways to respond immediately, making it difficult to provide efficient support. This leads to issues such as reduced convenience for users and a loss of satisfaction.

[0577] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0578] In this invention, the server includes a means for converting voice input into text using a voice recognition model, a means for generating a response based on the converted text input using a generative AI, a means for converting the generated text response into voice using a text-to-speech AI, a means for recording a response log, and a means for receiving voice input in the autonomous vehicle and outputting a voice response using a speaker in the vehicle, thereby enabling a user in the autonomous vehicle to receive prompt and appropriate support for technical issues or questions.

[0579] A "speech recognition model" is an algorithm or software for converting voice input into text data.

[0580] "Generative AI" is an artificial intelligence technology that generates appropriate responses based on input text data.

[0581] "Reading AI" refers to algorithms or software for converting text data into audio data.

[0582] A "response log" is data that records user input and the system's response to it.

[0583] An "autonomous vehicle" is a vehicle that drives autonomously using artificial intelligence and sensor technology.

[0584] A "speaker" is a device for outputting audio data as sound.

[0585] As an embodiment of the present invention, a technical support system in an autonomous vehicle is shown below. The system includes a speech recognition model, a generative AI, a voice-to-speech AI, a response log recording means, and a voice input and output means in the autonomous vehicle.

[0586] System Configuration

[0587] 1. Speech Recognition Model

[0588] A voice recognition model is an algorithm or software that converts a user's voice input into text data. Specifically, it uses a voice recognition API. It receives voice input through a microphone in the vehicle and converts it into text.

[0589] 2. Generative AI

[0590] Generative AI is an artificial intelligence technique that generates appropriate responses based on textual input. Specifically, it uses generative AI models to analyze text data and generate appropriate responses to user questions.

[0591] 3. Text-to-speech AI

[0592] Text-to-speech AI is an algorithm or software that converts the generated text response into audio data, specifically using a text-to-speech API, which then converts the generated text response into audio and outputs it to the user through the vehicle's speakers.

[0593] 4. Response logging methods

[0594] The response log recorder is a database that records user input and the system's response to that input, so that later follow-up by personnel may be required.

[0595] 5. Voice input and output means in autonomous vehicles

[0596] A microphone is used as the voice input means in an autonomous vehicle, and a speaker is used as the voice output means, allowing users to receive voice support in the vehicle.

[0597] System Operation

[0598] The server receives voice input from the vehicle's microphone and converts it to text using a speech recognition model. It then uses generative AI to analyze the text and generate an appropriate response. The generated text response is converted to audio using speech AI and output to the user through the vehicle's speakers. Response logs are recorded and follow-ups are provided by personnel as needed.

[0599] Examples

[0600] For example, if a user asks, "How do I turn up the air conditioner temperature?" the system would do the following:

[0601] 1. A microphone in the vehicle captures the user's voice.

[0602] 2. The speech recognition model converts the speech to text.

[0603] 3. Generative AI analyzes the text and generates an appropriate response.

[0604] 4. Text-to-speech AI converts the generated text response into audio.

[0605] 5. Speakers in the vehicle output a voice response to the user.

[0606] Examples of prompt statements

[0607] User: How do I turn up the heat on my air conditioner?

[0608] In this way, quick and appropriate voice responses can be provided to user questions within an autonomous vehicle.

[0609] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0610] Step 1:

[0611] The user inputs voice into a microphone in the vehicle.

[0612] Input: User's voice

[0613] Output: Audio data

[0614] Specific action: The user says, "Tell me how to turn up the air conditioner temperature."

[0615] Step 2:

[0616] The server converts the voice data into text data using a voice recognition model.

[0617] Input: Audio data

[0618] Output: Text data

[0619] Specific operation: The speech recognition API analyzes the voice data and generates the text "Tell me how to turn up the air conditioner temperature."

[0620] Step 3:

[0621] The server uses generative AI to analyze the text data and generate an appropriate response.

[0622] Input: Text data

[0623] Output: Response text

[0624] Specific behavior: The generative AI model analyzes the text "Tell me how to turn up the temperature on the air conditioner" and generates the response text "To turn up the temperature on the air conditioner, select the 'Air Conditioning' menu on the central touchscreen and increase the temperature setting."

[0625] Step 4:

[0626] The server uses a text-to-speech AI to convert the response text into audio data.

[0627] Input: Response text

[0628] Output: Audio data

[0629] Specific operation: The speech synthesis API analyzes the response text and generates speech data saying, "To increase the temperature of the air conditioner, select the 'Air Conditioning' menu on the central touch screen and increase the temperature setting."

[0630] Step 5:

[0631] The server outputs the audio data to the user using a speaker in the vehicle.

[0632] Input: Audio data

[0633] Output: Voice response

[0634] Specific operation: The speaker in the vehicle plays audio data, informing the user, "To increase the air conditioning temperature, select the 'Air Conditioning' menu on the central touchscreen and increase the temperature setting."

[0635] Step 6:

[0636] The server logs the response.

[0637] Input: User voice input, generated response text

[0638] Output: Response log

[0639] What happens: The server stores the user's voice input and the generated response text in a database in case a later human follow-up is required.

[0640] In addition, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0641] "Example 1"

[0642] As one embodiment of the present invention, a system incorporating an emotion engine is provided. This system converts voice input into text using a voice recognition model, generates a response based on the text input using a generative AI, converts the generated text response into voice using a speech-to-speech AI, and records a response log. Furthermore, this system includes an emotion engine that recognizes the user's emotion from the voice input. Specifically, when a user expresses an emotion such as anger or joy by voice, the emotion engine recognizes the emotion, and the generative AI generates a response based on the emotion. For example, when a user shows an emotion of anger, the generative AI is adjusted to generate a more polite response. In addition, the emotion engine can adjust the priority of responses based on the user's emotion. For example, when a user shows a strong emotion of anger, the system is adjusted to prioritize processing of inquiries from that user. This makes it possible to provide more appropriate support according to the user's emotion.

[0643] "Example 2"

[0644] As one embodiment of the present invention, a system incorporating an emotion engine is provided. This system converts voice input into text using a voice recognition model, generates a response based on the text input using a generative AI, converts the generated text response into voice using a speech-to-speech AI, and records a response log. Furthermore, this system includes an emotion engine that recognizes the user's emotion from the voice input. Specifically, when a user expresses an emotion such as anger or joy by voice, the emotion engine recognizes the emotion, and the generative AI generates a response based on the emotion. For example, when a user shows an emotion of anger, the generative AI is adjusted to generate a more polite response. In addition, the emotion engine can adjust the priority of responses based on the user's emotion. For example, when a user shows a strong emotion of anger, the system is adjusted to prioritize processing of inquiries from that user. This makes it possible to provide more appropriate support according to the user's emotion.

[0645] The process flow of each embodiment will be described below.

[0646] "Example 1"

[0647] Step 1: Receive voice input from the user. This voice input is received through a smartphone microphone or similar.

[0648] Step 2: Convert the received voice input into text using WhisperAPI. WhisperAPI is a speech recognition model that has the ability to convert speech to text.

[0649] Step 3: Recognize user emotions from the textual input using the emotion engine. The emotion engine has the ability to recognize user emotions (e.g., anger, joy) from the text.

[0650] Step 4: Generative AI generates a response based on the recognized emotion. Generative AI has the ability to generate an appropriate response based on the textual input and the recognized emotion.

[0651] Step 5: The generated text response is converted into speech using ReadSpeaker, a speech-to-speech AI capable of converting text to speech.

[0652] Step 6: Provide a spoken response to the user and simultaneously record the response log, which can be used for later follow-up and system improvement.

[0653] Example 1

[0654] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0655] Conventional technical support systems provide uniform responses without considering the user's feelings, which can lead to low user satisfaction. In addition, there are often waiting times, and responses are delayed even in situations where a quick response is required. This increases user stress and reduces support efficiency.

[0656] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0657] In this invention, the server includes a means for converting a voice input into text using a voice recognition model, a means for generating a response based on the input converted into text using generative artificial intelligence, a means for converting the generated text response into voice using a text-to-speech artificial intelligence, a means for recording a response log, a means for recognizing a user's emotion from the voice input using an emotion engine and adjusting the response based on the emotion, and a means for adjusting the priority of responses based on the user's emotion, thereby making it possible to quickly provide an appropriate response according to the user's emotion.

[0658] A "speech recognition model" is an algorithm or software for analyzing voice data and converting it into text data.

[0659] "Generative artificial intelligence" is an artificial intelligence technology that analyzes input data and generates appropriate responses or results.

[0660] "Read-to-speech artificial intelligence" is an artificial intelligence technology that converts text data into audio data and outputs it as audio.

[0661] A "response log" is data that records the content of inquiries from users and the responses generated by the system.

[0662] An "emotion engine" is a technology that analyzes user emotions from voice input and text data and recognizes those emotions.

[0663] "Means for adjusting priority" is a function for changing the order and importance of responses based on the user's emotions and the content of the inquiry.

[0664] This invention is a system for providing technical support for smartphones. Voice input from a user is converted to text by a voice recognition model. A voice recognition API is used as this voice recognition model. The text input is analyzed by generative artificial intelligence, and an appropriate response is generated. The generated text response is converted to voice by a speech-to-speech artificial intelligence, and output to the user. A voice synthesis API is used as this speech-to-speech artificial intelligence. In addition, the system records a response log, and when follow-up by a human is required, a proposal for a solution is made based on this. This enables voice support 24 hours a day, 365 days a year with no waiting time, reducing the burden on users and providing efficient support.

[0665] Furthermore, this system combines an emotion engine. The emotion engine recognizes the user's emotion from the voice input, and the generative AI generates a response based on that emotion. For example, if the user expresses anger, the generative AI is adjusted to generate a more polite response. The emotion engine can also adjust the priority of responses based on the user's emotion. For example, if a user expresses strong anger, the system is adjusted to prioritize inquiries from that user. This makes it possible to provide more appropriate support according to the user's emotion.

[0666] As a concrete example, consider the case where a user makes a voice inquiry to their smartphone saying, "Wi-Fi won't connect." The device converts this voice into text using a speech recognition API, generating the text "Wi-Fi won't connect." The server passes this text to a generative artificial intelligence system, which generates an appropriate response. For example, a response might be generated saying, "Please try restarting your Wi-Fi router." The speech synthesis API converts this response into speech, and the device outputs the response to the user in voice.

[0667] An example of a prompt sentence is, "The user spoke to his smartphone saying, 'I can't connect to Wi-Fi.' Please convert this speech into text, generate an appropriate response, and reply by voice."

[0668] This system allows users to receive voice support 24 hours a day, 365 days a year, without waiting, providing efficient support.

[0669] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0670] Step 1:

[0671] A user makes a voice inquiry into a smartphone. For example, they say, "I can't connect to Wi-Fi." The input is the user's voice data, and the output is the voice data sent to the device.

[0672] Step 2:

[0673] The device receives the user's voice input and converts the voice to text using a voice recognition API. Specifically, the device sends voice data to the voice recognition API, which analyzes the voice data and generates the text "Wi-Fi is not connected." The input is voice data, and the output is text data.

[0674] Step 3:

[0675] The server receives the text sent from the terminal. The server passes this text to the generative AI for analysis. The generative AI understands the content of the user's inquiry and creates a prompt sentence to generate an appropriate response. The input is text data, and the output is a prompt sentence.

[0676] Step 4:

[0677] The generative AI generates an appropriate response based on the prompt sentence. For example, a response such as "Please try restarting your Wi-Fi router" is generated. The server stores this response in text format. The input is the prompt sentence, and the output is the response text.

[0678] Step 5:

[0679] The server sends the generated text response to the device. The device uses a speech synthesis API to convert the text response into speech. Specifically, the device passes the text data to the speech synthesis API, which generates speech data. The input is text data, and the output is speech data.

[0680] Step 6:

[0681] The device outputs the generated voice data to the user. The user can hear the voice response, "Please try restarting your Wi-Fi router." The input is the voice data, and the output is the voice response to the user.

[0682] Step 7:

[0683] The server records all response logs, including the user's voice input, the transcribed input, and the generated response. If necessary, a human follow-up is performed and a resolution suggestion is provided. The input is the data, and the output is the recorded response log.

[0684] Step 8:

[0685] The server uses an emotion engine to recognize user emotions from voice input. For example, if a user expresses an emotion of anger, the emotion engine recognizes the emotion and adjusts the generative artificial intelligence to generate a more polite response. The emotion engine can also adjust the priority of responses. User queries that express a strong emotion of anger are given priority. The input is voice data and text data, and the output is the adjusted response and priority.

[0686] (Application example 1)

[0687] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0688] In autonomous vehicles, there is a need to provide prompt and appropriate support when users ask questions about how to operate the vehicle or troubleshooting. However, conventional systems are unable to respond in a way that takes into account the user's emotions, which can lead to a decrease in user satisfaction. In addition, the accuracy and speed of generating responses based on voice input are also issues.

[0689] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0690] In this invention, the server includes a means for converting voice input into text using a voice recognition model, a means for generating a response based on the text input using a generative AI, a means for converting the generated text response into voice using a text-to-speech AI, a means for recording a response log, a means for recognizing a user's emotion from the voice input using an emotion engine, and a means for adjusting the tone and priority of the response based on the user's emotion. This enables prompt and appropriate support according to the user's emotion.

[0691] A "speech recognition model" is an algorithm or software for converting voice input into text.

[0692] "Generative AI" is an artificial intelligence technology that generates appropriate responses based on text input.

[0693] "Reading AI" refers to algorithms or software that converts text into speech and outputs it.

[0694] A "response log" is a database that records user inquiries and the system's responses to them.

[0695] An "emotion engine" is an algorithm or software that analyzes and recognizes a user's emotions from voice input.

[0696] "Tone" refers to the vocal cadence or nuance of the response produced.

[0697] "Priority" is a criterion for determining the order in which multiple inquiries or tasks are processed.

[0698] The system for implementing this invention is composed of a combination of a speech recognition model, a generative AI, a reading AI, and an emotion engine. Below, we will explain the details of each component and how they work together.

[0699] Speech Recognition Model

[0700] The server converts the user's voice input into text using a speech recognition model, which is provided by a speech recognition API, that analyzes the speech data and generates corresponding text data.

[0701] Generative AI

[0702] The server generates responses based on the textual input using a generative AI, which uses advanced natural language processing models such as GPT-3, and leverages pre-trained data to generate appropriate responses to user questions.

[0703] Reading AI

[0704] The server uses a voice AI to convert the generated text response into voice. A voice synthesis engine is used as the voice AI. The voice synthesis engine converts the text data into natural voice and outputs it to the user.

[0705] Emotion Engine

[0706] The server recognizes the user's emotions from the voice input using an emotion engine. The emotion engine analyzes the tone, pitch, and rate of the voice data to identify the user's emotional state. If the user is expressing an emotion such as anger or joy, a response is generated according to that emotion.

[0707] Adjusting the tone and priority of your response

[0708] The server adjusts the tone and priority of the responses generated by the generative AI based on the user's emotions recognized by the emotion engine. For example, if the user expresses anger, the generative AI is adjusted to generate more polite responses. In addition, inquiries from users who show strong emotions are given priority.

[0709] Examples

[0710] Consider a case where a user asks how to operate the air conditioner in an autonomous vehicle. When the user asks verbally, "How do I adjust the temperature of the air conditioner?", a speech recognition model converts this speech into text. A generative AI then generates a response based on this text: "To adjust the temperature of the air conditioner, tap the temperature setting on the center display." The emotion engine analyzes the user's emotions, and if there is no particular change in emotion, the reading AI vocalizes this response in a normal tone and provides it to the user.

[0711] Examples of prompt statements

[0712] User sentiment: Neutral

[0713] User Question: How do I adjust the temperature on my air conditioner?

[0714] Generate an appropriate response.

[0715] In this way, it is possible to provide prompt and appropriate support according to the user's emotions.

[0716] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0717] Step 1:

[0718] The user performs voice input. For example, the user may ask a question in an autonomous vehicle, such as "How do I adjust the temperature of the air conditioner?" This voice data is sent to the server.

[0719] Step 2:

[0720] The server converts the voice input to text using a voice recognition model. The voice recognition API analyzes the voice data and generates the corresponding text data: "How do I adjust the temperature of the air conditioner?" The input is voice data, and the output is text data.

[0721] Step 3:

[0722] The server uses an emotion engine to recognize the user's emotion from the voice input. The emotion engine analyzes the tone, pitch, and speed of the voice data and identifies the user's emotional state as "neutral." The input is the voice data, and the output is the emotional state.

[0723] Step 4:

[0724] The server uses a generative AI to generate a response based on the textual input. The generative AI (e.g., GPT-3) uses the prompt sentence to generate an appropriate response: "To adjust the temperature of the air conditioner, tap the temperature setting on the center display." The input is the text data and the emotional state, and the output is the response text.

[0725] Step 5:

[0726] The server adjusts the tone and priority of the response generated by the generative AI based on the user's emotion recognized by the emotion engine. For example, because the user's emotion is "normal," a response is generated in a normal tone. The input is the response text and the emotional state, and the output is the adjusted response text.

[0727] Step 6:

[0728] The server uses a voice-to-speech AI to convert the generated text response into voice. The voice synthesis engine converts the text data into natural speech and outputs it to the user. The input is the adjusted response text, and the output is the voice data.

[0729] Step 7:

[0730] The server records the response log. User queries and the system's responses to them are stored in a database. Inputs are voice data, text data, emotional state, and response text, and output is the recorded log data.

[0731] In this way, prompt and appropriate support can be provided according to the user's feelings.

[0732] Example 2

[0733] Next, a description will be given of Example 2 of the embodiment 2. In the following description, the data processing device 12 is referred to as a "server" and the smart glasses 214 are referred to as a "terminal".

[0734] Conventional smartphone technical support systems convert users' voice input into text, generate responses using generative AI, and convert them into voice using text-to-speech AI, but they have a problem in that they cannot adjust responses to take the user's emotions into account. This can lead to lower user satisfaction and make it difficult to provide efficient support.

[0735] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0736] In this invention, the server includes a means for converting a voice input into text using a voice recognition model, a means for generating a response based on the converted text input using generative artificial intelligence, a means for converting the generated text response into voice using a text-to-speech artificial intelligence, a means for recording a response log, a means for recognizing a user's emotion from the voice input using an emotion engine, and a means for adjusting a response based on the user's emotion, thereby enabling more appropriate support according to the user's emotion.

[0737] A "speech recognition model" is an algorithm or software for converting voice input into text data.

[0738] "Generative artificial intelligence" is an artificial intelligence technology that analyzes input text data and generates appropriate responses.

[0739] "Read-to-speech artificial intelligence" is a technology that converts text data into voice data and provides a voice response to the user.

[0740] A "response log" is data for recording inquiries from users and responses to those inquiries.

[0741] An "emotion engine" is a technology for recognizing a user's emotions from voice input and adjusting responses based on those emotions.

[0742] "Text conversion" is the process of converting audio data into text data.

[0743] "Analysis" is the process of analyzing input data and understanding its meaning and intent.

[0744] "Adjustment" is the process of changing responses or behavior in response to specific conditions or situations.

[0745] This invention is a system for providing technical support for smartphones, which converts voice input from the user into text, analyzes the text to generate an appropriate response, and returns it to the user as voice. It also has a function to recognize the user's emotions and provide a response according to those emotions.

[0746] System Configuration

[0747] 1. Voice to text conversion

[0748] Hardware / Software: The server converts voice input to text using a speech recognition model, which is provided by a speech recognition API.

[0749] Example: When a user says, "My smartphone battery is running out quickly, what should I do?" the server converts this speech into text.

[0750] 2. Text Analysis and Response Generation

[0751] Hardware / Software: The server uses generative artificial intelligence (e.g., a generative AI model) to parse the textual input and generate an appropriate response.

[0752] Example: A server parses the text "My phone's battery is draining quickly, what should I do?" and generates a response that reads "To extend battery life, you can close apps running in the background and adjust the screen brightness."

[0753] 3. Audio Responses

[0754] Hardware / Software: The server converts the generated text response into speech using a speech-to-text AI, which uses a text-to-speech API.

[0755] Example: The server converts the text "To extend your battery life, you can close background apps and adjust your screen brightness." into speech, which the device then outputs to the user.

[0756] 4. Response logging

[0757] Hardware / Software: The server uses a database to log all responses.

[0758] Example: The server stores the user's inquiry and the generated response in a database. For example, the following log is recorded: "Inquiry: My smartphone's battery runs out quickly. What should I do?" "Response: To extend the battery life, it is effective to close apps running in the background and adjust the screen brightness."

[0759] 5. Emotion recognition and response adjustment

[0760] Hardware / Software: The server uses an emotion engine to recognize emotions from the user's voice.

[0761] Example: If a user expresses anger, the server uses the emotion engine to recognize the emotion. For example, if the user says in an angry tone, "Why is my battery running out so fast?", the generative AI generates a more polite response such as, "I'm sorry. We will give you a solution to your battery problem right away." The system is also adjusted so that inquiries from users who express strong emotions are given priority.

[0762] Examples of prompt statements

[0763] "My smartphone battery runs out quickly, what should I do?"

[0764] "The app keeps crashing, what should I do?"

[0765] "I can't connect to Wi-Fi, what should I do?"

[0766] This system allows users to receive voice support 24 hours a day, 365 days a year, without waiting, providing efficient support.

[0767] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0768] Step 1:

[0769] Receiving audio input

[0770] How it works: A user speaks a query into their smartphone.

[0771] Input: The user's voice input.

[0772] Output: The audio data.

[0773] Specific behavior: A user says, "My smartphone battery is running out quickly. What should I do?" The device receives this voice through the microphone.

[0774] Step 2:

[0775] Voice to text conversion

[0776] How it works: The server converts voice input into text using a speech recognition model.

[0777] Input: Audio data.

[0778] Output: Text data.

[0779] What happens: The server calls a speech recognition API to convert the voice data into text such as, "My smartphone's battery is running out quickly. What should I do?"

[0780] Step 3:

[0781] Text analysis and response generation

[0782] How it works: The server uses generative artificial intelligence to parse the textual input and generate an appropriate response.

[0783] Input: Text data.

[0784] Output: The response text.

[0785] How it works: The server inputs the text "My smartphone's battery is draining quickly, what should I do?" into the generative AI model, which generates a response saying "To extend battery life, it is effective to close apps running in the background and adjust the screen brightness."

[0786] Step 4:

[0787] Spoken responses

[0788] How it works: The server uses text-to-speech artificial intelligence to convert the generated text response into speech.

[0789] Input: The response text.

[0790] Output: The audio data.

[0791] Specific operation: The server calls the speech synthesis API to convert the text "To extend battery life, it is effective to close apps running in the background and adjust the screen brightness." into speech, and the device outputs the speech to the user.

[0792] Step 5:

[0793] Response logging

[0794] Behavior: The server logs all responses.

[0795] Input: The user's query and the generated response.

[0796] Output: The response log data.

[0797] Specific operation: The server records the following log in the database: "Query: My smartphone's battery is draining quickly. What should I do?" "Response: To extend battery life, it is effective to close apps running in the background and adjust the screen brightness."

[0798] Step 6:

[0799] Emotion recognition and response adjustment

[0800] How it works: The server uses an emotion engine to recognize emotions from the user's voice and tailor its response.

[0801] Input: The user's voice data.

[0802] Output: The adjusted response text.

[0803] Specific behavior: If a user expresses anger, the server uses the emotion engine to recognize the emotion. For example, if a user says in an angry tone, "Why is my battery running out so fast?", the generative AI generates a more polite response such as, "I'm sorry. We will give you a solution to your battery problem right away." The system is also adjusted so that inquiries from users who express strong emotions are given priority.

[0804] (Application example 2)

[0805] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the smart glasses 214 will be referred to as a "terminal".

[0806] In autonomous vehicles, when a user asks a question about vehicle operation or navigation by voice while driving, it is necessary to provide a quick and appropriate response. However, conventional systems are unable to respond taking into account the user's emotions, which may reduce user satisfaction. In addition, it is difficult to respond quickly to questions with high urgency because it is not possible to adjust the priority of responses based on emotions. To solve these issues, a system is needed that recognizes the user's emotions, generates responses based on them, and provides responses with an appropriate tone and priority.

[0807] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0808] In this invention, the server includes a means for converting a voice input into text using a voice recognition model, a means for generating a response based on the input converted into text using a generative AI, a means for converting the generated text response into voice using a text-to-speech AI, a means for recording a response log, a means for recognizing a user's emotion from the voice input using an emotion engine, a means for adjusting the tone and priority of the response based on the user's emotion, and a means for being installed in the infotainment system of the autonomous vehicle. This enables a quick and appropriate response that takes the user's emotion into consideration, thereby improving user satisfaction.

[0809] A "speech recognition model" is an algorithm or software for converting voice input into text.

[0810] "Generative AI" is an artificial intelligence technology that generates appropriate responses based on input text data.

[0811] "Read-to-speech AI" is an artificial intelligence technology that converts text data into audio and outputs it.

[0812] A "response log" is a database that records user inquiries and the system's responses to them.

[0813] An "emotion engine" is a technology for recognizing a user's emotions from voice input and adjusting responses based on those emotions.

[0814] An "infotainment system" is a system that provides information and entertainment within a vehicle.

[0815] An "autonomous vehicle" is a vehicle that drives itself using artificial intelligence and sensor technology.

[0816] As an embodiment of the present invention, a program installed in the infotainment system of an autonomous vehicle will be described. The server uses a speech recognition model, generative AI, text-to-speech AI, and emotion engine to generate a response based on the user's voice input, and outputs it by voice.

[0817] First, the user speaks to a vehicle operation or navigation question. The voice input is converted to text using a speech recognition model, which uses a speech recognition API. The text input is then analyzed by an emotion engine to recognize the user's emotion. The emotion engine recognizes the user's emotion from the voice input and adjusts the tone and priority of the response based on that emotion.

[0818] A generative AI model is then used to generate an appropriate response based on the textual input and the recognized emotion. The response is then converted to speech by a text-to-speech AI, which uses a text-to-speech API. The generated voice response is then output to the user through the infotainment system.

[0819] In addition, response logs are recorded on the server and can be viewed later, making it possible to track user inquiry history and response content, and to have personnel follow up if necessary.

[0820] As a concrete example, consider a user asking, "Where is the next gas station?" In this case, the emotion engine recognizes the user's impatience, and the generative AI generates a quick and polite response, such as, "The next gas station is 3 kilometers away on the right. Don't worry."

[0821] Example prompt:

[0822] User Question: Where is the next gas station?

[0823] User Sentiment: Impatience

[0824] Appropriate response:

[0825] In this way, a quick and appropriate response that takes the user's feelings into consideration is possible, thereby improving user satisfaction.

[0826] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0827] Step 1:

[0828] The user asks questions about vehicle operation and navigation by voice.

[0829] Input: User's voice input

[0830] Output: Audio data

[0831] Specific Action: A user asks a question to the infotainment system of an autonomous vehicle.

[0832] Step 2:

[0833] The server converts the voice input into text using a speech recognition model.

[0834] Input: Audio data

[0835] Output: Text data

[0836] Specific operation: The speech recognition API analyzes the speech data and generates corresponding text data.

[0837] Step 3:

[0838] The server uses an emotion engine to recognize the user's emotion from the text data.

[0839] Input: Text data

[0840] Output: Emotion data

[0841] Specific behavior: The emotion engine analyzes text data and recognizes the user's emotions (e.g., impatience, anger, joy).

[0842] Step 4:

[0843] The server uses generative AI to generate a response based on text data and emotion data.

[0844] Input: Text data, emotion data

[0845] Output: Response text

[0846] Specific operation: The generative AI model generates an appropriate response text based on text data and emotion data.

[0847] Step 5:

[0848] The server uses text-to-speech AI to convert the response text into speech.

[0849] Input: Response text

[0850] Output: Audio data

[0851] Specific operation: The speech synthesis API analyzes the response text and generates corresponding speech data.

[0852] Step 6:

[0853] The server outputs the generated voice data to the user through the infotainment system.

[0854] Input: Audio data

[0855] Output: A voice response to the user

[0856] Specific Action: The infotainment system plays the audio data and provides a response to the user.

[0857] Step 7:

[0858] The server logs the response.

[0859] Input: Text data, emotion data, response text

[0860] Output: Response log

[0861] Specific behavior: The server records the user's question, the recognized emotion, and the generated response in a database.

[0862] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0863] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0864] Another example of generative AI is Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) are mentioned.

[0865] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0866] [Third embodiment]

[0867] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0868] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0869] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0870] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0871] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[0872] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[0873] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[0874] Fig. 6 shows an example of main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0875] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0876] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0877] In the headset type terminal 314, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0878] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0879] "Example 1"

[0880] The present invention is a system for providing technical support for smartphones. Voice input from a user is converted to text by a voice recognition model. For example, WhisperAPI is used as this voice recognition model. The converted text input is analyzed by a generative AI, and an appropriate response is generated. The generated text response is converted to voice by a text-to-speech AI and output to the user. For example, ReadSpeaker is used as this text-to-speech AI. In addition, the system records a response log, and when follow-up by personnel is required, a proposal for a solution is made based on this. This enables voice response 24 hours a day, 365 days a year without waiting time, reducing the burden on users and providing efficient support.

[0881] "Example 2"

[0882] The present invention is a system for providing technical support for smartphones. Voice input from a user is converted to text by a voice recognition model. For example, WhisperAPI is used as this voice recognition model. The converted text input is analyzed by a generative AI, and an appropriate response is generated. The generated text response is converted to voice by a text-to-speech AI and output to the user. For example, ReadSpeaker is used as this text-to-speech AI. In addition, the system records a response log, and when follow-up by personnel is required, a proposal for a solution is made based on this. This enables voice response 24 hours a day, 365 days a year without waiting time, reducing the burden on users and providing efficient support.

[0883] The process flow of each embodiment will be described below.

[0884] "Example 1"

[0885] Step 1: Receive voice input from the user. This voice input is done through the smartphone microphone.

[0886] Step 2: The received voice input is converted into text using a voice recognition model, such as WhisperAPI.

[0887] Step 3: The textual input is analyzed by a generative AI to generate an appropriate response. This generative AI has the ability to understand the user's problem and suggest appropriate solutions.

[0888] Step 4: The generated text response is converted to audio by a speech AI and output to the user. For example, ReadSpeaker is used as the speech AI. Step 5: The system records the response log. This allows the system to make suggestions for a solution based on the log, even if follow-up by a human is required.

[0889] Example 1

[0890] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[0891] Conventional smartphone technical support systems often required users to wait a long time to resolve their issues, making it difficult to provide efficient support. In addition, the process of converting voice input into text, generating an appropriate response, and returning it by voice was sometimes inconsistent, which reduced the user experience.

[0892] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0893] In this invention, the server includes a means for converting voice input into text using a voice recognition model, a means for generating a response based on the converted text input using generative artificial intelligence, a means for converting the generated text response into voice using speech-to-speech artificial intelligence, a means for recording a response log, and a means for following up with personnel, thereby enabling efficient support 24 hours a day, 365 days a year without waiting times.

[0894] A "speech recognition model" is an algorithm or system for analyzing speech data and converting it into corresponding text data.

[0895] "Generative artificial intelligence" is an artificial intelligence technology that analyzes input data and generates appropriate responses or information.

[0896] "Read-to-speech artificial intelligence" is an artificial intelligence technology that converts text data into audio data and outputs it as audio.

[0897] A "response log" is a database or file that maintains a record of user interactions and the responses generated.

[0898] "Follow-up" is the process by which personnel respond when a user's problem is not resolved or additional support is required.

[0899] A "voice recognition API" is an application programming interface for converting voice data into text data.

[0900] A "speech synthesis API" is an application programming interface for converting text data into speech data.

[0901] This invention is a system for providing technical support for smartphones. Voice input from a user is converted to text by a voice recognition model. A voice recognition API is used as this voice recognition model. The text input is analyzed by a generative artificial intelligence, and an appropriate response is generated. The generated text response is converted to voice by a voice-to-speech artificial intelligence, and output to the user. A voice synthesis API is used as this voice-to-speech artificial intelligence. In addition, the system records a response log, and when follow-up by personnel is required, a proposal for a solution is made based on this. This enables voice response 24 hours a day, 365 days a year with no waiting time, reducing the burden on users and providing efficient support.

[0902] Specifically, a user speaks a question or problem into a smartphone. For example, they might say, "My smartphone's battery is running low." The device records this voice and sends the data to a server. The server uses a speech recognition API to convert the voice data into text. Specifically, it analyzes the voice signal and generates a corresponding string of characters.

[0903] The server then passes the converted user input to a generative AI, which uses natural language processing techniques, for example, to analyze the user's question or problem and generate an appropriate response, such as "To reduce battery drain, we recommend closing apps running in the background."

[0904] The generated text response is passed from the server to the speech-to-speech AI, which converts the text into speech and sends it to the device. The device then plays the speech to the user. Specifically, it generates an audio file and plays it through the smartphone speaker.

[0905] In addition, the server records a log of all responses. If the user's problem is not resolved or additional support is required, the server can use this log to send a follow-up call to the appropriate technician. For example, if a user says, "I can't connect to Wi-Fi," the server will check the log and escalate the call to the appropriate technician.

[0906] An example of a prompt sentence is, "What should you do if your smartphone battery runs out quickly?" By inputting this prompt sentence into a generative artificial intelligence system, an appropriate response is generated.

[0907] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0908] Step 1:

[0909] A user speaks a question or problem into a smartphone. For example, they might say, "My smartphone's battery is running low." The device records this voice and sends the data to a server. The input is the voice data, and the output is the voice data sent to the server.

[0910] Step 2:

[0911] The server uses a speech recognition API to convert this voice data into text: it analyzes the audio signal and generates a corresponding string of characters. The input is the voice data, and the output is the text data.

[0912] Step 3:

[0913] The server passes the textualized user input to the generative AI. The generative AI uses natural language processing techniques to analyze the user's question or problem and generate an appropriate response. For example, it generates a response such as "To reduce battery drain, we recommend closing apps running in the background." The input is text data, and the output is the response text.

[0914] Step 4:

[0915] The generated text response is passed from the server to the speech AI, which converts the text into speech and sends it to the terminal. Specifically, it generates an audio file and plays it through the smartphone speaker. The input is the response text, and the output is the audio data.

[0916] Step 5:

[0917] The server records all response logs. If the user's problem is not resolved or additional support is required, the server uses these logs to send personnel to follow up. For example, if a user says "Wi-Fi is not working," the server checks the logs and escalates the issue to the appropriate technician. The input is the response log, and the output is the information for follow-up.

[0918] (Application example 1)

[0919] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[0920] As autonomous vehicles become more widespread, users need to receive support for vehicle operation and troubleshooting quickly and efficiently. However, conventional systems require users to spend a lot of time and effort to solve problems, making it difficult to respond quickly, especially in emergencies.

[0921] The identification process by the identification processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means. In this invention, the server includes a means for converting a voice input into text using a voice recognition model, a means for generating a response based on the input converted into text using a generative AI, a means for converting the generated text response into voice using a text-to-speech AI, a means for recording a response log, and a means for analyzing a voice input of a user in an autonomous vehicle and providing support regarding vehicle operation and troubleshooting. This allows the user to receive support quickly and efficiently using voice in an autonomous vehicle.

[0922] A "speech recognition model" is an algorithm or software for converting voice input into text data.

[0923] "Generative AI" is an artificial intelligence technology that analyzes input text data and generates appropriate responses.

[0924] "Reading AI" refers to algorithms and software for converting text data into speech.

[0925] A "response log" is a database or file that stores records of user interactions.

[0926] An "autonomous vehicle" is a vehicle that uses artificial intelligence and sensor technology to drive autonomously.

[0927] "User's voice input" refers to voice data generated by a user through a voice input device such as a microphone.

[0928] "Vehicle operation" refers to the act of controlling various vehicle functions (e.g., air conditioning, navigation, audio system, etc.).

[0929] "Troubleshooting" is the process of identifying and resolving problems with a system or device.

[0930] "Support" is a service that provides appropriate solutions and information to problems or questions faced by users.

[0931] A system for implementing the invention includes a speech recognition model, a generative AI, a text-to-speech AI, and a means for recording a response log, as well as a means for analyzing a user's voice input in an autonomous vehicle and providing support for operating and troubleshooting the vehicle.

[0932] System Configuration

[0933] 1. Speech recognition model:

[0934] For the speech recognition model, we use a speech recognition API, which is an algorithm or software that converts a user's voice input into text data.

[0935] 2. Generative AI:

[0936] As a generative AI, we use a generative AI model, which is an artificial intelligence technology that analyzes textual input and generates an appropriate response.

[0937] 3. Text-to-speech AI:

[0938] For speech-to-speech AI, we use a text-to-speech engine, which is an algorithm or software that converts generated text responses into speech.

[0939] 4. Response log:

[0940] A response log is a database or file that stores a record of user interactions so that they can be viewed at a later time.

[0941] 5. Support measures in autonomous vehicles:

[0942] This includes a means for analyzing a user's voice input in an autonomous vehicle and providing assistance regarding vehicle operation and troubleshooting.

[0943] System Operation

[0944] When a user speaks to the server while in an autonomous vehicle, the server converts the speech into text data using a speech recognition model. Next, a generative AI analyzes the text data and generates an appropriate response. The generated response is converted into speech by a text-to-speech AI and provided to the user. Response logs are recorded, and follow-up is provided by personnel as necessary.

[0945] Examples

[0946] For example, consider a user asking how to operate the air conditioning in a self-driving vehicle.

[0947] User speaks: "How do I turn down the air conditioner?"

[0948] Example prompt: "How can I turn down the air conditioner?"

[0949] When this prompt sentence is input to a generative AI, the AI ​​generates the following response:

[0950] Generated response: "To turn down the air conditioning temperature, press the temperature control button located on the center dashboard. To turn down the temperature, press the minus button."

[0951] The response is then converted into audio by a speech-to-text AI and provided to the user, allowing them to receive support quickly and efficiently.

[0952] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0953] Step 1:

[0954] A user performs voice input in an autonomous vehicle. The user's voice input is transmitted to a server through a microphone in the vehicle. The input data is in the form of voice data.

[0955] Step 2:

[0956] The server converts the voice input into text data using a voice recognition model. Specifically, it uses a voice recognition API to analyze the voice data and generate corresponding text data. The input is voice data, and the output is text data.

[0957] Step 3:

[0958] The server uses generative AI to generate a response based on the textual input. A generative AI model is used to analyze the text data and generate an appropriate response. The input is the text data and the output is the response text.

[0959] Step 4:

[0960] The server converts the generated text response using the speech AI into voice data. It uses a speech synthesis engine to convert the text data into voice data. The input is the response text, and the output is the voice data.

[0961] Step 5:

[0962] The server provides the generated voice data to the user, specifically by playing a voice response through the vehicle's speakers. The input is the voice data and the output is the speech heard by the user.

[0963] Step 6:

[0964] The server records the response log. It saves the user's voice input, the converted text data, the generated response text, and the voice data in a database. This makes it possible to refer to them later. The input is various data, and the output is the recorded log data.

[0965] Step 7:

[0966] If necessary, the server will send a human to follow up. If the problem is not resolved or additional support is required based on the response log, a specialist staff member will respond. The input is the response log, and the output is the follow-up support.

[0967] Example 2

[0968] Next, a second embodiment of the second form will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[0969] Traditional technical support systems often required users to wait for long periods of time to resolve their issues, and the problem was that they did not provide efficient support. In addition, there was a lack of a system that could consistently convert voice input into text, generate an appropriate response, and return it by voice. Furthermore, there was insufficient follow-up when the user's problem was not resolved.

[0970] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0971] In this invention, the server includes a means for converting voice input to text using a speech recognition model, a means for generating a response based on the converted text input using generative artificial intelligence, a means for converting the generated text response to voice using text-to-speech artificial intelligence, a means for recording a response log, and a means for performing follow-up by personnel, thereby enabling users to receive fast and efficient technical support 24 hours a day, 365 days a year without waiting times.

[0972] A "speech recognition model" is an algorithm or software for converting voice data into text data.

[0973] "Generative artificial intelligence" is an artificial intelligence technology that analyzes input data and generates appropriate responses or information.

[0974] "Read-to-speech artificial intelligence" refers to algorithms and software for converting text data into audio data.

[0975] A "response log" is a database or file that records user interactions and the responses generated.

[0976] "Follow-up" is the process for providing additional support if the user's issue is not resolved.

[0977] This invention is a technical support system using a smartphone, which converts voice input from a user into text, generates an appropriate response using generative artificial intelligence, and returns the response by voice using a text-to-speech artificial intelligence. A specific embodiment of this system will be described below.

[0978] First, the user speaks a question or problem into the smartphone. The device receives this voice input through the microphone and sends it to the server. The server uses a speech recognition model to convert the voice data into text data. For example, a speech recognition API is used as this speech recognition model.

[0979] The server then passes the converted user input to a generative AI, which analyzes the input text and understands the user's intent. For example, a model using natural language processing technology is used as the generative AI. Based on the analysis results, the generative AI generates an appropriate response.

[0980] The generated text response is passed from the server to a voice-to-speech AI, which converts the text into speech. For example, a voice synthesis API is used as the voice-to-speech AI. The converted voice data is sent to the terminal, which plays the voice data and provides the response to the user.

[0981] In addition, the server records all response logs. If the user's problem is not resolved or additional support is required, personnel will follow up based on the recorded logs. This allows users to receive fast and efficient technical support 24 hours a day, 365 days a year, without waiting.

[0982] As a concrete example, consider the case where a user says "Wi-Fi is not connecting" to a smartphone. The device sends this voice to the server, and the speech recognition API converts it into text "Wi-Fi is not connecting." The generative AI analyzes this text and generates a response saying "Please try restarting the router." The text-to-speech AI converts this response into speech, and the device tells the user aloud, "Please try restarting the router."

[0983] An example of a prompt sentence is, "If a user says to their smartphone, 'I can't connect to Wi-Fi,' how would you respond?"

[0984] This system allows users to receive technical support quickly and efficiently.

[0985] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0986] Step 1:

[0987] A user speaks a question or problem into a smartphone. The device receives this voice input through a microphone. The input is voice data, and the output is voice data as is. A specific example of how this works is when a user says, "I can't connect to Wi-Fi."

[0988] Step 2:

[0989] The device sends the received voice data to the server. The server uses a voice recognition model to convert the voice data into text data. The input is voice data, and the output is text data. In concrete terms, the voice recognition API converts the voice saying "Wi-Fi is not connecting" into the text "Wi-Fi is not connecting."

[0990] Step 3:

[0991] The server passes the converted user input to the generative AI. The generative AI analyzes the input text and understands the user's intent. The input is text data, and the output is the analysis result. In concrete terms, the generative AI analyzes the text "Wi-Fi is not connecting" and determines that the problem is with the network connection.

[0992] Step 4:

[0993] The server uses generative AI to generate an appropriate response based on the analysis results. The input is the analysis results, and the output is the response text. The generative AI generates the response "Try restarting your router."

[0994] Step 5:

[0995] The generated text response is passed from the server to the speech AI, which converts the text to speech. The input is the response text, and the output is the voice data. In concrete terms, the speech synthesis API converts the text "Please try restarting the router" into voice data.

[0996] Step 6:

[0997] The server sends voice data to the terminal. The terminal plays the voice data and provides a response to the user. The input is voice data, and the output is voice output. In concrete terms, the terminal tells the user by voice, "Please try restarting the router."

[0998] Step 7:

[0999] The server records all response logs. If the user's problem is not resolved or additional support is required, personnel will follow up based on the recorded logs. The input is the response log, and the output is follow-up suggestions. In concrete terms, if the user inquires again saying "still can't connect," the server will refer to the log and suggest the next step, such as "check your network settings."

[1000] (Application example 2)

[1001] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1002] There is a demand for prompt and appropriate support for technical problems and questions that arise inside autonomous vehicles. However, with conventional systems, when a user faces a technical problem inside the vehicle, there are limited ways to respond immediately, making it difficult to provide efficient support. This leads to issues such as reduced convenience for users and a loss of satisfaction.

[1003] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1004] In this invention, the server includes a means for converting voice input into text using a voice recognition model, a means for generating a response based on the converted text input using a generative AI, a means for converting the generated text response into voice using a text-to-speech AI, a means for recording a response log, and a means for receiving voice input in the autonomous vehicle and outputting a voice response using a speaker in the vehicle, thereby enabling a user in the autonomous vehicle to receive prompt and appropriate support for technical issues or questions.

[1005] A "speech recognition model" is an algorithm or software for converting voice input into text data.

[1006] "Generative AI" is an artificial intelligence technology that generates appropriate responses based on input text data.

[1007] "Reading AI" refers to algorithms or software for converting text data into audio data.

[1008] A "response log" is data that records user input and the system's response to it.

[1009] An "autonomous vehicle" is a vehicle that drives autonomously using artificial intelligence and sensor technology.

[1010] A "speaker" is a device for outputting audio data as sound.

[1011] As an embodiment of the present invention, a technical support system in an autonomous vehicle is shown below. The system includes a speech recognition model, a generative AI, a voice-to-speech AI, a response log recording means, and a voice input and output means in the autonomous vehicle.

[1012] System Configuration

[1013] 1. Speech Recognition Model

[1014] A voice recognition model is an algorithm or software that converts a user's voice input into text data. Specifically, it uses a voice recognition API. It receives voice input through a microphone in the vehicle and converts it into text.

[1015] 2. Generative AI

[1016] Generative AI is an artificial intelligence technique that generates appropriate responses based on textual input. Specifically, it uses generative AI models to analyze text data and generate appropriate responses to user questions.

[1017] 3. Text-to-speech AI

[1018] Text-to-speech AI is an algorithm or software that converts the generated text response into audio data, specifically using a text-to-speech API, which then converts the generated text response into audio and outputs it to the user through the vehicle's speakers.

[1019] 4. Response logging methods

[1020] The response log recorder is a database that records user input and the system's response to that input, so that later follow-up by personnel may be required.

[1021] 5. Voice input and output means in autonomous vehicles

[1022] A microphone is used as the voice input means in an autonomous vehicle, and a speaker is used as the voice output means, allowing users to receive voice support in the vehicle.

[1023] System Operation

[1024] The server receives voice input from the vehicle's microphone and converts it to text using a speech recognition model. It then uses generative AI to analyze the text and generate an appropriate response. The generated text response is converted to audio using speech AI and output to the user through the vehicle's speakers. Response logs are recorded and follow-ups are provided by personnel as needed.

[1025] Examples

[1026] For example, if a user asks, "How do I turn up the air conditioner temperature?" the system would do the following:

[1027] 1. A microphone in the vehicle captures the user's voice.

[1028] 2. The speech recognition model converts the speech to text.

[1029] 3. Generative AI analyzes the text and generates an appropriate response.

[1030] 4. Text-to-speech AI converts the generated text response into audio.

[1031] 5. Speakers in the vehicle output a voice response to the user.

[1032] Examples of prompt statements

[1033] User: How do I turn up the heat on my air conditioner?

[1034] In this way, quick and appropriate voice responses can be provided to user questions within an autonomous vehicle.

[1035] The flow of the specific process in the application example 2 will be described with reference to FIG.

[1036] Step 1:

[1037] The user inputs voice into a microphone in the vehicle.

[1038] Input: User's voice

[1039] Output: Audio data

[1040] Specific action: The user says, "Tell me how to turn up the air conditioner temperature."

[1041] Step 2:

[1042] The server converts the voice data into text data using a voice recognition model.

[1043] Input: Audio data

[1044] Output: Text data

[1045] Specific operation: The speech recognition API analyzes the voice data and generates the text "Tell me how to turn up the air conditioner temperature."

[1046] Step 3:

[1047] The server uses generative AI to analyze the text data and generate an appropriate response.

[1048] Input: Text data

[1049] Output: Response text

[1050] Specific behavior: The generative AI model analyzes the text "Tell me how to turn up the temperature on the air conditioner" and generates the response text "To turn up the temperature on the air conditioner, select the 'Air Conditioning' menu on the central touchscreen and increase the temperature setting."

[1051] Step 4:

[1052] The server uses a text-to-speech AI to convert the response text into audio data.

[1053] Input: Response text

[1054] Output: Audio data

[1055] Specific operation: The speech synthesis API analyzes the response text and generates speech data saying, "To increase the temperature of the air conditioner, select the 'Air Conditioning' menu on the central touch screen and increase the temperature setting."

[1056] Step 5:

[1057] The server outputs the audio data to the user using a speaker in the vehicle.

[1058] Input: Audio data

[1059] Output: Voice response

[1060] Specific operation: The speaker in the vehicle plays audio data, informing the user, "To increase the air conditioning temperature, select the 'Air Conditioning' menu on the central touchscreen and increase the temperature setting."

[1061] Step 6:

[1062] The server logs the response.

[1063] Input: User voice input, generated response text

[1064] Output: Response log

[1065] What happens: The server stores the user's voice input and the generated response text in a database in case a later human follow-up is required.

[1066] In addition, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1067] "Example 1"

[1068] As one embodiment of the present invention, a system incorporating an emotion engine is provided. This system converts voice input into text using a voice recognition model, generates a response based on the text input using a generative AI, converts the generated text response into voice using a speech-to-speech AI, and records a response log. Furthermore, this system includes an emotion engine that recognizes the user's emotion from the voice input. Specifically, when a user expresses an emotion such as anger or joy by voice, the emotion engine recognizes the emotion, and the generative AI generates a response based on the emotion. For example, when a user shows an emotion of anger, the generative AI is adjusted to generate a more polite response. In addition, the emotion engine can adjust the priority of responses based on the user's emotion. For example, when a user shows a strong emotion of anger, the system is adjusted to prioritize processing of inquiries from that user. This makes it possible to provide more appropriate support according to the user's emotion.

[1069] "Example 2"

[1070] As one embodiment of the present invention, a system incorporating an emotion engine is provided. This system converts voice input into text using a voice recognition model, generates a response based on the text input using a generative AI, converts the generated text response into voice using a speech-to-speech AI, and records a response log. Furthermore, this system includes an emotion engine that recognizes the user's emotion from the voice input. Specifically, when a user expresses an emotion such as anger or joy by voice, the emotion engine recognizes the emotion, and the generative AI generates a response based on the emotion. For example, when a user shows an emotion of anger, the generative AI is adjusted to generate a more polite response. In addition, the emotion engine can adjust the priority of responses based on the user's emotion. For example, when a user shows a strong emotion of anger, the system is adjusted to prioritize processing of inquiries from that user. This makes it possible to provide more appropriate support according to the user's emotion.

[1071] The process flow of each embodiment will be described below.

[1072] "Example 1"

[1073] Step 1: Receive voice input from the user. This voice input is received through a smartphone microphone or similar.

[1074] Step 2: Convert the received voice input into text using WhisperAPI. WhisperAPI is a speech recognition model that has the ability to convert speech to text.

[1075] Step 3: Recognize user emotions from the textual input using the emotion engine. The emotion engine has the ability to recognize user emotions (e.g., anger, joy) from the text.

[1076] Step 4: Generative AI generates a response based on the recognized emotion. Generative AI has the ability to generate an appropriate response based on the textual input and the recognized emotion.

[1077] Step 5: The generated text response is converted into speech using ReadSpeaker, a speech-to-speech AI capable of converting text to speech.

[1078] Step 6: Provide a spoken response to the user and simultaneously record the response log, which can be used for later follow-up and system improvement.

[1079] Example 1

[1080] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1081] Conventional technical support systems provide uniform responses without considering the user's feelings, which can lead to low user satisfaction. In addition, there are often waiting times, and responses are delayed even in situations where a quick response is required. This increases user stress and reduces support efficiency.

[1082] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1083] In this invention, the server includes a means for converting a voice input into text using a voice recognition model, a means for generating a response based on the input converted into text using generative artificial intelligence, a means for converting the generated text response into voice using a text-to-speech artificial intelligence, a means for recording a response log, a means for recognizing a user's emotion from the voice input using an emotion engine and adjusting the response based on the emotion, and a means for adjusting the priority of responses based on the user's emotion, thereby making it possible to quickly provide an appropriate response according to the user's emotion.

[1084] A "speech recognition model" is an algorithm or software for analyzing voice data and converting it into text data.

[1085] "Generative artificial intelligence" is an artificial intelligence technology that analyzes input data and generates appropriate responses or results.

[1086] "Read-to-speech artificial intelligence" is an artificial intelligence technology that converts text data into audio data and outputs it as audio.

[1087] A "response log" is data that records the content of inquiries from users and the responses generated by the system.

[1088] An "emotion engine" is a technology that analyzes user emotions from voice input and text data and recognizes those emotions.

[1089] "Means for adjusting priority" is a function for changing the order and importance of responses based on the user's emotions and the content of the inquiry.

[1090] This invention is a system for providing technical support for smartphones. Voice input from a user is converted to text by a voice recognition model. A voice recognition API is used as this voice recognition model. The text input is analyzed by generative artificial intelligence, and an appropriate response is generated. The generated text response is converted to voice by a speech-to-speech artificial intelligence, and output to the user. A voice synthesis API is used as this speech-to-speech artificial intelligence. In addition, the system records a response log, and when follow-up by a human is required, a proposal for a solution is made based on this. This enables voice support 24 hours a day, 365 days a year with no waiting time, reducing the burden on users and providing efficient support.

[1091] Furthermore, this system combines an emotion engine. The emotion engine recognizes the user's emotion from the voice input, and the generative AI generates a response based on that emotion. For example, if the user expresses anger, the generative AI is adjusted to generate a more polite response. The emotion engine can also adjust the priority of responses based on the user's emotion. For example, if a user expresses strong anger, the system is adjusted to prioritize inquiries from that user. This makes it possible to provide more appropriate support according to the user's emotion.

[1092] As a concrete example, consider the case where a user makes a voice inquiry to their smartphone saying, "Wi-Fi won't connect." The device converts this voice into text using a speech recognition API, generating the text "Wi-Fi won't connect." The server passes this text to a generative artificial intelligence system, which generates an appropriate response. For example, a response might be generated saying, "Please try restarting your Wi-Fi router." The speech synthesis API converts this response into speech, and the device outputs the response to the user in voice.

[1093] An example of a prompt sentence is, "The user spoke to his smartphone saying, 'I can't connect to Wi-Fi.' Please convert this speech into text, generate an appropriate response, and reply by voice."

[1094] This system allows users to receive voice support 24 hours a day, 365 days a year, without waiting, providing efficient support.

[1095] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1096] Step 1:

[1097] A user makes a voice inquiry into a smartphone. For example, they say, "I can't connect to Wi-Fi." The input is the user's voice data, and the output is the voice data sent to the device.

[1098] Step 2:

[1099] The device receives the user's voice input and converts the voice to text using a voice recognition API. Specifically, the device sends voice data to the voice recognition API, which analyzes the voice data and generates the text "Wi-Fi is not connected." The input is voice data, and the output is text data.

[1100] Step 3:

[1101] The server receives the text sent from the terminal. The server passes this text to the generative AI for analysis. The generative AI understands the content of the user's inquiry and creates a prompt sentence to generate an appropriate response. The input is text data, and the output is a prompt sentence.

[1102] Step 4:

[1103] The generative AI generates an appropriate response based on the prompt sentence. For example, a response such as "Please try restarting your Wi-Fi router" is generated. The server stores this response in text format. The input is the prompt sentence, and the output is the response text.

[1104] Step 5:

[1105] The server sends the generated text response to the device. The device uses a speech synthesis API to convert the text response into speech. Specifically, the device passes the text data to the speech synthesis API, which generates speech data. The input is text data, and the output is speech data.

[1106] Step 6:

[1107] The device outputs the generated voice data to the user. The user can hear the voice response, "Please try restarting your Wi-Fi router." The input is the voice data, and the output is the voice response to the user.

[1108] Step 7:

[1109] The server records all response logs, including the user's voice input, the transcribed input, and the generated response. If necessary, a human follow-up is performed and a resolution suggestion is provided. The input is the data, and the output is the recorded response log.

[1110] Step 8:

[1111] The server uses an emotion engine to recognize user emotions from voice input. For example, if a user expresses an emotion of anger, the emotion engine recognizes the emotion and adjusts the generative artificial intelligence to generate a more polite response. The emotion engine can also adjust the priority of responses. User queries that express a strong emotion of anger are given priority. The input is voice data and text data, and the output is the adjusted response and priority.

[1112] (Application example 1)

[1113] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1114] In autonomous vehicles, there is a need to provide prompt and appropriate support when users ask questions about how to operate the vehicle or troubleshooting. However, conventional systems are unable to respond in a way that takes into account the user's emotions, which can lead to a decrease in user satisfaction. In addition, the accuracy and speed of generating responses based on voice input are also issues.

[1115] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1116] In this invention, the server includes a means for converting voice input into text using a voice recognition model, a means for generating a response based on the text input using a generative AI, a means for converting the generated text response into voice using a text-to-speech AI, a means for recording a response log, a means for recognizing a user's emotion from the voice input using an emotion engine, and a means for adjusting the tone and priority of the response based on the user's emotion. This enables prompt and appropriate support according to the user's emotion.

[1117] A "speech recognition model" is an algorithm or software for converting voice input into text.

[1118] "Generative AI" is an artificial intelligence technology that generates appropriate responses based on text input.

[1119] "Reading AI" refers to algorithms or software that converts text into speech and outputs it.

[1120] A "response log" is a database that records user inquiries and the system's responses to them.

[1121] An "emotion engine" is an algorithm or software that analyzes and recognizes a user's emotions from voice input.

[1122] "Tone" refers to the vocal cadence or nuance of the response produced.

[1123] "Priority" is a criterion for determining the order in which multiple inquiries or tasks are processed.

[1124] The system for implementing this invention is composed of a combination of a speech recognition model, a generative AI, a reading AI, and an emotion engine. Below, we will explain the details of each component and how they work together.

[1125] Speech Recognition Model

[1126] The server converts the user's voice input into text using a speech recognition model, which is provided by a speech recognition API, that analyzes the speech data and generates corresponding text data.

[1127] Generative AI

[1128] The server generates responses based on the textual input using a generative AI, which uses advanced natural language processing models such as GPT-3, and leverages pre-trained data to generate appropriate responses to user questions.

[1129] Reading AI

[1130] The server uses a voice AI to convert the generated text response into voice. A voice synthesis engine is used as the voice AI. The voice synthesis engine converts the text data into natural voice and outputs it to the user.

[1131] Emotion Engine

[1132] The server recognizes the user's emotions from the voice input using an emotion engine. The emotion engine analyzes the tone, pitch, and rate of the voice data to identify the user's emotional state. If the user is expressing an emotion such as anger or joy, a response is generated according to that emotion.

[1133] Adjusting the tone and priority of your response

[1134] The server adjusts the tone and priority of the responses generated by the generative AI based on the user's emotions recognized by the emotion engine. For example, if the user expresses anger, the generative AI is adjusted to generate more polite responses. In addition, inquiries from users who show strong emotions are given priority.

[1135] Examples

[1136] Consider a case where a user asks how to operate the air conditioner in an autonomous vehicle. When the user asks verbally, "How do I adjust the temperature of the air conditioner?", a speech recognition model converts this speech into text. A generative AI then generates a response based on this text: "To adjust the temperature of the air conditioner, tap the temperature setting on the center display." The emotion engine analyzes the user's emotions, and if there is no particular change in emotion, the reading AI vocalizes this response in a normal tone and provides it to the user.

[1137] Examples of prompt statements

[1138] User sentiment: Neutral

[1139] User Question: How do I adjust the temperature on my air conditioner?

[1140] Generate an appropriate response.

[1141] In this way, it is possible to provide prompt and appropriate support according to the user's emotions.

[1142] The flow of the specific process in the application example 1 will be described with reference to FIG.

[1143] Step 1:

[1144] The user performs voice input. For example, the user may ask a question in an autonomous vehicle, such as "How do I adjust the temperature of the air conditioner?" This voice data is sent to the server.

[1145] Step 2:

[1146] The server converts the voice input to text using a voice recognition model. The voice recognition API analyzes the voice data and generates the corresponding text data: "How do I adjust the temperature of the air conditioner?" The input is voice data, and the output is text data.

[1147] Step 3:

[1148] The server uses an emotion engine to recognize the user's emotion from the voice input. The emotion engine analyzes the tone, pitch, and speed of the voice data and identifies the user's emotional state as "neutral." The input is the voice data, and the output is the emotional state.

[1149] Step 4:

[1150] The server uses a generative AI to generate a response based on the textual input. The generative AI (e.g., GPT-3) uses the prompt sentence to generate an appropriate response: "To adjust the temperature of the air conditioner, tap the temperature setting on the center display." The input is the text data and the emotional state, and the output is the response text.

[1151] Step 5:

[1152] The server adjusts the tone and priority of the response generated by the generative AI based on the user's emotion recognized by the emotion engine. For example, because the user's emotion is "normal," a response is generated in a normal tone. The input is the response text and the emotional state, and the output is the adjusted response text.

[1153] Step 6:

[1154] The server uses a voice-to-speech AI to convert the generated text response into voice. The voice synthesis engine converts the text data into natural speech and outputs it to the user. The input is the adjusted response text, and the output is the voice data.

[1155] Step 7:

[1156] The server records the response log. User queries and the system's responses to them are stored in a database. Inputs are voice data, text data, emotional state, and response text, and output is the recorded log data.

[1157] In this way, prompt and appropriate support can be provided according to the user's feelings.

[1158] Example 2

[1159] Next, a second embodiment of the second form will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1160] Conventional smartphone technical support systems convert users' voice input into text, generate responses using generative AI, and convert them into voice using text-to-speech AI, but they have a problem in that they cannot adjust responses to take the user's emotions into account. This can lead to lower user satisfaction and make it difficult to provide efficient support.

[1161] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1162] In this invention, the server includes a means for converting a voice input into text using a voice recognition model, a means for generating a response based on the converted text input using generative artificial intelligence, a means for converting the generated text response into voice using a text-to-speech artificial intelligence, a means for recording a response log, a means for recognizing a user's emotion from the voice input using an emotion engine, and a means for adjusting a response based on the user's emotion, thereby enabling more appropriate support according to the user's emotion.

[1163] A "speech recognition model" is an algorithm or software for converting voice input into text data.

[1164] "Generative artificial intelligence" is an artificial intelligence technology that analyzes input text data and generates appropriate responses.

[1165] "Read-to-speech artificial intelligence" is a technology that converts text data into voice data and provides a voice response to the user.

[1166] A "response log" is data for recording inquiries from users and responses to those inquiries.

[1167] An "emotion engine" is a technology for recognizing a user's emotions from voice input and adjusting responses based on those emotions.

[1168] "Text conversion" is the process of converting audio data into text data.

[1169] "Analysis" is the process of analyzing input data and understanding its meaning and intent.

[1170] "Adjustment" is the process of changing responses or behavior in response to specific conditions or situations.

[1171] This invention is a system for providing technical support for smartphones, which converts voice input from the user into text, analyzes the text to generate an appropriate response, and returns it to the user as voice. It also has a function to recognize the user's emotions and provide a response according to those emotions.

[1172] System Configuration

[1173] 1. Voice to text conversion

[1174] Hardware / Software: The server converts voice input to text using a speech recognition model, which is provided by a speech recognition API.

[1175] Example: When a user says, "My smartphone battery is running out quickly, what should I do?" the server converts this speech into text.

[1176] 2. Text Analysis and Response Generation

[1177] Hardware / Software: The server uses generative artificial intelligence (e.g., a generative AI model) to parse the textual input and generate an appropriate response.

[1178] Example: A server parses the text "My phone's battery is draining quickly, what should I do?" and generates a response that reads "To extend battery life, you can close apps running in the background and adjust the screen brightness."

[1179] 3. Audio Responses

[1180] Hardware / Software: The server converts the generated text response into speech using a speech-to-text AI, which uses a text-to-speech API.

[1181] Example: The server converts the text "To extend battery life, you can close background apps and adjust your screen brightness." into speech, which the device then outputs to the user.

[1182] 4. Response logging

[1183] Hardware / Software: The server uses a database to log all responses.

[1184] Example: The server stores the user's inquiry and the generated response in a database. For example, the following log is recorded: "Inquiry: My smartphone's battery runs out quickly. What should I do?" "Response: To extend the battery life, it is effective to close apps running in the background and adjust the screen brightness."

[1185] 5. Emotion recognition and response adjustment

[1186] Hardware / Software: The server uses an emotion engine to recognize emotions from the user's voice.

[1187] Example: If a user expresses anger, the server uses the emotion engine to recognize the emotion. For example, if the user says in an angry tone, "Why is my battery running out so fast?", the generative AI generates a more polite response such as, "I'm sorry. We will give you a solution to your battery problem right away." The system is also adjusted so that inquiries from users who express strong emotions are given priority.

[1188] Examples of prompt statements

[1189] "My smartphone battery runs out quickly, what should I do?"

[1190] "The app keeps crashing, what should I do?"

[1191] "I can't connect to Wi-Fi, what should I do?"

[1192] This system allows users to receive voice support 24 hours a day, 365 days a year, without waiting, providing efficient support.

[1193] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1194] Step 1:

[1195] Receiving audio input

[1196] How it works: A user speaks a query into their smartphone.

[1197] Input: The user's voice input.

[1198] Output: The audio data.

[1199] Specific behavior: A user says, "My smartphone battery is running out quickly. What should I do?" The device receives this voice through the microphone.

[1200] Step 2:

[1201] Voice to text conversion

[1202] How it works: The server converts voice input into text using a speech recognition model.

[1203] Input: Audio data.

[1204] Output: Text data.

[1205] What happens: The server calls a speech recognition API to convert the voice data into text such as, "My smartphone's battery is running out quickly. What should I do?"

[1206] Step 3:

[1207] Text analysis and response generation

[1208] How it works: The server uses generative artificial intelligence to parse the textual input and generate an appropriate response.

[1209] Input: Text data.

[1210] Output: The response text.

[1211] How it works: The server inputs the text "My smartphone's battery is draining quickly, what should I do?" into the generative AI model, which generates a response saying "To extend battery life, it is effective to close apps running in the background and adjust the screen brightness."

[1212] Step 4:

[1213] Spoken responses

[1214] How it works: The server uses text-to-speech artificial intelligence to convert the generated text response into speech.

[1215] Input: The response text.

[1216] Output: The audio data.

[1217] Specific operation: The server calls the speech synthesis API to convert the text "To extend battery life, it is effective to close apps running in the background and adjust the screen brightness." into speech, and the device outputs the speech to the user.

[1218] Step 5:

[1219] Response logging

[1220] Behavior: The server logs all responses.

[1221] Input: The user's query and the generated response.

[1222] Output: The response log data.

[1223] Specific operation: The server records the following log in the database: "Query: My smartphone's battery is draining quickly. What should I do?" "Response: To extend battery life, it is effective to close apps running in the background and adjust the screen brightness."

[1224] Step 6:

[1225] Emotion recognition and response adjustment

[1226] How it works: The server uses an emotion engine to recognize emotions from the user's voice and tailor its response.

[1227] Input: The user's voice data.

[1228] Output: The adjusted response text.

[1229] Specific behavior: If a user expresses anger, the server uses the emotion engine to recognize the emotion. For example, if a user says in an angry tone, "Why is my battery running out so fast?", the generative AI generates a more polite response such as, "I'm sorry. We will give you a solution to your battery problem right away." The system is also adjusted so that inquiries from users who express strong emotions are given priority.

[1230] (Application example 2)

[1231] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1232] In autonomous vehicles, when a user asks a question about vehicle operation or navigation by voice while driving, it is necessary to provide a quick and appropriate response. However, conventional systems are unable to respond taking into account the user's emotions, which may reduce user satisfaction. In addition, it is difficult to respond quickly to questions with high urgency because it is not possible to adjust the priority of responses based on emotions. To solve these issues, a system is needed that recognizes the user's emotions, generates responses based on them, and provides responses with an appropriate tone and priority.

[1233] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1234] In this invention, the server includes a means for converting a voice input into text using a voice recognition model, a means for generating a response based on the input converted into text using a generative AI, a means for converting the generated text response into voice using a text-to-speech AI, a means for recording a response log, a means for recognizing a user's emotion from the voice input using an emotion engine, a means for adjusting the tone and priority of the response based on the user's emotion, and a means for being installed in the infotainment system of the autonomous vehicle. This enables a quick and appropriate response that takes the user's emotion into consideration, thereby improving user satisfaction.

[1235] A "speech recognition model" is an algorithm or software for converting voice input into text.

[1236] "Generative AI" is an artificial intelligence technology that generates appropriate responses based on input text data.

[1237] "Read-to-speech AI" is an artificial intelligence technology that converts text data into audio and outputs it.

[1238] A "response log" is a database that records user inquiries and the system's responses to them.

[1239] An "emotion engine" is a technology for recognizing a user's emotions from voice input and adjusting responses based on those emotions.

[1240] An "infotainment system" is a system that provides information and entertainment within a vehicle.

[1241] An "autonomous vehicle" is a vehicle that drives itself using artificial intelligence and sensor technology.

[1242] As an embodiment of the present invention, a program installed in the infotainment system of an autonomous vehicle will be described. The server uses a speech recognition model, generative AI, text-to-speech AI, and emotion engine to generate a response based on the user's voice input, and outputs it by voice.

[1243] First, the user speaks to a vehicle operation or navigation question. The voice input is converted to text using a speech recognition model, which uses a speech recognition API. The text input is then analyzed by an emotion engine to recognize the user's emotion. The emotion engine recognizes the user's emotion from the voice input and adjusts the tone and priority of the response based on that emotion.

[1244] A generative AI model is then used to generate an appropriate response based on the textual input and the recognized emotion. The response is then converted to speech by a text-to-speech AI, which uses a text-to-speech API. The generated voice response is then output to the user through the infotainment system.

[1245] In addition, response logs are recorded on the server and can be viewed later, making it possible to track user inquiry history and response content, and to have personnel follow up if necessary.

[1246] As a concrete example, consider a user asking, "Where is the next gas station?" In this case, the emotion engine recognizes the user's impatience, and the generative AI generates a quick and polite response, such as, "The next gas station is 3 kilometers away on the right. Don't worry."

[1247] Example prompt:

[1248] User Question: Where is the next gas station?

[1249] User Sentiment: Impatience

[1250] Appropriate response:

[1251] In this way, a quick and appropriate response that takes the user's feelings into consideration is possible, thereby improving user satisfaction.

[1252] The flow of the specific process in the application example 2 will be described with reference to FIG.

[1253] Step 1:

[1254] The user asks questions about vehicle operation and navigation by voice.

[1255] Input: User's voice input

[1256] Output: Audio data

[1257] Specific Action: A user asks a question to the infotainment system of an autonomous vehicle.

[1258] Step 2:

[1259] The server converts the voice input into text using a speech recognition model.

[1260] Input: Audio data

[1261] Output: Text data

[1262] Specific operation: The speech recognition API analyzes the speech data and generates corresponding text data.

[1263] Step 3:

[1264] The server uses an emotion engine to recognize the user's emotion from the text data.

[1265] Input: Text data

[1266] Output: Emotion data

[1267] Specific behavior: The emotion engine analyzes text data and recognizes the user's emotions (e.g., impatience, anger, joy).

[1268] Step 4:

[1269] The server uses generative AI to generate a response based on text data and emotion data.

[1270] Input: Text data, emotion data

[1271] Output: Response text

[1272] Specific operation: The generative AI model generates an appropriate response text based on text data and emotion data.

[1273] Step 5:

[1274] The server uses text-to-speech AI to convert the response text into speech.

[1275] Input: Response text

[1276] Output: Audio data

[1277] Specific operation: The speech synthesis API analyzes the response text and generates corresponding speech data.

[1278] Step 6:

[1279] The server outputs the generated voice data to the user through the infotainment system.

[1280] Input: Audio data

[1281] Output: A voice response to the user

[1282] Specific Action: The infotainment system plays the audio data and provides a response to the user.

[1283] Step 7:

[1284] The server logs the response.

[1285] Input: Text data, emotion data, response text

[1286] Output: Response log

[1287] Specific behavior: The server records the user's question, the recognized emotion, and the generated response in a database.

[1288] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input for the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1289] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1290] Another example of generative AI is Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) are mentioned.

[1291] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1292] [Fourth embodiment]

[1293] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1294] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1295] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[1296] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. In addition, the microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1297] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[1298] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[1299] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[1300] The control target 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, legs, etc. The posture and behavior of the robot 414 are controlled by controlling the motors of the arms, hands, legs, etc. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1301] Fig. 8 shows an example of main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1302] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1303] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1304] In the robot 414, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1305] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1306] "Example 1"

[1307] The present invention is a system for providing technical support for smartphones. Voice input from a user is converted to text by a voice recognition model. For example, WhisperAPI is used as this voice recognition model. The converted text input is analyzed by a generative AI, and an appropriate response is generated. The generated text response is converted to voice by a text-to-speech AI and output to the user. For example, ReadSpeaker is used as this text-to-speech AI. In addition, the system records a response log, and when follow-up by personnel is required, a proposal for a solution is made based on this. This enables voice response 24 hours a day, 365 days a year without waiting time, reducing the burden on users and providing efficient support.

[1308] "Example 2"

[1309] The present invention is a system for providing technical support for smartphones. Voice input from a user is converted to text by a voice recognition model. For example, WhisperAPI is used as this voice recognition model. The converted text input is analyzed by a generative AI, and an appropriate response is generated. The generated text response is converted to voice by a text-to-speech AI and output to the user. For example, ReadSpeaker is used as this text-to-speech AI. In addition, the system records a response log, and when follow-up by personnel is required, a proposal for a solution is made based on this. This enables voice response 24 hours a day, 365 days a year without waiting time, reducing the burden on users and providing efficient support.

[1310] The process flow of each embodiment will be described below.

[1311] "Example 1"

[1312] Step 1: Receive voice input from the user. This voice input is done through the smartphone microphone.

[1313] Step 2: The received voice input is converted into text using a voice recognition model, such as WhisperAPI.

[1314] Step 3: The textual input is analyzed by a generative AI to generate an appropriate response. This generative AI has the ability to understand the user's problem and suggest appropriate solutions.

[1315] Step 4: The generated text response is converted to audio by a speech AI and output to the user. For example, ReadSpeaker is used as the speech AI. Step 5: The system records the response log. This allows the system to make suggestions for a solution based on the log, even if follow-up by a human is required.

[1316] Example 1

[1317] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1318] Conventional smartphone technical support systems often required users to wait a long time to resolve their issues, making it difficult to provide efficient support. In addition, the process of converting voice input into text, generating an appropriate response, and returning it by voice was sometimes inconsistent, which reduced the user experience.

[1319] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1320] In this invention, the server includes a means for converting voice input into text using a voice recognition model, a means for generating a response based on the converted text input using generative artificial intelligence, a means for converting the generated text response into voice using speech-to-speech artificial intelligence, a means for recording a response log, and a means for following up with personnel, thereby enabling efficient support 24 hours a day, 365 days a year without waiting times.

[1321] A "speech recognition model" is an algorithm or system for analyzing speech data and converting it into corresponding text data.

[1322] "Generative artificial intelligence" is an artificial intelligence technology that analyzes input data and generates appropriate responses or information.

[1323] "Read-to-speech artificial intelligence" is an artificial intelligence technology that converts text data into audio data and outputs it as audio.

[1324] A "response log" is a database or file that maintains a record of user interactions and the responses generated.

[1325] "Follow-up" is the process by which personnel respond when a user's problem is not resolved or additional support is required.

[1326] A "voice recognition API" is an application programming interface for converting voice data into text data.

[1327] A "speech synthesis API" is an application programming interface for converting text data into speech data.

[1328] This invention is a system for providing technical support for smartphones. Voice input from a user is converted to text by a voice recognition model. A voice recognition API is used as this voice recognition model. The text input is analyzed by a generative artificial intelligence, and an appropriate response is generated. The generated text response is converted to voice by a voice-to-speech artificial intelligence, and output to the user. A voice synthesis API is used as this voice-to-speech artificial intelligence. In addition, the system records a response log, and when follow-up by personnel is required, a proposal for a solution is made based on this. This enables voice response 24 hours a day, 365 days a year with no waiting time, reducing the burden on users and providing efficient support.

[1329] Specifically, a user speaks a question or problem into a smartphone. For example, they might say, "My smartphone's battery is running low." The device records this voice and sends the data to a server. The server uses a speech recognition API to convert the voice data into text. Specifically, it analyzes the voice signal and generates a corresponding string of characters.

[1330] The server then passes the converted user input to a generative AI, which uses natural language processing techniques, for example, to analyze the user's question or problem and generate an appropriate response, such as "To reduce battery drain, we recommend closing apps running in the background."

[1331] The generated text response is passed from the server to the speech-to-speech AI, which converts the text into speech and sends it to the device. The device then plays the speech to the user. Specifically, it generates an audio file and plays it through the smartphone speaker.

[1332] In addition, the server records a log of all responses. If the user's problem is not resolved or additional support is required, the server can use this log to send a follow-up call to the appropriate technician. For example, if a user says, "I can't connect to Wi-Fi," the server will check the log and escalate the call to the appropriate technician.

[1333] An example of a prompt sentence is, "What should you do if your smartphone battery runs out quickly?" By inputting this prompt sentence into a generative artificial intelligence system, an appropriate response is generated.

[1334] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1335] Step 1:

[1336] A user speaks a question or problem into a smartphone. For example, they might say, "My smartphone's battery is running low." The device records this voice and sends the data to a server. The input is the voice data, and the output is the voice data sent to the server.

[1337] Step 2:

[1338] The server uses a speech recognition API to convert this voice data into text: it analyzes the audio signal and generates a corresponding string of characters. The input is the voice data, and the output is the text data.

[1339] Step 3:

[1340] The server passes the textualized user input to the generative AI. The generative AI uses natural language processing techniques to analyze the user's question or problem and generate an appropriate response. For example, it generates a response such as "To reduce battery drain, we recommend closing apps running in the background." The input is text data, and the output is the response text.

[1341] Step 4:

[1342] The generated text response is passed from the server to the speech AI, which converts the text into speech and sends it to the terminal. Specifically, it generates an audio file and plays it through the smartphone speaker. The input is the response text, and the output is the audio data.

[1343] Step 5:

[1344] The server records all response logs. If the user's problem is not resolved or additional support is required, the server uses these logs to send personnel to follow up. For example, if a user says "Wi-Fi is not working," the server checks the logs and escalates the issue to the appropriate technician. The input is the response log, and the output is the information for follow-up.

[1345] (Application example 1)

[1346] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1347] As autonomous vehicles become more widespread, users need to receive support for vehicle operation and troubleshooting quickly and efficiently. However, conventional systems require users to spend a lot of time and effort to solve problems, making it difficult to respond quickly, especially in emergencies.

[1348] The identification process by the identification processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means. In this invention, the server includes a means for converting a voice input into text using a voice recognition model, a means for generating a response based on the input converted into text using a generative AI, a means for converting the generated text response into voice using a text-to-speech AI, a means for recording a response log, and a means for analyzing a voice input of a user in an autonomous vehicle and providing support regarding vehicle operation and troubleshooting. This allows the user to receive support quickly and efficiently using voice in an autonomous vehicle.

[1349] A "speech recognition model" is an algorithm or software for converting voice input into text data.

[1350] "Generative AI" is an artificial intelligence technology that analyzes input text data and generates appropriate responses.

[1351] "Reading AI" refers to algorithms and software for converting text data into speech.

[1352] A "response log" is a database or file that stores records of user interactions.

[1353] An "autonomous vehicle" is a vehicle that uses artificial intelligence and sensor technology to drive autonomously.

[1354] "User's voice input" refers to voice data generated by a user through a voice input device such as a microphone.

[1355] "Vehicle operation" refers to the act of controlling various vehicle functions (e.g., air conditioning, navigation, audio system, etc.).

[1356] "Troubleshooting" is the process of identifying and resolving problems with a system or device.

[1357] "Support" is a service that provides appropriate solutions and information to problems or questions faced by users.

[1358] A system for implementing the invention includes a speech recognition model, a generative AI, a text-to-speech AI, and a means for recording a response log, as well as a means for analyzing a user's voice input in an autonomous vehicle and providing support for operating and troubleshooting the vehicle.

[1359] System Configuration

[1360] 1. Speech recognition model:

[1361] For the speech recognition model, we use a speech recognition API, which is an algorithm or software that converts a user's voice input into text data.

[1362] 2. Generative AI:

[1363] As a generative AI, we use a generative AI model, which is an artificial intelligence technology that analyzes textual input and generates an appropriate response.

[1364] 3. Text-to-speech AI:

[1365] For speech-to-speech AI, we use a text-to-speech engine, which is an algorithm or software that converts generated text responses into speech.

[1366] 4. Response log:

[1367] A response log is a database or file that stores a record of user interactions so that they can be viewed at a later time.

[1368] 5. Support measures in autonomous vehicles:

[1369] This includes a means for analyzing a user's voice input in an autonomous vehicle and providing assistance regarding vehicle operation and troubleshooting.

[1370] System Operation

[1371] When a user speaks to the server while in an autonomous vehicle, the server converts the speech into text data using a speech recognition model. Next, a generative AI analyzes the text data and generates an appropriate response. The generated response is converted into speech by a text-to-speech AI and provided to the user. Response logs are recorded, and follow-up is provided by personnel as necessary.

[1372] Examples

[1373] For example, consider a user asking how to operate the air conditioning in a self-driving vehicle.

[1374] User speaks: "How do I turn down the air conditioner?"

[1375] Example prompt: "How can I turn down the air conditioner?"

[1376] When this prompt sentence is input to a generative AI, the AI ​​generates the following response:

[1377] Generated response: "To turn down the air conditioning temperature, press the temperature control button located on the center dashboard. To turn down the temperature, press the minus button."

[1378] The response is then converted into audio by a speech-to-text AI and provided to the user, allowing them to receive support quickly and efficiently.

[1379] The flow of the specific process in the application example 1 will be described with reference to FIG.

[1380] Step 1:

[1381] A user performs voice input in an autonomous vehicle. The user's voice input is transmitted to a server through a microphone in the vehicle. The input data is in the form of voice data.

[1382] Step 2:

[1383] The server converts the voice input into text data using a voice recognition model. Specifically, it uses a voice recognition API to analyze the voice data and generate corresponding text data. The input is voice data, and the output is text data.

[1384] Step 3:

[1385] The server uses generative AI to generate a response based on the textual input. It uses a generative AI model to analyze the text data and generate an appropriate response. The input is the text data and the output is the response text.

[1386] Step 4:

[1387] The server converts the generated text response using the speech AI into voice data. It uses a speech synthesis engine to convert the text data into voice data. The input is the response text, and the output is the voice data.

[1388] Step 5:

[1389] The server provides the generated voice data to the user, specifically by playing a voice response through the vehicle's speakers. The input is the voice data and the output is the speech heard by the user.

[1390] Step 6:

[1391] The server records the response log. It saves the user's voice input, the converted text data, the generated response text, and the voice data in a database. This makes it possible to refer to them later. The input is various data, and the output is the recorded log data.

[1392] Step 7:

[1393] If necessary, the server will send a human to follow up. If the problem is not resolved or additional support is required based on the response log, a specialist staff member will respond. The input is the response log, and the output is the follow-up support.

[1394] Example 2

[1395] Next, a second embodiment of the second embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1396] Traditional technical support systems often required users to wait for long periods of time to resolve their issues, and the problem was that they did not provide efficient support. In addition, there was a lack of a system that could consistently convert voice input into text, generate an appropriate response, and return it by voice. Furthermore, there was insufficient follow-up when the user's problem was not resolved.

[1397] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1398] In this invention, the server includes a means for converting voice input to text using a speech recognition model, a means for generating a response based on the converted text input using generative artificial intelligence, a means for converting the generated text response to voice using text-to-speech artificial intelligence, a means for recording a response log, and a means for performing follow-up by personnel, thereby enabling users to receive fast and efficient technical support 24 hours a day, 365 days a year without waiting times.

[1399] A "speech recognition model" is an algorithm or software for converting voice data into text data.

[1400] "Generative artificial intelligence" is an artificial intelligence technology that analyzes input data and generates appropriate responses or information.

[1401] "Read-to-speech artificial intelligence" refers to algorithms and software for converting text data into audio data.

[1402] A "response log" is a database or file that records user interactions and the responses generated.

[1403] "Follow-up" is the process for providing additional support if the user's issue is not resolved.

[1404] This invention is a technical support system using a smartphone, which converts voice input from a user into text, generates an appropriate response using generative artificial intelligence, and returns the response by voice using a text-to-speech artificial intelligence. A specific embodiment of this system will be described below.

[1405] First, the user speaks a question or problem into the smartphone. The device receives this voice input through the microphone and sends it to the server. The server uses a speech recognition model to convert the voice data into text data. For example, a speech recognition API is used as this speech recognition model.

[1406] The server then passes the converted user input to a generative AI, which analyzes the input text and understands the user's intent. For example, a model using natural language processing technology is used as the generative AI. Based on the analysis results, the generative AI generates an appropriate response.

[1407] The generated text response is passed from the server to a voice-to-speech AI, which converts the text into speech. For example, a voice synthesis API is used as the voice-to-speech AI. The converted voice data is sent to the terminal, which plays the voice data and provides the response to the user.

[1408] In addition, the server records all response logs. If the user's problem is not resolved or additional support is required, personnel will follow up based on the recorded logs. This allows users to receive fast and efficient technical support 24 hours a day, 365 days a year, without waiting.

[1409] As a concrete example, consider the case where a user says "Wi-Fi is not connecting" to a smartphone. The device sends this voice to the server, and the speech recognition API converts it into text "Wi-Fi is not connecting." The generative AI analyzes this text and generates a response saying "Please try restarting the router." The text-to-speech AI converts this response into speech, and the device tells the user aloud, "Please try restarting the router."

[1410] An example of a prompt sentence is, "If a user says to their smartphone, 'I can't connect to Wi-Fi,' how would you respond?"

[1411] This system allows users to receive technical support quickly and efficiently.

[1412] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1413] Step 1:

[1414] A user speaks a question or problem into a smartphone. The device receives this voice input through a microphone. The input is voice data, and the output is voice data as is. A specific example of how this works is when a user says, "I can't connect to Wi-Fi."

[1415] Step 2:

[1416] The device sends the received voice data to the server. The server uses a voice recognition model to convert the voice data into text data. The input is voice data, and the output is text data. In concrete terms, the voice recognition API converts the voice saying "Wi-Fi is not connecting" into the text "Wi-Fi is not connecting."

[1417] Step 3:

[1418] The server passes the converted user input to the generative AI. The generative AI analyzes the input text and understands the user's intent. The input is text data, and the output is the analysis result. In concrete terms, the generative AI analyzes the text "Wi-Fi is not connecting" and determines that the problem is with the network connection.

[1419] Step 4:

[1420] The server uses generative AI to generate an appropriate response based on the analysis results. The input is the analysis results, and the output is the response text. The generative AI generates the response "Try restarting your router."

[1421] Step 5:

[1422] The generated text response is passed from the server to the speech AI, which converts the text to speech. The input is the response text, and the output is the voice data. In concrete terms, the speech synthesis API converts the text "Please try restarting the router" into voice data.

[1423] Step 6:

[1424] The server sends voice data to the terminal. The terminal plays the voice data and provides a response to the user. The input is voice data, and the output is voice output. In concrete terms, the terminal tells the user by voice, "Please try restarting the router."

[1425] Step 7:

[1426] The server records all response logs. If the user's problem is not resolved or additional support is required, personnel will follow up based on the recorded logs. The input is the response log, and the output is follow-up suggestions. In concrete terms, if the user inquires again saying "still can't connect," the server will refer to the log and suggest the next step, such as "check your network settings."

[1427] (Application example 2)

[1428] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1429] There is a demand for prompt and appropriate support for technical problems and questions that arise inside autonomous vehicles. However, with conventional systems, when a user faces a technical problem inside the vehicle, there are limited ways to respond immediately, making it difficult to provide efficient support. This leads to issues such as reduced convenience for users and a loss of satisfaction.

[1430] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1431] In this invention, the server includes a means for converting voice input into text using a voice recognition model, a means for generating a response based on the converted text input using a generative AI, a means for converting the generated text response into voice using a text-to-speech AI, a means for recording a response log, and a means for receiving voice input in the autonomous vehicle and outputting a voice response using a speaker in the vehicle, thereby enabling a user in the autonomous vehicle to receive prompt and appropriate support for technical issues or questions.

[1432] A "speech recognition model" is an algorithm or software for converting voice input into text data.

[1433] "Generative AI" is an artificial intelligence technology that generates appropriate responses based on input text data.

[1434] "Reading AI" refers to algorithms or software for converting text data into audio data.

[1435] A "response log" is data that records user input and the system's response to it.

[1436] An "autonomous vehicle" is a vehicle that drives autonomously using artificial intelligence and sensor technology.

[1437] A "speaker" is a device for outputting audio data as sound.

[1438] As an embodiment of the present invention, a technical support system in an autonomous vehicle is shown below. The system includes a speech recognition model, a generative AI, a voice-to-speech AI, a response log recording means, and a voice input and output means in the autonomous vehicle.

[1439] System Configuration

[1440] 1. Speech Recognition Model

[1441] A voice recognition model is an algorithm or software that converts a user's voice input into text data. Specifically, it uses a voice recognition API. It receives voice input through a microphone in the vehicle and converts it into text.

[1442] 2. Generative AI

[1443] Generative AI is an artificial intelligence technique that generates appropriate responses based on textual input. Specifically, it uses generative AI models to analyze text data and generate appropriate responses to user questions.

[1444] 3. Text-to-speech AI

[1445] Text-to-speech AI is an algorithm or software that converts the generated text response into audio data, specifically using a text-to-speech API, which then converts the generated text response into audio and outputs it to the user through the vehicle's speakers.

[1446] 4. Response logging methods

[1447] The response log recorder is a database that records user input and the system's response to that input, so that later follow-up by personnel may be required.

[1448] 5. Voice input and output means in autonomous vehicles

[1449] A microphone is used as the voice input means in an autonomous vehicle, and a speaker is used as the voice output means, allowing users to receive voice support in the vehicle.

[1450] System Operation

[1451] The server receives voice input from the vehicle's microphone and converts it to text using a speech recognition model. It then uses generative AI to analyze the text and generate an appropriate response. The generated text response is converted to audio using speech AI and output to the user through the vehicle's speakers. Response logs are recorded and follow-ups are provided by personnel as needed.

[1452] Examples

[1453] For example, if a user asks, "How do I turn up the air conditioner temperature?" the system would do the following:

[1454] 1. A microphone in the vehicle captures the user's voice.

[1455] 2. The speech recognition model converts the speech to text.

[1456] 3. Generative AI analyzes the text and generates an appropriate response.

[1457] 4. Text-to-speech AI converts the generated text response into audio.

[1458] 5. Speakers in the vehicle output a voice response to the user.

[1459] Examples of prompt statements

[1460] User: How do I turn up the heat on my air conditioner?

[1461] In this way, quick and appropriate voice responses can be provided to user questions within an autonomous vehicle.

[1462] The flow of the specific process in the application example 2 will be described with reference to FIG.

[1463] Step 1:

[1464] The user inputs voice into a microphone in the vehicle.

[1465] Input: User's voice

[1466] Output: Audio data

[1467] Specific action: The user says, "Tell me how to turn up the air conditioner temperature."

[1468] Step 2:

[1469] The server converts the voice data into text data using a voice recognition model.

[1470] Input: Audio data

[1471] Output: Text data

[1472] Specific operation: The speech recognition API analyzes the voice data and generates the text "Tell me how to turn up the air conditioner temperature."

[1473] Step 3:

[1474] The server uses generative AI to analyze the text data and generate an appropriate response.

[1475] Input: Text data

[1476] Output: Response text

[1477] Specific behavior: The generative AI model analyzes the text "Tell me how to turn up the temperature on the air conditioner" and generates the response text "To turn up the temperature on the air conditioner, select the 'Air Conditioning' menu on the central touchscreen and increase the temperature setting."

[1478] Step 4:

[1479] The server uses a text-to-speech AI to convert the response text into audio data.

[1480] Input: Response text

[1481] Output: Audio data

[1482] Specific operation: The speech synthesis API analyzes the response text and generates speech data saying, "To increase the temperature of the air conditioner, select the 'Air Conditioning' menu on the central touch screen and increase the temperature setting."

[1483] Step 5:

[1484] The server outputs the audio data to the user using a speaker in the vehicle.

[1485] Input: Audio data

[1486] Output: Voice response

[1487] Specific operation: The speaker in the vehicle plays audio data, informing the user, "To increase the air conditioning temperature, select the 'Air Conditioning' menu on the central touchscreen and increase the temperature setting."

[1488] Step 6:

[1489] The server logs the response.

[1490] Input: User voice input, generated response text

[1491] Output: Response log

[1492] What happens: The server stores the user's voice input and the generated response text in a database in case a later human follow-up is required.

[1493] In addition, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1494] "Example 1"

[1495] As one embodiment of the present invention, a system incorporating an emotion engine is provided. This system converts voice input into text using a voice recognition model, generates a response based on the text input using a generative AI, converts the generated text response into voice using a speech-to-speech AI, and records a response log. Furthermore, this system includes an emotion engine that recognizes the user's emotion from the voice input. Specifically, when a user expresses an emotion such as anger or joy by voice, the emotion engine recognizes the emotion, and the generative AI generates a response based on the emotion. For example, when a user shows an emotion of anger, the generative AI is adjusted to generate a more polite response. In addition, the emotion engine can adjust the priority of responses based on the user's emotion. For example, when a user shows a strong emotion of anger, the system is adjusted to prioritize processing of inquiries from that user. This makes it possible to provide more appropriate support according to the user's emotion.

[1496] "Example 2"

[1497] As one embodiment of the present invention, a system incorporating an emotion engine is provided. This system converts voice input into text using a voice recognition model, generates a response based on the text input using a generative AI, converts the generated text response into voice using a speech-to-speech AI, and records a response log. Furthermore, this system includes an emotion engine that recognizes the user's emotion from the voice input. Specifically, when a user expresses an emotion such as anger or joy by voice, the emotion engine recognizes the emotion, and the generative AI generates a response based on the emotion. For example, when a user shows an emotion of anger, the generative AI is adjusted to generate a more polite response. In addition, the emotion engine can adjust the priority of responses based on the user's emotion. For example, when a user shows a strong emotion of anger, the system is adjusted to prioritize processing of inquiries from that user. This makes it possible to provide more appropriate support according to the user's emotion.

[1498] The process flow of each embodiment will be described below.

[1499] "Example 1"

[1500] Step 1: Receive voice input from the user. This voice input is received through a smartphone microphone or similar.

[1501] Step 2: Convert the received voice input into text using WhisperAPI. WhisperAPI is a speech recognition model that has the ability to convert speech to text.

[1502] Step 3: Recognize user emotions from the textual input using the emotion engine. The emotion engine has the ability to recognize user emotions (anger, joy, etc.) from the text.

[1503] Step 4: Generative AI generates a response based on the recognized emotion. Generative AI has the ability to generate an appropriate response based on the textual input and the recognized emotion.

[1504] Step 5: The generated text response is converted into speech using ReadSpeaker, a speech-to-speech AI capable of converting text to speech.

[1505] Step 6: Provide a spoken response to the user and simultaneously record the response log, which can be used for later follow-up and system improvement.

[1506] Example 1

[1507] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1508] Conventional technical support systems provide uniform responses without considering the user's feelings, which can lead to low user satisfaction. In addition, there are often waiting times, and responses are delayed even in situations where a quick response is required. This increases user stress and reduces support efficiency.

[1509] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1510] In this invention, the server includes a means for converting a voice input into text using a voice recognition model, a means for generating a response based on the input converted into text using generative artificial intelligence, a means for converting the generated text response into voice using a text-to-speech artificial intelligence, a means for recording a response log, a means for recognizing a user's emotion from the voice input using an emotion engine and adjusting the response based on the emotion, and a means for adjusting the priority of responses based on the user's emotion, thereby making it possible to quickly provide an appropriate response according to the user's emotion.

[1511] A "speech recognition model" is an algorithm or software for analyzing voice data and converting it into text data.

[1512] "Generative artificial intelligence" is an artificial intelligence technology that analyzes input data and generates appropriate responses or results.

[1513] "Read-to-speech artificial intelligence" is an artificial intelligence technology that converts text data into audio data and outputs it as audio.

[1514] A "response log" is data that records the content of inquiries from users and the responses generated by the system.

[1515] An "emotion engine" is a technology that analyzes user emotions from voice input and text data and recognizes those emotions.

[1516] "Means for adjusting priority" is a function for changing the order and importance of responses based on the user's emotions and the content of the inquiry.

[1517] This invention is a system for providing technical support for smartphones. Voice input from a user is converted to text by a voice recognition model. A voice recognition API is used as this voice recognition model. The text input is analyzed by generative artificial intelligence, and an appropriate response is generated. The generated text response is converted to voice by a speech-to-speech artificial intelligence, and output to the user. A voice synthesis API is used as this speech-to-speech artificial intelligence. In addition, the system records a response log, and when follow-up by a human is required, a proposal for a solution is made based on this. This enables voice support 24 hours a day, 365 days a year with no waiting time, reducing the burden on users and providing efficient support.

[1518] Furthermore, this system combines an emotion engine. The emotion engine recognizes the user's emotion from the voice input, and the generative AI generates a response based on that emotion. For example, if the user expresses anger, the generative AI is adjusted to generate a more polite response. The emotion engine can also adjust the priority of responses based on the user's emotion. For example, if a user expresses strong anger, the system is adjusted to prioritize inquiries from that user. This makes it possible to provide more appropriate support according to the user's emotion.

[1519] As a concrete example, consider the case where a user makes a voice inquiry to their smartphone saying, "Wi-Fi won't connect." The device converts this voice into text using a speech recognition API, generating the text "Wi-Fi won't connect." The server passes this text to a generative artificial intelligence system, which generates an appropriate response. For example, a response might be generated saying, "Please try restarting your Wi-Fi router." The speech synthesis API converts this response into speech, and the device outputs the response to the user in voice.

[1520] An example of a prompt sentence is, "The user spoke to his smartphone saying, 'I can't connect to Wi-Fi.' Please convert this speech into text, generate an appropriate response, and reply by voice."

[1521] This system allows users to receive voice support 24 hours a day, 365 days a year, without waiting, providing efficient support.

[1522] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1523] Step 1:

[1524] A user makes a voice inquiry into a smartphone. For example, they say, "I can't connect to Wi-Fi." The input is the user's voice data, and the output is the voice data sent to the device.

[1525] Step 2:

[1526] The device receives the user's voice input and converts the voice to text using a voice recognition API. Specifically, the device sends voice data to the voice recognition API, which analyzes the voice data and generates the text "Wi-Fi is not connected." The input is voice data, and the output is text data.

[1527] Step 3:

[1528] The server receives the text sent from the terminal. The server passes this text to the generative AI for analysis. The generative AI understands the content of the user's inquiry and creates a prompt sentence to generate an appropriate response. The input is text data, and the output is a prompt sentence.

[1529] Step 4:

[1530] The generative AI generates an appropriate response based on the prompt sentence. For example, a response such as "Please try restarting your Wi-Fi router" is generated. The server stores this response in text format. The input is the prompt sentence, and the output is the response text.

[1531] Step 5:

[1532] The server sends the generated text response to the device. The device uses a speech synthesis API to convert the text response into speech. Specifically, the device passes the text data to the speech synthesis API, which generates speech data. The input is text data, and the output is speech data.

[1533] Step 6:

[1534] The device outputs the generated voice data to the user. The user can hear the voice response, "Please try restarting your Wi-Fi router." The input is the voice data, and the output is the voice response to the user.

[1535] Step 7:

[1536] The server records all response logs, including the user's voice input, the transcribed input, and the generated response. If necessary, a human follow-up is performed and a resolution suggestion is provided. The input is the data, and the output is the recorded response log.

[1537] Step 8:

[1538] The server uses an emotion engine to recognize user emotions from voice input. For example, if a user expresses an emotion of anger, the emotion engine recognizes the emotion and adjusts the generative artificial intelligence to generate a more polite response. The emotion engine can also adjust the priority of responses. User queries that express a strong emotion of anger are given priority. The input is voice data and text data, and the output is the adjusted response and priority.

[1539] (Application example 1)

[1540] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1541] In autonomous vehicles, there is a need to provide prompt and appropriate support when users ask questions about how to operate the vehicle or troubleshooting. However, conventional systems are unable to respond in a way that takes into account the user's emotions, which can lead to a decrease in user satisfaction. In addition, the accuracy and speed of generating responses based on voice input are also issues.

[1542] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1543] In this invention, the server includes a means for converting voice input into text using a voice recognition model, a means for generating a response based on the text input using a generative AI, a means for converting the generated text response into voice using a text-to-speech AI, a means for recording a response log, a means for recognizing a user's emotion from the voice input using an emotion engine, and a means for adjusting the tone and priority of the response based on the user's emotion. This enables prompt and appropriate support according to the user's emotion.

[1544] A "speech recognition model" is an algorithm or software for converting voice input into text.

[1545] "Generative AI" is an artificial intelligence technology that generates appropriate responses based on text input.

[1546] "Reading AI" refers to algorithms or software that converts text into speech and outputs it.

[1547] A "response log" is a database that records user inquiries and the system's responses to them.

[1548] An "emotion engine" is an algorithm or software that analyzes and recognizes a user's emotions from voice input.

[1549] "Tone" refers to the vocal cadence or nuance of the response produced.

[1550] "Priority" is a criterion for determining the order in which multiple inquiries or tasks are processed.

[1551] The system for implementing this invention is composed of a combination of a speech recognition model, a generative AI, a reading AI, and an emotion engine. Below, we will explain the details of each component and how they work together.

[1552] Speech Recognition Model

[1553] The server converts the user's voice input into text using a speech recognition model, which is provided by a speech recognition API, that analyzes the speech data and generates corresponding text data.

[1554] Generative AI

[1555] The server generates responses based on the textual input using a generative AI, which uses advanced natural language processing models such as GPT-3, and leverages pre-trained data to generate appropriate responses to user questions.

[1556] Reading AI

[1557] The server uses a voice AI to convert the generated text response into voice. A voice synthesis engine is used as the voice AI. The voice synthesis engine converts the text data into natural voice and outputs it to the user.

[1558] Emotion Engine

[1559] The server recognizes the user's emotions from the voice input using an emotion engine. The emotion engine analyzes the tone, pitch, and rate of the voice data to identify the user's emotional state. If the user is expressing an emotion such as anger or joy, a response is generated according to that emotion.

[1560] Adjusting the tone and priority of your response

[1561] The server adjusts the tone and priority of the responses generated by the generative AI based on the user's emotions recognized by the emotion engine. For example, if the user expresses anger, the generative AI is adjusted to generate more polite responses. In addition, inquiries from users who show strong emotions are given priority.

[1562] Examples

[1563] Consider a case where a user asks how to operate the air conditioner in an autonomous vehicle. When the user asks verbally, "How do I adjust the temperature of the air conditioner?", a speech recognition model converts this speech into text. A generative AI then generates a response based on this text: "To adjust the temperature of the air conditioner, tap the temperature setting on the center display." The emotion engine analyzes the user's emotions, and if there is no particular change in emotion, the reading AI vocalizes this response in a normal tone and provides it to the user.

[1564] Examples of prompt statements

[1565] User sentiment: Neutral

[1566] User Question: How do I adjust the temperature on my air conditioner?

[1567] Generate an appropriate response.

[1568] In this way, it is possible to provide prompt and appropriate support according to the user's emotions.

[1569] The flow of the specific process in the application example 1 will be described with reference to FIG.

[1570] Step 1:

[1571] The user performs voice input. For example, the user may ask a question in an autonomous vehicle, such as "How do I adjust the temperature of the air conditioner?" This voice data is sent to the server.

[1572] Step 2:

[1573] The server converts the voice input to text using a voice recognition model. The voice recognition API analyzes the voice data and generates the corresponding text data: "How do I adjust the temperature of the air conditioner?" The input is voice data, and the output is text data.

[1574] Step 3:

[1575] The server uses an emotion engine to recognize the user's emotion from the voice input. The emotion engine analyzes the tone, pitch, and speed of the voice data and identifies the user's emotional state as "neutral." The input is the voice data, and the output is the emotional state.

[1576] Step 4:

[1577] The server uses a generative AI to generate a response based on the textual input. The generative AI (e.g., GPT-3) uses the prompt sentence to generate an appropriate response: "To adjust the temperature of the air conditioner, tap the temperature setting on the center display." The input is the text data and the emotional state, and the output is the response text.

[1578] Step 5:

[1579] The server adjusts the tone and priority of the response generated by the generative AI based on the user's emotion recognized by the emotion engine. For example, because the user's emotion is "normal," a response is generated in a normal tone. The input is the response text and the emotional state, and the output is the adjusted response text.

[1580] Step 6:

[1581] The server uses a voice-to-speech AI to convert the generated text response into voice. The voice synthesis engine converts the text data into natural speech and outputs it to the user. The input is the adjusted response text, and the output is the voice data.

[1582] Step 7:

[1583] The server records the response log. User queries and the system's responses to them are stored in a database. Inputs are voice data, text data, emotional state, and response text, and output is the recorded log data.

[1584] In this way, prompt and appropriate support can be provided according to the user's feelings.

[1585] Example 2

[1586] Next, a second embodiment of the second embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1587] Conventional smartphone technical support systems convert users' voice input into text, generate responses using generative AI, and convert them into voice using text-to-speech AI, but they have a problem in that they cannot adjust responses to take the user's emotions into account. This can lead to lower user satisfaction and make it difficult to provide efficient support.

[1588] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1589] In this invention, the server includes a means for converting a voice input into text using a voice recognition model, a means for generating a response based on the converted text input using generative artificial intelligence, a means for converting the generated text response into voice using a text-to-speech artificial intelligence, a means for recording a response log, a means for recognizing a user's emotion from the voice input using an emotion engine, and a means for adjusting a response based on the user's emotion, thereby enabling more appropriate support according to the user's emotion.

[1590] A "speech recognition model" is an algorithm or software for converting voice input into text data.

[1591] "Generative artificial intelligence" is an artificial intelligence technology that analyzes input text data and generates appropriate responses.

[1592] "Read-to-speech artificial intelligence" is a technology that converts text data into voice data and provides a voice response to the user.

[1593] A "response log" is data for recording inquiries from users and responses to those inquiries.

[1594] An "emotion engine" is a technology for recognizing a user's emotions from voice input and adjusting responses based on those emotions.

[1595] "Text conversion" is the process of converting audio data into text data.

[1596] "Analysis" is the process of analyzing input data and understanding its meaning and intent.

[1597] "Adjustment" is the process of changing responses or behavior in response to specific conditions or situations.

[1598] This invention is a system for providing technical support for smartphones, which converts voice input from the user into text, analyzes the text to generate an appropriate response, and returns it to the user as voice. It also has a function to recognize the user's emotions and provide a response according to those emotions.

[1599] System Configuration

[1600] 1. Voice to text conversion

[1601] Hardware / Software: The server converts voice input to text using a speech recognition model, which is provided by a speech recognition API.

[1602] Example: When a user says, "My smartphone battery is running out quickly, what should I do?" the server converts this speech into text.

[1603] 2. Text Analysis and Response Generation

[1604] Hardware / Software: The server uses generative artificial intelligence (e.g., a generative AI model) to parse the textual input and generate an appropriate response.

[1605] Example: A server parses the text "My phone's battery is draining quickly, what should I do?" and generates a response that reads "To extend battery life, you can close apps running in the background and adjust the screen brightness."

[1606] 3. Audio Responses

[1607] Hardware / Software: The server converts the generated text response into speech using a speech-to-text AI, which uses a text-to-speech API.

[1608] Example: The server converts the text "To extend battery life, you can close background apps and adjust your screen brightness." into speech, which the device then outputs to the user.

[1609] 4. Response logging

[1610] Hardware / Software: The server uses a database to log all responses.

[1611] Example: The server stores the user's inquiry and the generated response in a database. For example, the following log is recorded: "Inquiry: My smartphone's battery runs out quickly. What should I do?" "Response: To extend the battery life, it is effective to close apps running in the background and adjust the screen brightness."

[1612] 5. Emotion recognition and response adjustment

[1613] Hardware / Software: The server uses an emotion engine to recognize emotions from the user's voice.

[1614] Example: If a user expresses anger, the server uses the emotion engine to recognize the emotion. For example, if the user says in an angry tone, "Why is my battery running out so fast?", the generative AI generates a more polite response such as, "I'm sorry. We will give you a solution to your battery problem right away." The system is also adjusted so that inquiries from users who express strong emotions are given priority.

[1615] Examples of prompt statements

[1616] "My smartphone battery runs out quickly, what should I do?"

[1617] "The app keeps crashing, what should I do?"

[1618] "I can't connect to Wi-Fi, what should I do?"

[1619] This system allows users to receive voice support 24 hours a day, 365 days a year, without waiting, providing efficient support.

[1620] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1621] Step 1:

[1622] Receiving audio input

[1623] How it works: A user speaks a query into their smartphone.

[1624] Input: The user's voice input.

[1625] Output: The audio data.

[1626] Specific behavior: A user says, "My smartphone battery is running out quickly. What should I do?" The device receives this voice through the microphone.

[1627] Step 2:

[1628] Voice to text conversion

[1629] How it works: The server converts voice input into text using a speech recognition model.

[1630] Input: Audio data.

[1631] Output: Text data.

[1632] What happens: The server calls a speech recognition API to convert the voice data into text such as, "My smartphone's battery is running out quickly. What should I do?"

[1633] Step 3:

[1634] Text analysis and response generation

[1635] How it works: The server uses generative artificial intelligence to parse the textual input and generate an appropriate response.

[1636] Input: Text data.

[1637] Output: The response text.

[1638] How it works: The server inputs the text "My smartphone's battery is draining quickly, what should I do?" into the generative AI model, which generates a response saying "To extend battery life, it is effective to close apps running in the background and adjust the screen brightness."

[1639] Step 4:

[1640] Spoken responses

[1641] How it works: The server uses text-to-speech artificial intelligence to convert the generated text response into speech.

[1642] Input: The response text.

[1643] Output: The audio data.

[1644] Specific operation: The server calls the speech synthesis API to convert the text "To extend battery life, it is effective to close apps running in the background and adjust the screen brightness." into speech, and the device outputs the speech to the user.

[1645] Step 5:

[1646] Response logging

[1647] Behavior: The server logs all responses.

[1648] Input: The user's query and the generated response.

[1649] Output: The response log data.

[1650] Specific operation: The server records the following log in the database: "Query: My smartphone's battery is draining quickly. What should I do?" "Response: To extend battery life, it is effective to close apps running in the background and adjust the screen brightness."

[1651] Step 6:

[1652] Emotion recognition and response adjustment

[1653] How it works: The server uses an emotion engine to recognize emotions from the user's voice and tailor its response.

[1654] Input: The user's voice data.

[1655] Output: The adjusted response text.

[1656] Specific behavior: If a user expresses anger, the server uses the emotion engine to recognize the emotion. For example, if a user says in an angry tone, "Why is my battery running out so fast?", the generative AI generates a more polite response such as, "I'm sorry. We will give you a solution to your battery problem right away." The system is also adjusted so that inquiries from users who express strong emotions are given priority.

[1657] (Application example 2)

[1658] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1659] In autonomous vehicles, when a user asks a question about vehicle operation or navigation by voice while driving, it is necessary to provide a quick and appropriate response. However, conventional systems are unable to respond taking into account the user's emotions, which may reduce user satisfaction. In addition, it is difficult to respond quickly to questions with high urgency because it is not possible to adjust the priority of responses based on emotions. To solve these issues, a system is needed that recognizes the user's emotions, generates responses based on them, and provides responses with an appropriate tone and priority.

[1660] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1661] In this invention, the server includes a means for converting a voice input into text using a voice recognition model, a means for generating a response based on the input converted into text using a generative AI, a means for converting the generated text response into voice using a text-to-speech AI, a means for recording a response log, a means for recognizing a user's emotion from the voice input using an emotion engine, a means for adjusting the tone and priority of the response based on the user's emotion, and a means for being installed in the infotainment system of the autonomous vehicle. This enables a quick and appropriate response that takes the user's emotion into consideration, thereby improving user satisfaction.

[1662] A "speech recognition model" is an algorithm or software for converting voice input into text.

[1663] "Generative AI" is an artificial intelligence technology that generates appropriate responses based on input text data.

[1664] "Read-to-speech AI" is an artificial intelligence technology that converts text data into audio and outputs it.

[1665] A "response log" is a database that records user inquiries and the system's responses to them.

[1666] An "emotion engine" is a technology for recognizing a user's emotions from voice input and adjusting responses based on those emotions.

[1667] An "infotainment system" is a system that provides information and entertainment within a vehicle.

[1668] An "autonomous vehicle" is a vehicle that drives itself using artificial intelligence and sensor technology.

[1669] As an embodiment of the present invention, a program installed in the infotainment system of an autonomous vehicle will be described. The server uses a speech recognition model, generative AI, text-to-speech AI, and emotion engine to generate a response based on the user's voice input, and outputs it by voice.

[1670] First, the user speaks to a vehicle operation or navigation question. The voice input is converted to text using a speech recognition model, which uses a speech recognition API. The text input is then analyzed by an emotion engine to recognize the user's emotion. The emotion engine recognizes the user's emotion from the voice input and adjusts the tone and priority of the response based on that emotion.

[1671] A generative AI model is then used to generate an appropriate response based on the textual input and the recognized emotion. The response is then converted to speech by a text-to-speech AI, which uses a text-to-speech API. The generated voice response is then output to the user through the infotainment system.

[1672] In addition, response logs are recorded on the server and can be viewed later, making it possible to track user inquiry history and response content, and to have personnel follow up if necessary.

[1673] As a concrete example, consider a user asking, "Where is the next gas station?" In this case, the emotion engine recognizes the user's impatience, and the generative AI generates a quick and polite response, such as, "The next gas station is 3 kilometers away on the right. Don't worry."

[1674] Example prompt:

[1675] User Question: Where is the next gas station?

[1676] User Sentiment: Impatience

[1677] Appropriate response:

[1678] In this way, a quick and appropriate response that takes the user's feelings into consideration is possible, thereby improving user satisfaction.

[1679] The flow of the specific process in the application example 2 will be described with reference to FIG.

[1680] Step 1:

[1681] The user asks questions about vehicle operation and navigation by voice.

[1682] Input: User's voice input

[1683] Output: Audio data

[1684] Specific Action: A user asks a question to the infotainment system of an autonomous vehicle.

[1685] Step 2:

[1686] The server converts the voice input into text using a speech recognition model.

[1687] Input: Audio data

[1688] Output: Text data

[1689] Specific operation: The speech recognition API analyzes the speech data and generates corresponding text data.

[1690] Step 3:

[1691] The server uses an emotion engine to recognize the user's emotion from the text data.

[1692] Input: Text data

[1693] Output: Emotion data

[1694] Specific behavior: The emotion engine analyzes text data and recognizes the user's emotions (e.g., impatience, anger, joy).

[1695] Step 4:

[1696] The server uses generative AI to generate a response based on text data and emotion data.

[1697] Input: Text data, emotion data

[1698] Output: Response text

[1699] Specific operation: The generative AI model generates an appropriate response text based on text data and emotion data.

[1700] Step 5:

[1701] The server uses text-to-speech AI to convert the response text into speech.

[1702] Input: Response text

[1703] Output: Audio data

[1704] Specific operation: The speech synthesis API analyzes the response text and generates corresponding speech data.

[1705] Step 6:

[1706] The server outputs the generated voice data to the user through the infotainment system.

[1707] Input: Audio data

[1708] Output: A voice response to the user

[1709] Specific Action: The infotainment system plays the audio data and provides a response to the user.

[1710] Step 7:

[1711] The server logs the response.

[1712] Input: Text data, emotion data, response text

[1713] Output: Response log

[1714] Specific behavior: The server records the user's question, the recognized emotion, and the generated response in a database.

[1715] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1716] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1717] Another example of generative AI is Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) are mentioned.

[1718] In the above embodiment, an example was given in which the specific process was performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the robot 414.

[1719] The emotion identification model 59 as an emotion engine may determine the emotion of the user according to a specific mapping. Specifically, the emotion identification model 59 may determine the emotion of the user according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the emotion of the robot, and the identification processing unit 290 may perform identification processing using the emotion of the robot.

[1720] FIG. 9 is a diagram showing an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive emotions are arranged. The more outside the concentric circles, the more emotions that represent states and actions that arise from a state of mind are arranged. Emotions are a concept that includes emotions and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions that occur in the brain are arranged. On the right side of the concentric circles, emotions that are generally induced by situational judgment are arranged. On the upper and lower sides of the concentric circles, emotions that are generally generated from reactions that occur in the brain and are induced by situational judgment are arranged. In addition, on the upper side of the concentric circles, emotions of "pleasure" are arranged, and on the lower side, emotions of "discomfort" are arranged. In this way, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1721] These emotions are distributed in the 3 o'clock direction of emotion map 400 and usually fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1722] The inside of emotion map 400 represents what is going on inside one's mind, and the outside of emotion map 400 represents behavior, so the further out you go on emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1723] Here, human emotions are based on various balances such as posture and blood sugar level, and when these balances are far from the ideal, it indicates an unpleasant state, and when they are close to the ideal, it indicates a pleasant state. Emotions can also be created for robots, cars, motorcycles, etc., based on various balances such as posture and battery level, so that when these balances are far from the ideal, it indicates an unpleasant state, and when they are close to the ideal, it indicates a pleasant state. The emotion map may be generated, for example, based on the emotion map of Dr. Mitsuyoshi (Research on speech emotion recognition and emotion brain physiological signal analysis system, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). On the left half of the emotion map, emotions belonging to an area called "reaction" where sensation is dominant are lined up. On the right half of the emotion map, emotions belonging to an area called "situation" where situation recognition is dominant are lined up.

[1724] The emotion map defines two emotions that promote learning. The first is the negative emotion around the middle of "repentance" or "remorse" on the situation side. In other words, this is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the positive emotion around "desire" on the response side. In other words, this is when the robot has positive feelings such as "I want more" or "I want to know more."

[1725] The emotion identification model 59 inputs the user input to a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the emotion of the user. This neural network is pre-trained based on multiple learning data that are combinations of the user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in Fig. 10. Fig. 10 shows an example in which multiple emotions, "relief," "calm," and "encouraging," have similar emotion values.

[1726] In the above embodiment, an example was given in which a specific process is performed by one computer 22, but the technology disclosed herein is not limited to this, and distributed processing of the specific process may be performed by multiple computers, including computer 22.

[1727] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable computer-readable non-transitory storage medium such as a Universal Serial Bus (USB) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1728] In addition, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 upon request from the data processing device 12.

[1729] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1730] As the hardware resource for executing the specific process, various processors as shown below can be used. An example of the processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing the specific process by executing software, i.e., a program. Another example of the processor is a dedicated electric circuit, which is a processor having a circuit configuration designed exclusively for executing the specific process, such as a Field-Programmable Gate Array (FPGA), a Programmable Logic Device (PLD), or an Application Specific Integrated Circuit (ASIC). Each processor has a built-in or connected memory, and each processor executes the specific process by using the memory.

[1731] The hardware resource that executes the specific process may be one of these various processors, or may be a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific process may be a single processor.

[1732] As an example of a configuration using one processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a configuration in which a processor is used that realizes the functions of the entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1733] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements. The specific processes described above are merely examples. It goes without saying that unnecessary steps may be deleted, new steps may be added, or the order of processes may be changed without departing from the spirit of the invention.

[1734] The above description ...

Claims

1. A first means for accepting a voice query from a user in an autonomous vehicle; A second means for generating a prompt sentence for generating an answer to a question from the user based on the content of the question; a third means for inputting the generated prompt sentence into a generative AI model to generate the answer; a fourth means for outputting the generated answer to the user using a speaker in the autonomous vehicle; A system including:

2. a fifth means for recognizing an emotion of the user from the user's voice using an emotion engine; and a sixth means for adjusting a tone or a priority of the response based on a sentiment of the user. The system of claim 1 .

3. The sixth means adjusts the processing so that a question from a first user who shows a strong emotion is processed with a higher priority than a question from a second user who shows a weak emotion. The system of claim 2.

Images