System

Abstract

A system is provided.SOLUTION: A system, comprising: means for receiving a problem or a request provided by a user through audio input; means for converting the audio input into text data; means for transmitting the converted text data to a server; means for the server receiving the text data and generating a solution or an advice using a generation AI module; means for transmitting the generated solution or advice to a user equipment; and means for the user equipment displaying and playing back the received solution or advice.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] As mobile phones have become more widespread, they have become more multifunctional, but there is also a growing number of users who find it difficult to operate and set up these functions, and users who want to effectively use new functions.It is often difficult for users to quickly and accurately set up and operate their mobile phones on their own, which often results in hassle and stress. [Means for solving the problem]

[0005] This invention provides a system that allows users to input problems and requests about their mobile phones through voice input. This system includes a means for converting voice input into text data, a means for sending the converted text data to a server, and a means for generating solutions and advice using a generation AI module using the text data received on the server side. The system transmits, displays, and plays back the generated solutions and advice on the user's terminal, allowing the user to easily and quickly operate and configure their mobile phone.

[0006] "Voice input" is the process of capturing the user's speech as a digital signal into a terminal.

[0007] "Text data" is text information converted from voice input using voice recognition technology.

[0008] A "server" is a remote computer system that receives and processes data and generates a response.

[0009] "Generative AI Module" refers to an artificial intelligence algorithm that processes text data and generates solutions or advice.

[0010] "User Terminal" means an electronic device that receives voice input, communicates with the server, and displays and outputs the results.

[0011] "Solutions or advice" refers to information such as operating procedures and settings suggested by the generating AI module in response to problems or requests presented by the user.

[0012] "Speech recognition means" is a general term for processes and techniques for generating text data from speech input.

[0013] "Display means" refers to a display or screen device that visually displays solutions or advice on a user's device.

[0014] "Audio playback means" refers to a speaker or voice synthesis function that outputs audio data on the user terminal.

[0015] "Transmission means" refers to a communication module or protocol that allows a user terminal to send data to a server.

[0016] The "receiving means" refers to a communication module or protocol that allows the server to receive data sent from the user terminal.

[0017] "User's past operation history" refers to a record of the user's past settings and usage of the mobile phone.

[0018] "Setting information" is data relating to various functions and option settings of the mobile phone.

[0019] "Personalization" is the process of providing individualized information and solutions based on each user's specific conditions and history. [Brief explanation of the drawings]

[0020] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0021] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0022] First, the terms used in the following description will be explained.

[0023] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0024] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0025] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0026] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0027] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0028] [First embodiment]

[0029] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0030] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0031] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0032] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0033] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0034] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0035] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0036] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0037] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0038] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0039] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0040] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0041] System Overview

[0042] This invention is a system for easily resolving problems users may encounter when operating or configuring a mobile phone. This system involves a series of processes: converting user voice input into text data, sending it to a server, generating solutions or advice using a generation AI module, and returning it to the user's device.

[0043] Key Components of the System

[0044] User device:

[0045] A device that allows users to input voice information. It includes a voice recognition module, a transmission / reception module, a display, and a speaker.

[0046] server:

[0047] A remote system that receives voice-recognized text data and generates solutions or advice using a generative AI module.

[0048] Communication Interface:

[0049] A network module for sending and receiving data between a user terminal and a server.

[0050] Program processing overview

[0051] 1. Voice input

[0052] Users communicate their problems and requests via voice.

[0053] 2. Voice Recognition

[0054] The device converts the user's voice input into text data.

[0055] 3. Sending text data

[0056] The terminal transmits the converted text data to the server.

[0057] 4. Receiving text data

[0058] The server receives the text data sent from the user terminal.

[0059] 5. Processing by the Generative AI Module

[0060] The server provides the received text to a generative AI module, which generates an appropriate solution or advice.

[0061] 6. Submitting Solutions or Advice

[0062] The server transmits the generated solution or advice to the user terminal.

[0063] 7. Displaying and playing audio solutions or advice

[0064] The device displays the received solution or advice on the display and plays it aloud through the speaker.

[0065] Specific examples

[0066] Example 1: Wi-Fi settings

[0067] When a user says, "I don't know how to set up Wi-Fi," the following process occurs:

[0068] Audio Input:

[0069] The user says, "I don't know how to set up Wi-Fi."

[0070] Voice Recognition:

[0071] The device converts the voice into text data: "I don't know how to set up Wi-Fi."

[0072] Sending text data:

[0073] The device sends text data to the server saying "I don't know how to set up Wi-Fi."

[0074] Receiving text data:

[0075] The server receives the text data.

[0076] Processing by the Generative AI module:

[0077] The server uses the text data to generate a solution that includes instructions for setting up Wi-Fi.

[0078] Example response from the generative AI module: "Open the Settings app and go to the Wi-Fi settings section. From the list of Wi-Fi networks, select the network you want to connect to, enter the password, and tap Connect."

[0079] Submit a solution or advice:

[0080] The server sends the generated solution to the user terminal.

[0081] View and listen to solutions or advice:

[0082] The device will show the solution it received on the display and play a voice message saying, "Open the Settings app and go to the Wi-Fi settings section. From the list of Wi-Fi networks, select the network you want to connect to, enter the password, and tap Connect."

[0083] This allows users to solve problems themselves and smoothly set up their mobile phones. The advantage of this system is that it provides fast and accurate support.

[0084] The processing flow will be explained below.

[0085] Step 1:

[0086] The user verbally states their problem or request, for example, "I don't know how to set up Wi-Fi."

[0087] Step 2:

[0088] The device receives the user's voice input and passes it to the speech recognition module, which converts the voice data into text data. Example: "I don't know how to set up Wi-Fi."

[0089] Step 3:

[0090] The terminal transmits the converted text data to the server.

[0091] Step 4:

[0092] The server receives the text data sent from the user terminal.

[0093] Step 5:

[0094] The server inputs the received text data into a generation AI module to generate a solution or advice. For example, a generated solution might be, "Open the Settings app and go to the Wi-Fi settings section. Select the network you want to connect to from the list of Wi-Fi networks, enter the password, and tap Connect."

[0095] Step 6:

[0096] The server transmits the generated solution or advice to the user terminal.

[0097] Step 7:

[0098] The terminal receives the solution or advice received from the server.

[0099] Step 8:

[0100] The device displays the received solution or advice on the display, and simultaneously plays back the solution aloud using a speech synthesis module.

[0101] Step 9:

[0102] Users can follow the solution displayed on their device to configure their phone: open the Settings app, go to the Wi-Fi settings section, select the network they want to connect to from the list, enter the password, and tap Connect.

[0103] Example 1

[0104] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0105] Conventional support systems for mobile phone operation and settings make it difficult for users to solve problems directly and quickly. In particular, systems that allow users to solve problems through voice input are limited, and in many cases, complex operating procedures are required. As a result, users are more likely to make operating or setting errors. In response to this, the present invention aims to provide a system that allows users to easily report problems through voice input and quickly provides solutions using a generative AI module.

[0106] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0107] In this invention, the server includes a means for receiving a problem or request provided by a user through voice input, a means for converting the voice input into text data, and a means for transmitting the converted text data to the server, so that the user can report a problem through voice, which is converted into text data and transmitted to the server, thereby enabling a solution to be generated and provided to the user for quickly and accurately resolving the problem.

[0108] "User" refers to the entity that operates the system and provides problems or requests using voice input.

[0109] "Voice input" refers to a means by which a user communicates problems or requests to a system using their voice.

[0110] "Text data" refers to data that has been converted from voice input into a character string format.

[0111] "Server" refers to the remote system that receives the text data converted from the voice input and generates solutions or advice using a generative AI module.

[0112] "Generative AI module" refers to an artificial intelligence algorithm that generates appropriate solutions or advice based on text data received from the user.

[0113] A "prompt sentence" refers to an instruction sentence for generating an output for solving a problem based on text data input to the generation AI module.

[0114] "Communication interface" refers to a network module for transmitting and receiving data between a user terminal and a server.

[0115] "User terminal" refers to a device through which a user makes voice input, and which is equipped with a voice recognition module, a display, and a speaker.

[0116] The present invention is a system that allows users to report problems related to mobile phone operation and settings through voice input, and uses a generative AI module to quickly and accurately provide solutions. The system includes the following main components: a user terminal, a server, and a communication interface.

[0117] System configuration

[0118] User Device

[0119] A user terminal is a device that allows users to input voice. It includes a microphone, display, and speaker as hardware, and a speech recognition module as software. Specifically, speech recognition software such as Google Voice API or IBM Watson is used.

[0120] server

[0121] The server is a remote system that receives text data sent from the user's device and generates solutions and advice using a generative AI model, such as OpenAI's GPT-4.

[0122] Communication Interface

[0123] The communication interface serves to send and receive data between the user device and the server, specifically via Wi-Fi or 4G / 5G communication.

[0124] Processing the data

[0125] 1. Voice input: The user voices the problem they are having with their phone's operation or settings. For example, they say, "I don't know how to set up Wi-Fi."

[0126] 2. Speech recognition: The device's speech recognition module records the user's voice and converts it into text data in real time. The Google Voice API converts the voice into text data such as "I don't know how to set up Wi-Fi."

[0127] 3. Sending text data: The converted text data is sent to the server, and the device's transceiver module transmits the data via Wi-Fi or 4G / 5G communication.

[0128] 4. Receiving text data: The server receives the text data and stores the received data in a processing queue.

[0129] 5. Processing by the generative AI module: The server generates a prompt and provides it to the generative AI model. For example, it provides the following prompt:

[0130] When a user inquires about how to set up Wi-Fi, please provide a clear explanation of the steps to set up Wi-Fi.

[0131] A generative AI model (e.g., OpenAI's GPT-4) generates a solution, such as "Open the Settings app and go to the Wi-Fi settings section. From the list of Wi-Fi networks, select the network you want to connect to, enter the password, and tap Connect."

[0132] 6. Sending the solution: The generated solution is sent to the user terminal, and the data is sent back through the communication interface.

[0133] 7. Display and audio solution: Your device will display and audio the solution using a Text-to-Speech (TTS) engine (e.g., Google TTS), for example, "Open the Settings app and go to the Wi-Fi settings section. From the list of Wi-Fi networks, select the network you want to connect to, enter the password, and tap Connect."

[0134] Specific examples

[0135] Example 1: Wi-Fi settings

[0136] When a user says, "I don't know how to set up Wi-Fi," the following sequence of events occurs:

[0137] 1. Voice input: The user says, "I don't know how to set up Wi-Fi."

[0138] 2. Speech recognition: The device uses the Google Voice API to convert the speech into text data such as "I don't know how to set up Wi-Fi."

[0139] 3. Sending text data: The device sends text data saying "I don't know how to set up Wi-Fi" to the server.

[0140] 4. Receiving text data: The server receives the text data.

[0141] 5. Processing by the generative AI module: The server provides the received text data to the generative AI model, which generates a solution including instructions for setting up Wi-Fi.

[0142] 6. Sending the solution: The server sends the generated solution to the user terminal.

[0143] 7. Displaying the solution and playing it aloud: The device displays the solution it received on the display and plays it aloud using Google TTS.

[0144] As described above, the system of the present invention allows a user to report a problem by voice, which is converted into text data and sent to a server, and then a solution to quickly and accurately resolve the problem can be generated and provided to the user.

[0145] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0146] Step 1:

[0147] Voice input

[0148] The user communicates problems or requests regarding the settings of their mobile phone by voice. The input is the user's voice data. Specifically, the user says, "I don't know how to set up Wi-Fi." This causes the voice data to be captured by the microphone in the device.

[0149] Step 2:

[0150] Voice Recognition

[0151] The device converts the voice data it receives into text data. The input is voice data, and the output is text data. Specifically, the device's voice recognition module uses the Google Voice API to convert the voice data into text data such as "I don't know how to set up Wi-Fi."

[0152] Step 3:

[0153] Sending text data

[0154] The device sends the converted text data to the server. The input is text data, and the output is data transmission to the server. Specifically, the device's transceiver module sends the text data "I don't know how to set up Wi-Fi" to the server using Wi-Fi or 4G / 5G communication.

[0155] Step 4:

[0156] Receiving text data

[0157] The server receives text data sent from the user terminal. The input is the text data sent from the terminal, and the output is data storage within the server. Specifically, the server's communication interface receives the text data and stores it in a processing queue.

[0158] Step 5:

[0159] Processing by generative AI module

[0160] The server provides the received text data to a generative AI model to generate a solution or advice. The input is the text data and a prompt, and the output is a solution or advice. Specifically, the server sends the following prompt to the generative AI model (e.g., OpenAI's GPT-4):

[0161] When a user inquires about how to set up Wi-Fi, please provide a clear explanation of the steps to set up Wi-Fi.

[0162] The generative AI model generates the following solution: "Open the Settings app and go to the Wi-Fi settings section. Select the network you want to connect to from the list of Wi-Fi networks, enter the password, and tap Connect."

[0163] Step 6:

[0164] Submit a solution

[0165] The server sends the generated solutions and advice to the user's device. The input is the generated solution, and the output is data transmission to the user's device. Specifically, the server's transmission module prepares the generated text data and transmits it to the device via Wi-Fi or 4G / 5G communication.

[0166] Step 7:

[0167] View and listen to the solution

[0168] The device displays the received solution on the display and plays it back aloud using a Text-to-Speech (TTS) engine. The input is the solution sent from the server, and the output is the display and audio playback. Specifically, the device's receiving module receives the solution text data and displays it on the display as follows:

[0169] Open the Settings app and go to the Wi-Fi settings section. Select the network you want to connect to from the list of Wi-Fi networks, enter the password, and tap Connect.

[0170] In addition, the Google TTS engine converts the text data into speech and plays it back through the speaker.

[0171] (Application example 1)

[0172] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0173] Modern security systems have sophisticated and complex functions, and it is often difficult for average users to understand how to configure and use them. Furthermore, when a configuration error or problem occurs, it is difficult to quickly find a solution, potentially increasing security risks. Traditional support systems have the problem of taking a long time to respond, increasing the hassle and stress for users.

[0174] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0175] In this invention, the server includes means for receiving a problem or request provided by a user through voice input, means for converting the voice input into text data, means for transmitting the converted text data to the server, means for the server to receive the text data and generate a solution or advice using a generation AI module, means for transmitting the generated solution or advice to the user terminal, means for the user terminal to display and play the received solution or advice by voice, means for receiving a problem or request regarding the configuration and use of the security system, and means for generating a solution or advice regarding the configuration and use of the security system, thereby enabling a user to quickly and easily configure and troubleshoot the security system.

[0176] A "user terminal" is an electronic device that allows a user to voice-input a problem or request and receive a solution or advice.

[0177] "Server" means a remote system that receives text data sent from a user terminal and generates a solution or advice using a generative AI module.

[0178] "Voice input" refers to the act of a user providing a problem or request to a device by voice.

[0179] "Text data" is character information generated using voice recognition technology based on voice input.

[0180] The "generative AI module" is an artificial intelligence module that analyzes text data and generates appropriate solutions and advice.

[0181] "Solutions or Advice" refers to specific responses or instructions generated by the generative AI module in response to a user's problem or request.

[0182] "Security system" is a general term for surveillance cameras, alarms, access control devices, and other equipment used to keep a home or office safe.

[0183] "Display" is the act of visually presenting information on the display of a user terminal.

[0184] "Audio playback" refers to the act of outputting audio information using the speaker of the user device.

[0185] An "operational procedure" is a set of specific steps that a user must perform to achieve a particular goal.

[0186] "Configuration Instructions" are instructions on how to properly configure the security system.

[0187] "Personalization" is the act of generating solutions and advice optimized for each individual user based on the user's past operation history and settings information.

[0188] "Speech recognition" is a technology that converts voice input into text data.

[0189] The present invention is a system that uses a user terminal, a server, and a generation AI module to provide appropriate solutions and advice for problems and requests related to the configuration and use of a security system. Specific embodiments for implementing this system are described below.

[0190] 1. User Device

[0191] The user device is equipped with a voice recognition module that receives voice input and converts it into text data. This module can use the Google Speech-to-Text API. It also has a communication interface that transmits the input text data to the server. This allows users to input security system issues and requests by voice, and the content is converted into text data and sent to the server.

[0192] 2. Server

[0193] The server receives text data sent from the user's device and passes it to the generation AI module. The received text data is analyzed by the generation AI module, and optimal solutions and advice are generated. OpenAI's GPT-4 can be used as the generation AI module. This generated data is then sent back to the user's device.

[0194] 3. Generative AI Module

[0195] The generative AI module generates solutions and advice on security system configuration and usage based on the text data received by the server. This module has learned from a huge data set and can provide the most appropriate information for the user's problem.

[0196] 4. User Interface

[0197] The user device has the ability to display the received solutions and advice on the screen and play them back aloud using the Google Text-to-Speech API, allowing users to confirm the solutions and advice both visually and audibly.

[0198] Specific examples

[0199] For example, if a user says "my security camera isn't working," the following will be processed:

[0200] 1. The user speaks "My security camera isn't working."

[0201] 2. The user device converts the voice into text data and sends it to the server.

[0202] 3. The server receives the text data and provides it to the generation AI module.

[0203] 4. The generative AI module generates a solution such as "Please check the power supply of the security camera and restart it."

[0204] 5. The server sends the generated solution to the user device.

[0205] 6. The user device displays the received solution on the display and plays it back aloud.

[0206] Prompt Sentence Examples

[0207] "How do I reset the settings on my security cameras?"

[0208] "My alarm system is malfunctioning. Can you help me fix it?"

[0209] This allows users to quickly and easily troubleshoot security system problems.

[0210] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0211] Step 1:

[0212] Users input their security system problems and requests by voice. The input voice is picked up through the microphone of the user's device. Based on the input, the user's voice data is collected.

[0213] Step 2:

[0214] The voice data captured by the user device is converted into text data using voice recognition software (e.g., Google Speech-to-Text API). In this step, voice signals are analyzed using voice recognition technology and converted into corresponding strings of characters. The output is the converted text data.

[0215] Step 3:

[0216] The user terminal sends the converted text data to the server. Here, the data is sent to the server using the HTTPS protocol to ensure security. The input is the text data generated by speech recognition, and the output is the text data sent to the server.

[0217] Step 4:

[0218] The server receives the text data sent from the user terminal. In this step, the server is prepared to analyze the text data. The input is the text data sent from the user terminal, and the output is the provision of the text data to the generation AI module.

[0219] Step 5:

[0220] The server provides the received text data to a generative AI module (e.g., OpenAI GPT-4) to generate an optimal solution or advice. In this step, the AI ​​model analyzes the text data and generates a solution. The output is the generated solution or advice text data.

[0221] Step 6:

[0222] The server sends the solutions and advice generated by the generative AI module to the user's device. Here, data is transferred securely using the HTTPS protocol again. The input is the text data of the generated solutions and advice, and the output is the transmission of the solutions and advice to the user's device.

[0223] Step 7:

[0224] The user device displays the received solution or advice on the display and plays it back aloud. In this step, the Google Text-to-Speech API is used to convert the text data into speech and output it as speech. The input is the text data of the solution or advice received from the server, and the output is the text displayed on the display and the solution or advice played back aloud.

[0225] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0226] System Overview

[0227] The present invention relates to a system that recognizes a user's voice input and emotions and provides optimal solutions and advice to the user regarding the operation and settings of a mobile phone. By combining this with the recognition of the user's emotions, the system can provide more personalized support.

[0228] Key Components of the System

[0229] User device:

[0230] This device allows users to input voice and converts that voice into text data and emotion data. It is equipped with a voice recognition module, emotion engine, transmission / reception module, display, speaker, etc.

[0231] server:

[0232] It is a remote system that receives text data and emotion data, generates solutions and advice using a generative AI module, and adjusts them based on the emotion data.

[0233] Communication Interface:

[0234] A network module for sending and receiving data between a user terminal and a server.

[0235] Program processing overview

[0236] 1. Voice input

[0237] The user voices their problem or request.

[0238] 2. Speech and Emotion Recognition

[0239] The device converts the user's voice input into text data and simultaneously recognizes the user's emotions using an emotion engine.

[0240] 3. Data transmission

[0241] The terminal transmits the text data and the emotion data to the server.

[0242] 4. Receiving Data

[0243] The server receives the text data and emotion data sent from the user terminal.

[0244] 5. Processing by the Generative AI Module

[0245] The server provides the received text data to a generation AI module, which generates an appropriate solution or advice.

[0246] 6. Adjustments based on emotional data

[0247] The server tailors the generated solution or advice based on the received emotional data, for example providing more detailed explanations if the user is feeling anxious.

[0248] 7. Submitting Solutions or Advice

[0249] The server sends the tailored solution or advice to the user terminal.

[0250] 8. Displaying and playing audio solutions or advice

[0251] The solution or advice received by the terminal is displayed on the display and played aloud using a speech synthesis module.

[0252] Specific examples

[0253] Example 1: Wi-Fi settings

[0254] When a user says, "I don't know how to set up Wi-Fi," the device converts the user's voice into text data and simultaneously recognizes the user's anxiety using an emotion engine.

[0255] Audio Input:

[0256] The user says, "I don't know how to set up Wi-Fi."

[0257] Speech and emotion recognition:

[0258] The voice recognition module converts this into text data such as "I don't know how to set up Wi-Fi," and the emotion engine recognizes the user's anxiety.

[0259] Sending data:

[0260] The device sends text data such as "I don't know how to set up Wi-Fi" and emotional data such as "anxiety" to the server.

[0261] Receiving data:

[0262] The server receives the text data and the emotion data.

[0263] Processing by the Generative AI module:

[0264] The server generates a solution based on the text data, including instructions for setting up Wi-Fi.

[0265] Example response from the generative AI module: "Open the Settings app and go to the Wi-Fi settings section. From the list of Wi-Fi networks, select the network you want to connect to, enter the password, and tap Connect."

[0266] Adjustments based on sentiment data:

[0267] The server adjusts the generated solution based on the anxiety sentiment and provides a detailed explanation.

[0268] Sample tailored response: "Don't worry! First, open the Settings app on your phone. Then, go to the Wi-Fi settings section. Next, select your home network from the list of Wi-Fi networks that appears. Finally, enter your network password and tap the Connect button."

[0269] Submit a solution or advice:

[0270] The server sends the adjusted solution to the user terminal.

[0271] View and listen to solutions or advice:

[0272] The device will display the solution it receives on the display and play a voice message saying, "Don't worry. First, open your phone's Settings app..."

[0273] This example allows users to easily set up Wi-Fi without any worries. Combined with the emotion engine, it improves the user experience.

[0274] The processing flow will be explained below.

[0275] Step 1:

[0276] The user verbally states their problem or request, for example, "I don't know how to set up Wi-Fi."

[0277] Step 2:

[0278] The device receives the user's voice input and passes it to the speech recognition module, which converts the voice data into text data. Example: "I don't know how to set up Wi-Fi."

[0279] Step 3:

[0280] The device passes the user's voice input to the emotion engine, which analyzes the voice tone, speed, intonation, etc. to recognize the user's emotion. Example: Recognizes "anxiety"

[0281] Step 4:

[0282] The terminal transmits the converted text data and the recognized emotion data to the server.

[0283] Step 5:

[0284] The server receives the text data and emotion data sent from the user terminal.

[0285] Step 6:

[0286] The server inputs the received text data into a generation AI module, which generates an appropriate solution or advice. For example, a generated solution might be, "Open the Settings app and go to the Wi-Fi settings section. Select the network you want to connect to from the list of Wi-Fi networks, enter the password, and tap Connect."

[0287] Step 7:

[0288] The server tailors the generated solution or advice based on the emotion data received. For example, if the user is perceived as "anxious," it adds a detailed explanation to the solution. Example: "Don't worry. First, open the Settings app on your phone..."

[0289] Step 8:

[0290] The server sends the tailored solution or advice to the user terminal.

[0291] Step 9:

[0292] The terminal receives the solution or advice received from the server.

[0293] Step 10:

[0294] The device displays the received solution or advice on a display, and simultaneously plays the adjusted solution aloud using a speech synthesis module.

[0295] Step 11:

[0296] Users should follow the on-screen and audio-guided solutions to set up their phone by opening the Settings app, navigating to the Wi-Fi settings section, selecting the network they want to connect to from the list, entering the password, and tapping Connect.

[0297] Example 2

[0298] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0299] When operating or setting up a conventional mobile phone, users often become confused because they do not know the operating procedures or how to set it up. Furthermore, they may feel anxious or stressed because they are not provided with personalized support that is tailored to the user's emotions and situation. This can lead to a poor user experience and make it difficult to use the mobile phone.

[0300] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0301] In this invention, the server includes means for receiving a problem or request provided by a user through voice input, means for converting the voice input into text data, means for transmitting the converted text data and emotion data to the server, means for the server to receive the text data and emotion data and generate a solution or advice using a generative AI model, means for adjusting the generated solution or advice based on the received emotion data, means for transmitting the adjusted solution or advice to the user terminal, and means for the user terminal to display the received solution or advice and play it back by voice. This makes it possible to provide personalized support for mobile phone operations and settings while taking into consideration the user's emotions.

[0302] "Voice input" is the act of a user providing a problem or request by speaking into a mobile phone or other device.

[0303] "Text data" is data that has been converted from voice input into text information, and specifically expresses the content of a problem or request.

[0304] "Emotional data" is information about emotions extracted from the user's voice tone and expressions, and is data that represents the user's feelings and psychological state.

[0305] A "server" is a remote computer system that receives data from a user terminal via a communications network, processes the data using a generative AI model or the like, and transmits the results to the user terminal.

[0306] A "generative AI model" is an artificial intelligence model that automatically generates solutions and advice based on received text data, and is a program for generating specific answers and instructions.

[0307] "User device" refers to a device used by a user, such as a mobile phone or computer, for voice input and for receiving and displaying solutions and advice.

[0308] "Solutions or Advice" refers to instructions or guides on how to solve a user's problem or procedures that are generated by a generative AI model based on text data.

[0309] "Adjustment" refers to the act of changing the content and expression of the generated solution or advice to match the user's emotions based on the received emotion data.

[0310] The present invention relates to a system that recognizes a user's voice input and emotions and provides optimal solutions and advice to the user regarding the operation and settings of a mobile phone. By combining this with the recognition of the user's emotions, the system can provide more personalized support.

[0311] Hardware and Software Configuration

[0312] User Device

[0313] Speech recognition module: converts speech into text data. For example, you can use the Google Speech-to-Text API.

[0314] Emotion engine: Extracts emotion data from the user's voice. For example, it can use the Microsoft Azure Emotion API.

[0315] Transmitting / receiving module: Sends text data and emotion data to the server and receives responses from the server.

[0316] Display and speech synthesis module: displays solutions and advice and plays them aloud.

[0317] server

[0318] Communication interface: processes data received from the user terminal.

[0319] Generative AI models: Generate solutions and advice based on the received text data. For example, you can use the OpenAI GPT-3 model.

[0320] Data adjustment module: Adjusts the generated solutions and advice based on the received sentiment data.

[0321] Processing flow

[0322] 1. Receiving voice input

[0323] The user verbally describes their problem or request into their mobile phone.

[0324] 2. Speech and Emotion Recognition

[0325] The device uses a voice recognition module to convert voice into text data and an emotion engine to recognize the user's emotions.

[0326] 3. Data transmission

[0327] The text data and emotion data are transmitted to the server using a transmitting / receiving module.

[0328] 4. Processing by generative AI models

[0329] The server provides the received text data to a generative AI model, which generates appropriate solutions and advice.

[0330] For example, the prompt is:

[0331] If a user says they don't know how to set up Wi-Fi, explain the process in detail. Be sensitive to their concerns.

[0332] 5. Adjustments based on emotional data

[0333] The server adjusts the generated solutions and advice based on the received emotion data.

[0334] 6. Submitting and Displaying Solutions or Advice

[0335] The tailored solutions and advice are sent from the server to the user's device, which displays them on the display and plays them aloud using a speech synthesis module.

[0336] Specific examples

[0337] Example 1: Wi-Fi settings

[0338] When a user says, "I don't know how to set up Wi-Fi," the device converts the user's voice into text data and detects their anxiety.

[0339] 1. Voice input

[0340] The user says, "I don't know how to set up Wi-Fi."

[0341] 2. Speech and Emotion Recognition

[0342] The voice recognition module converts "I don't know how to set up Wi-Fi" into text data, and the emotion engine recognizes the user's anxiety.

[0343] 3. Data transmission

[0344] The device sends the text data "I don't know how to set up Wi-Fi" and the emotional data "anxiety" to the server.

[0345] 4. Processing by generative AI models

[0346] The server generates Wi-Fi setting instructions based on the text data.

[0347] Example response: "Open the Settings app and go to the Wi-Fi settings section. From the list of Wi-Fi networks, select the network you want to connect to, enter the password, and tap Connect."

[0348] 5. Adjustments based on emotional data

[0349] The server adjusts the generated solutions based on anxiety sentiment and provides detailed explanations.

[0350] Sample tailored response: "Don't worry! First, open the Settings app on your phone. Then, go to the Wi-Fi settings section. Next, select your home network from the list of Wi-Fi networks that appears. Finally, enter your network password and tap the Connect button."

[0351] 6. Displaying and playing audio solutions or advice

[0352] The device will display the received solution on its display and play a voice message saying, "Don't worry. First, open the Settings app on your phone..."

[0353] This system allows users to set up Wi-Fi seamlessly and without worry. The combination of an emotion engine and generative AI models improves the user experience.

[0354] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0355] Step 1:

[0356] Receiving audio input

[0357] user:

[0358] The user verbally describes their problem or request into their mobile phone.

[0359] For example, say, "I don't know how to set up Wi-Fi."

[0360] Input: User speech.

[0361] Output: Audio data.

[0362] Step 2:

[0363] Speech and Emotion Recognition

[0364] Device:

[0365] A speech recognition module converts the user's voice data into text data, for example, using the Google Speech-to-Text API.

[0366] The emotion engine analyzes the voice data and extracts emotion data, for example, using the Microsoft Azure Emotion API.

[0367] Input: Audio data.

[0368] Data processing: The voice data is analyzed, and the voice recognition module converts the voice into text data. The emotion engine also extracts emotional information from the voice.

[0369] Output: Text and sentiment data.

[0370] Step 3:

[0371] Sending data

[0372] Device:

[0373] The generated text data and emotion data are sent to the server using a transmission / reception module.

[0374] Input: Text data and sentiment data.

[0375] Data processing: Assembling text data and emotion data into data packets.

[0376] Output: Send the data packet to the server over the communication network.

[0377] Step 4:

[0378] Receiving data

[0379] server:

[0380] A data packet is received from a user terminal through a communication interface.

[0381] Input: Data packet.

[0382] Data processing: Analyze data packets and extract text and sentiment data.

[0383] Output: Extracted text and sentiment data.

[0384] Step 5:

[0385] Processing by generative AI models

[0386] server:

[0387] The received text data is fed into a generative AI model (e.g., OpenAI GPT-3) to generate solutions or advice.

[0388] Example prompt: "If the user says they don't know how to set up Wi-Fi, explain the process in detail. Be sensitive to their concerns."

[0389] Input: Text data.

[0390] Data processing: Input text data into a generative AI model to obtain generated solutions or advice.

[0391] Output: The generated solution or advice.

[0392] Step 6:

[0393] Adjustments based on emotional data

[0394] server:

[0395] Tailor generated solutions and advice based on sentiment data.

[0396] For example: If the user is feeling unsure, add a detailed explanation.

[0397] Input: Generated solution or advice, sentiment data.

[0398] Data processing: Analyzing sentiment data and making adjustments based on generated solutions and advice.

[0399] Output: Tailored solutions or advice.

[0400] Step 7:

[0401] Submit a solution or advice

[0402] server:

[0403] Send tailored solutions and advice to your device.

[0404] Input: Tailored solutions or advice.

[0405] Data processing: Consolidating tailored solutions and advice into data packets.

[0406] Output: Sends data packets over the communications network to the user terminal.

[0407] Step 8:

[0408] Display and audio playback of solutions or advice

[0409] Device:

[0410] The received solutions and advice are displayed on the screen and played aloud using a speech synthesis module.

[0411] Example: Display and play the following: "Don't worry. First, open the Settings app on your phone..."

[0412] Input: Tailored solutions or advice.

[0413] Data processing: Converting tailored solutions and advice into display and speech synthesis.

[0414] Output: Display and audio playback.

[0415] (Application example 2)

[0416] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0417] Conventional support systems for setting up and operating mobile phones and communication devices simply convert the user's voice input into text data and provide solutions and advice. However, because they do not take the user's emotions into consideration, the support content often does not match the user's current situation, resulting in an unsatisfactory user experience. Furthermore, users who are feeling particularly anxious or confused need appropriate advice that addresses their emotions. To solve these issues, a more personalized support system that incorporates the user's emotions is needed.

[0418] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0419] In this invention, the server includes means for receiving a problem or request provided by a user through voice input, means for converting the voice input into text data and emotional data, means for transmitting the converted text data and emotional data to the server, means for the server to receive the text data and emotional data and generate a solution or advice using a generation AI module, means for adjusting the solution or advice based on the user's emotional data, means for transmitting the generated solution or advice to a user terminal, and means for the user terminal to display the received solution or advice and play it back by voice, thereby enabling personalized support that reflects the user's emotions.

[0420] "Voice input" is voice data provided by the user through a microphone.

[0421] "Emotion data" is data that indicates the user's emotions analyzed from voice input.

[0422] "User terminal" refers to a device operated by a user that accepts voice input and displays and plays solutions and advice.

[0423] A "generative AI module" is a program that runs within the server and includes artificial intelligence that generates solutions and advice based on received text data.

[0424] The "server" is a remote system that receives text data and emotion data sent from a user terminal, generates solutions or advice using a generative AI module, and sends the solutions or advice to the user terminal.

[0425] "Text data" is character data converted from voice input using voice recognition technology.

[0426] "Solutions or Advice" refers to problem-solving steps or advice provided by the generative AI module based on text data and emotion data.

[0427] A "voice recognition module" is a program or hardware that converts voice input into text data.

[0428] A "communication interface" is a network means for transmitting and receiving data between a user terminal and a server.

[0429] "Personalization" is a means of providing settings and advice optimized for each user.

[0430] System configuration

[0431] The present invention is a system including: means for receiving a problem or request provided by a user through voice input; means for converting the voice input into text data and emotional data; means for transmitting the text data and emotional data to a server; means for the server to receive the text data and emotional data and generate a solution or advice using a generation AI module; means for adjusting the solution or advice based on the user's emotional data; means for transmitting the generated solution or advice to a user terminal; and means for the user terminal to display the received solution or advice and play it back aloud.

[0432] Hardware and Software Configuration

[0433] User device:

[0434] The user terminal is a device equipped with a voice recognition module, an emotion engine, a transmission / reception module, a display, a speaker, etc.

[0435] server:

[0436] The server is a remote system that receives text data and emotion data, generates solutions and advice using a generative AI module, and sends them to the user's device.

[0437] Communication Interface:

[0438] The communication interface is a network module for transmitting and receiving data between the user terminal and the server.

[0439] Program processing overview

[0440] First, the user inputs voice. The user device uses a voice recognition module to convert the voice input into text data, and then uses an emotion engine to analyze the user's emotions. The acquired text data and emotion data are sent to the server via a communication interface.

[0441] The server receives text data and emotion data sent from the user's device. The generative AI module generates problem-solving procedures and advice based on the received text data. The generated solutions and advice are adjusted based on the user's emotion data.

[0442] The adjusted solution or advice is then sent back to the user terminal via the communication interface, where it is displayed on the display and played aloud using a speech synthesis module.

[0443] Examples of concrete examples and prompts

[0444] For example, if a user says, "I want a new smartphone," the speech recognition module converts the speech into text and generates the text data, "I want a new smartphone." At the same time, the emotion engine recognizes the user's excited emotion.

[0445] An example of user input is:

[0446] Prompt Sentence Examples

[0447] User input: "I want a new smartphone"

[0448] User Sentiment: Excitement

[0449] The generative AI module generates the following recommendations based on this text data and emotion data:

[0450] "The best smartphones include those with the latest camera technology and long battery life. Our most popular models take your experience to the next level. Want to learn more?"

[0451] In this way, the system provides personalized support that reflects the user's emotions.

[0452] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0453] Step 1:

[0454] The user provides voice input.

[0455] Input: User speech (e.g., "I want a new smartphone")

[0456] Action: The user speaks into the device's microphone.

[0457] Output: User's voice data

[0458] Step 2:

[0459] The terminal uses a voice recognition module to convert the voice input into text data.

[0460] Input: Audio data

[0461] Data processing: Converting voice into text using voice recognition technology

[0462] How it works: The speech recognition module analyzes the audio signal and generates corresponding text.

[0463] Output: Text data (e.g., "I want a new smartphone")

[0464] Step 3:

[0465] The device uses an emotion engine to recognize the user's emotions.

[0466] Input: Audio data

[0467] Data calculation: Extracting features such as pitch, intonation, and speed from speech to estimate emotions

[0468] How it works: The emotion engine analyzes voice data and identifies the user's emotion.

[0469] Output: Emotion data (e.g., excitement)

[0470] Step 4:

[0471] The terminal transmits the text data and the emotion data to the server.

[0472] Input: Text data, emotion data

[0473] Operation: The sending and receiving module assembles data into packets and sends them to the server through the communication interface.

[0474] Output: Text data and emotion data sent to the server

[0475] Step 5:

[0476] A server receives the text data and the emotion data.

[0477] Input: Text data, emotion data

[0478] Operation: The server receives data through the communication interface and stores it in storage.

[0479] Output: Text data and emotion data stored in the server storage

[0480] Step 6:

[0481] The server uses a generative AI module to generate a solution or advice based on the text data.

[0482] Input: Text data

[0483] Data calculation: Generative AI module analyzes text data and generates appropriate solutions or advice

[0484] How it works: The generative AI module takes text data and uses algorithms to generate solutions or advice.

[0485] Output: Solution or advice (e.g., "We recommend a smartphone with the latest camera technology.")

[0486] Step 7:

[0487] The server tailors the solution or advice based on the user's emotional data.

[0488] Input: Solution or advice, sentiment data

[0489] Data processing: Using sentiment data to enhance and modify solutions and advice

[0490] How it works: The server references the emotion data and adjusts the content of the generated solution or advice to match the emotion.

[0491] Output: Tailored solution or advice (e.g., "You seem excited. In that case, we recommend a smartphone with the latest camera technology.")

[0492] Step 8:

[0493] The server sends the tailored solution or advice to the user terminal.

[0494] Input: Tailored solution or advice

[0495] Operation: The server assembles the adjusted solution or advice into a packet via the sending and receiving module and sends it to the user terminal through the communication interface.

[0496] Output: Tailored solution or advice sent to user terminal

[0497] Step 9:

[0498] The user device displays and plays audibly the tailored solution or advice received.

[0499] Input: Tailored solution or advice

[0500] How it works: The user device displays the solution or advice on the display and plays it aloud using a speech synthesis module.

[0501] Output: Displayed solution or advice, audio played (e.g., "You seem excited. In that case, we recommend a smartphone with the latest camera technology.")

[0502] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0503] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0504] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0505] [Second embodiment]

[0506] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0507] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0508] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0509] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0510] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0511] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0512] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0513] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0514] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0515] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0516] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0517] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0518] System Overview

[0519] This invention is a system for easily resolving problems users may encounter when operating or configuring a mobile phone. This system involves a series of processes: converting user voice input into text data, sending it to a server, generating solutions or advice using a generation AI module, and returning it to the user's device.

[0520] Key Components of the System

[0521] User device:

[0522] A device that allows users to input voice information. It includes a voice recognition module, a transmission / reception module, a display, and a speaker.

[0523] server:

[0524] A remote system that receives voice-recognized text data and generates solutions or advice using a generative AI module.

[0525] Communication Interface:

[0526] A network module for sending and receiving data between a user terminal and a server.

[0527] Program processing overview

[0528] 1. Voice input

[0529] Users communicate their problems and requests via voice.

[0530] 2. Voice Recognition

[0531] The device converts the user's voice input into text data.

[0532] 3. Sending text data

[0533] The terminal transmits the converted text data to the server.

[0534] 4. Receiving text data

[0535] The server receives the text data sent from the user terminal.

[0536] 5. Processing by the Generative AI Module

[0537] The server provides the received text to a generative AI module, which generates an appropriate solution or advice.

[0538] 6. Submitting Solutions or Advice

[0539] The server transmits the generated solution or advice to the user terminal.

[0540] 7. Displaying and playing audio solutions or advice

[0541] The device displays the received solution or advice on the display and plays it aloud through the speaker.

[0542] Specific examples

[0543] Example 1: Wi-Fi settings

[0544] When a user says, "I don't know how to set up Wi-Fi," the following process occurs:

[0545] Audio Input:

[0546] The user says, "I don't know how to set up Wi-Fi."

[0547] Voice Recognition:

[0548] The device converts the voice into text data: "I don't know how to set up Wi-Fi."

[0549] Sending text data:

[0550] The device sends text data to the server saying "I don't know how to set up Wi-Fi."

[0551] Receiving text data:

[0552] The server receives the text data.

[0553] Processing by the Generative AI module:

[0554] The server uses the text data to generate a solution that includes instructions for setting up Wi-Fi.

[0555] Example response from the generative AI module: "Open the Settings app and go to the Wi-Fi settings section. From the list of Wi-Fi networks, select the network you want to connect to, enter the password, and tap Connect."

[0556] Submit a solution or advice:

[0557] The server sends the generated solution to the user terminal.

[0558] View and listen to solutions or advice:

[0559] The device will show the solution it received on the display and play a voice message saying, "Open the Settings app and go to the Wi-Fi settings section. From the list of Wi-Fi networks, select the network you want to connect to, enter the password, and tap Connect."

[0560] This allows users to solve problems themselves and smoothly set up their mobile phones. The advantage of this system is that it provides fast and accurate support.

[0561] The processing flow will be explained below.

[0562] Step 1:

[0563] The user verbally states their problem or request, for example, "I don't know how to set up Wi-Fi."

[0564] Step 2:

[0565] The device receives the user's voice input and passes it to the speech recognition module, which converts the voice data into text data. Example: "I don't know how to set up Wi-Fi."

[0566] Step 3:

[0567] The terminal transmits the converted text data to the server.

[0568] Step 4:

[0569] The server receives the text data sent from the user terminal.

[0570] Step 5:

[0571] The server inputs the received text data into a generation AI module to generate a solution or advice. For example, a generated solution might be, "Open the Settings app and go to the Wi-Fi settings section. Select the network you want to connect to from the list of Wi-Fi networks, enter the password, and tap Connect."

[0572] Step 6:

[0573] The server transmits the generated solution or advice to the user terminal.

[0574] Step 7:

[0575] The terminal receives the solution or advice received from the server.

[0576] Step 8:

[0577] The device displays the received solution or advice on the display, and simultaneously plays back the solution aloud using a speech synthesis module.

[0578] Step 9:

[0579] Users can follow the solution displayed on their device to configure their phone: open the Settings app, go to the Wi-Fi settings section, select the network they want to connect to from the list, enter the password, and tap Connect.

[0580] Example 1

[0581] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0582] Conventional support systems for mobile phone operation and settings make it difficult for users to solve problems directly and quickly. In particular, systems that allow users to solve problems through voice input are limited, and in many cases, complex operating procedures are required. As a result, users are more likely to make operating or setting errors. In response to this, the present invention aims to provide a system that allows users to easily report problems through voice input and quickly provides solutions using a generative AI module.

[0583] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0584] In this invention, the server includes a means for receiving a problem or request provided by a user through voice input, a means for converting the voice input into text data, and a means for transmitting the converted text data to the server, so that the user can report a problem through voice, which is converted into text data and transmitted to the server, thereby enabling a solution to be generated and provided to the user for quickly and accurately resolving the problem.

[0585] "User" refers to the entity that operates the system and provides problems or requests using voice input.

[0586] "Voice input" refers to a means by which a user communicates problems or requests to a system using their voice.

[0587] "Text data" refers to data that has been converted from voice input into a character string format.

[0588] "Server" refers to the remote system that receives the text data converted from the voice input and generates solutions or advice using a generative AI module.

[0589] "Generative AI module" refers to an artificial intelligence algorithm that generates appropriate solutions or advice based on text data received from the user.

[0590] A "prompt sentence" refers to an instruction sentence for generating an output for solving a problem based on text data input to the generation AI module.

[0591] "Communication interface" refers to a network module for transmitting and receiving data between a user terminal and a server.

[0592] "User terminal" refers to a device through which a user makes voice input, and which is equipped with a voice recognition module, a display, and a speaker.

[0593] The present invention is a system that allows users to report problems related to mobile phone operation and settings through voice input, and uses a generative AI module to quickly and accurately provide solutions. The system includes the following main components: a user terminal, a server, and a communication interface.

[0594] System configuration

[0595] User Device

[0596] A user terminal is a device that allows users to input voice. It includes a microphone, display, and speaker as hardware, and a speech recognition module as software. Specifically, speech recognition software such as Google Voice API or IBM Watson is used.

[0597] server

[0598] The server is a remote system that receives text data sent from the user's device and generates solutions and advice using a generative AI model, such as OpenAI's GPT-4.

[0599] Communication Interface

[0600] The communication interface serves to send and receive data between the user device and the server, specifically via Wi-Fi or 4G / 5G communication.

[0601] Processing the data

[0602] 1. Voice input: The user voices the problem they are having with their phone's operation or settings. For example, they say, "I don't know how to set up Wi-Fi."

[0603] 2. Speech recognition: The device's speech recognition module records the user's voice and converts it into text data in real time. The Google Voice API converts the voice into text data such as "I don't know how to set up Wi-Fi."

[0604] 3. Sending text data: The converted text data is sent to the server, and the device's transceiver module transmits the data via Wi-Fi or 4G / 5G communication.

[0605] 4. Receiving text data: The server receives the text data and stores the received data in a processing queue.

[0606] 5. Processing by the generative AI module: The server generates a prompt and provides it to the generative AI model. For example, it provides the following prompt:

[0607] When a user inquires about how to set up Wi-Fi, please provide a clear explanation of the steps to set up Wi-Fi.

[0608] A generative AI model (e.g., OpenAI's GPT-4) generates a solution, such as "Open the Settings app and go to the Wi-Fi settings section. From the list of Wi-Fi networks, select the network you want to connect to, enter the password, and tap Connect."

[0609] 6. Sending the solution: The generated solution is sent to the user terminal, and the data is sent back through the communication interface.

[0610] 7. Display and audio solution: Your device will display and audio the solution using a Text-to-Speech (TTS) engine (e.g., Google TTS), for example, "Open the Settings app and go to the Wi-Fi settings section. From the list of Wi-Fi networks, select the network you want to connect to, enter the password, and tap Connect."

[0611] Specific examples

[0612] Example 1: Wi-Fi settings

[0613] When a user says, "I don't know how to set up Wi-Fi," the following sequence of events occurs:

[0614] 1. Voice input: The user says, "I don't know how to set up Wi-Fi."

[0615] 2. Speech recognition: The device uses the Google Voice API to convert the speech into text data such as "I don't know how to set up Wi-Fi."

[0616] 3. Sending text data: The device sends text data saying "I don't know how to set up Wi-Fi" to the server.

[0617] 4. Receiving text data: The server receives the text data.

[0618] 5. Processing by the generative AI module: The server provides the received text data to the generative AI model, which generates a solution including instructions for setting up Wi-Fi.

[0619] 6. Sending the solution: The server sends the generated solution to the user terminal.

[0620] 7. Displaying the solution and playing it aloud: The device displays the solution it received on the display and plays it aloud using Google TTS.

[0621] As described above, the system of the present invention allows a user to report a problem by voice, which is converted into text data and sent to a server, and then a solution to quickly and accurately resolve the problem can be generated and provided to the user.

[0622] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0623] Step 1:

[0624] Voice input

[0625] The user communicates problems or requests regarding the settings of their mobile phone by voice. The input is the user's voice data. Specifically, the user says, "I don't know how to set up Wi-Fi." This causes the voice data to be captured by the microphone in the device.

[0626] Step 2:

[0627] Voice Recognition

[0628] The device converts the voice data it receives into text data. The input is voice data, and the output is text data. Specifically, the device's voice recognition module uses the Google Voice API to convert the voice data into text data such as "I don't know how to set up Wi-Fi."

[0629] Step 3:

[0630] Sending text data

[0631] The device sends the converted text data to the server. The input is text data, and the output is data transmission to the server. Specifically, the device's transceiver module sends the text data "I don't know how to set up Wi-Fi" to the server using Wi-Fi or 4G / 5G communication.

[0632] Step 4:

[0633] Receiving text data

[0634] The server receives text data sent from the user terminal. The input is the text data sent from the terminal, and the output is data storage within the server. Specifically, the server's communication interface receives the text data and stores it in a processing queue.

[0635] Step 5:

[0636] Processing by generative AI module

[0637] The server provides the received text data to a generative AI model to generate a solution or advice. The input is the text data and a prompt, and the output is a solution or advice. Specifically, the server sends the following prompt to the generative AI model (e.g., OpenAI's GPT-4):

[0638] When a user inquires about how to set up Wi-Fi, please provide a clear explanation of the steps to set up Wi-Fi.

[0639] The generative AI model generates the following solution: "Open the Settings app and go to the Wi-Fi settings section. Select the network you want to connect to from the list of Wi-Fi networks, enter the password, and tap Connect."

[0640] Step 6:

[0641] Submit a solution

[0642] The server sends the generated solutions and advice to the user's device. The input is the generated solution, and the output is data transmission to the user's device. Specifically, the server's transmission module prepares the generated text data and transmits it to the device via Wi-Fi or 4G / 5G communication.

[0643] Step 7:

[0644] View and listen to the solution

[0645] The device displays the received solution on the display and plays it back aloud using a Text-to-Speech (TTS) engine. The input is the solution sent from the server, and the output is the display and audio playback. Specifically, the device's receiving module receives the solution text data and displays it on the display as follows:

[0646] Open the Settings app and go to the Wi-Fi settings section. Select the network you want to connect to from the list of Wi-Fi networks, enter the password, and tap Connect.

[0647] In addition, the Google TTS engine converts the text data into speech and plays it back through the speaker.

[0648] (Application example 1)

[0649] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0650] Modern security systems have sophisticated and complex functions, and it is often difficult for average users to understand how to configure and use them. Furthermore, when a configuration error or problem occurs, it is difficult to quickly find a solution, potentially increasing security risks. Traditional support systems have the problem of taking a long time to respond, increasing the hassle and stress for users.

[0651] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0652] In this invention, the server includes means for receiving a problem or request provided by a user through voice input, means for converting the voice input into text data, means for transmitting the converted text data to the server, means for the server to receive the text data and generate a solution or advice using a generation AI module, means for transmitting the generated solution or advice to the user terminal, means for the user terminal to display and play the received solution or advice by voice, means for receiving a problem or request regarding the configuration and use of the security system, and means for generating a solution or advice regarding the configuration and use of the security system, thereby enabling a user to quickly and easily configure and troubleshoot the security system.

[0653] A "user terminal" is an electronic device that allows a user to voice-input a problem or request and receive a solution or advice.

[0654] "Server" means a remote system that receives text data sent from a user terminal and generates a solution or advice using a generative AI module.

[0655] "Voice input" refers to the act of a user providing a problem or request to a device by voice.

[0656] "Text data" is character information generated using voice recognition technology based on voice input.

[0657] The "generative AI module" is an artificial intelligence module that analyzes text data and generates appropriate solutions and advice.

[0658] "Solutions or Advice" refers to specific responses or instructions generated by the generative AI module in response to a user's problem or request.

[0659] "Security system" is a general term for surveillance cameras, alarms, access control devices, and other equipment used to keep a home or office safe.

[0660] "Display" is the act of visually presenting information on the display of a user terminal.

[0661] "Audio playback" refers to the act of outputting audio information using the speaker of the user device.

[0662] An "operational procedure" is a set of specific steps that a user must perform to achieve a particular goal.

[0663] "Configuration Instructions" are instructions on how to properly configure the security system.

[0664] "Personalization" is the act of generating solutions and advice optimized for each individual user based on the user's past operation history and settings information.

[0665] "Speech recognition" is a technology that converts voice input into text data.

[0666] The present invention is a system that uses a user terminal, a server, and a generation AI module to provide appropriate solutions and advice for problems and requests related to the configuration and use of a security system. Specific embodiments for implementing this system are described below.

[0667] 1. User Device

[0668] The user device is equipped with a voice recognition module that receives voice input and converts it into text data. This module can use the Google Speech-to-Text API. It also has a communication interface that transmits the input text data to the server. This allows users to input security system issues and requests by voice, and the content is converted into text data and sent to the server.

[0669] 2. Server

[0670] The server receives text data sent from the user's device and passes it to the generation AI module. The received text data is analyzed by the generation AI module, and optimal solutions and advice are generated. OpenAI's GPT-4 can be used as the generation AI module. This generated data is then sent back to the user's device.

[0671] 3. Generative AI Module

[0672] The generative AI module generates solutions and advice on security system configuration and usage based on the text data received by the server. This module has learned from a huge data set and can provide the most appropriate information for the user's problem.

[0673] 4. User Interface

[0674] The user device has the ability to display the received solutions and advice on the screen and play them back aloud using the Google Text-to-Speech API, allowing users to confirm the solutions and advice both visually and audibly.

[0675] Specific examples

[0676] For example, if a user says "my security camera isn't working," the following will be processed:

[0677] 1. The user speaks "My security camera isn't working."

[0678] 2. The user device converts the voice into text data and sends it to the server.

[0679] 3. The server receives the text data and provides it to the generation AI module.

[0680] 4. The generative AI module generates a solution such as "Please check the power supply of the security camera and restart it."

[0681] 5. The server sends the generated solution to the user device.

[0682] 6. The user device displays the received solution on the display and plays it back aloud.

[0683] Prompt Sentence Examples

[0684] "How do I reset the settings on my security cameras?"

[0685] "My alarm system is malfunctioning. Can you help me fix it?"

[0686] This allows users to quickly and easily troubleshoot security system problems.

[0687] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0688] Step 1:

[0689] Users input their security system problems and requests by voice. The input voice is picked up through the microphone of the user's device. Based on the input, the user's voice data is collected.

[0690] Step 2:

[0691] The voice data captured by the user device is converted into text data using voice recognition software (e.g., Google Speech-to-Text API). In this step, voice signals are analyzed using voice recognition technology and converted into corresponding strings of characters. The output is the converted text data.

[0692] Step 3:

[0693] The user terminal sends the converted text data to the server. Here, the data is sent to the server using the HTTPS protocol to ensure security. The input is the text data generated by speech recognition, and the output is the text data sent to the server.

[0694] Step 4:

[0695] The server receives the text data sent from the user terminal. In this step, the server is prepared to analyze the text data. The input is the text data sent from the user terminal, and the output is the provision of the text data to the generation AI module.

[0696] Step 5:

[0697] The server provides the received text data to a generative AI module (e.g., OpenAI GPT-4) to generate an optimal solution or advice. In this step, the AI ​​model analyzes the text data and generates a solution. The output is the generated solution or advice text data.

[0698] Step 6:

[0699] The server sends the solutions and advice generated by the generative AI module to the user's device. Here, data is transferred securely using the HTTPS protocol again. The input is the text data of the generated solutions and advice, and the output is the transmission of the solutions and advice to the user's device.

[0700] Step 7:

[0701] The user device displays the received solution or advice on the display and plays it back aloud. In this step, the Google Text-to-Speech API is used to convert the text data into speech and output it as speech. The input is the text data of the solution or advice received from the server, and the output is the text displayed on the display and the solution or advice played back aloud.

[0702] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0703] System Overview

[0704] The present invention relates to a system that recognizes a user's voice input and emotions and provides optimal solutions and advice to the user regarding the operation and settings of a mobile phone. By combining this with the recognition of the user's emotions, the system can provide more personalized support.

[0705] Key Components of the System

[0706] User device:

[0707] This device allows users to input voice and converts that voice into text data and emotion data. It is equipped with a voice recognition module, emotion engine, transmission / reception module, display, speaker, etc.

[0708] server:

[0709] It is a remote system that receives text data and emotion data, generates solutions and advice using a generative AI module, and adjusts them based on the emotion data.

[0710] Communication Interface:

[0711] A network module for sending and receiving data between a user terminal and a server.

[0712] Program processing overview

[0713] 1. Voice input

[0714] The user voices their problem or request.

[0715] 2. Speech and Emotion Recognition

[0716] The device converts the user's voice input into text data and simultaneously recognizes the user's emotions using an emotion engine.

[0717] 3. Data transmission

[0718] The terminal transmits the text data and the emotion data to the server.

[0719] 4. Receiving Data

[0720] The server receives the text data and emotion data sent from the user terminal.

[0721] 5. Processing by the Generative AI Module

[0722] The server provides the received text data to a generation AI module, which generates an appropriate solution or advice.

[0723] 6. Adjustments based on emotional data

[0724] The server tailors the generated solution or advice based on the received emotional data, for example providing more detailed explanations if the user is feeling anxious.

[0725] 7. Submitting Solutions or Advice

[0726] The server sends the tailored solution or advice to the user terminal.

[0727] 8. Displaying and playing audio solutions or advice

[0728] The solution or advice received by the terminal is displayed on the display and played aloud using a speech synthesis module.

[0729] Specific examples

[0730] Example 1: Wi-Fi settings

[0731] When a user says, "I don't know how to set up Wi-Fi," the device converts the user's voice into text data and simultaneously recognizes the user's anxiety using an emotion engine.

[0732] Audio Input:

[0733] The user says, "I don't know how to set up Wi-Fi."

[0734] Speech and emotion recognition:

[0735] The voice recognition module converts this into text data such as "I don't know how to set up Wi-Fi," and the emotion engine recognizes the user's anxiety.

[0736] Sending data:

[0737] The device sends text data such as "I don't know how to set up Wi-Fi" and emotional data such as "anxiety" to the server.

[0738] Receiving data:

[0739] The server receives the text data and the emotion data.

[0740] Processing by the Generative AI module:

[0741] The server generates a solution based on the text data, including instructions for setting up Wi-Fi.

[0742] Example response from the generative AI module: "Open the Settings app and go to the Wi-Fi settings section. From the list of Wi-Fi networks, select the network you want to connect to, enter the password, and tap Connect."

[0743] Adjustments based on sentiment data:

[0744] The server adjusts the generated solution based on the anxiety sentiment and provides a detailed explanation.

[0745] Sample tailored response: "Don't worry! First, open the Settings app on your phone. Then, go to the Wi-Fi settings section. Next, select your home network from the list of Wi-Fi networks that appears. Finally, enter your network password and tap the Connect button."

[0746] Submit a solution or advice:

[0747] The server sends the adjusted solution to the user terminal.

[0748] View and listen to solutions or advice:

[0749] The device will display the solution it receives on the display and play a voice message saying, "Don't worry. First, open your phone's Settings app..."

[0750] This example allows users to easily set up Wi-Fi without any worries. Combined with the emotion engine, it improves the user experience.

[0751] The processing flow will be explained below.

[0752] Step 1:

[0753] The user verbally states their problem or request, for example, "I don't know how to set up Wi-Fi."

[0754] Step 2:

[0755] The device receives the user's voice input and passes it to the speech recognition module, which converts the voice data into text data. Example: "I don't know how to set up Wi-Fi."

[0756] Step 3:

[0757] The device passes the user's voice input to the emotion engine, which analyzes the voice tone, speed, intonation, etc. to recognize the user's emotion. Example: Recognizes "anxiety"

[0758] Step 4:

[0759] The terminal transmits the converted text data and the recognized emotion data to the server.

[0760] Step 5:

[0761] The server receives the text data and emotion data sent from the user terminal.

[0762] Step 6:

[0763] The server inputs the received text data into a generation AI module, which generates an appropriate solution or advice. For example, a generated solution might be, "Open the Settings app and go to the Wi-Fi settings section. Select the network you want to connect to from the list of Wi-Fi networks, enter the password, and tap Connect."

[0764] Step 7:

[0765] The server tailors the generated solution or advice based on the emotion data received. For example, if the user is perceived as "anxious," it adds a detailed explanation to the solution. Example: "Don't worry. First, open the Settings app on your phone..."

[0766] Step 8:

[0767] The server sends the tailored solution or advice to the user terminal.

[0768] Step 9:

[0769] The terminal receives the solution or advice received from the server.

[0770] Step 10:

[0771] The device displays the received solution or advice on a display, and simultaneously plays the adjusted solution aloud using a speech synthesis module.

[0772] Step 11:

[0773] Users should follow the on-screen and audio-guided solutions to set up their phone by opening the Settings app, navigating to the Wi-Fi settings section, selecting the network they want to connect to from the list, entering the password, and tapping Connect.

[0774] Example 2

[0775] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0776] When operating or setting up a conventional mobile phone, users often become confused because they do not know the operating procedures or how to set it up. Furthermore, they may feel anxious or stressed because they are not provided with personalized support that is tailored to the user's emotions and situation. This can lead to a poor user experience and make it difficult to use the mobile phone.

[0777] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0778] In this invention, the server includes means for receiving a problem or request provided by a user through voice input, means for converting the voice input into text data, means for transmitting the converted text data and emotion data to the server, means for the server to receive the text data and emotion data and generate a solution or advice using a generative AI model, means for adjusting the generated solution or advice based on the received emotion data, means for transmitting the adjusted solution or advice to the user terminal, and means for the user terminal to display the received solution or advice and play it back by voice. This makes it possible to provide personalized support for mobile phone operations and settings while taking into consideration the user's emotions.

[0779] "Voice input" is the act of a user providing a problem or request by speaking into a mobile phone or other device.

[0780] "Text data" is data that has been converted from voice input into text information, and specifically expresses the content of a problem or request.

[0781] "Emotional data" is information about emotions extracted from the user's voice tone and expressions, and is data that represents the user's feelings and psychological state.

[0782] A "server" is a remote computer system that receives data from a user terminal via a communications network, processes the data using a generative AI model or the like, and transmits the results to the user terminal.

[0783] A "generative AI model" is an artificial intelligence model that automatically generates solutions and advice based on received text data, and is a program for generating specific answers and instructions.

[0784] "User device" refers to a device used by a user, such as a mobile phone or computer, for voice input and for receiving and displaying solutions and advice.

[0785] "Solutions or Advice" refers to instructions or guides on how to solve a user's problem or procedures that are generated by a generative AI model based on text data.

[0786] "Adjustment" refers to the act of changing the content and expression of the generated solution or advice to match the user's emotions based on the received emotion data.

[0787] The present invention relates to a system that recognizes a user's voice input and emotions and provides optimal solutions and advice to the user regarding the operation and settings of a mobile phone. By combining this with the recognition of the user's emotions, the system can provide more personalized support.

[0788] Hardware and Software Configuration

[0789] User Device

[0790] Speech recognition module: converts speech into text data. For example, you can use the Google Speech-to-Text API.

[0791] Emotion engine: Extracts emotion data from the user's voice. For example, it can use the Microsoft Azure Emotion API.

[0792] Transmitting / receiving module: Sends text data and emotion data to the server and receives responses from the server.

[0793] Display and speech synthesis module: displays solutions and advice and plays them aloud.

[0794] server

[0795] Communication interface: processes data received from the user terminal.

[0796] Generative AI models: Generate solutions and advice based on the received text data. For example, you can use the OpenAI GPT-3 model.

[0797] Data adjustment module: Adjusts the generated solutions and advice based on the received sentiment data.

[0798] Processing flow

[0799] 1. Receiving voice input

[0800] The user verbally describes their problem or request into their mobile phone.

[0801] 2. Speech and Emotion Recognition

[0802] The device uses a voice recognition module to convert voice into text data and an emotion engine to recognize the user's emotions.

[0803] 3. Data transmission

[0804] The text data and emotion data are transmitted to the server using a transmitting / receiving module.

[0805] 4. Processing by generative AI models

[0806] The server provides the received text data to a generative AI model, which generates appropriate solutions and advice.

[0807] For example, the prompt is:

[0808] If a user says they don't know how to set up Wi-Fi, explain the process in detail. Be sensitive to their concerns.

[0809] 5. Adjustments based on emotional data

[0810] The server adjusts the generated solutions and advice based on the received emotion data.

[0811] 6. Submitting and Displaying Solutions or Advice

[0812] The tailored solutions and advice are sent from the server to the user's device, which displays them on the display and plays them aloud using a speech synthesis module.

[0813] Specific examples

[0814] Example 1: Wi-Fi settings

[0815] When a user says, "I don't know how to set up Wi-Fi," the device converts the user's voice into text data and detects their anxiety.

[0816] 1. Voice input

[0817] The user says, "I don't know how to set up Wi-Fi."

[0818] 2. Speech and Emotion Recognition

[0819] The voice recognition module converts "I don't know how to set up Wi-Fi" into text data, and the emotion engine recognizes the user's anxiety.

[0820] 3. Data transmission

[0821] The device sends the text data "I don't know how to set up Wi-Fi" and the emotional data "anxiety" to the server.

[0822] 4. Processing by generative AI models

[0823] The server generates Wi-Fi setting instructions based on the text data.

[0824] Example response: "Open the Settings app and go to the Wi-Fi settings section. From the list of Wi-Fi networks, select the network you want to connect to, enter the password, and tap Connect."

[0825] 5. Adjustments based on emotional data

[0826] The server adjusts the generated solutions based on anxiety sentiment and provides detailed explanations.

[0827] Sample tailored response: "Don't worry! First, open the Settings app on your phone. Then, go to the Wi-Fi settings section. Next, select your home network from the list of Wi-Fi networks that appears. Finally, enter your network password and tap the Connect button."

[0828] 6. Displaying and playing audio solutions or advice

[0829] The device will display the received solution on its display and play a voice message saying, "Don't worry. First, open the Settings app on your phone..."

[0830] This system allows users to set up Wi-Fi seamlessly and without worry. The combination of an emotion engine and generative AI models improves the user experience.

[0831] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0832] Step 1:

[0833] Receiving audio input

[0834] user:

[0835] The user verbally describes their problem or request into their mobile phone.

[0836] For example, say, "I don't know how to set up Wi-Fi."

[0837] Input: User speech.

[0838] Output: Audio data.

[0839] Step 2:

[0840] Speech and Emotion Recognition

[0841] Device:

[0842] A speech recognition module converts the user's voice data into text data, for example, using the Google Speech-to-Text API.

[0843] The emotion engine analyzes the voice data and extracts emotion data, for example, using the Microsoft Azure Emotion API.

[0844] Input: Audio data.

[0845] Data processing: The voice data is analyzed, and the voice recognition module converts the voice into text data. The emotion engine also extracts emotional information from the voice.

[0846] Output: Text and sentiment data.

[0847] Step 3:

[0848] Sending data

[0849] Device:

[0850] The generated text data and emotion data are sent to the server using a transmission / reception module.

[0851] Input: Text data and sentiment data.

[0852] Data processing: Assembling text data and emotion data into data packets.

[0853] Output: Send the data packet to the server over the communication network.

[0854] Step 4:

[0855] Receiving data

[0856] server:

[0857] A data packet is received from a user terminal through a communication interface.

[0858] Input: Data packet.

[0859] Data processing: Analyze data packets and extract text and sentiment data.

[0860] Output: Extracted text and sentiment data.

[0861] Step 5:

[0862] Processing by generative AI models

[0863] server:

[0864] The received text data is fed into a generative AI model (e.g., OpenAI GPT-3) to generate solutions or advice.

[0865] Example prompt: "If the user says they don't know how to set up Wi-Fi, explain the process in detail. Be sensitive to their concerns."

[0866] Input: Text data.

[0867] Data processing: Input text data into a generative AI model to obtain generated solutions or advice.

[0868] Output: The generated solution or advice.

[0869] Step 6:

[0870] Adjustments based on emotional data

[0871] server:

[0872] Tailor generated solutions and advice based on sentiment data.

[0873] For example: If the user is feeling unsure, add a detailed explanation.

[0874] Input: Generated solution or advice, sentiment data.

[0875] Data processing: Analyzing sentiment data and making adjustments based on generated solutions and advice.

[0876] Output: Tailored solutions or advice.

[0877] Step 7:

[0878] Submit a solution or advice

[0879] server:

[0880] Send tailored solutions and advice to your device.

[0881] Input: Tailored solutions or advice.

[0882] Data processing: Consolidating tailored solutions and advice into data packets.

[0883] Output: Sends data packets over the communications network to the user terminal.

[0884] Step 8:

[0885] Display and audio playback of solutions or advice

[0886] Device:

[0887] The received solutions and advice are displayed on the screen and played aloud using a speech synthesis module.

[0888] Example: Display and play the following: "Don't worry. First, open the Settings app on your phone..."

[0889] Input: Tailored solutions or advice.

[0890] Data processing: Converting tailored solutions and advice into display and speech synthesis.

[0891] Output: Display and audio playback.

[0892] (Application example 2)

[0893] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0894] Conventional support systems for setting up and operating mobile phones and communication devices simply convert the user's voice input into text data and provide solutions and advice. However, because they do not take the user's emotions into consideration, the support content often does not match the user's current situation, resulting in an unsatisfactory user experience. Furthermore, users who are feeling particularly anxious or confused need appropriate advice that addresses their emotions. To solve these issues, a more personalized support system that incorporates the user's emotions is needed.

[0895] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0896] In this invention, the server includes means for receiving a problem or request provided by a user through voice input, means for converting the voice input into text data and emotional data, means for transmitting the converted text data and emotional data to the server, means for the server to receive the text data and emotional data and generate a solution or advice using a generation AI module, means for adjusting the solution or advice based on the user's emotional data, means for transmitting the generated solution or advice to a user terminal, and means for the user terminal to display the received solution or advice and play it back by voice, thereby enabling personalized support that reflects the user's emotions.

[0897] "Voice input" is voice data provided by the user through a microphone.

[0898] "Emotion data" is data that indicates the user's emotions analyzed from voice input.

[0899] "User terminal" refers to a device operated by a user that accepts voice input and displays and plays solutions and advice.

[0900] A "generative AI module" is a program that runs within the server and includes artificial intelligence that generates solutions and advice based on received text data.

[0901] The "server" is a remote system that receives text data and emotion data sent from a user terminal, generates solutions or advice using a generative AI module, and sends the solutions or advice to the user terminal.

[0902] "Text data" is character data converted from voice input using voice recognition technology.

[0903] "Solutions or Advice" refers to problem-solving steps or advice provided by the generative AI module based on text data and emotion data.

[0904] A "voice recognition module" is a program or hardware that converts voice input into text data.

[0905] A "communication interface" is a network means for transmitting and receiving data between a user terminal and a server.

[0906] "Personalization" is a means of providing settings and advice optimized for each user.

[0907] System configuration

[0908] The present invention is a system including: means for receiving a problem or request provided by a user through voice input; means for converting the voice input into text data and emotional data; means for transmitting the text data and emotional data to a server; means for the server to receive the text data and emotional data and generate a solution or advice using a generation AI module; means for adjusting the solution or advice based on the user's emotional data; means for transmitting the generated solution or advice to a user terminal; and means for the user terminal to display the received solution or advice and play it back aloud.

[0909] Hardware and Software Configuration

[0910] User device:

[0911] The user terminal is a device equipped with a voice recognition module, an emotion engine, a transmission / reception module, a display, a speaker, etc.

[0912] server:

[0913] The server is a remote system that receives text data and emotion data, generates solutions and advice using a generative AI module, and sends them to the user's device.

[0914] Communication Interface:

[0915] The communication interface is a network module for transmitting and receiving data between the user terminal and the server.

[0916] Program processing overview

[0917] First, the user inputs voice. The user device uses a voice recognition module to convert the voice input into text data, and then uses an emotion engine to analyze the user's emotions. The acquired text data and emotion data are sent to the server via a communication interface.

[0918] The server receives text data and emotion data sent from the user's device. The generative AI module generates problem-solving procedures and advice based on the received text data. The generated solutions and advice are adjusted based on the user's emotion data.

[0919] The adjusted solution or advice is then sent back to the user terminal via the communication interface, where it is displayed on the display and played aloud using a speech synthesis module.

[0920] Examples of concrete examples and prompts

[0921] For example, if a user says, "I want a new smartphone," the speech recognition module converts the speech into text and generates the text data, "I want a new smartphone." At the same time, the emotion engine recognizes the user's excited emotion.

[0922] An example of user input is:

[0923] Prompt Sentence Examples

[0924] User input: "I want a new smartphone"

[0925] User Sentiment: Excitement

[0926] The generative AI module generates the following recommendations based on this text data and emotion data:

[0927] "The best smartphones include those with the latest camera technology and long battery life. Our most popular models take your experience to the next level. Want to learn more?"

[0928] In this way, the system provides personalized support that reflects the user's emotions.

[0929] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0930] Step 1:

[0931] The user provides voice input.

[0932] Input: User speech (e.g., "I want a new smartphone")

[0933] Action: The user speaks into the device's microphone.

[0934] Output: User's voice data

[0935] Step 2:

[0936] The terminal uses a voice recognition module to convert the voice input into text data.

[0937] Input: Audio data

[0938] Data processing: Converting voice into text using voice recognition technology

[0939] How it works: The speech recognition module analyzes the audio signal and generates corresponding text.

[0940] Output: Text data (e.g., "I want a new smartphone")

[0941] Step 3:

[0942] The device uses an emotion engine to recognize the user's emotions.

[0943] Input: Audio data

[0944] Data calculation: Extracting features such as pitch, intonation, and speed from speech to estimate emotions

[0945] How it works: The emotion engine analyzes voice data and identifies the user's emotion.

[0946] Output: Emotion data (e.g., excitement)

[0947] Step 4:

[0948] The terminal transmits the text data and the emotion data to the server.

[0949] Input: Text data, emotion data

[0950] Operation: The sending and receiving module assembles data into packets and sends them to the server through the communication interface.

[0951] Output: Text data and emotion data sent to the server

[0952] Step 5:

[0953] A server receives the text data and the emotion data.

[0954] Input: Text data, emotion data

[0955] Operation: The server receives data through the communication interface and stores it in storage.

[0956] Output: Text data and emotion data stored in the server storage

[0957] Step 6:

[0958] The server uses a generative AI module to generate a solution or advice based on the text data.

[0959] Input: Text data

[0960] Data calculation: Generative AI module analyzes text data and generates appropriate solutions or advice

[0961] How it works: The generative AI module takes text data and uses algorithms to generate solutions or advice.

[0962] Output: Solution or advice (e.g., "We recommend a smartphone with the latest camera technology.")

[0963] Step 7:

[0964] The server tailors the solution or advice based on the user's emotional data.

[0965] Input: Solution or advice, sentiment data

[0966] Data processing: Using sentiment data to enhance and modify solutions and advice

[0967] How it works: The server references the emotion data and adjusts the content of the generated solution or advice to match the emotion.

[0968] Output: Tailored solution or advice (e.g., "You seem excited. In that case, we recommend a smartphone with the latest camera technology.")

[0969] Step 8:

[0970] The server sends the tailored solution or advice to the user terminal.

[0971] Input: Tailored solution or advice

[0972] Operation: The server assembles the adjusted solution or advice into a packet via the sending and receiving module and sends it to the user terminal through the communication interface.

[0973] Output: Tailored solution or advice sent to user terminal

[0974] Step 9:

[0975] The user device displays and plays audibly the tailored solution or advice received.

[0976] Input: Tailored solution or advice

[0977] How it works: The user device displays the solution or advice on the display and plays it aloud using a speech synthesis module.

[0978] Output: Displayed solution or advice, audio played (e.g., "You seem excited. In that case, we recommend a smartphone with the latest camera technology.")

[0979] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0980] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0981] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0982] [Third embodiment]

[0983] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0984] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0985] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0986] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0987] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0988] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0989] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0990] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0991] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0992] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0993] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0994] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0995] System Overview

[0996] This invention is a system for easily resolving problems users may encounter when operating or configuring a mobile phone. This system involves a series of processes: converting user voice input into text data, sending it to a server, generating solutions or advice using a generation AI module, and returning it to the user's device.

[0997] Key Components of the System

[0998] User device:

[0999] A device that allows users to input voice information. It includes a voice recognition module, a transmission / reception module, a display, and a speaker.

[1000] server:

[1001] A remote system that receives voice-recognized text data and generates solutions or advice using a generative AI module.

[1002] Communication Interface:

[1003] A network module for sending and receiving data between a user terminal and a server.

[1004] Program processing overview

[1005] 1. Voice input

[1006] Users communicate their problems and requests via voice.

[1007] 2. Voice Recognition

[1008] The device converts the user's voice input into text data.

[1009] 3. Sending text data

[1010] The terminal transmits the converted text data to the server.

[1011] 4. Receiving text data

[1012] The server receives the text data sent from the user terminal.

[1013] 5. Processing by the Generative AI Module

[1014] The server provides the received text to a generative AI module, which generates an appropriate solution or advice.

[1015] 6. Submitting Solutions or Advice

[1016] The server transmits the generated solution or advice to the user terminal.

[1017] 7. Displaying and playing audio solutions or advice

[1018] The device displays the received solution or advice on the display and plays it aloud through the speaker.

[1019] Specific examples

[1020] Example 1: Wi-Fi settings

[1021] When a user says, "I don't know how to set up Wi-Fi," the following process occurs:

[1022] Audio Input:

[1023] The user says, "I don't know how to set up Wi-Fi."

[1024] Voice Recognition:

[1025] The device converts the voice into text data: "I don't know how to set up Wi-Fi."

[1026] Sending text data:

[1027] The device sends text data to the server saying "I don't know how to set up Wi-Fi."

[1028] Receiving text data:

[1029] The server receives the text data.

[1030] Processing by the Generative AI module:

[1031] The server uses the text data to generate a solution that includes instructions for setting up Wi-Fi.

[1032] Example response from the generative AI module: "Open the Settings app and go to the Wi-Fi settings section. From the list of Wi-Fi networks, select the network you want to connect to, enter the password, and tap Connect."

[1033] Submit a solution or advice:

[1034] The server sends the generated solution to the user terminal.

[1035] View and listen to solutions or advice:

[1036] The device will show the solution it received on the display and play a voice message saying, "Open the Settings app and go to the Wi-Fi settings section. From the list of Wi-Fi networks, select the network you want to connect to, enter the password, and tap Connect."

[1037] This allows users to solve problems themselves and smoothly set up their mobile phones. The advantage of this system is that it provides fast and accurate support.

[1038] The processing flow will be explained below.

[1039] Step 1:

[1040] The user verbally states their problem or request, for example, "I don't know how to set up Wi-Fi."

[1041] Step 2:

[1042] The device receives the user's voice input and passes it to the speech recognition module, which converts the voice data into text data. Example: "I don't know how to set up Wi-Fi."

[1043] Step 3:

[1044] The terminal transmits the converted text data to the server.

[1045] Step 4:

[1046] The server receives the text data sent from the user terminal.

[1047] Step 5:

[1048] The server inputs the received text data into a generation AI module to generate a solution or advice. For example, a generated solution might be, "Open the Settings app and go to the Wi-Fi settings section. Select the network you want to connect to from the list of Wi-Fi networks, enter the password, and tap Connect."

[1049] Step 6:

[1050] The server transmits the generated solution or advice to the user terminal.

[1051] Step 7:

[1052] The terminal receives the solution or advice received from the server.

[1053] Step 8:

[1054] The device displays the received solution or advice on the display, and simultaneously plays back the solution aloud using a speech synthesis module.

[1055] Step 9:

[1056] Users can follow the solution displayed on their device to configure their phone: open the Settings app, go to the Wi-Fi settings section, select the network they want to connect to from the list, enter the password, and tap Connect.

[1057] Example 1

[1058] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1059] Conventional support systems for mobile phone operation and settings make it difficult for users to solve problems directly and quickly. In particular, systems that allow users to solve problems through voice input are limited, and in many cases, complex operating procedures are required. As a result, users are more likely to make operating or setting errors. In response to this, the present invention aims to provide a system that allows users to easily report problems through voice input and quickly provides solutions using a generative AI module.

[1060] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1061] In this invention, the server includes a means for receiving a problem or request provided by a user through voice input, a means for converting the voice input into text data, and a means for transmitting the converted text data to the server, so that the user can report a problem through voice, which is converted into text data and transmitted to the server, thereby enabling a solution to be generated and provided to the user for quickly and accurately resolving the problem.

[1062] "User" refers to the entity that operates the system and provides problems or requests using voice input.

[1063] "Voice input" refers to a means by which a user communicates problems or requests to a system using their voice.

[1064] "Text data" refers to data that has been converted from voice input into a character string format.

[1065] "Server" refers to the remote system that receives the text data converted from the voice input and generates solutions or advice using a generative AI module.

[1066] "Generative AI module" refers to an artificial intelligence algorithm that generates appropriate solutions or advice based on text data received from the user.

[1067] A "prompt sentence" refers to an instruction sentence for generating an output for solving a problem based on text data input to the generation AI module.

[1068] "Communication interface" refers to a network module for transmitting and receiving data between a user terminal and a server.

[1069] "User terminal" refers to a device through which a user makes voice input, and which is equipped with a voice recognition module, a display, and a speaker.

[1070] The present invention is a system that allows users to report problems related to mobile phone operation and settings through voice input, and uses a generative AI module to quickly and accurately provide solutions. The system includes the following main components: a user terminal, a server, and a communication interface.

[1071] System configuration

[1072] User Device

[1073] A user terminal is a device that allows users to input voice. It includes a microphone, display, and speaker as hardware, and a speech recognition module as software. Specifically, speech recognition software such as Google Voice API or IBM Watson is used.

[1074] server

[1075] The server is a remote system that receives text data sent from the user's device and generates solutions and advice using a generative AI model, such as OpenAI's GPT-4.

[1076] Communication Interface

[1077] The communication interface serves to send and receive data between the user device and the server, specifically via Wi-Fi or 4G / 5G communication.

[1078] Processing the data

[1079] 1. Voice input: The user voices the problem they are having with their phone's operation or settings. For example, they say, "I don't know how to set up Wi-Fi."

[1080] 2. Speech recognition: The device's speech recognition module records the user's voice and converts it into text data in real time. The Google Voice API converts the voice into text data such as "I don't know how to set up Wi-Fi."

[1081] 3. Sending text data: The converted text data is sent to the server, and the device's transceiver module transmits the data via Wi-Fi or 4G / 5G communication.

[1082] 4. Receiving text data: The server receives the text data and stores the received data in a processing queue.

[1083] 5. Processing by the generative AI module: The server generates a prompt and provides it to the generative AI model. For example, it provides the following prompt:

[1084] When a user inquires about how to set up Wi-Fi, please provide a clear explanation of the steps to set up Wi-Fi.

[1085] A generative AI model (e.g., OpenAI's GPT-4) generates a solution, such as "Open the Settings app and go to the Wi-Fi settings section. From the list of Wi-Fi networks, select the network you want to connect to, enter the password, and tap Connect."

[1086] 6. Sending the solution: The generated solution is sent to the user terminal, and the data is sent back through the communication interface.

[1087] 7. Display and audio solution: Your device will display and audio the solution using a Text-to-Speech (TTS) engine (e.g., Google TTS), for example, "Open the Settings app and go to the Wi-Fi settings section. From the list of Wi-Fi networks, select the network you want to connect to, enter the password, and tap Connect."

[1088] Specific examples

[1089] Example 1: Wi-Fi settings

[1090] When a user says, "I don't know how to set up Wi-Fi," the following sequence of events occurs:

[1091] 1. Voice input: The user says, "I don't know how to set up Wi-Fi."

[1092] 2. Speech recognition: The device uses the Google Voice API to convert the speech into text data such as "I don't know how to set up Wi-Fi."

[1093] 3. Sending text data: The device sends text data saying "I don't know how to set up Wi-Fi" to the server.

[1094] 4. Receiving text data: The server receives the text data.

[1095] 5. Processing by the generative AI module: The server provides the received text data to the generative AI model, which generates a solution including instructions for setting up Wi-Fi.

[1096] 6. Sending the solution: The server sends the generated solution to the user terminal.

[1097] 7. Displaying the solution and playing it aloud: The device displays the solution it received on the display and plays it aloud using Google TTS.

[1098] As described above, the system of the present invention allows a user to report a problem by voice, which is converted into text data and sent to a server, and then a solution to quickly and accurately resolve the problem can be generated and provided to the user.

[1099] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1100] Step 1:

[1101] Voice input

[1102] The user communicates problems or requests regarding the settings of their mobile phone by voice. The input is the user's voice data. Specifically, the user says, "I don't know how to set up Wi-Fi." This causes the voice data to be captured by the microphone in the device.

[1103] Step 2:

[1104] Voice Recognition

[1105] The device converts the voice data it receives into text data. The input is voice data, and the output is text data. Specifically, the device's voice recognition module uses the Google Voice API to convert the voice data into text data such as "I don't know how to set up Wi-Fi."

[1106] Step 3:

[1107] Sending text data

[1108] The device sends the converted text data to the server. The input is text data, and the output is data transmission to the server. Specifically, the device's transceiver module sends the text data "I don't know how to set up Wi-Fi" to the server using Wi-Fi or 4G / 5G communication.

[1109] Step 4:

[1110] Receiving text data

[1111] The server receives text data sent from the user terminal. The input is the text data sent from the terminal, and the output is data storage within the server. Specifically, the server's communication interface receives the text data and stores it in a processing queue.

[1112] Step 5:

[1113] Processing by generative AI module

[1114] The server provides the received text data to a generative AI model to generate a solution or advice. The input is the text data and a prompt, and the output is a solution or advice. Specifically, the server sends the following prompt to the generative AI model (e.g., OpenAI's GPT-4):

[1115] When a user inquires about how to set up Wi-Fi, please provide a clear explanation of the steps to set up Wi-Fi.

[1116] The generative AI model generates the following solution: "Open the Settings app and go to the Wi-Fi settings section. Select the network you want to connect to from the list of Wi-Fi networks, enter the password, and tap Connect."

[1117] Step 6:

[1118] Submit a solution

[1119] The server sends the generated solutions and advice to the user's device. The input is the generated solution, and the output is data transmission to the user's device. Specifically, the server's transmission module prepares the generated text data and transmits it to the device via Wi-Fi or 4G / 5G communication.

[1120] Step 7:

[1121] View and listen to the solution

[1122] The device displays the received solution on the display and plays it back aloud using a Text-to-Speech (TTS) engine. The input is the solution sent from the server, and the output is the display and audio playback. Specifically, the device's receiving module receives the solution text data and displays it on the display as follows:

[1123] Open the Settings app and go to the Wi-Fi settings section. Select the network you want to connect to from the list of Wi-Fi networks, enter the password, and tap Connect.

[1124] In addition, the Google TTS engine converts the text data into speech and plays it back through the speaker.

[1125] (Application example 1)

[1126] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1127] Modern security systems have sophisticated and complex functions, and it is often difficult for average users to understand how to configure and use them. Furthermore, when a configuration error or problem occurs, it is difficult to quickly find a solution, potentially increasing security risks. Traditional support systems have the problem of taking a long time to respond, increasing the hassle and stress for users.

[1128] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1129] In this invention, the server includes means for receiving a problem or request provided by a user through voice input, means for converting the voice input into text data, means for transmitting the converted text data to the server, means for the server to receive the text data and generate a solution or advice using a generation AI module, means for transmitting the generated solution or advice to the user terminal, means for the user terminal to display and play the received solution or advice by voice, means for receiving a problem or request regarding the configuration and use of the security system, and means for generating a solution or advice regarding the configuration and use of the security system, thereby enabling a user to quickly and easily configure and troubleshoot the security system.

[1130] A "user terminal" is an electronic device that allows a user to voice-input a problem or request and receive a solution or advice.

[1131] "Server" means a remote system that receives text data sent from a user terminal and generates a solution or advice using a generative AI module.

[1132] "Voice input" refers to the act of a user providing a problem or request to a device by voice.

[1133] "Text data" is character information generated using voice recognition technology based on voice input.

[1134] The "generative AI module" is an artificial intelligence module that analyzes text data and generates appropriate solutions and advice.

[1135] "Solutions or Advice" refers to specific responses or instructions generated by the generative AI module in response to a user's problem or request.

[1136] "Security system" is a general term for surveillance cameras, alarms, access control devices, and other equipment used to keep a home or office safe.

[1137] "Display" is the act of visually presenting information on the display of a user terminal.

[1138] "Audio playback" refers to the act of outputting audio information using the speaker of the user device.

[1139] An "operational procedure" is a set of specific steps that a user must perform to achieve a particular goal.

[1140] "Configuration Instructions" are instructions on how to properly configure the security system.

[1141] "Personalization" is the act of generating solutions and advice optimized for each individual user based on the user's past operation history and settings information.

[1142] "Speech recognition" is a technology that converts voice input into text data.

[1143] The present invention is a system that uses a user terminal, a server, and a generation AI module to provide appropriate solutions and advice for problems and requests related to the configuration and use of a security system. Specific embodiments for implementing this system are described below.

[1144] 1. User Device

[1145] The user device is equipped with a voice recognition module that receives voice input and converts it into text data. This module can use the Google Speech-to-Text API. It also has a communication interface that transmits the input text data to the server. This allows users to input security system issues and requests by voice, and the content is converted into text data and sent to the server.

[1146] 2. Server

[1147] The server receives text data sent from the user's device and passes it to the generation AI module. The received text data is analyzed by the generation AI module, and optimal solutions and advice are generated. OpenAI's GPT-4 can be used as the generation AI module. This generated data is then sent back to the user's device.

[1148] 3. Generative AI Module

[1149] The generative AI module generates solutions and advice on security system configuration and usage based on the text data received by the server. This module has learned from a huge data set and can provide the most appropriate information for the user's problem.

[1150] 4. User Interface

[1151] The user device has the ability to display the received solutions and advice on the screen and play them back aloud using the Google Text-to-Speech API, allowing users to confirm the solutions and advice both visually and audibly.

[1152] Specific examples

[1153] For example, if a user says "my security camera isn't working," the following will be processed:

[1154] 1. The user speaks "My security camera isn't working."

[1155] 2. The user device converts the voice into text data and sends it to the server.

[1156] 3. The server receives the text data and provides it to the generation AI module.

[1157] 4. The generative AI module generates a solution such as "Please check the power supply of the security camera and restart it."

[1158] 5. The server sends the generated solution to the user device.

[1159] 6. The user device displays the received solution on the display and plays it back aloud.

[1160] Prompt Sentence Examples

[1161] "How do I reset the settings on my security cameras?"

[1162] "My alarm system is malfunctioning. Can you help me fix it?"

[1163] This allows users to quickly and easily troubleshoot security system problems.

[1164] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1165] Step 1:

[1166] Users input their security system problems and requests by voice. The input voice is picked up through the microphone of the user's device. Based on the input, the user's voice data is collected.

[1167] Step 2:

[1168] The voice data captured by the user device is converted into text data using voice recognition software (e.g., Google Speech-to-Text API). In this step, voice signals are analyzed using voice recognition technology and converted into corresponding strings of characters. The output is the converted text data.

[1169] Step 3:

[1170] The user terminal sends the converted text data to the server. Here, the data is sent to the server using the HTTPS protocol to ensure security. The input is the text data generated by speech recognition, and the output is the text data sent to the server.

[1171] Step 4:

[1172] The server receives the text data sent from the user terminal. In this step, the server is prepared to analyze the text data. The input is the text data sent from the user terminal, and the output is the provision of the text data to the generation AI module.

[1173] Step 5:

[1174] The server provides the received text data to a generative AI module (e.g., OpenAI GPT-4) to generate an optimal solution or advice. In this step, the AI ​​model analyzes the text data and generates a solution. The output is the generated solution or advice text data.

[1175] Step 6:

[1176] The server sends the solutions and advice generated by the generative AI module to the user's device. Here, data is transferred securely using the HTTPS protocol again. The input is the text data of the generated solutions and advice, and the output is the transmission of the solutions and advice to the user's device.

[1177] Step 7:

[1178] The user device displays the received solution or advice on the display and plays it back aloud. In this step, the Google Text-to-Speech API is used to convert the text data into speech and output it as speech. The input is the text data of the solution or advice received from the server, and the output is the text displayed on the display and the solution or advice played back aloud.

[1179] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1180] System Overview

[1181] The present invention relates to a system that recognizes a user's voice input and emotions and provides optimal solutions and advice to the user regarding the operation and settings of a mobile phone. By combining this with the recognition of the user's emotions, the system can provide more personalized support.

[1182] Key Components of the System

[1183] User device:

[1184] This device allows users to input voice and converts that voice into text data and emotion data. It is equipped with a voice recognition module, emotion engine, transmission / reception module, display, speaker, etc.

[1185] server:

[1186] It is a remote system that receives text data and emotion data, generates solutions and advice using a generative AI module, and adjusts them based on the emotion data.

[1187] Communication Interface:

[1188] A network module for sending and receiving data between a user terminal and a server.

[1189] Program processing overview

[1190] 1. Voice input

[1191] The user voices their problem or request.

[1192] 2. Speech and Emotion Recognition

[1193] The device converts the user's voice input into text data and simultaneously recognizes the user's emotions using an emotion engine.

[1194] 3. Data transmission

[1195] The terminal transmits the text data and the emotion data to the server.

[1196] 4. Receiving Data

[1197] The server receives the text data and emotion data sent from the user terminal.

[1198] 5. Processing by the Generative AI Module

[1199] The server provides the received text data to a generation AI module, which generates an appropriate solution or advice.

[1200] 6. Adjustments based on emotional data

[1201] The server tailors the generated solution or advice based on the received emotional data, for example providing more detailed explanations if the user is feeling anxious.

[1202] 7. Submitting Solutions or Advice

[1203] The server sends the tailored solution or advice to the user terminal.

[1204] 8. Displaying and playing audio solutions or advice

[1205] The solution or advice received by the terminal is displayed on the display and played aloud using a speech synthesis module.

[1206] Specific examples

[1207] Example 1: Wi-Fi settings

[1208] When a user says, "I don't know how to set up Wi-Fi," the device converts the user's voice into text data and simultaneously recognizes the user's anxiety using an emotion engine.

[1209] Audio Input:

[1210] The user says, "I don't know how to set up Wi-Fi."

[1211] Speech and emotion recognition:

[1212] The voice recognition module converts this into text data such as "I don't know how to set up Wi-Fi," and the emotion engine recognizes the user's anxiety.

[1213] Sending data:

[1214] The device sends text data such as "I don't know how to set up Wi-Fi" and emotional data such as "anxiety" to the server.

[1215] Receiving data:

[1216] The server receives the text data and the emotion data.

[1217] Processing by the Generative AI module:

[1218] The server generates a solution based on the text data, including instructions for setting up Wi-Fi.

[1219] Example response from the generative AI module: "Open the Settings app and go to the Wi-Fi settings section. From the list of Wi-Fi networks, select the network you want to connect to, enter the password, and tap Connect."

[1220] Adjustments based on sentiment data:

[1221] The server adjusts the generated solution based on the anxiety sentiment and provides a detailed explanation.

[1222] Sample tailored response: "Don't worry! First, open the Settings app on your phone. Then, go to the Wi-Fi settings section. Next, select your home network from the list of Wi-Fi networks that appears. Finally, enter your network password and tap the Connect button."

[1223] Submit a solution or advice:

[1224] The server sends the adjusted solution to the user terminal.

[1225] View and listen to solutions or advice:

[1226] The device will display the solution it receives on the display and play a voice message saying, "Don't worry. First, open your phone's Settings app..."

[1227] This example allows users to easily set up Wi-Fi without any worries. Combined with the emotion engine, it improves the user experience.

[1228] The processing flow will be explained below.

[1229] Step 1:

[1230] The user verbally states their problem or request, for example, "I don't know how to set up Wi-Fi."

[1231] Step 2:

[1232] The device receives the user's voice input and passes it to the speech recognition module, which converts the voice data into text data. Example: "I don't know how to set up Wi-Fi."

[1233] Step 3:

[1234] The device passes the user's voice input to the emotion engine, which analyzes the voice tone, speed, intonation, etc. to recognize the user's emotion. Example: Recognizes "anxiety"

[1235] Step 4:

[1236] The terminal transmits the converted text data and the recognized emotion data to the server.

[1237] Step 5:

[1238] The server receives the text data and emotion data sent from the user terminal.

[1239] Step 6:

[1240] The server inputs the received text data into a generation AI module, which generates an appropriate solution or advice. For example, a generated solution might be, "Open the Settings app and go to the Wi-Fi settings section. Select the network you want to connect to from the list of Wi-Fi networks, enter the password, and tap Connect."

[1241] Step 7:

[1242] The server tailors the generated solution or advice based on the emotion data received. For example, if the user is perceived as "anxious," it adds a detailed explanation to the solution. Example: "Don't worry. First, open the Settings app on your phone..."

[1243] Step 8:

[1244] The server sends the tailored solution or advice to the user terminal.

[1245] Step 9:

[1246] The terminal receives the solution or advice received from the server.

[1247] Step 10:

[1248] The device displays the received solution or advice on a display, and simultaneously plays the adjusted solution aloud using a speech synthesis module.

[1249] Step 11:

[1250] Users should follow the on-screen and audio-guided solutions to set up their phone by opening the Settings app, navigating to the Wi-Fi settings section, selecting the network they want to connect to from the list, entering the password, and tapping Connect.

[1251] Example 2

[1252] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1253] When operating or setting up a conventional mobile phone, users often become confused because they do not know the operating procedures or how to set it up. Furthermore, they may feel anxious or stressed because they are not provided with personalized support that is tailored to the user's emotions and situation. This can lead to a poor user experience and make it difficult to use the mobile phone.

[1254] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1255] In this invention, the server includes means for receiving a problem or request provided by a user through voice input, means for converting the voice input into text data, means for transmitting the converted text data and emotion data to the server, means for the server to receive the text data and emotion data and generate a solution or advice using a generative AI model, means for adjusting the generated solution or advice based on the received emotion data, means for transmitting the adjusted solution or advice to the user terminal, and means for the user terminal to display the received solution or advice and play it back by voice. This makes it possible to provide personalized support for mobile phone operations and settings while taking into consideration the user's emotions.

[1256] "Voice input" is the act of a user providing a problem or request by speaking into a mobile phone or other device.

[1257] "Text data" is data that has been converted from voice input into text information, and specifically expresses the content of a problem or request.

[1258] "Emotional data" is information about emotions extracted from the user's voice tone and expressions, and is data that represents the user's feelings and psychological state.

[1259] A "server" is a remote computer system that receives data from a user terminal via a communications network, processes the data using a generative AI model or the like, and transmits the results to the user terminal.

[1260] A "generative AI model" is an artificial intelligence model that automatically generates solutions and advice based on received text data, and is a program for generating specific answers and instructions.

[1261] "User device" refers to a device used by a user, such as a mobile phone or computer, for voice input and for receiving and displaying solutions and advice.

[1262] "Solutions or Advice" refers to instructions or guides on how to solve a user's problem or procedures that are generated by a generative AI model based on text data.

[1263] "Adjustment" refers to the act of changing the content and expression of the generated solution or advice to match the user's emotions based on the received emotion data.

[1264] The present invention relates to a system that recognizes a user's voice input and emotions and provides optimal solutions and advice to the user regarding the operation and settings of a mobile phone. By combining this with the recognition of the user's emotions, the system can provide more personalized support.

[1265] Hardware and Software Configuration

[1266] User Device

[1267] Speech recognition module: converts speech into text data. For example, you can use the Google Speech-to-Text API.

[1268] Emotion engine: Extracts emotion data from the user's voice. For example, it can use the Microsoft Azure Emotion API.

[1269] Transmitting / receiving module: Sends text data and emotion data to the server and receives responses from the server.

[1270] Display and speech synthesis module: displays solutions and advice and plays them aloud.

[1271] server

[1272] Communication interface: processes data received from the user terminal.

[1273] Generative AI models: Generate solutions and advice based on the received text data. For example, you can use the OpenAI GPT-3 model.

[1274] Data adjustment module: Adjusts the generated solutions and advice based on the received sentiment data.

[1275] Processing flow

[1276] 1. Receiving voice input

[1277] The user verbally describes their problem or request into their mobile phone.

[1278] 2. Speech and Emotion Recognition

[1279] The device uses a voice recognition module to convert voice into text data and an emotion engine to recognize the user's emotions.

[1280] 3. Data transmission

[1281] The text data and emotion data are transmitted to the server using a transmitting / receiving module.

[1282] 4. Processing by generative AI models

[1283] The server provides the received text data to a generative AI model, which generates appropriate solutions and advice.

[1284] For example, the prompt is:

[1285] If a user says they don't know how to set up Wi-Fi, explain the process in detail. Be sensitive to their concerns.

[1286] 5. Adjustments based on emotional data

[1287] The server adjusts the generated solutions and advice based on the received emotion data.

[1288] 6. Submitting and Displaying Solutions or Advice

[1289] The tailored solutions and advice are sent from the server to the user's device, which displays them on the display and plays them aloud using a speech synthesis module.

[1290] Specific examples

[1291] Example 1: Wi-Fi settings

[1292] When a user says, "I don't know how to set up Wi-Fi," the device converts the user's voice into text data and detects their anxiety.

[1293] 1. Voice input

[1294] The user says, "I don't know how to set up Wi-Fi."

[1295] 2. Speech and Emotion Recognition

[1296] The voice recognition module converts "I don't know how to set up Wi-Fi" into text data, and the emotion engine recognizes the user's anxiety.

[1297] 3. Data transmission

[1298] The device sends the text data "I don't know how to set up Wi-Fi" and the emotional data "anxiety" to the server.

[1299] 4. Processing by generative AI models

[1300] The server generates Wi-Fi setting instructions based on the text data.

[1301] Example response: "Open the Settings app and go to the Wi-Fi settings section. From the list of Wi-Fi networks, select the network you want to connect to, enter the password, and tap Connect."

[1302] 5. Adjustments based on emotional data

[1303] The server adjusts the generated solutions based on anxiety sentiment and provides detailed explanations.

[1304] Sample tailored response: "Don't worry! First, open the Settings app on your phone. Then, go to the Wi-Fi settings section. Next, select your home network from the list of Wi-Fi networks that appears. Finally, enter your network password and tap the Connect button."

[1305] 6. Displaying and playing audio solutions or advice

[1306] The device will display the received solution on its display and play a voice message saying, "Don't worry. First, open the Settings app on your phone..."

[1307] This system allows users to set up Wi-Fi seamlessly and without worry. The combination of an emotion engine and generative AI models improves the user experience.

[1308] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1309] Step 1:

[1310] Receiving audio input

[1311] user:

[1312] The user verbally describes their problem or request into their mobile phone.

[1313] For example, say, "I don't know how to set up Wi-Fi."

[1314] Input: User speech.

[1315] Output: Audio data.

[1316] Step 2:

[1317] Speech and Emotion Recognition

[1318] Device:

[1319] A speech recognition module converts the user's voice data into text data, for example, using the Google Speech-to-Text API.

[1320] The emotion engine analyzes the voice data and extracts emotion data, for example, using the Microsoft Azure Emotion API.

[1321] Input: Audio data.

[1322] Data processing: The voice data is analyzed, and the voice recognition module converts the voice into text data. The emotion engine also extracts emotional information from the voice.

[1323] Output: Text and sentiment data.

[1324] Step 3:

[1325] Sending data

[1326] Device:

[1327] The generated text data and emotion data are sent to the server using a transmission / reception module.

[1328] Input: Text data and sentiment data.

[1329] Data processing: Assembling text data and emotion data into data packets.

[1330] Output: Send the data packet to the server over the communication network.

[1331] Step 4:

[1332] Receiving data

[1333] server:

[1334] A data packet is received from a user terminal through a communication interface.

[1335] Input: Data packet.

[1336] Data processing: Analyze data packets and extract text and sentiment data.

[1337] Output: Extracted text and sentiment data.

[1338] Step 5:

[1339] Processing by generative AI models

[1340] server:

[1341] The received text data is fed into a generative AI model (e.g., OpenAI GPT-3) to generate solutions or advice.

[1342] Example prompt: "If the user says they don't know how to set up Wi-Fi, explain the process in detail. Be sensitive to their concerns."

[1343] Input: Text data.

[1344] Data processing: Input text data into a generative AI model to obtain generated solutions or advice.

[1345] Output: The generated solution or advice.

[1346] Step 6:

[1347] Adjustments based on emotional data

[1348] server:

[1349] Tailor generated solutions and advice based on sentiment data.

[1350] For example: If the user is feeling unsure, add a detailed explanation.

[1351] Input: Generated solution or advice, sentiment data.

[1352] Data processing: Analyzing sentiment data and making adjustments based on generated solutions and advice.

[1353] Output: Tailored solutions or advice.

[1354] Step 7:

[1355] Submit a solution or advice

[1356] server:

[1357] Send tailored solutions and advice to your device.

[1358] Input: Tailored solutions or advice.

[1359] Data processing: Consolidating tailored solutions and advice into data packets.

[1360] Output: Sends data packets over the communications network to the user terminal.

[1361] Step 8:

[1362] Display and audio playback of solutions or advice

[1363] Device:

[1364] The received solutions and advice are displayed on the screen and played aloud using a speech synthesis module.

[1365] Example: Display and play the following: "Don't worry. First, open the Settings app on your phone..."

[1366] Input: Tailored solutions or advice.

[1367] Data processing: Converting tailored solutions and advice into display and speech synthesis.

[1368] Output: Display and audio playback.

[1369] (Application example 2)

[1370] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1371] Conventional support systems for setting up and operating mobile phones and communication devices simply convert the user's voice input into text data and provide solutions and advice. However, because they do not take the user's emotions into consideration, the support content often does not match the user's current situation, resulting in an unsatisfactory user experience. Furthermore, users who are feeling particularly anxious or confused need appropriate advice that addresses their emotions. To solve these issues, a more personalized support system that incorporates the user's emotions is needed.

[1372] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1373] In this invention, the server includes means for receiving a problem or request provided by a user through voice input, means for converting the voice input into text data and emotional data, means for transmitting the converted text data and emotional data to the server, means for the server to receive the text data and emotional data and generate a solution or advice using a generation AI module, means for adjusting the solution or advice based on the user's emotional data, means for transmitting the generated solution or advice to a user terminal, and means for the user terminal to display the received solution or advice and play it back by voice, thereby enabling personalized support that reflects the user's emotions.

[1374] "Voice input" is voice data provided by the user through a microphone.

[1375] "Emotion data" is data that indicates the user's emotions analyzed from voice input.

[1376] "User terminal" refers to a device operated by a user that accepts voice input and displays and plays solutions and advice.

[1377] A "generative AI module" is a program that runs within the server and includes artificial intelligence that generates solutions and advice based on received text data.

[1378] The "server" is a remote system that receives text data and emotion data sent from a user terminal, generates solutions or advice using a generative AI module, and sends the solutions or advice to the user terminal.

[1379] "Text data" is character data converted from voice input using voice recognition technology.

[1380] "Solutions or Advice" refers to problem-solving steps or advice provided by the generative AI module based on text data and emotion data.

[1381] A "voice recognition module" is a program or hardware that converts voice input into text data.

[1382] A "communication interface" is a network means for transmitting and receiving data between a user terminal and a server.

[1383] "Personalization" is a means of providing settings and advice optimized for each user.

[1384] System configuration

[1385] The present invention is a system including: means for receiving a problem or request provided by a user through voice input; means for converting the voice input into text data and emotional data; means for transmitting the text data and emotional data to a server; means for the server to receive the text data and emotional data and generate a solution or advice using a generation AI module; means for adjusting the solution or advice based on the user's emotional data; means for transmitting the generated solution or advice to a user terminal; and means for the user terminal to display the received solution or advice and play it back aloud.

[1386] Hardware and Software Configuration

[1387] User device:

[1388] The user terminal is a device equipped with a voice recognition module, an emotion engine, a transmission / reception module, a display, a speaker, etc.

[1389] server:

[1390] The server is a remote system that receives text data and emotion data, generates solutions and advice using a generative AI module, and sends them to the user's device.

[1391] Communication Interface:

[1392] The communication interface is a network module for transmitting and receiving data between the user terminal and the server.

[1393] Program processing overview

[1394] First, the user inputs voice. The user device uses a voice recognition module to convert the voice input into text data, and then uses an emotion engine to analyze the user's emotions. The acquired text data and emotion data are sent to the server via a communication interface.

[1395] The server receives text data and emotion data sent from the user's device. The generative AI module generates problem-solving procedures and advice based on the received text data. The generated solutions and advice are adjusted based on the user's emotion data.

[1396] The adjusted solution or advice is then sent back to the user terminal via the communication interface, where it is displayed on the display and played aloud using a speech synthesis module.

[1397] Examples of concrete examples and prompts

[1398] For example, if a user says, "I want a new smartphone," the speech recognition module converts the speech into text and generates the text data, "I want a new smartphone." At the same time, the emotion engine recognizes the user's excited emotion.

[1399] An example of user input is:

[1400] Prompt Sentence Examples

[1401] User input: "I want a new smartphone"

[1402] User Sentiment: Excitement

[1403] The generative AI module generates the following recommendations based on this text data and emotion data:

[1404] "The best smartphones include those with the latest camera technology and long battery life. Our most popular models take your experience to the next level. Want to learn more?"

[1405] In this way, the system provides personalized support that reflects the user's emotions.

[1406] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1407] Step 1:

[1408] The user provides voice input.

[1409] Input: User speech (e.g., "I want a new smartphone")

[1410] Action: The user speaks into the device's microphone.

[1411] Output: User's voice data

[1412] Step 2:

[1413] The terminal uses a voice recognition module to convert the voice input into text data.

[1414] Input: Audio data

[1415] Data processing: Converting voice into text using voice recognition technology

[1416] How it works: The speech recognition module analyzes the audio signal and generates corresponding text.

[1417] Output: Text data (e.g., "I want a new smartphone")

[1418] Step 3:

[1419] The device uses an emotion engine to recognize the user's emotions.

[1420] Input: Audio data

[1421] Data calculation: Extracting features such as pitch, intonation, and speed from speech to estimate emotions

[1422] How it works: The emotion engine analyzes voice data and identifies the user's emotion.

[1423] Output: Emotion data (e.g., excitement)

[1424] Step 4:

[1425] The terminal transmits the text data and the emotion data to the server.

[1426] Input: Text data, emotion data

[1427] Operation: The sending and receiving module assembles data into packets and sends them to the server through the communication interface.

[1428] Output: Text data and emotion data sent to the server

[1429] Step 5:

[1430] A server receives the text data and the emotion data.

[1431] Input: Text data, emotion data

[1432] Operation: The server receives data through the communication interface and stores it in storage.

[1433] Output: Text data and emotion data stored in the server storage

[1434] Step 6:

[1435] The server uses a generative AI module to generate a solution or advice based on the text data.

[1436] Input: Text data

[1437] Data calculation: Generative AI module analyzes text data and generates appropriate solutions or advice

[1438] How it works: The generative AI module takes text data and uses algorithms to generate solutions or advice.

[1439] Output: Solution or advice (e.g., "We recommend a smartphone with the latest camera technology.")

[1440] Step 7:

[1441] The server tailors the solution or advice based on the user's emotional data.

[1442] Input: Solution or advice, sentiment data

[1443] Data processing: Using sentiment data to enhance and modify solutions and advice

[1444] How it works: The server references the emotion data and adjusts the content of the generated solution or advice to match the emotion.

[1445] Output: Tailored solution or advice (e.g., "You seem excited. In that case, we recommend a smartphone with the latest camera technology.")

[1446] Step 8:

[1447] The server sends the tailored solution or advice to the user terminal.

[1448] Input: Tailored solution or advice

[1449] Operation: The server assembles the adjusted solution or advice into a packet via the sending and receiving module and sends it to the user terminal through the communication interface.

[1450] Output: Tailored solution or advice sent to user terminal

[1451] Step 9:

[1452] The user device displays and plays audibly the tailored solution or advice received.

[1453] Input: Tailored solution or advice

[1454] How it works: The user device displays the solution or advice on the display and plays it aloud using a speech synthesis module.

[1455] Output: Displayed solution or advice, audio played (e.g., "You seem excited. In that case, we recommend a smartphone with the latest camera technology.")

[1456] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1457] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1458] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1459] [Fourth embodiment]

[1460] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1461] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1462] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1463] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1464] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1465] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1466] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1467] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1468] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1469] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1470] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1471] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1472] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1473] System Overview

[1474] This invention is a system for easily resolving problems users may encounter when operating or configuring a mobile phone. This system involves a series of processes: converting user voice input into text data, sending it to a server, generating solutions or advice using a generation AI module, and returning it to the user's device.

[1475] Key Components of the System

[1476] User device:

[1477] A device that allows users to input voice information. It includes a voice recognition module, a transmission / reception module, a display, and a speaker.

[1478] server:

[1479] A remote system that receives voice-recognized text data and generates solutions or advice using a generative AI module.

[1480] Communication Interface:

[1481] A network module for sending and receiving data between a user terminal and a server.

[1482] Program processing overview

[1483] 1. Voice input

[1484] Users communicate their problems and requests via voice.

[1485] 2. Voice Recognition

[1486] The device converts the user's voice input into text data.

[1487] 3. Sending text data

[1488] The terminal transmits the converted text data to the server.

[1489] 4. Receiving text data

[1490] The server receives the text data sent from the user terminal.

[1491] 5. Processing by the Generative AI Module

[1492] The server provides the received text to a generative AI module, which generates an appropriate solution or advice.

[1493] 6. Submitting Solutions or Advice

[1494] The server transmits the generated solution or advice to the user terminal.

[1495] 7. Displaying and playing audio solutions or advice

[1496] The device displays the received solution or advice on the display and plays it aloud through the speaker.

[1497] Specific examples

[1498] Example 1: Wi-Fi settings

[1499] When a user says, "I don't know how to set up Wi-Fi," the following process occurs:

[1500] Audio Input:

[1501] The user says, "I don't know how to set up Wi-Fi."

[1502] Voice Recognition:

[1503] The device converts the voice into text data: "I don't know how to set up Wi-Fi."

[1504] Sending text data:

[1505] The device sends text data to the server saying "I don't know how to set up Wi-Fi."

[1506] Receiving text data:

[1507] The server receives the text data.

[1508] Processing by the Generative AI module:

[1509] The server uses the text data to generate a solution that includes instructions for setting up Wi-Fi.

[1510] Example response from the generative AI module: "Open the Settings app and go to the Wi-Fi settings section. From the list of Wi-Fi networks, select the network you want to connect to, enter the password, and tap Connect."

[1511] Submit a solution or advice:

[1512] The server sends the generated solution to the user terminal.

[1513] View and listen to solutions or advice:

[1514] The device will show the solution it received on the display and play a voice message saying, "Open the Settings app and go to the Wi-Fi settings section. From the list of Wi-Fi networks, select the network you want to connect to, enter the password, and tap Connect."

[1515] This allows users to solve problems themselves and smoothly set up their mobile phones. The advantage of this system is that it provides fast and accurate support.

[1516] The processing flow will be explained below.

[1517] Step 1:

[1518] The user verbally states their problem or request, for example, "I don't know how to set up Wi-Fi."

[1519] Step 2:

[1520] The device receives the user's voice input and passes it to the speech recognition module, which converts the voice data into text data. Example: "I don't know how to set up Wi-Fi."

[1521] Step 3:

[1522] The terminal transmits the converted text data to the server.

[1523] Step 4:

[1524] The server receives the text data sent from the user terminal.

[1525] Step 5:

[1526] The server inputs the received text data into a generation AI module to generate a solution or advice. For example, a generated solution might be, "Open the Settings app and go to the Wi-Fi settings section. Select the network you want to connect to from the list of Wi-Fi networks, enter the password, and tap Connect."

[1527] Step 6:

[1528] The server transmits the generated solution or advice to the user terminal.

[1529] Step 7:

[1530] The terminal receives the solution or advice received from the server.

[1531] Step 8:

[1532] The device displays the received solution or advice on the display, and simultaneously plays back the solution aloud using a speech synthesis module.

[1533] Step 9:

[1534] Users can follow the solution displayed on their device to configure their phone: open the Settings app, go to the Wi-Fi settings section, select the network they want to connect to from the list, enter the password, and tap Connect.

[1535] Example 1

[1536] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1537] Conventional support systems for mobile phone operation and settings make it difficult for users to solve problems directly and quickly. In particular, systems that allow users to solve problems through voice input are limited, and in many cases, complex operating procedures are required. As a result, users are more likely to make operating or setting errors. In response to this, the present invention aims to provide a system that allows users to easily report problems through voice input and quickly provides solutions using a generative AI module.

[1538] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1539] In this invention, the server includes a means for receiving a problem or request provided by a user through voice input, a means for converting the voice input into text data, and a means for transmitting the converted text data to the server, so that the user can report a problem through voice, which is converted into text data and transmitted to the server, thereby enabling a solution to be generated and provided to the user for quickly and accurately resolving the problem.

[1540] "User" refers to the entity that operates the system and provides problems or requests using voice input.

[1541] "Voice input" refers to a means by which a user communicates problems or requests to a system using their voice.

[1542] "Text data" refers to data that has been converted from voice input into a character string format.

[1543] "Server" refers to the remote system that receives the text data converted from the voice input and generates solutions or advice using a generative AI module.

[1544] "Generative AI module" refers to an artificial intelligence algorithm that generates appropriate solutions or advice based on text data received from the user.

[1545] A "prompt sentence" refers to an instruction sentence for generating an output for solving a problem based on text data input to the generation AI module.

[1546] "Communication interface" refers to a network module for transmitting and receiving data between a user terminal and a server.

[1547] "User terminal" refers to a device through which a user makes voice input, and which is equipped with a voice recognition module, a display, and a speaker.

[1548] The present invention is a system that allows users to report problems related to mobile phone operation and settings through voice input, and uses a generative AI module to quickly and accurately provide solutions. The system includes the following main components: a user terminal, a server, and a communication interface.

[1549] System configuration

[1550] User Device

[1551] A user terminal is a device that allows users to input voice. It includes a microphone, display, and speaker as hardware, and a speech recognition module as software. Specifically, speech recognition software such as Google Voice API or IBM Watson is used.

[1552] server

[1553] The server is a remote system that receives text data sent from the user's device and generates solutions and advice using a generative AI model, such as OpenAI's GPT-4.

[1554] Communication Interface

[1555] The communication interface serves to send and receive data between the user device and the server, specifically via Wi-Fi or 4G / 5G communication.

[1556] Processing the data

[1557] 1. Voice input: The user voices the problem they are having with their phone's operation or settings. For example, they say, "I don't know how to set up Wi-Fi."

[1558] 2. Speech recognition: The device's speech recognition module records the user's voice and converts it into text data in real time. The Google Voice API converts the voice into text data such as "I don't know how to set up Wi-Fi."

[1559] 3. Sending text data: The converted text data is sent to the server, and the device's transceiver module transmits the data via Wi-Fi or 4G / 5G communication.

[1560] 4. Receiving text data: The server receives the text data and stores the received data in a processing queue.

[1561] 5. Processing by the generative AI module: The server generates a prompt and provides it to the generative AI model. For example, it provides the following prompt:

[1562] When a user inquires about how to set up Wi-Fi, please provide a clear explanation of the steps to set up Wi-Fi.

[1563] A generative AI model (e.g., OpenAI's GPT-4) generates a solution, such as "Open the Settings app and go to the Wi-Fi settings section. From the list of Wi-Fi networks, select the network you want to connect to, enter the password, and tap Connect."

[1564] 6. Sending the solution: The generated solution is sent to the user terminal, and the data is sent back through the communication interface.

[1565] 7. Display and audio solution: Your device will display and audio the solution using a Text-to-Speech (TTS) engine (e.g., Google TTS), for example, "Open the Settings app and go to the Wi-Fi settings section. From the list of Wi-Fi networks, select the network you want to connect to, enter the password, and tap Connect."

[1566] Specific examples

[1567] Example 1: Wi-Fi settings

[1568] When a user says, "I don't know how to set up Wi-Fi," the following sequence of events occurs:

[1569] 1. Voice input: The user says, "I don't know how to set up Wi-Fi."

[1570] 2. Speech recognition: The device uses the Google Voice API to convert the speech into text data such as "I don't know how to set up Wi-Fi."

[1571] 3. Sending text data: The device sends text data saying "I don't know how to set up Wi-Fi" to the server.

[1572] 4. Receiving text data: The server receives the text data.

[1573] 5. Processing by the generative AI module: The server provides the received text data to the generative AI model, which generates a solution including instructions for setting up Wi-Fi.

[1574] 6. Sending the solution: The server sends the generated solution to the user terminal.

[1575] 7. Displaying the solution and playing it aloud: The device displays the solution it received on the display and plays it aloud using Google TTS.

[1576] As described above, the system of the present invention allows a user to report a problem by voice, which is converted into text data and sent to a server, and then a solution to quickly and accurately resolve the problem can be generated and provided to the user.

[1577] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1578] Step 1:

[1579] Voice input

[1580] The user communicates problems or requests regarding the settings of their mobile phone by voice. The input is the user's voice data. Specifically, the user says, "I don't know how to set up Wi-Fi." This causes the voice data to be captured by the microphone in the device.

[1581] Step 2:

[1582] Voice Recognition

[1583] The device converts the voice data it receives into text data. The input is voice data, and the output is text data. Specifically, the device's voice recognition module uses the Google Voice API to convert the voice data into text data such as "I don't know how to set up Wi-Fi."

[1584] Step 3:

[1585] Sending text data

[1586] The device sends the converted text data to the server. The input is text data, and the output is data transmission to the server. Specifically, the device's transceiver module sends the text data "I don't know how to set up Wi-Fi" to the server using Wi-Fi or 4G / 5G communication.

[1587] Step 4:

[1588] Receiving text data

[1589] The server receives text data sent from the user terminal. The input is the text data sent from the terminal, and the output is data storage within the server. Specifically, the server's communication interface receives the text data and stores it in a processing queue.

[1590] Step 5:

[1591] Processing by generative AI module

[1592] The server provides the received text data to a generative AI model to generate a solution or advice. The input is the text data and a prompt, and the output is a solution or advice. Specifically, the server sends the following prompt to the generative AI model (e.g., OpenAI's GPT-4):

[1593] When a user inquires about how to set up Wi-Fi, please provide a clear explanation of the steps to set up Wi-Fi.

[1594] The generative AI model generates the following solution: "Open the Settings app and go to the Wi-Fi settings section. Select the network you want to connect to from the list of Wi-Fi networks, enter the password, and tap Connect."

[1595] Step 6:

[1596] Submit a solution

[1597] The server sends the generated solutions and advice to the user's device. The input is the generated solution, and the output is data transmission to the user's device. Specifically, the server's transmission module prepares the generated text data and transmits it to the device via Wi-Fi or 4G / 5G communication.

[1598] Step 7:

[1599] View and listen to the solution

[1600] The device displays the received solution on the display and plays it back aloud using a Text-to-Speech (TTS) engine. The input is the solution sent from the server, and the output is the display and audio playback. Specifically, the device's receiving module receives the solution text data and displays it on the display as follows:

[1601] Open the Settings app and go to the Wi-Fi settings section. Select the network you want to connect to from the list of Wi-Fi networks, enter the password, and tap Connect.

[1602] In addition, the Google TTS engine converts the text data into speech and plays it back through the speaker.

[1603] (Application example 1)

[1604] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1605] Modern security systems have sophisticated and complex functions, and it is often difficult for average users to understand how to configure and use them. Furthermore, when a configuration error or problem occurs, it is difficult to quickly find a solution, potentially increasing security risks. Traditional support systems have the problem of taking a long time to respond, increasing the hassle and stress for users.

[1606] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1607] In this invention, the server includes means for receiving a problem or request provided by a user through voice input, means for converting the voice input into text data, means for transmitting the converted text data to the server, means for the server to receive the text data and generate a solution or advice using a generation AI module, means for transmitting the generated solution or advice to the user terminal, means for the user terminal to display and play the received solution or advice by voice, means for receiving a problem or request regarding the configuration and use of the security system, and means for generating a solution or advice regarding the configuration and use of the security system, thereby enabling a user to quickly and easily configure and troubleshoot the security system.

[1608] A "user terminal" is an electronic device that allows a user to voice-input a problem or request and receive a solution or advice.

[1609] "Server" means a remote system that receives text data sent from a user terminal and generates a solution or advice using a generative AI module.

[1610] "Voice input" refers to the act of a user providing a problem or request to a device by voice.

[1611] "Text data" is character information generated using voice recognition technology based on voice input.

[1612] The "generative AI module" is an artificial intelligence module that analyzes text data and generates appropriate solutions and advice.

[1613] "Solutions or Advice" refers to specific responses or instructions generated by the generative AI module in response to a user's problem or request.

[1614] "Security system" is a general term for surveillance cameras, alarms, access control devices, and other equipment used to keep a home or office safe.

[1615] "Display" is the act of visually presenting information on the display of a user terminal.

[1616] "Audio playback" refers to the act of outputting audio information using the speaker of the user device.

[1617] An "operational procedure" is a set of specific steps that a user must perform to achieve a particular goal.

[1618] "Configuration Instructions" are instructions on how to properly configure the security system.

[1619] "Personalization" is the act of generating solutions and advice optimized for each individual user based on the user's past operation history and settings information.

[1620] "Speech recognition" is a technology that converts voice input into text data.

[1621] The present invention is a system that uses a user terminal, a server, and a generation AI module to provide appropriate solutions and advice for problems and requests related to the configuration and use of a security system. Specific embodiments for implementing this system are described below.

[1622] 1. User Device

[1623] The user device is equipped with a voice recognition module that receives voice input and converts it into text data. This module can use the Google Speech-to-Text API. It also has a communication interface that transmits the input text data to the server. This allows users to input security system issues and requests by voice, and the content is converted into text data and sent to the server.

[1624] 2. Server

[1625] The server receives text data sent from the user's device and passes it to the generation AI module. The received text data is analyzed by the generation AI module, and optimal solutions and advice are generated. OpenAI's GPT-4 can be used as the generation AI module. This generated data is then sent back to the user's device.

[1626] 3. Generative AI Module

[1627] The generative AI module generates solutions and advice on security system configuration and usage based on the text data received by the server. This module has learned from a huge data set and can provide the most appropriate information for the user's problem.

[1628] 4. User Interface

[1629] The user device has the ability to display the received solutions and advice on the screen and play them back aloud using the Google Text-to-Speech API, allowing users to confirm the solutions and advice both visually and audibly.

[1630] Specific examples

[1631] For example, if a user says "my security camera isn't working," the following will be processed:

[1632] 1. The user speaks "My security camera isn't working."

[1633] 2. The user device converts the voice into text data and sends it to the server.

[1634] 3. The server receives the text data and provides it to the generation AI module.

[1635] 4. The generative AI module generates a solution such as "Please check the power supply of the security camera and restart it."

[1636] 5. The server sends the generated solution to the user device.

[1637] 6. The user device displays the received solution on the display and plays it back aloud.

[1638] Prompt Sentence Examples

[1639] "How do I reset the settings on my security cameras?"

[1640] "My alarm system is malfunctioning. Can you help me fix it?"

[1641] This allows users to quickly and easily troubleshoot security system problems.

[1642] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1643] Step 1:

[1644] Users input their security system problems and requests by voice. The input voice is picked up through the microphone of the user's device. Based on the input, the user's voice data is collected.

[1645] Step 2:

[1646] The voice data captured by the user device is converted into text data using voice recognition software (e.g., Google Speech-to-Text API). In this step, voice signals are analyzed using voice recognition technology and converted into corresponding strings of characters. The output is the converted text data.

[1647] Step 3:

[1648] The user terminal sends the converted text data to the server. Here, the data is sent to the server using the HTTPS protocol to ensure security. The input is the text data generated by speech recognition, and the output is the text data sent to the server.

[1649] Step 4:

[1650] The server receives the text data sent from the user terminal. In this step, the server is prepared to analyze the text data. The input is the text data sent from the user terminal, and the output is the provision of the text data to the generation AI module.

[1651] Step 5:

[1652] The server provides the received text data to a generative AI module (e.g., OpenAI GPT-4) to generate an optimal solution or advice. In this step, the AI ​​model analyzes the text data and generates a solution. The output is the generated solution or advice text data.

[1653] Step 6:

[1654] The server sends the solutions and advice generated by the generative AI module to the user's device. Here, data is transferred securely using the HTTPS protocol again. The input is the text data of the generated solutions and advice, and the output is the transmission of the solutions and advice to the user's device.

[1655] Step 7:

[1656] The user device displays the received solution or advice on the display and plays it back aloud. In this step, the Google Text-to-Speech API is used to convert the text data into speech and output it as speech. The input is the text data of the solution or advice received from the server, and the output is the text displayed on the display and the solution or advice played back aloud.

[1657] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1658] System Overview

[1659] The present invention relates to a system that recognizes a user's voice input and emotions and provides optimal solutions and advice to the user regarding the operation and settings of a mobile phone. By combining this with the recognition of the user's emotions, the system can provide more personalized support.

[1660] Key Components of the System

[1661] User device:

[1662] This device allows users to input voice and converts that voice into text data and emotion data. It is equipped with a voice recognition module, emotion engine, transmission / reception module, display, speaker, etc.

[1663] server:

[1664] It is a remote system that receives text data and emotion data, generates solutions and advice using a generative AI module, and adjusts them based on the emotion data.

[1665] Communication Interface:

[1666] A network module for sending and receiving data between a user terminal and a server.

[1667] Program processing overview

[1668] 1. Voice input

[1669] The user voices their problem or request.

[1670] 2. Speech and Emotion Recognition

[1671] The device converts the user's voice input into text data and simultaneously recognizes the user's emotions using an emotion engine.

[1672] 3. Data transmission

[1673] The terminal transmits the text data and the emotion data to the server.

[1674] 4. Receiving Data

[1675] The server receives the text data and emotion data sent from the user terminal.

[1676] 5. Processing by the Generative AI Module

[1677] The server provides the received text data to a generation AI module, which generates an appropriate solution or advice.

[1678] 6. Adjustments based on emotional data

[1679] The server tailors the generated solution or advice based on the received emotional data, for example providing more detailed explanations if the user is feeling anxious.

[1680] 7. Submitting Solutions or Advice

[1681] The server sends the tailored solution or advice to the user terminal.

[1682] 8. Displaying and playing audio solutions or advice

[1683] The solution or advice received by the terminal is displayed on the display and played aloud using a speech synthesis module.

[1684] Specific examples

[1685] Example 1: Wi-Fi settings

[1686] When a user says, "I don't know how to set up Wi-Fi," the device converts the user's voice into text data and simultaneously recognizes the user's anxiety using an emotion engine.

[1687] Audio Input:

[1688] The user says, "I don't know how to set up Wi-Fi."

[1689] Speech and emotion recognition:

[1690] The voice recognition module converts this into text data such as "I don't know how to set up Wi-Fi," and the emotion engine recognizes the user's anxiety.

[1691] Sending data:

[1692] The device sends text data such as "I don't know how to set up Wi-Fi" and emotional data such as "anxiety" to the server.

[1693] Receiving data:

[1694] The server receives the text data and the emotion data.

[1695] Processing by the Generative AI module:

[1696] The server generates a solution based on the text data, including instructions for setting up Wi-Fi.

[1697] Example response from the generative AI module: "Open the Settings app and go to the Wi-Fi settings section. From the list of Wi-Fi networks, select the network you want to connect to, enter the password, and tap Connect."

[1698] Adjustments based on sentiment data:

[1699] The server adjusts the generated solution based on the anxiety sentiment and provides a detailed explanation.

[1700] Sample tailored response: "Don't worry! First, open the Settings app on your phone. Then, go to the Wi-Fi settings section. Next, select your home network from the list of Wi-Fi networks that appears. Finally, enter your network password and tap the Connect button."

[1701] Submit a solution or advice:

[1702] The server sends the adjusted solution to the user terminal.

[1703] View and listen to solutions or advice:

[1704] The device will display the solution it receives on the display and play a voice message saying, "Don't worry. First, open your phone's Settings app..."

[1705] This example allows users to easily set up Wi-Fi without any worries. Combined with the emotion engine, it improves the user experience.

[1706] The processing flow will be explained below.

[1707] Step 1:

[1708] The user verbally states their problem or request, for example, "I don't know how to set up Wi-Fi."

[1709] Step 2:

[1710] The device receives the user's voice input and passes it to the speech recognition module, which converts the voice data into text data. Example: "I don't know how to set up Wi-Fi."

[1711] Step 3:

[1712] The device passes the user's voice input to the emotion engine, which analyzes the voice tone, speed, intonation, etc. to recognize the user's emotion. Example: Recognizes "anxiety"

[1713] Step 4:

[1714] The terminal transmits the converted text data and the recognized emotion data to the server.

[1715] Step 5:

[1716] The server receives the text data and emotion data sent from the user terminal.

[1717] Step 6:

[1718] The server inputs the received text data into a generation AI module, which generates an appropriate solution or advice. For example, a generated solution might be, "Open the Settings app and go to the Wi-Fi settings section. Select the network you want to connect to from the list of Wi-Fi networks, enter the password, and tap Connect."

[1719] Step 7:

[1720] The server tailors the generated solution or advice based on the emotion data received. For example, if the user is perceived as "anxious," it adds a detailed explanation to the solution. Example: "Don't worry. First, open the Settings app on your phone..."

[1721] Step 8:

[1722] The server sends the tailored solution or advice to the user terminal.

[1723] Step 9:

[1724] The terminal receives the solution or advice received from the server.

[1725] Step 10:

[1726] The device displays the received solution or advice on a display, and simultaneously plays the adjusted solution aloud using a speech synthesis module.

[1727] Step 11:

[1728] Users should follow the on-screen and audio-guided solutions to set up their phone by opening the Settings app, navigating to the Wi-Fi settings section, selecting the network they want to connect to from the list, entering the password, and tapping Connect.

[1729] Example 2

[1730] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1731] When operating or setting up a conventional mobile phone, users often become confused because they do not know the operating procedures or how to set it up. Furthermore, they may feel anxious or stressed because they are not provided with personalized support that is tailored to the user's emotions and situation. This can lead to a poor user experience and make it difficult to use the mobile phone.

[1732] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1733] In this invention, the server includes means for receiving a problem or request provided by a user through voice input, means for converting the voice input into text data, means for transmitting the converted text data and emotion data to the server, means for the server to receive the text data and emotion data and generate a solution or advice using a generative AI model, means for adjusting the generated solution or advice based on the received emotion data, means for transmitting the adjusted solution or advice to the user terminal, and means for the user terminal to display the received solution or advice and play it back by voice. This makes it possible to provide personalized support for mobile phone operations and settings while taking into consideration the user's emotions.

[1734] "Voice input" is the act of a user providing a problem or request by speaking into a mobile phone or other device.

[1735] "Text data" is data that has been converted from voice input into text information, and specifically expresses the content of a problem or request.

[1736] "Emotional data" is information about emotions extracted from the user's voice tone and expressions, and is data that represents the user's feelings and psychological state.

[1737] A "server" is a remote computer system that receives data from a user terminal via a communications network, processes the data using a generative AI model or the like, and transmits the results to the user terminal.

[1738] A "generative AI model" is an artificial intelligence model that automatically generates solutions and advice based on received text data, and is a program for generating specific answers and instructions.

[1739] "User device" refers to a device used by a user, such as a mobile phone or computer, for voice input and for receiving and displaying solutions and advice.

[1740] "Solutions or Advice" refers to instructions or guides on how to solve a user's problem or procedures that are generated by a generative AI model based on text data.

[1741] "Adjustment" refers to the act of changing the content and expression of the generated solution or advice to match the user's emotions based on the received emotion data.

[1742] The present invention relates to a system that recognizes a user's voice input and emotions and provides optimal solutions and advice to the user regarding the operation and settings of a mobile phone. By combining this with the recognition of the user's emotions, the system can provide more personalized support.

[1743] Hardware and Software Configuration

[1744] User Device

[1745] Speech recognition module: converts speech into text data. For example, you can use the Google Speech-to-Text API.

[1746] Emotion engine: Extracts emotion data from the user's voice. For example, it can use the Microsoft Azure Emotion API.

[1747] Transmitting / receiving module: Sends text data and emotion data to the server and receives responses from the server.

[1748] Display and speech synthesis module: displays solutions and advice and plays them aloud.

[1749] server

[1750] Communication interface: processes data received from the user terminal.

[1751] Generative AI models: Generate solutions and advice based on the received text data. For example, you can use the OpenAI GPT-3 model.

[1752] Data adjustment module: Adjusts the generated solutions and advice based on the received sentiment data.

[1753] Processing flow

[1754] 1. Receiving voice input

[1755] The user verbally describes their problem or request into their mobile phone.

[1756] 2. Speech and Emotion Recognition

[1757] The device uses a voice recognition module to convert voice into text data and an emotion engine to recognize the user's emotions.

[1758] 3. Data transmission

[1759] The text data and emotion data are transmitted to the server using a transmitting / receiving module.

[1760] 4. Processing by generative AI models

[1761] The server provides the received text data to a generative AI model, which generates appropriate solutions and advice.

[1762] For example, the prompt is:

[1763] If a user says they don't know how to set up Wi-Fi, explain the process in detail. Be sensitive to their concerns.

[1764] 5. Adjustments based on emotional data

[1765] The server adjusts the generated solutions and advice based on the received emotion data.

[1766] 6. Submitting and Displaying Solutions or Advice

[1767] The tailored solutions and advice are sent from the server to the user's device, which displays them on the display and plays them aloud using a speech synthesis module.

[1768] Specific examples

[1769] Example 1: Wi-Fi settings

[1770] When a user says, "I don't know how to set up Wi-Fi," the device converts the user's voice into text data and detects their anxiety.

[1771] 1. Voice input

[1772] The user says, "I don't know how to set up Wi-Fi."

[1773] 2. Speech and Emotion Recognition

[1774] The voice recognition module converts "I don't know how to set up Wi-Fi" into text data, and the emotion engine recognizes the user's anxiety.

[1775] 3. Data transmission

[1776] The device sends the text data "I don't know how to set up Wi-Fi" and the emotional data "anxiety" to the server.

[1777] 4. Processing by generative AI models

[1778] The server generates Wi-Fi setting instructions based on the text data.

[1779] Example response: "Open the Settings app and go to the Wi-Fi settings section. From the list of Wi-Fi networks, select the network you want to connect to, enter the password, and tap Connect."

[1780] 5. Adjustments based on emotional data

[1781] The server adjusts the generated solutions based on anxiety sentiment and provides detailed explanations.

[1782] Sample tailored response: "Don't worry! First, open the Settings app on your phone. Then, go to the Wi-Fi settings section. Next, select your home network from the list of Wi-Fi networks that appears. Finally, enter your network password and tap the Connect button."

[1783] 6. Displaying and playing audio solutions or advice

[1784] The device will display the received solution on its display and play a voice message saying, "Don't worry. First, open the Settings app on your phone..."

[1785] This system allows users to set up Wi-Fi seamlessly and without worry. The combination of an emotion engine and generative AI models improves the user experience.

[1786] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1787] Step 1:

[1788] Receiving audio input

[1789] user:

[1790] The user verbally describes their problem or request into their mobile phone.

[1791] For example, say, "I don't know how to set up Wi-Fi."

[1792] Input: User speech.

[1793] Output: Audio data.

[1794] Step 2:

[1795] Speech and Emotion Recognition

[1796] Device:

[1797] A speech recognition module converts the user's voice data into text data, for example, using the Google Speech-to-Text API.

[1798] The emotion engine analyzes the voice data and extracts emotion data, for example, using the Microsoft Azure Emotion API.

[1799] Input: Audio data.

[1800] Data processing: The voice data is analyzed, and the voice recognition module converts the voice into text data. The emotion engine also extracts emotional information from the voice.

[1801] Output: Text and sentiment data.

[1802] Step 3:

[1803] Sending data

[1804] Device:

[1805] The generated text data and emotion data are sent to the server using a transmission / reception module.

[1806] Input: Text data and sentiment data.

[1807] Data processing: Assembling text data and emotion data into data packets.

[1808] Output: Send the data packet to the server over the communication network.

[1809] Step 4:

[1810] Receiving data

[1811] server:

[1812] A data packet is received from a user terminal through a communication interface.

[1813] Input: Data packet.

[1814] Data processing: Analyze data packets and extract text and sentiment data.

[1815] Output: Extracted text and sentiment data.

[1816] Step 5:

[1817] Processing by generative AI models

[1818] server:

[1819] The received text data is fed into a generative AI model (e.g., OpenAI GPT-3) to generate solutions or advice.

[1820] Example prompt: "If the user says they don't know how to set up Wi-Fi, explain the process in detail. Be sensitive to their concerns."

[1821] Input: Text data.

[1822] Data processing: Input text data into a generative AI model to obtain generated solutions or advice.

[1823] Output: The generated solution or advice.

[1824] Step 6:

[1825] Adjustments based on emotional data

[1826] server:

[1827] Tailor generated solutions and advice based on sentiment data.

[1828] For example: If the user is feeling unsure, add a detailed explanation.

[1829] Input: Generated solution or advice, sentiment data.

[1830] Data processing: Analyzing sentiment data and making adjustments based on generated solutions and advice.

[1831] Output: Tailored solutions or advice.

[1832] Step 7:

[1833] Submit a solution or advice

[1834] server:

[1835] Send tailored solutions and advice to your device.

[1836] Input: Tailored solutions or advice.

[1837] Data processing: Consolidating tailored solutions and advice into data packets.

[1838] Output: Sends data packets over the communications network to the user terminal.

[1839] Step 8:

[1840] Display and audio playback of solutions or advice

[1841] Device:

[1842] The received solutions and advice are displayed on the screen and played aloud using a speech synthesis module.

[1843] Example: Display and play the following: "Don't worry. First, open the Settings app on your phone..."

[1844] Input: Tailored solutions or advice.

[1845] Data processing: Converting tailored solutions and advice into display and speech synthesis.

[1846] Output: Display and audio playback.

[1847] (Application example 2)

[1848] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1849] Conventional support systems for setting up and operating mobile phones and communication devices simply convert the user's voice input into text data and provide solutions and advice. However, because they do not take the user's emotions into consideration, the support content often does not match the user's current situation, resulting in an unsatisfactory user experience. Furthermore, users who are feeling particularly anxious or confused need appropriate advice that addresses their emotions. To solve these issues, a more personalized support system that incorporates the user's emotions is needed.

[1850] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1851] In this invention, the server includes means for receiving a problem or request provided by a user through voice input, means for converting the voice input into text data and emotional data, means for transmitting the converted text data and emotional data to the server, means for the server to receive the text data and emotional data and generate a solution or advice using a generation AI module, means for adjusting the solution or advice based on the user's emotional data, means for transmitting the generated solution or advice to a user terminal, and means for the user terminal to display the received solution or advice and play it back by voice, thereby enabling personalized support that reflects the user's emotions.

[1852] "Voice input" is voice data provided by the user through a microphone.

[1853] "Emotion data" is data that indicates the user's emotions analyzed from voice input.

[1854] "User terminal" refers to a device operated by a user that accepts voice input and displays and plays solutions and advice.

[1855] A "generative AI module" is a program that runs within the server and includes artificial intelligence that generates solutions and advice based on received text data.

[1856] The "server" is a remote system that receives text data and emotion data sent from a user terminal, generates solutions or advice using a generative AI module, and sends the solutions or advice to the user terminal.

[1857] "Text data" is character data converted from voice input using voice recognition technology.

[1858] "Solutions or Advice" refers to problem-solving steps or advice provided by the generative AI module based on text data and emotion data.

[1859] A "voice recognition module" is a program or hardware that converts voice input into text data.

[1860] A "communication interface" is a network means for transmitting and receiving data between a user terminal and a server.

[1861] "Personalization" is a means of providing settings and advice optimized for each user.

[1862] System configuration

[1863] The present invention is a system including: means for receiving a problem or request provided by a user through voice input; means for converting the voice input into text data and emotional data; means for transmitting the text data and emotional data to a server; means for the server to receive the text data and emotional data and generate a solution or advice using a generation AI module; means for adjusting the solution or advice based on the user's emotional data; means for transmitting the generated solution or advice to a user terminal; and means for the user terminal to display the received solution or advice and play it back aloud.

[1864] Hardware and Software Configuration

[1865] User device:

[1866] The user terminal is a device equipped with a voice recognition module, an emotion engine, a transmission / reception module, a display, a speaker, etc.

[1867] server:

[1868] The server is a remote system that receives text data and emotion data, generates solutions and advice using a generative AI module, and sends them to the user's device.

[1869] Communication Interface:

[1870] The communication interface is a network module for transmitting and receiving data between the user terminal and the server.

[1871] Program processing overview

[1872] First, the user inputs voice. The user device uses a voice recognition module to convert the voice input into text data, and then uses an emotion engine to analyze the user's emotions. The acquired text data and emotion data are sent to the server via a communication interface.

[1873] The server receives text data and emotion data sent from the user's device. The generative AI module generates problem-solving procedures and advice based on the received text data. The generated solutions and advice are adjusted based on the user's emotion data.

[1874] The adjusted solution or advice is then sent back to the user terminal via the communication interface, where it is displayed on the display and played aloud using a speech synthesis module.

[1875] Examples of concrete examples and prompts

[1876] For example, if a user says, "I want a new smartphone," the speech recognition module converts the speech into text and generates the text data, "I want a new smartphone." At the same time, the emotion engine recognizes the user's excited emotion.

[1877] An example of user input is:

[1878] Prompt Sentence Examples

[1879] User input: "I want a new smartphone"

[1880] User Sentiment: Excitement

[1881] The generative AI module generates the following recommendations based on this text data and emotion data:

[1882] "The best smartphones include those with the latest camera technology and long battery life. Our most popular models take your experience to the next level. Want to learn more?"

[1883] In this way, the system provides personalized support that reflects the user's emotions.

[1884] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1885] Step 1:

[1886] The user provides voice input.

[1887] Input: User speech (e.g., "I want a new smartphone")

[1888] Action: The user speaks into the device's microphone.

[1889] Output: User's voice data

[1890] Step 2:

[1891] The terminal uses a voice recognition module to convert the voice input into text data.

[1892] Input: Audio data

[1893] Data processing: Converting voice into text using voice recognition technology

[1894] How it works: The speech recognition module analyzes the audio signal and generates corresponding text.

[1895] Output: Text data (e.g., "I want a new smartphone")

[1896] Step 3:

[1897] The device uses an emotion engine to recognize the user's emotions.

[1898] Input: Audio data

[1899] Data calculation: Extracting features such as pitch, intonation, and speed from speech to estimate emotions

[1900] How it works: The emotion engine analyzes voice data and identifies the user's emotion.

[1901] Output: Emotion data (e.g., excitement)

[1902] Step 4:

[1903] The terminal transmits the text data and the emotion data to the server.

[1904] Input: Text data, emotion data

[1905] Operation: The sending and receiving module assembles data into packets and sends them to the server through the communication interface.

[1906] Output: Text data and emotion data sent to the server

[1907] Step 5:

[1908] A server receives the text data and the emotion data.

[1909] Input: Text data, emotion data

[1910] Operation: The server receives data through the communication interface and stores it in storage.

[1911] Output: Text data and emotion data stored in the server storage

[1912] Step 6:

[1913] The server uses a generative AI module to generate a solution or advice based on the text data.

[1914] Input: Text data

[1915] Data calculation: Generative AI module analyzes text data and generates appropriate solutions or advice

[1916] How it works: The generative AI module takes text data and uses algorithms to generate solutions or advice.

[1917] Output: Solution or advice (e.g., "We recommend a smartphone with the latest camera technology.")

[1918] Step 7:

[1919] The server tailors the solution or advice based on the user's emotional data.

[1920] Input: Solution or advice, sentiment data

[1921] Data processing: Using sentiment data to enhance and modify solutions and advice

[1922] How it works: The server references the emotion data and adjusts the content of the generated solution or advice to match the emotion.

[1923] Output: Tailored solution or advice (e.g., "You seem excited. In that case, we recommend a smartphone with the latest camera technology.")

[1924] Step 8:

[1925] The server sends the tailored solution or advice to the user terminal.

[1926] Input: Tailored solution or advice

[1927] Operation: The server assembles the adjusted solution or advice into a packet via the sending and receiving module and sends it to the user terminal through the communication interface.

[1928] Output: Tailored solution or advice sent to user terminal

[1929] Step 9:

[1930] The user device displays and plays audibly the tailored solution or advice received.

[1931] Input: Tailored solution or advice

[1932] How it works: The user device displays the solution or advice on the display and plays it aloud using a speech synthesis module.

[1933] Output: Displayed solution or advice, audio played (e.g., "You seem excited. In that case, we recommend a smartphone with the latest camera technology.")

[1934] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1935] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1936] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1937] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1938] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1939] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1940] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1941] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1942] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1943] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1944] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1945] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1946] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1947] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1948] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1949] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1950] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1951] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1952] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1953] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1954] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1955] The following is further disclosed regarding the above embodiment.

[1956] (Claim 1)

[1957] a means for receiving a problem or request provided by a user via voice input;

[1958] means for converting voice input into text data;

[1959] means for transmitting the converted text data to a server;

[1960] a server receiving the text data and generating a solution or advice using a generation AI module;

[1961] means for transmitting the generated solution or advice to a user terminal;

[1962] a means for the user terminal to display and play the received solution or advice by voice;

[1963] A system including:

[1964] (Claim 2)

[1965] 2. The system of claim 1, wherein the solution or advice generated by the generation AI module includes an explanation of the user's operating procedures and settings.

[1966] (Claim 3)

[1967] The system of claim 1, wherein the results of the speech recognition and the solutions or advice generated by the generation AI module are personalized based on the user's past operation history and setting information.

[1968] "Example 1"

[1969] (Claim 1)

[1970] a means for receiving a problem or request provided by a user via voice input;

[1971] means for converting voice input into text data;

[1972] means for transmitting the converted text data to a server;

[1973] a server receiving the text data and generating a solution or advice using a generation AI module;

[1974] means for transmitting the generated solution or advice to a user terminal;

[1975] means for displaying and audibly playing the received solution or advice on the user terminal;

[1976] a means for providing a prompt to the generative AI module to generate an appropriate solution to the user's problem;

[1977] A system including:

[1978] (Claim 2)

[1979] 2. The system of claim 1, wherein the solution or advice generated by the generation AI module includes an explanation of the user's operating procedures and settings.

[1980] (Claim 3)

[1981] The system of claim 1, wherein the results of the speech recognition and the solutions or advice generated by the generation AI module are personalized based on the user's past operation history and setting information.

[1982] "Application Example 1"

[1983] (Claim 1)

[1984] a means for receiving a problem or request provided by a user via voice input;

[1985] means for converting voice input into text data;

[1986] means for transmitting the converted text data to a server;

[1987] a server receiving the text data and generating a solution or advice using a generation AI module;

[1988] means for transmitting the generated solution or advice to a user terminal;

[1989] means for displaying and audibly playing the received solution or advice on the user terminal;

[1990] A means for receiving problems or requests regarding the configuration and use of the security system;

[1991] means for generating solutions or advice regarding the configuration and use of security systems;

[1992] A system including:

[1993] (Claim 2)

[1994] 2. The system of claim 1, wherein the solutions or advice generated by the generative AI module include instructions for operating and configuring a security system.

[1995] (Claim 3)

[1996] The system of claim 1, wherein the results of the speech recognition and the solutions or advice generated by the generation AI module are personalized based on the user's past operation history and setting information.

[1997] "Example 2: Combining Emotion Engines"

[1998] (Claim 1)

[1999] a means for receiving a problem or request provided by a user via voice input;

[2000] means for converting voice input into text data;

[2001] means for transmitting the converted text data and emotion data to a server;

[2002] a server receiving the text data and emotion data and generating a solution or advice using a generative AI model;

[2003] a means for adjusting the generated solution or advice based on the received sentiment data;

[2004] means for transmitting the tailored solution or advice to the user terminal;

[2005] a means for the user terminal to display and play the received solution or advice by voice;

[2006] A system including:

[2007] (Claim 2)

[2008] 10. The system of claim 1, wherein the solution or advice generated by the generative AI model includes a description of the user's operating procedures and settings, with details adjusted based on the received emotional data.

[2009] (Claim 3)

[2010] The system of claim 1, wherein the results of the speech recognition and the solutions or advice generated by the generative AI model are personalized based on the user's past operation history and setting information.

[2011] "Application example 2 when combining emotion engines"

[2012] (Claim 1)

[2013] a means for receiving a problem or request provided by a user via voice input;

[2014] means for converting voice input into text data;

[2015] means for transmitting the converted text data and the user's emotion data to a server;

[2016] a server receiving the text data and emotion data and generating a solution or advice using a generative AI module;

[2017] a means for tailoring solutions or advice based on user sentiment data;

[2018] means for transmitting the generated solution or advice to a user terminal;

[2019] a means for the user terminal to display and play the received solution or advice by voice;

[2020] A system including:

[2021] (Claim 2)

[2022] 2. The system of claim 1, wherein the solution or advice generated by the generative AI module includes an explanation of the user's operating procedures and settings and is adjusted based on emotional data.

[2023] (Claim 3)

[2024] The system of claim 1, wherein the results of the speech recognition and the solutions or advice generated by the generation AI module are personalized based on the user's emotional data in addition to the user's past operation history and setting information. [Explanation of symbols]

[2025] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for receiving a problem or request provided by a user via voice input; means for converting voice input into text data; means for transmitting the converted text data to a server; a server receiving the text data and generating a solution or advice using a generation AI module; means for transmitting the generated solution or advice to a user terminal; a means for the user terminal to display and play the received solution or advice by voice; A system including:

2. The system of claim 1 , wherein the solution or advice generated by the generation AI module includes an explanation of user operation procedures and settings.

3. The system of claim 1 , wherein the results of the speech recognition and the solutions or advice generated by the generation AI module are personalized based on the user's past operation history and setting information.

Images