system

Abstract

The system according to the embodiment aims to enable a user to easily set up and operate a mobile phone simply by issuing instructions in natural language. [Solution] A system according to an embodiment includes a reception unit, an analysis unit, and an execution unit. The reception unit receives instructions in natural language from a user. The analysis unit analyzes the instructions in natural language received by the reception unit and identifies appropriate settings and operations. The execution unit executes the settings and operations identified by the analysis unit.

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional technologies have had the problem that users are required to go through complicated procedures and have specialized knowledge when setting up and operating their mobile phones, making them difficult to use.

[0005] The system according to the embodiment aims to enable a user to easily set up and operate a mobile phone simply by issuing instructions in natural language. [Means for solving the problem]

[0006] The system according to the embodiment includes a reception unit, an analysis unit, and an execution unit. The reception unit receives instructions in natural language from a user. The analysis unit analyzes the instructions in natural language received by the reception unit and identifies appropriate settings and operations. The execution unit executes the settings and operations identified by the analysis unit. [Effects of the Invention]

[0007] The system according to the embodiment allows the user to easily set up and operate a mobile phone simply by issuing instructions in natural language. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. DETAILED DESCRIPTION OF THE INVENTION

[0009] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0010] First, the terms used in the following description will be explained.

[0011] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, the processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (Tensor Processing Unit).

[0012] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0013] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0014] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), and Bluetooth (registered trademark).

[0015] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0016] [First embodiment] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0017] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0018] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0019] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0020] The reception device 38 includes a touch panel 38A and a microphone 38B, and receives user input. The touch panel 38A detects contact with a pointer (for example, a pen or a finger) to receive user input by the touch of the pointer. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 (see FIG. 2) acquires the data indicating the user input.

[0021] Output device 40 includes a display 40A and a speaker 40B, and presents data to a user by outputting the data in a form of expression that the user can perceive (e.g., audio and / or text). Display 40A displays visible information such as text and images in accordance with instructions from processor 46. Speaker 40B outputs audio in accordance with instructions from processor 46. Camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0022] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0023] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0024] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0025] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate a user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotion, including estimation and prediction of the user's emotion, but is not limited to these examples. Furthermore, the estimation and prediction of emotion also includes, for example, emotion analysis.

[0026] In the smart device 14, the specific processing is performed by the processor 46. The storage 50 stores a specific processing program 60. The specific processing program 60 is used together with the specific processing program 56 by the data processing system 10. The processor 46 reads the specific processing program 60 from the storage 50 and executes the read specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as the control unit 46A in accordance with the specific processing program 60 executed on the RAM 48. Note that the smart device 14 has a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59, and can also perform processing similar to that of the specific processing unit 290 using these models.

[0027] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device (e.g., a generation server) may have the data generation model 58. In this case, the data processing device 12 obtains a processing result (prediction result, etc.) using the data generation model 58 by communicating with the server device having the data generation model 58. Furthermore, the data processing device 12 may be a server device, or may be a terminal device owned by a user (e.g., a mobile phone, a robot, a home appliance, etc.). Next, an example of processing by the data processing system 10 according to the first embodiment will be described.

[0028] (Example 1) A mobile phone setting support system according to an embodiment of the present invention utilizes a generation AI to simplify the setup and operation of a mobile phone. This mobile phone setting support system allows a user to input setup and operation instructions in natural language, and the generation AI analyzes the instructions and executes the appropriate setup and operation. For example, when a user inputs instructions such as "Turn on Wi-Fi" or "Maximize screen brightness," the generation AI analyzes the instructions and executes the operation of turning on Wi-Fi or maximizing screen brightness. In this way, users can easily set up and operate their mobile phones without performing complex operations. For example, even non-technical users or elderly users can set up and operate their mobile phones simply by inputting instructions in natural language. Furthermore, even busy users can complete the setup and operation in a short amount of time. This allows the mobile phone setting support system to easily allow users to set up and operate their mobile phones simply by inputting instructions in natural language.

[0029] A mobile phone setting assistance system according to an embodiment includes a reception unit, an analysis unit, and an execution unit. The reception unit receives natural language instructions from a user. Natural language instructions from a user include, but are not limited to, voice instructions and text instructions. The reception unit can convert voice instructions into text data using, for example, voice recognition technology. The reception unit can also allow a user to directly input text instructions using a text input interface. The analysis unit uses a generation AI to analyze the natural language instructions received by the reception unit and identify appropriate settings and operations. The analysis unit analyzes the instructions using natural language processing technology such as morphological analysis, grammatical analysis, and semantic analysis. For example, the generation AI analyzes the user's instructions using a text generation AI (e.g., GPT-4 (registered trademark)) to identify appropriate settings and operations. The analysis unit can also use the generation AI to understand the intent of the user's instructions and identify appropriate settings and operations. For example, the generation AI analyzes the instruction "Turn on Wi-Fi" and identifies the operation to turn on Wi-Fi. The execution unit executes the settings and operations identified by the analysis unit. For example, the execution unit can access the settings menu of the mobile phone and execute an operation to turn on Wi-Fi. The execution unit can also execute an operation to maximize screen brightness. For example, the execution unit accesses the settings menu of the mobile phone and executes an operation to maximize screen brightness. This allows the mobile phone setting support system according to the embodiment to easily configure and operate the mobile phone simply by the user inputting instructions in natural language. Some or all of the above-described processing in the execution unit may be performed using, for example, a generation AI, or may be performed without using a generation AI. For example, the execution unit can execute the settings and operations using a generation AI model that receives the settings and operations identified by the analysis unit as input and outputs the settings and operations.

[0030] The analysis unit can analyze natural language instructions using a generation AI and identify appropriate settings and operations. The generation AI can analyze user instructions and identify appropriate settings and operations using, for example, a text generation AI (e.g., GPT-4). For example, the generation AI can analyze the instruction "Turn on Wi-Fi" and identify the operation to turn on Wi-Fi. The generation AI can also understand the intent of the user's instructions and identify appropriate settings and operations. For example, the generation AI can analyze the instruction "Maximize screen brightness" and identify the operation to maximize screen brightness. In this way, the generation AI can accurately analyze natural language instructions and identify appropriate settings and operations. Some or all of the above-described processing in the analysis unit can be performed using, for example, a generation AI, or can be performed without using a generation AI. For example, the analysis unit can analyze instructions using a generation AI model that receives user instructions as input and outputs appropriate settings and operations.

[0031] The execution unit can execute an operation to turn Wi-Fi on or off or adjust screen brightness. For example, the execution unit accesses a settings menu of a mobile phone and executes an operation to turn Wi-Fi on. For example, the execution unit turns on the Wi-Fi switch from the settings menu. The execution unit can also execute an operation to turn Wi-Fi off. For example, the execution unit turns off the Wi-Fi switch from the settings menu. The execution unit can also execute an operation to maximize screen brightness. For example, the execution unit moves the screen brightness slider to the maximum from the settings menu. The execution unit can also execute an operation to minimize screen brightness. For example, the execution unit moves the screen brightness slider to the minimum from the settings menu. This allows a user to easily turn Wi-Fi on or off or adjust screen brightness. Some or all of the above-described processing in the execution unit may be performed using, for example, a generative AI, or may be performed without using a generative AI. For example, the execution unit can execute settings and operations using a generative AI model that receives settings and operations identified by the analysis unit as input and outputs settings and operations.

[0032] The execution unit can perform operations to install or uninstall an app and set notifications. For example, the execution unit accesses an app store and performs an operation to install an app. For example, the execution unit searches for a specific app from the app store and clicks an install button. The execution unit can also perform an operation to uninstall an app. For example, the execution unit displays a list of apps from a settings menu and clicks an uninstall button. The execution unit can also perform an operation to set notifications. For example, the execution unit switches notifications on and off from the settings menu. The execution unit can also perform an operation to set notification sounds. For example, the execution unit selects a notification sound from the settings menu. This allows a user to easily install or uninstall an app and set notifications. Some or all of the above-described processing in the execution unit may be performed using, for example, a generation AI, or may be performed without using a generation AI. For example, the execution unit can perform settings and operations using a generation AI model that receives settings and operations identified by the analysis unit as input and outputs settings and operations.

[0033] The analysis unit may include a feedback unit that analyzes a user's instruction and feeds back the analysis result to the user. The feedback unit provides the analysis result to the user in text format, for example. For example, the feedback unit displays the analysis result as a text message. The feedback unit may also provide the analysis result to the user in audio format. For example, the feedback unit plays back the analysis result as an audio message. The feedback unit may also provide the analysis result in a visually easy-to-understand format. For example, the feedback unit displays the analysis result as a graph or chart. This allows the user to receive the analysis result as feedback. Some or all of the above-described processing in the feedback unit may be performed using, for example, a generation AI, or may be performed without using a generation AI. For example, the feedback unit may provide feedback using a generation AI model that receives the analysis result identified by the analysis unit as input and outputs feedback.

[0034] The execution unit may include a history unit that records the operation history. The history unit records the operation history in the form of, for example, date and time, operation content, operation result, etc. For example, the history unit records the date and time when the operation was performed. The history unit can also record the operation content. For example, the history unit records operation content such as turning Wi-Fi on and off or adjusting the screen brightness. The history unit can also record the operation result. For example, the history unit records whether the operation was successful. This allows the user to check the past operation history. Some or all of the above-mentioned processing in the history unit may be performed using, for example, a generation AI, or may be performed without using a generation AI. For example, the history unit can record the operation history using a generation AI model that inputs the operation content identified by the execution unit and outputs the operation history.

[0035] The reception unit can analyze the user's past instruction history and select the optimal reception method. The reception unit, for example, prioritizes reception of instruction methods (such as voice and text) that the user has frequently used in the past. For example, the reception unit analyzes the user's past instruction history and, if the user frequently uses voice instructions, prioritizes reception of voice instructions. The reception unit can also predict and suggest an instruction method to be used in a specific time period based on the user's past instruction history. For example, if the user frequently uses text instructions in the morning, the reception unit prioritizes reception of text instructions in the morning. The reception unit can also analyze patterns of instructions given by the user in the past and select the optimal reception method. For example, if the user uses a specific instruction method when performing a specific operation, the reception unit prioritizes reception of that instruction method. In this way, the optimal reception method can be selected by analyzing the user's past instruction history. Some or all of the above-described processing in the reception unit may be performed using, for example, a generation AI, or may be performed without using a generation AI. For example, the reception unit can select a reception method using a generation AI model that inputs the user's past instruction history and outputs the optimal reception method.

[0036] When receiving instructions, the reception unit can filter instructions based on the user's current situation and areas of interest. For example, if the user is currently driving, the reception unit allows the generation AI to only accept voice instructions. For example, the reception unit may detect the user's current situation using a camera or sensor and determine that the user is driving. Furthermore, if the user is interested in a specific app, the reception unit may preferentially accept instructions related to that app. For example, the reception unit may analyze the user's past operation history or social media activity to determine that the user is interested in a specific app. Furthermore, the reception unit may predict operations the user will perform during a specific time period and preferentially accept instructions related to that time period. For example, the reception unit may analyze the user's past operation history and predict operations the user will perform during a specific time period. This allows instructions to be filtered based on the user's current situation and areas of interest, thereby enabling more appropriate instructions to be accepted. Some or all of the above-described processing in the reception unit may be performed using, or without, the generation AI. For example, the reception unit can filter instructions using a generative AI model that takes the user's current situation and areas of interest as input and outputs filtered instructions.

[0037] When receiving instructions, the reception unit can prioritize receiving highly relevant instructions by taking into account the user's geographical location information. For example, when the user is at home, the reception unit causes the generation AI to prioritize receiving instructions related to home settings. For example, the reception unit acquires the user's geographical location information from GPS data, Wi-Fi location information, etc., and determines that the user is at home. Furthermore, when the user is at work, the reception unit can cause the generation AI to prioritize receiving work-related instructions. For example, the reception unit acquires the user's geographical location information and determines that the user is at work. Furthermore, when the user is traveling, the reception unit can cause the generation AI to prioritize receiving travel-related instructions. For example, the reception unit acquires the user's geographical location information and determines that the user is traveling. In this way, by taking the user's geographical location information into account, highly relevant instructions can be prioritized. Some or all of the above-described processing in the reception unit may be performed using, for example, the generation AI, or may be performed without using the generation AI. For example, the reception unit can receive instructions using a generation AI model that inputs the user's geographical location information and outputs highly relevant instructions.

[0038] When receiving an instruction, the reception unit can analyze the user's social media activity and receive related instructions. For example, if the user mentions a specific app on social media, the reception unit can preferentially receive instructions related to the app. For example, the reception unit can analyze the user's social media posts and determine that the specific app is mentioned. Furthermore, if the user is participating in a specific event on social media, the reception unit can preferentially receive instructions related to the event. For example, the reception unit can analyze the user's social media activity and determine that the user is participating in a specific event. Furthermore, if the user has checked in to a specific location on social media, the reception unit can preferentially receive instructions related to the location. For example, the reception unit can analyze the user's social media activity and determine that the user has checked in to a specific location. In this way, by analyzing the user's social media activity, related instructions can be preferentially received. Some or all of the above-described processing by the reception unit may be performed using, for example, a generative AI, or may be performed without using a generative AI. For example, the reception unit can receive instructions using a generative AI model that inputs the user's social media activity and outputs related instructions.

[0039] During analysis, the analysis unit can adjust the level of detail of the analysis based on the importance of the instruction. For example, the analysis unit may have the generation AI perform a detailed analysis of an instruction with high importance. For example, the analysis unit may analyze the content of a user's instruction and perform a detailed analysis if it determines that the instruction is of high importance. The analysis unit may also have the generation AI perform a concise analysis of an instruction with low importance. For example, the analysis unit may analyze the content of a user's instruction and perform a concise analysis if it determines that the instruction is of low importance. The analysis unit may also have the generation AI perform an analysis with an appropriate level of detail of an instruction with medium importance. For example, the analysis unit may analyze the content of a user's instruction and perform an analysis with an appropriate level of detail if it determines that the instruction is of medium importance. This allows for adjusting the level of detail of the analysis based on the importance of the instruction, thereby providing a more appropriate analysis result. Some or all of the above-described processing in the analysis unit may be performed using, for example, the generation AI, or may be performed without using the generation AI. For example, the analysis unit may adjust the level of detail of the analysis using a generation AI model that inputs the content of a user's instruction and outputs the level of detail of the analysis.

[0040] During analysis, the analysis unit can apply different analysis algorithms depending on the category of the instruction. For example, the generation AI applies a specific analysis algorithm to an instruction regarding system settings. For example, when the analysis unit receives an instruction regarding system settings, it applies a specific analysis algorithm. Furthermore, the generation AI can apply a different analysis algorithm to an instruction regarding app operation. For example, when the analysis unit receives an instruction regarding app operation, it applies a different analysis algorithm. Furthermore, the analysis unit can apply yet another analysis algorithm to an instruction regarding notification settings. For example, when the analysis unit receives an instruction regarding notification settings, it applies yet another analysis algorithm. In this way, by applying different analysis algorithms depending on the category of the instruction, more appropriate analysis results can be provided. Some or all of the above-mentioned processing in the analysis unit may be performed using, for example, the generation AI, or may be performed without using the generation AI. For example, the analysis unit can apply the analysis algorithm using a generation AI model that inputs the category of the instruction and outputs an analysis algorithm.

[0041] During analysis, the analysis unit can determine the priority of analysis based on the time when the instructions were submitted. The analysis unit, for example, prioritizes analysis of recently submitted instructions. For example, the analysis unit records the submission date and time of instructions and prioritizes analysis of recently submitted instructions. The analysis unit can also postpone instructions submitted earlier. For example, the analysis unit postpones instructions submitted earlier. The analysis unit can also moderately analyze instructions submitted at a medium time. For example, the analysis unit moderately analyzes instructions submitted at a medium time. This allows for determining the priority of analysis based on the time when the instructions were submitted, thereby providing more appropriate analysis results. Some or all of the above-described processing in the analysis unit may be performed using, for example, a generation AI, or may be performed without using a generation AI. For example, the analysis unit can determine the priority of analysis using a generation AI model that inputs the submission date and time of instructions and outputs the priority of analysis.

[0042] During analysis, the analysis unit can adjust the order of analysis based on the relevance of instructions. The analysis unit, for example, prioritizes analysis of highly relevant instructions. For example, the analysis unit analyzes the content of instructions and prioritizes analysis of instructions determined to be highly relevant. The analysis unit can also postpone instructions with low relevance. For example, the analysis unit analyzes the content of instructions and postpones instructions determined to be low relevance. The analysis unit can also moderately analyze instructions with medium relevance. For example, the analysis unit analyzes the content of instructions and moderately analyzes instructions determined to be medium relevance. This allows for adjusting the order of analysis based on the relevance of instructions, thereby providing more appropriate analysis results. Some or all of the above-described processing in the analysis unit may be performed using, for example, a generative AI, or may be performed without using a generative AI. For example, the analysis unit can adjust the order of analysis using a generative AI model that inputs the relevance of instructions and outputs the order of analysis.

[0043] During execution, the execution unit can analyze the user's past operation history and select an appropriate execution method. For example, the execution unit prioritizes execution of operation methods that the user frequently performed in the past. For example, the execution unit analyzes the user's past operation history and prioritizes execution of operation methods that the user frequently performed. The execution unit can also predict and suggest operation methods to be performed in a specific time period based on the user's past operation history. For example, the execution unit analyzes the user's past operation history and predicts operation methods to be performed in a specific time period. The execution unit can also analyze patterns of operations performed by the user in the past and select an optimal execution method. For example, the execution unit analyzes the user's past operation history, identifies operation patterns, and selects an optimal execution method. In this way, the optimal execution method can be selected by analyzing the user's past operation history. Some or all of the above-described processing in the execution unit may be performed using, for example, a generation AI, or may be performed without using a generation AI. For example, the execution unit can select an execution method using a generation AI model that inputs the user's past operation history and outputs an optimal execution method.

[0044] The execution unit can customize the means of execution based on the user's current situation during execution. For example, if the user is currently driving, the execution unit causes the generation AI to execute only voice instructions. For example, the execution unit detects the user's current situation using a camera or sensor and determines that the user is driving. Furthermore, if the user shows interest in a particular app, the execution unit can prioritize execution of operations related to that app. For example, the execution unit analyzes the user's past operation history and social media activity to determine that the user shows interest in a particular app. Furthermore, the execution unit can predict operations the user will perform during a particular time period and prioritize execution of operations related to that time period. For example, the execution unit analyzes the user's past operation history and predicts operations to be performed during a particular time period. This allows customizing the means of execution based on the user's current situation, thereby providing a more appropriate execution method. Some or all of the above-described processing in the execution unit may be performed using, for example, a generation AI. For example, the execution unit can customize the means of execution using a generation AI model that inputs the user's current situation and outputs a means of execution.

[0045] During execution, the execution unit can select an appropriate execution method by taking into account the user's geographical location information. For example, when the user is at home, the execution unit causes the generation AI to prioritize operations related to home settings. For example, the execution unit acquires the user's geographical location information from GPS data, Wi-Fi location information, etc., and determines that the user is at home. Furthermore, when the user is at work, the execution unit can cause the generation AI to prioritize work-related operations. For example, the execution unit acquires the user's geographical location information and determines that the user is at work. Furthermore, when the user is traveling, the generation AI can prioritize travel-related operations. For example, the execution unit acquires the user's geographical location information and determines that the user is traveling. This allows the optimal execution method to be selected by taking the user's geographical location information into account. Some or all of the above-described processing in the execution unit may be performed using, for example, the generation AI, or may be performed without using the generation AI. For example, the execution unit can select an execution method using a generation AI model that inputs the user's geographical location information and outputs an appropriate execution method.

[0046] During execution, the execution unit can analyze the user's social media activity and suggest a means of execution. For example, if the user mentions a specific app on social media, the execution unit can prioritize executing an operation related to the app. For example, the execution unit can analyze the user's social media posts and determine that the user mentions a specific app. Furthermore, if the user is participating in a specific event on social media, the execution unit can prioritize executing an operation related to the event. For example, the execution unit can analyze the user's social media activity and determine that the user is participating in a specific event. Furthermore, if the user checks in to a specific location on social media, the execution unit can prioritize executing an operation related to the location. For example, the execution unit can analyze the user's social media activity and determine that the user checks in to a specific location. Thus, by analyzing the user's social media activity, related operations can be prioritized. Some or all of the above-described processing in the execution unit can be performed, for example, using a generative AI, or can be performed without using a generative AI. For example, the execution unit can suggest a means of execution using a generative AI model that inputs the user's social media activity and outputs a means of execution.

[0047] The feedback unit can adjust the level of detail of the feedback based on the importance of the analysis result. For example, the feedback unit provides detailed feedback for an analysis result with high importance. For example, the feedback unit evaluates the importance of the analysis result and provides detailed feedback if it is determined to be high. The feedback unit can also provide brief feedback for an analysis result with low importance. For example, the feedback unit evaluates the importance of the analysis result and provides brief feedback if it is determined to be low. The feedback unit can also provide feedback with an appropriate level of detail for an analysis result with medium importance. For example, the feedback unit evaluates the importance of the analysis result and provides feedback with an appropriate level of detail if it is determined to be medium. In this way, by adjusting the level of detail of the feedback based on the importance of the analysis result, more appropriate feedback can be provided. Some or all of the above-described processing in the feedback unit may be performed, for example, using a generation AI, or may be performed without using a generation AI. For example, the feedback unit can adjust the level of detail of the feedback using a generation AI model that inputs the importance of the analysis result and outputs the level of detail of the feedback.

[0048] The feedback unit can determine the priority of feedback based on the submission time of the analysis results. For example, the feedback unit prioritizes providing feedback to recently submitted analysis results. For example, the feedback unit records the submission date and time of the analysis results and prioritizes providing feedback to recently submitted analysis results. The feedback unit can also provide feedback later to analysis results submitted earlier. For example, the feedback unit provides feedback later to analysis results submitted earlier. The feedback unit can also provide appropriate feedback to analysis results submitted later. For example, the feedback unit provides appropriate feedback to analysis results submitted later. In this way, by determining the priority of feedback based on the submission time of the analysis results, more appropriate feedback can be provided. Some or all of the above-described processing in the feedback unit may be performed using, for example, a generation AI, or may be performed without using a generation AI. For example, the feedback unit can determine the priority of feedback using a generation AI model that inputs the submission time of the analysis results and outputs a feedback priority.

[0049] When recording history, the history unit can select an optimal recording method by referring to past history data. For example, if a user has preferred detailed history in the past, the history unit records detailed history. For example, the history unit can refer to past history data and determine that the user has preferred detailed history. Furthermore, if a user has preferred concise history in the past, the history unit can record concise history. For example, the history unit can refer to past history data and determine that the user has preferred concise history. Furthermore, the history unit can select an optimal recording method from the user's past history data. For example, the history unit can analyze past history data and select an optimal recording method. In this way, the optimal recording method can be selected by referring to the past history data. Some or all of the above-described processing in the history unit may be performed, for example, using a generation AI, or may be performed without using a generation AI. For example, the history unit can select a recording method using a generation AI model that inputs past history data and outputs an optimal recording method.

[0050] When recording history, the history unit can select the optimal recording method by taking into account the user's geographical location information. For example, when the user is at home, the history unit prioritizes recording history related to home settings. For example, the history unit acquires the user's geographical location information from GPS data, Wi-Fi location information, etc., and determines that the user is at home. Furthermore, when the user is at work, the history unit can prioritize recording work-related history. For example, the history unit acquires the user's geographical location information and determines that the user is at work. Furthermore, when the user is traveling, the history unit can prioritize recording travel-related history. For example, the history unit acquires the user's geographical location information and determines that the user is traveling. In this way, the optimal recording method can be selected by taking the user's geographical location information into account. Some or all of the above-described processing in the history unit may be performed using, for example, a generation AI, or may be performed without using a generation AI. For example, the history unit can select the recording method using a generation AI model that inputs the user's geographical location information and outputs the optimal recording method.

[0051] The system according to the embodiment is not limited to the above-described example, and various modifications are possible, for example, as follows.

[0052] The reception unit can analyze the user's past instruction history and select the optimal reception method. For example, it prioritizes reception of instruction methods (such as voice and text) that the user has frequently used in the past. For example, the reception unit analyzes the user's past instruction history and, if the user has frequently used voice instructions, prioritizes reception of voice instructions. The reception unit can also predict and suggest an instruction method to be used in a specific time period based on the user's past instruction history. For example, if the user frequently uses text instructions in the morning, the reception unit prioritizes reception of text instructions in the morning. The reception unit can also analyze patterns of instructions given by the user in the past and select the optimal reception method. For example, if the user uses a specific instruction method when performing a specific operation, the reception unit prioritizes reception of that instruction method. In this way, the optimal reception method can be selected by analyzing the user's past instruction history. Some or all of the above-described processing in the reception unit may be performed using, for example, a generation AI, or may be performed without using a generation AI. For example, the reception unit can select a reception method using a generation AI model that inputs the user's past instruction history and outputs the optimal reception method.

[0053] During analysis, the analysis unit can adjust the level of detail of the analysis based on the importance of the instruction. For example, the generation AI performs a detailed analysis for an instruction with high importance. For example, the analysis unit analyzes the content of the user's instruction and performs a detailed analysis if it determines that the instruction is highly important. The analysis unit can also perform a concise analysis for an instruction with low importance. For example, the analysis unit analyzes the content of the user's instruction and performs a concise analysis if it determines that the instruction is low important. The analysis unit can also perform an analysis with an appropriate level of detail for an instruction with medium importance. For example, the analysis unit analyzes the content of the user's instruction and performs an analysis with an appropriate level of detail if it determines that the instruction is medium important. This allows for adjusting the level of detail of the analysis based on the importance of the instruction, thereby providing a more appropriate analysis result. Some or all of the above-described processing in the analysis unit may be performed using, for example, the generation AI, or may be performed without using the generation AI. For example, the analysis unit can adjust the level of detail of the analysis using a generation AI model that inputs the content of the user's instruction and outputs the level of detail of the analysis.

[0054] During execution, the execution unit can analyze the user's past operation history and select an appropriate execution method. For example, the execution unit can prioritize execution of operation methods that the user frequently used in the past. For example, the execution unit can analyze the user's past operation history and prioritize execution of operation methods that the user frequently used. The execution unit can also predict and suggest operation methods to be performed in a specific time period based on the user's past operation history. For example, the execution unit can analyze the user's past operation history and predict operation methods to be performed in a specific time period. The execution unit can also analyze patterns of operations performed by the user in the past and select an optimal execution method. For example, the execution unit can analyze the user's past operation history, identify operation patterns, and select an optimal execution method. In this way, the optimal execution method can be selected by analyzing the user's past operation history. Some or all of the above-described processing in the execution unit can be performed using, for example, a generation AI, or can be performed without using a generation AI. For example, the execution unit can select an execution method using a generation AI model that inputs the user's past operation history and outputs an optimal execution method.

[0055] The feedback unit can adjust the level of detail of the feedback based on the importance of the analysis result. For example, detailed feedback is provided for an analysis result with high importance. For example, the feedback unit evaluates the importance of the analysis result and provides detailed feedback if it is determined to be high. The feedback unit can also provide brief feedback for an analysis result with low importance. For example, the feedback unit evaluates the importance of the analysis result and provides brief feedback if it is determined to be low. The feedback unit can also provide feedback with an appropriate level of detail for an analysis result with medium importance. For example, the feedback unit evaluates the importance of the analysis result and provides feedback with an appropriate level of detail if it is determined to be medium. In this way, by adjusting the level of detail of the feedback based on the importance of the analysis result, more appropriate feedback can be provided. Some or all of the above-described processing in the feedback unit may be performed, for example, using a generation AI, or may be performed without using a generation AI. For example, the feedback unit can adjust the level of detail of the feedback using a generation AI model that inputs the importance of the analysis result and outputs the level of detail of the feedback.

[0056] When recording history, the history unit can select the optimal recording method by referring to past history data. For example, if a user has preferred detailed history in the past, the history unit records detailed history. For example, the history unit can refer to past history data and determine that the user has preferred detailed history. Furthermore, if a user has preferred concise history in the past, the history unit can also record concise history. For example, the history unit can refer to past history data and determine that the user has preferred concise history. Furthermore, the history unit can select the optimal recording method from the user's past history data. For example, the history unit can analyze past history data and select the optimal recording method. In this way, the optimal recording method can be selected by referring to the past history data. Some or all of the above-described processing in the history unit may be performed, for example, using a generation AI, or may be performed without using a generation AI. For example, the history unit can select the recording method using a generation AI model that inputs past history data and outputs the optimal recording method.

[0057] The processing flow of the first embodiment will be briefly explained below.

[0058] Step 1: The reception unit receives instructions in natural language from the user. The natural language instructions from the user include voice instructions and text instructions. The reception unit can convert the voice instructions into text data using voice recognition technology, and the user can also directly input text instructions using a text input interface. Step 2: The analysis unit analyzes the natural language instructions received by the reception unit and identifies appropriate settings and operations. The analysis unit analyzes the instructions using natural language processing techniques such as morphological analysis, grammatical analysis, and semantic analysis, and analyzes the user's instructions using a generative AI (e.g., GPT-4). This allows the system to understand the intent of the user's instructions and identify appropriate settings and operations. Step 3: The execution unit executes the settings and operations identified by the analysis unit. The execution unit can access the phone's settings menu and execute operations such as turning on Wi-Fi or maximizing screen brightness. The processing in the execution unit can be performed with or without the aid of the generation AI.

[0059] (Example 2) A mobile phone setting support system according to an embodiment of the present invention utilizes a generation AI to simplify the setup and operation of a mobile phone. This mobile phone setting support system allows a user to input setup and operation instructions in natural language, and the generation AI analyzes the instructions and executes the appropriate setup and operation. For example, when a user inputs instructions such as "Turn on Wi-Fi" or "Maximize screen brightness," the generation AI analyzes the instructions and executes the operation of turning on Wi-Fi or maximizing screen brightness. In this way, users can easily set up and operate their mobile phones without performing complex operations. For example, even non-technical users or elderly users can set up and operate their mobile phones simply by inputting instructions in natural language. Furthermore, even busy users can complete the setup and operation in a short amount of time. This allows the mobile phone setting support system to easily allow users to set up and operate their mobile phones simply by inputting instructions in natural language.

[0060] A mobile phone setting assistance system according to an embodiment includes a reception unit, an analysis unit, and an execution unit. The reception unit receives natural language instructions from a user. Natural language instructions from a user include, but are not limited to, voice instructions and text instructions. The reception unit can convert voice instructions into text data using, for example, voice recognition technology. The reception unit can also allow a user to directly input text instructions using a text input interface. The analysis unit uses a generation AI to analyze the natural language instructions received by the reception unit and identify appropriate settings and operations. The analysis unit analyzes the instructions using natural language processing technology such as morphological analysis, grammatical analysis, and semantic analysis. For example, the generation AI can analyze the user's instructions using a text generation AI (e.g., GPT-4) to identify appropriate settings and operations. The analysis unit can also use the generation AI to understand the intent of the user's instructions and identify appropriate settings and operations. For example, the generation AI can analyze the instruction "Turn on Wi-Fi" and identify the operation to turn on Wi-Fi. The execution unit executes the settings and operations identified by the analysis unit. For example, the execution unit can access the settings menu of the mobile phone and execute an operation to turn on Wi-Fi. The execution unit can also execute an operation to maximize screen brightness. For example, the execution unit accesses the settings menu of the mobile phone and executes an operation to maximize screen brightness. This allows the mobile phone setting support system according to the embodiment to easily configure and operate the mobile phone simply by the user inputting instructions in natural language. Some or all of the above-described processing in the execution unit may be performed using, for example, a generation AI, or may be performed without using a generation AI. For example, the execution unit can execute the settings and operations using a generation AI model that receives the settings and operations identified by the analysis unit as input and outputs the settings and operations.

[0061] The analysis unit can analyze natural language instructions using a generation AI and identify appropriate settings and operations. The generation AI can analyze user instructions and identify appropriate settings and operations using, for example, a text generation AI (e.g., GPT-4). For example, the generation AI can analyze the instruction "Turn on Wi-Fi" and identify the operation to turn on Wi-Fi. The generation AI can also understand the intent of the user's instructions and identify appropriate settings and operations. For example, the generation AI can analyze the instruction "Maximize screen brightness" and identify the operation to maximize screen brightness. In this way, the generation AI can accurately analyze natural language instructions and identify appropriate settings and operations. Some or all of the above-described processing in the analysis unit can be performed using, for example, a generation AI, or can be performed without using a generation AI. For example, the analysis unit can analyze instructions using a generation AI model that receives user instructions as input and outputs appropriate settings and operations.

[0062] The execution unit can execute an operation to turn Wi-Fi on or off or adjust screen brightness. For example, the execution unit accesses a settings menu of a mobile phone and executes an operation to turn Wi-Fi on. For example, the execution unit turns on the Wi-Fi switch from the settings menu. The execution unit can also execute an operation to turn Wi-Fi off. For example, the execution unit turns off the Wi-Fi switch from the settings menu. The execution unit can also execute an operation to maximize screen brightness. For example, the execution unit moves the screen brightness slider to the maximum from the settings menu. The execution unit can also execute an operation to minimize screen brightness. For example, the execution unit moves the screen brightness slider to the minimum from the settings menu. This allows a user to easily turn Wi-Fi on or off or adjust screen brightness. Some or all of the above-described processing in the execution unit may be performed using, for example, a generative AI, or may be performed without using a generative AI. For example, the execution unit can execute settings and operations using a generative AI model that receives settings and operations identified by the analysis unit as input and outputs settings and operations.

[0063] The execution unit can perform operations to install or uninstall an app and set notifications. For example, the execution unit accesses an app store and performs an operation to install an app. For example, the execution unit searches for a specific app from the app store and clicks an install button. The execution unit can also perform an operation to uninstall an app. For example, the execution unit displays a list of apps from a settings menu and clicks an uninstall button. The execution unit can also perform an operation to set notifications. For example, the execution unit switches notifications on and off from the settings menu. The execution unit can also perform an operation to set notification sounds. For example, the execution unit selects a notification sound from the settings menu. This allows a user to easily install or uninstall an app and set notifications. Some or all of the above-described processing in the execution unit may be performed using, for example, a generation AI, or may be performed without using a generation AI. For example, the execution unit can perform settings and operations using a generation AI model that receives settings and operations identified by the analysis unit as input and outputs settings and operations.

[0064] The analysis unit may include a feedback unit that analyzes a user's instruction and feeds back the analysis result to the user. The feedback unit provides the analysis result to the user in text format, for example. For example, the feedback unit displays the analysis result as a text message. The feedback unit may also provide the analysis result to the user in audio format. For example, the feedback unit plays back the analysis result as an audio message. The feedback unit may also provide the analysis result in a visually easy-to-understand format. For example, the feedback unit displays the analysis result as a graph or chart. This allows the user to receive the analysis result as feedback. Some or all of the above-described processing in the feedback unit may be performed using, for example, a generation AI, or may be performed without using a generation AI. For example, the feedback unit may provide feedback using a generation AI model that receives the analysis result identified by the analysis unit as input and outputs feedback.

[0065] The execution unit may include a history unit that records the operation history. The history unit records the operation history in the form of, for example, date and time, operation content, operation result, etc. For example, the history unit records the date and time when the operation was performed. The history unit can also record the operation content. For example, the history unit records operation content such as turning Wi-Fi on and off or adjusting the screen brightness. The history unit can also record the operation result. For example, the history unit records whether the operation was successful. This allows the user to check the past operation history. Some or all of the above-mentioned processing in the history unit may be performed using, for example, a generation AI, or may be performed without using a generation AI. For example, the history unit can record the operation history using a generation AI model that inputs the operation content identified by the execution unit and outputs the operation history.

[0066] The reception unit can estimate the user's emotions and adjust the timing of instruction acceptance based on the estimated user emotions. For example, if the user is feeling stressed, the generation AI can delay accepting instructions and wait until the user is relaxed. For example, the reception unit can capture the user's facial expression with a camera and estimate the emotion using an emotion estimation algorithm. Furthermore, if the user is relaxed, the reception unit can have the generation AI immediately accept instructions and quickly start processing. For example, the reception unit can record the user's voice and estimate the emotion using voice analysis technology. Furthermore, if the user is in a hurry, the reception unit can have the generation AI prioritize instruction acceptance and quickly process it. For example, the reception unit can collect the user's biometric data (heart rate and electrodermal activity) with a sensor and estimate the emotion using an emotion estimation algorithm. This allows the timing of instruction acceptance to be adjusted according to the user's emotions, thereby allowing instructions to be accepted at a more appropriate time. Emotion estimation is realized using an emotion estimation function, for example, an emotion engine or a generation AI. The generation AI may be, but is not limited to, a text generation AI (e.g., GPT-4) or a multimodal generation AI. Some or all of the above-described processing in the reception unit may be performed using, for example, the generation AI. For example, the reception unit may adjust the timing of receiving instructions using a generation AI model that receives user emotion data as input and outputs the timing of receiving instructions.

[0067] The reception unit can analyze the user's past instruction history and select the optimal reception method. The reception unit, for example, prioritizes reception of instruction methods (such as voice and text) that the user has frequently used in the past. For example, the reception unit analyzes the user's past instruction history and, if the user frequently uses voice instructions, prioritizes reception of voice instructions. The reception unit can also predict and suggest an instruction method to be used in a specific time period based on the user's past instruction history. For example, if the user frequently uses text instructions in the morning, the reception unit prioritizes reception of text instructions in the morning. The reception unit can also analyze patterns of instructions given by the user in the past and select the optimal reception method. For example, if the user uses a specific instruction method when performing a specific operation, the reception unit prioritizes reception of that instruction method. In this way, the optimal reception method can be selected by analyzing the user's past instruction history. Some or all of the above-described processing in the reception unit may be performed using, for example, a generation AI, or may be performed without using a generation AI. For example, the reception unit can select a reception method using a generation AI model that inputs the user's past instruction history and outputs the optimal reception method.

[0068] When receiving instructions, the reception unit can filter instructions based on the user's current situation and areas of interest. For example, if the user is currently driving, the reception unit allows the generation AI to only accept voice instructions. For example, the reception unit may detect the user's current situation using a camera or sensor and determine that the user is driving. Furthermore, if the user is interested in a specific app, the reception unit may preferentially accept instructions related to that app. For example, the reception unit may analyze the user's past operation history or social media activity to determine that the user is interested in a specific app. Furthermore, the reception unit may predict operations the user will perform during a specific time period and preferentially accept instructions related to that time period. For example, the reception unit may analyze the user's past operation history and predict operations the user will perform during a specific time period. This allows instructions to be filtered based on the user's current situation and areas of interest, thereby enabling more appropriate instructions to be accepted. Some or all of the above-described processing in the reception unit may be performed using, or without, the generation AI. For example, the reception unit can filter instructions using a generative AI model that takes the user's current situation and areas of interest as input and outputs filtered instructions.

[0069] The reception unit can estimate the user's emotions and determine the priority of instructions to be received based on the estimated user emotions. For example, if the user is feeling stressed, the reception unit allows the generation AI to postpone instructions of lower importance. For example, the reception unit can capture the user's facial expression with a camera and estimate the emotion using an emotion estimation algorithm. Alternatively, if the user is relaxed, the reception unit can allow the generation AI to process all instructions equally. For example, the reception unit can record the user's voice and estimate the emotion using voice analysis technology. Alternatively, if the user is in a hurry, the generation AI can prioritize instructions of higher importance. For example, the reception unit can collect the user's biometric data (heart rate and electrodermal activity) with a sensor and estimate the emotion using an emotion estimation algorithm. This allows the priority of instructions to be determined according to the user's emotions, allowing more appropriate instructions to be processed preferentially. Emotion estimation is achieved using an emotion estimation function, for example, an emotion engine or a generation AI. The generation AI may be, but is not limited to, a text generation AI (e.g., GPT-4) or a multimodal generation AI. Some or all of the above-described processing in the reception unit may be performed using, for example, the generation AI, or may be performed without using the generation AI. For example, the reception unit may determine the priority of instructions using a generation AI model that receives user emotion data as input and outputs the priority of instructions.

[0070] When receiving instructions, the reception unit can prioritize receiving highly relevant instructions by taking into account the user's geographical location information. For example, when the user is at home, the reception unit causes the generation AI to prioritize receiving instructions related to home settings. For example, the reception unit acquires the user's geographical location information from GPS data, Wi-Fi location information, etc., and determines that the user is at home. Furthermore, when the user is at work, the reception unit can cause the generation AI to prioritize receiving work-related instructions. For example, the reception unit acquires the user's geographical location information and determines that the user is at work. Furthermore, when the user is traveling, the reception unit can cause the generation AI to prioritize receiving travel-related instructions. For example, the reception unit acquires the user's geographical location information and determines that the user is traveling. In this way, by taking the user's geographical location information into account, highly relevant instructions can be prioritized. Some or all of the above-described processing in the reception unit may be performed using, for example, the generation AI, or may be performed without using the generation AI. For example, the reception unit can receive instructions using a generation AI model that inputs the user's geographical location information and outputs highly relevant instructions.

[0071] When receiving an instruction, the reception unit can analyze the user's social media activity and receive related instructions. For example, if the user mentions a specific app on social media, the reception unit can preferentially receive instructions related to the app. For example, the reception unit can analyze the user's social media posts and determine that the specific app is mentioned. Furthermore, if the user is participating in a specific event on social media, the reception unit can preferentially receive instructions related to the event. For example, the reception unit can analyze the user's social media activity and determine that the user is participating in a specific event. Furthermore, if the user has checked in to a specific location on social media, the reception unit can preferentially receive instructions related to the location. For example, the reception unit can analyze the user's social media activity and determine that the user has checked in to a specific location. In this way, by analyzing the user's social media activity, related instructions can be preferentially received. Some or all of the above-described processing by the reception unit may be performed using, for example, a generative AI, or may be performed without using a generative AI. For example, the reception unit can receive instructions using a generative AI model that inputs the user's social media activity and outputs related instructions.

[0072] The analysis unit can estimate the user's emotions and adjust the presentation method of the analysis based on the estimated user emotions. For example, if the user is relaxed, the generation AI can provide detailed analysis results. For example, the analysis unit can capture the user's facial expressions with a camera and estimate the emotions using an emotion estimation algorithm. Alternatively, if the user is in a hurry, the generation AI can provide concise analysis results. For example, the analysis unit can record the user's voice and estimate the emotions using voice analysis technology. Alternatively, if the user is feeling stressed, the generation AI can provide visually easy-to-understand analysis results. For example, the analysis unit can collect the user's biometric data (heart rate and electrodermal activity) using a sensor and estimate the emotions using an emotion estimation algorithm. This allows the system to adjust the presentation method of the analysis according to the user's emotions, thereby providing more appropriate analysis results. Emotion estimation is achieved using an emotion estimation function, such as an emotion engine or a generation AI. The generation AI can be, but is not limited to, a text generation AI (e.g., GPT-4) or a multimodal generation AI. Some or all of the above-described processing in the analysis unit may be performed using, for example, a generative AI, or may be performed without using a generative AI. For example, the analysis unit may adjust the expression method of the analysis using a generative AI model that receives user emotion data as input and outputs the expression method of the analysis.

[0073] During analysis, the analysis unit can adjust the level of detail of the analysis based on the importance of the instruction. For example, the analysis unit may have the generation AI perform a detailed analysis of an instruction with high importance. For example, the analysis unit may analyze the content of a user's instruction and perform a detailed analysis if it determines that the instruction is of high importance. The analysis unit may also have the generation AI perform a concise analysis of an instruction with low importance. For example, the analysis unit may analyze the content of a user's instruction and perform a concise analysis if it determines that the instruction is of low importance. The analysis unit may also have the generation AI perform an analysis with an appropriate level of detail of an instruction with medium importance. For example, the analysis unit may analyze the content of a user's instruction and perform an analysis with an appropriate level of detail if it determines that the instruction is of medium importance. This allows for adjusting the level of detail of the analysis based on the importance of the instruction, thereby providing a more appropriate analysis result. Some or all of the above-described processing in the analysis unit may be performed using, for example, the generation AI, or may be performed without using the generation AI. For example, the analysis unit may adjust the level of detail of the analysis using a generation AI model that inputs the content of a user's instruction and outputs the level of detail of the analysis.

[0074] During analysis, the analysis unit can apply different analysis algorithms depending on the category of the instruction. For example, the generation AI applies a specific analysis algorithm to an instruction regarding system settings. For example, when the analysis unit receives an instruction regarding system settings, it applies a specific analysis algorithm. Furthermore, the generation AI can apply a different analysis algorithm to an instruction regarding app operation. For example, when the analysis unit receives an instruction regarding app operation, it applies a different analysis algorithm. Furthermore, the analysis unit can apply yet another analysis algorithm to an instruction regarding notification settings. For example, when the analysis unit receives an instruction regarding notification settings, it applies yet another analysis algorithm. In this way, by applying different analysis algorithms depending on the category of the instruction, more appropriate analysis results can be provided. Some or all of the above-mentioned processing in the analysis unit may be performed using, for example, the generation AI, or may be performed without using the generation AI. For example, the analysis unit can apply the analysis algorithm using a generation AI model that inputs the category of the instruction and outputs an analysis algorithm.

[0075] The analysis unit can estimate the user's emotions and adjust the length of the analysis based on the estimated user emotions. For example, if the user is in a hurry, the analysis unit allows the generation AI to perform a short, concise analysis. For example, the analysis unit captures the user's facial expressions with a camera and estimates their emotions using an emotion estimation algorithm. Furthermore, if the user is relaxed, the analysis unit can allow the generation AI to perform a detailed analysis. For example, the analysis unit records the user's voice and estimates their emotions using voice analysis technology. Furthermore, if the user is feeling stressed, the generation AI can perform a visually easy-to-understand analysis. For example, the analysis unit collects the user's biometric data (heart rate and electrodermal activity) using a sensor and estimates their emotions using an emotion estimation algorithm. This allows the length of the analysis to be adjusted according to the user's emotions, thereby providing more appropriate analysis results. Emotion estimation is achieved using an emotion estimation function, for example, an emotion engine or a generation AI. The generation AI can be, but is not limited to, a text generation AI (e.g., GPT-4) or a multimodal generation AI. Some or all of the above-described processing in the analysis unit may be performed using, for example, a generation AI, or may be performed without using a generation AI. For example, the analysis unit can adjust the length of the analysis using a generation AI model that inputs user emotion data and outputs the length of the analysis.

[0076] During analysis, the analysis unit can determine the priority of analysis based on the time when the instructions were submitted. The analysis unit, for example, prioritizes analysis of recently submitted instructions. For example, the analysis unit records the submission date and time of instructions and prioritizes analysis of recently submitted instructions. The analysis unit can also postpone instructions submitted earlier. For example, the analysis unit postpones instructions submitted earlier. The analysis unit can also moderately analyze instructions submitted at a medium time. For example, the analysis unit moderately analyzes instructions submitted at a medium time. This allows for determining the priority of analysis based on the time when the instructions were submitted, thereby providing more appropriate analysis results. Some or all of the above-described processing in the analysis unit may be performed using, for example, a generation AI, or may be performed without using a generation AI. For example, the analysis unit can determine the priority of analysis using a generation AI model that inputs the submission date and time of instructions and outputs the priority of analysis.

[0077] During analysis, the analysis unit can adjust the order of analysis based on the relevance of instructions. The analysis unit, for example, prioritizes analysis of highly relevant instructions. For example, the analysis unit analyzes the content of instructions and prioritizes analysis of instructions determined to be highly relevant. The analysis unit can also postpone instructions with low relevance. For example, the analysis unit analyzes the content of instructions and postpones instructions determined to be low relevance. The analysis unit can also moderately analyze instructions with medium relevance. For example, the analysis unit analyzes the content of instructions and moderately analyzes instructions determined to be medium relevance. This allows for adjusting the order of analysis based on the relevance of instructions, thereby providing more appropriate analysis results. Some or all of the above-described processing in the analysis unit may be performed using, for example, a generative AI, or may be performed without using a generative AI. For example, the analysis unit can adjust the order of analysis using a generative AI model that inputs the relevance of instructions and outputs the order of analysis.

[0078] The execution unit can estimate the user's emotions and adjust the execution method based on the estimated user emotions. For example, if the user is relaxed, the execution unit executes the execution while the generation AI provides detailed instructions. For example, the execution unit captures the user's facial expressions with a camera and estimates the user's emotions using an emotion estimation algorithm. Furthermore, if the user is in a hurry, the execution unit can execute the execution while the generation AI provides quick instructions. For example, the execution unit records the user's voice and estimates the user's emotions using voice analysis technology. Furthermore, if the user is feeling stressed, the execution unit can execute the execution while the generation AI provides easy-to-understand visual instructions. For example, the execution unit collects the user's biometric data (heart rate and electrodermal activity) with a sensor and estimates the user's emotions using an emotion estimation algorithm. This allows the execution method to be adjusted according to the user's emotions, thereby providing a more appropriate execution method. Emotion estimation is realized using an emotion estimation function, for example, using an emotion engine or generation AI. The generation AI can be, but is not limited to, a text generation AI (e.g., GPT-4) or a multimodal generation AI. Some or all of the above-described processing in the execution unit may be performed using, for example, a generative AI, or may be performed without using a generative AI. For example, the execution unit may adjust the execution method using a generative AI model that receives user emotion data as input and outputs an execution method.

[0079] During execution, the execution unit can analyze the user's past operation history and select an appropriate execution method. For example, the execution unit prioritizes execution of operation methods that the user frequently performed in the past. For example, the execution unit analyzes the user's past operation history and prioritizes execution of operation methods that the user frequently performed. The execution unit can also predict and suggest operation methods to be performed in a specific time period based on the user's past operation history. For example, the execution unit analyzes the user's past operation history and predicts operation methods to be performed in a specific time period. The execution unit can also analyze patterns of operations performed by the user in the past and select an optimal execution method. For example, the execution unit analyzes the user's past operation history, identifies operation patterns, and selects an optimal execution method. In this way, the optimal execution method can be selected by analyzing the user's past operation history. Some or all of the above-described processing in the execution unit may be performed using, for example, a generation AI, or may be performed without using a generation AI. For example, the execution unit can select an execution method using a generation AI model that inputs the user's past operation history and outputs an optimal execution method.

[0080] The execution unit can customize the means of execution based on the user's current situation during execution. For example, if the user is currently driving, the execution unit causes the generation AI to execute only voice instructions. For example, the execution unit detects the user's current situation using a camera or sensor and determines that the user is driving. Furthermore, if the user shows interest in a particular app, the execution unit can prioritize execution of operations related to that app. For example, the execution unit analyzes the user's past operation history and social media activity to determine that the user shows interest in a particular app. Furthermore, the execution unit can predict operations the user will perform during a particular time period and prioritize execution of operations related to that time period. For example, the execution unit analyzes the user's past operation history and predicts operations to be performed during a particular time period. This allows customizing the means of execution based on the user's current situation, thereby providing a more appropriate execution method. Some or all of the above-described processing in the execution unit may be performed using, for example, a generation AI. For example, the execution unit can customize the means of execution using a generation AI model that inputs the user's current situation and outputs a means of execution.

[0081] The execution unit can estimate the user's emotions and determine the priority of execution based on the estimated user emotions. For example, if the user is feeling stressed, the generation AI postpones less important operations. For example, the execution unit captures the user's facial expression with a camera and estimates their emotions using an emotion estimation algorithm. Alternatively, if the user is relaxed, the generation AI can execute all operations equally. For example, the execution unit records the user's voice and estimates their emotions using voice analysis technology. Alternatively, if the user is in a hurry, the generation AI can prioritize more important operations. For example, the execution unit collects the user's biometric data (heart rate and electrodermal activity) using a sensor and estimates their emotions using an emotion estimation algorithm. This allows the execution priority to be determined according to the user's emotions, allowing more appropriate operations to be executed preferentially. Emotion estimation is achieved using an emotion estimation function, for example, an emotion engine or a generation AI. The generative AI may be, but is not limited to, a text generative AI (e.g., GPT-4) or a multimodal generative AI. Some or all of the above-described processing in the execution unit may be performed using, for example, the generative AI, or may be performed without using the generative AI. For example, the execution unit may determine the priority of execution using a generative AI model that receives user emotion data as input and outputs the priority of execution.

[0082] During execution, the execution unit can select an appropriate execution method by taking into account the user's geographical location information. For example, when the user is at home, the execution unit causes the generation AI to prioritize operations related to home settings. For example, the execution unit acquires the user's geographical location information from GPS data, Wi-Fi location information, etc., and determines that the user is at home. Furthermore, when the user is at work, the execution unit can cause the generation AI to prioritize work-related operations. For example, the execution unit acquires the user's geographical location information and determines that the user is at work. Furthermore, when the user is traveling, the generation AI can prioritize travel-related operations. For example, the execution unit acquires the user's geographical location information and determines that the user is traveling. This allows the optimal execution method to be selected by taking the user's geographical location information into account. Some or all of the above-described processing in the execution unit may be performed using, for example, the generation AI, or may be performed without using the generation AI. For example, the execution unit can select an execution method using a generation AI model that inputs the user's geographical location information and outputs an appropriate execution method.

[0083] During execution, the execution unit can analyze the user's social media activity and suggest a means of execution. For example, if the user mentions a specific app on social media, the execution unit can prioritize executing an operation related to the app. For example, the execution unit can analyze the user's social media posts and determine that the user mentions a specific app. Furthermore, if the user is participating in a specific event on social media, the execution unit can prioritize executing an operation related to the event. For example, the execution unit can analyze the user's social media activity and determine that the user is participating in a specific event. Furthermore, if the user checks in to a specific location on social media, the execution unit can prioritize executing an operation related to the location. For example, the execution unit can analyze the user's social media activity and determine that the user checks in to a specific location. Thus, by analyzing the user's social media activity, related operations can be prioritized. Some or all of the above-described processing in the execution unit can be performed, for example, using a generative AI, or can be performed without using a generative AI. For example, the execution unit can suggest a means of execution using a generative AI model that inputs the user's social media activity and outputs a means of execution.

[0084] The feedback unit can estimate the user's emotions and adjust the way the feedback is expressed based on the estimated user emotions. For example, the feedback unit provides detailed feedback when the user is relaxed. For example, the feedback unit captures the user's facial expressions with a camera and estimates the user's emotions using an emotion estimation algorithm. The feedback unit can also provide concise feedback when the user is in a hurry. For example, the feedback unit records the user's voice and estimates the user's emotions using voice analysis technology. The feedback unit can also provide visually easy-to-understand feedback when the user is stressed. For example, the feedback unit collects the user's biometric data (heart rate and electrodermal activity) using a sensor and estimates the user's emotions using an emotion estimation algorithm. This allows the system to adjust the way the feedback is expressed based on the user's emotions, thereby providing more appropriate feedback. The emotion estimation is achieved using an emotion estimation function, such as an emotion engine or a generative AI. The generative AI can be, but is not limited to, a text generation AI (e.g., GPT-4) or a multimodal generation AI. Some or all of the above-described processing in the feedback unit may be performed using, for example, a generation AI, or may be performed without using a generation AI. For example, the feedback unit may adjust the feedback expression method using a generation AI model that receives user emotion data as input and outputs a feedback expression method.

[0085] The feedback unit can adjust the level of detail of the feedback based on the importance of the analysis result. For example, the feedback unit provides detailed feedback for an analysis result with high importance. For example, the feedback unit evaluates the importance of the analysis result and provides detailed feedback if it is determined to be high. The feedback unit can also provide brief feedback for an analysis result with low importance. For example, the feedback unit evaluates the importance of the analysis result and provides brief feedback if it is determined to be low. The feedback unit can also provide feedback with an appropriate level of detail for an analysis result with medium importance. For example, the feedback unit evaluates the importance of the analysis result and provides feedback with an appropriate level of detail if it is determined to be medium. In this way, by adjusting the level of detail of the feedback based on the importance of the analysis result, more appropriate feedback can be provided. Some or all of the above-described processing in the feedback unit may be performed, for example, using a generation AI, or may be performed without using a generation AI. For example, the feedback unit can adjust the level of detail of the feedback using a generation AI model that inputs the importance of the analysis result and outputs the level of detail of the feedback.

[0086] The feedback unit can estimate the user's emotions and adjust the length of the feedback based on the estimated user emotions. For example, if the user is in a hurry, the feedback unit provides short, to-the-point feedback. For example, the feedback unit captures the user's facial expressions with a camera and estimates the user's emotions using an emotion estimation algorithm. The feedback unit can also provide detailed feedback if the user is relaxed. For example, the feedback unit records the user's voice and estimates the user's emotions using voice analysis technology. The feedback unit can also provide visually easy-to-understand feedback if the user is feeling stressed. For example, the feedback unit collects the user's biometric data (heart rate and electrodermal activity) with a sensor and estimates the user's emotions using an emotion estimation algorithm. This allows the length of the feedback to be adjusted according to the user's emotions, thereby providing more appropriate feedback. The emotion estimation is achieved using an emotion estimation function, for example, an emotion engine or a generative AI. The generative AI can be, but is not limited to, a text generation AI (e.g., GPT-4) or a multimodal generation AI. Some or all of the above-described processing in the feedback unit may be performed using, for example, a generation AI, or may be performed without using a generation AI. For example, the feedback unit may adjust the length of the feedback using a generation AI model that receives user emotion data as input and outputs the length of the feedback.

[0087] The feedback unit can determine the priority of feedback based on the submission time of the analysis results. For example, the feedback unit prioritizes providing feedback to recently submitted analysis results. For example, the feedback unit records the submission date and time of the analysis results and prioritizes providing feedback to recently submitted analysis results. The feedback unit can also provide feedback later to analysis results submitted earlier. For example, the feedback unit provides feedback later to analysis results submitted earlier. The feedback unit can also provide appropriate feedback to analysis results submitted later. For example, the feedback unit provides appropriate feedback to analysis results submitted later. In this way, by determining the priority of feedback based on the submission time of the analysis results, more appropriate feedback can be provided. Some or all of the above-described processing in the feedback unit may be performed using, for example, a generation AI, or may be performed without using a generation AI. For example, the feedback unit can determine the priority of feedback using a generation AI model that inputs the submission time of the analysis results and outputs a feedback priority.

[0088] The history unit can estimate the user's emotions and adjust the history recording method based on the estimated user emotions. For example, the history unit records detailed history when the user is relaxed. For example, the history unit captures the user's facial expressions with a camera and estimates the user's emotions using an emotion estimation algorithm. The history unit can also record concise history when the user is in a hurry. For example, the history unit records the user's voice and estimates the user's emotions using voice analysis technology. The history unit can also record visually easy-to-understand history when the user is stressed. For example, the history unit collects the user's biometric data (heart rate and electrodermal activity) with a sensor and estimates the user's emotions using an emotion estimation algorithm. This allows for more appropriate history recording by adjusting the history recording method according to the user's emotions. Emotion estimation is achieved using an emotion estimation function, such as an emotion engine or generative AI. Generative AI can be, but is not limited to, text generation AI (e.g., GPT-4) or multimodal generation AI. Some or all of the above-described processing in the history unit may be performed using, for example, a generative AI, or may be performed without using a generative AI. For example, the history unit may adjust the history recording method using a generative AI model that receives user emotion data as input and outputs a history recording method.

[0089] When recording history, the history unit can select an optimal recording method by referring to past history data. For example, if a user has preferred detailed history in the past, the history unit records detailed history. For example, the history unit can refer to past history data and determine that the user has preferred detailed history. Furthermore, if a user has preferred concise history in the past, the history unit can record concise history. For example, the history unit can refer to past history data and determine that the user has preferred concise history. Furthermore, the history unit can select an optimal recording method from the user's past history data. For example, the history unit can analyze past history data and select an optimal recording method. In this way, the optimal recording method can be selected by referring to the past history data. Some or all of the above-described processing in the history unit may be performed, for example, using a generation AI, or may be performed without using a generation AI. For example, the history unit can select a recording method using a generation AI model that inputs past history data and outputs an optimal recording method.

[0090] The history unit can estimate the user's emotions and prioritize history entries based on the estimated user emotions. For example, if the user is feeling stressed, the history unit postpones history entries of lower importance. For example, the history unit captures the user's facial expressions with a camera and estimates their emotions using an emotion estimation algorithm. The history unit can also record all history entries evenly when the user is relaxed. For example, the history unit records the user's voice and estimates their emotions using voice analysis technology. The history unit can also prioritize history entries of higher importance when the user is in a hurry. For example, the history unit collects the user's biometric data (heart rate and electrodermal activity) using a sensor and estimates their emotions using an emotion estimation algorithm. This allows the system to prioritize history entries based on the user's emotions, resulting in more appropriate history entries. Emotion estimation is achieved using an emotion estimation function, such as an emotion engine or generative AI. Generative AI can be, but is not limited to, text generation AI (e.g., GPT-4) or multimodal generation AI. Some or all of the above-described processing in the history unit may be performed using, for example, a generation AI, or may be performed without using a generation AI. For example, the history unit may determine the priority of history using a generation AI model that receives user emotion data as input and outputs the priority of history.

[0091] When recording history, the history unit can select the optimal recording method by taking into account the user's geographical location information. For example, when the user is at home, the history unit prioritizes recording history related to home settings. For example, the history unit acquires the user's geographical location information from GPS data, Wi-Fi location information, etc., and determines that the user is at home. Furthermore, when the user is at work, the history unit can prioritize recording work-related history. For example, the history unit acquires the user's geographical location information and determines that the user is at work. Furthermore, when the user is traveling, the history unit can prioritize recording travel-related history. For example, the history unit acquires the user's geographical location information and determines that the user is traveling. In this way, the optimal recording method can be selected by taking the user's geographical location information into account. Some or all of the above-described processing in the history unit may be performed using, for example, a generation AI, or may be performed without using a generation AI. For example, the history unit can select the recording method using a generation AI model that inputs the user's geographical location information and outputs the optimal recording method. === Hard Collateral 1-1 === Each of the multiple elements including the above-mentioned reception unit, analysis unit, and execution unit is realized, for example, by at least one of the smart device 14 and the data processing device 12. For example, the reception unit receives natural language instructions from a user using the microphone 38B or touch panel 38A of the smart device 14. The analysis unit is realized by the identification processing unit 290 of the data processing device 12, analyzes the user's instructions using a generation AI, and identifies appropriate settings and operations. The execution unit is realized by the control unit 46A of the smart device 14, and executes the settings and operations identified by the analysis unit. === Hard Collateral 1-2 === Each of the multiple elements including the above-described reception unit, analysis unit, and execution unit is realized, for example, by at least one of the smart glasses 214 and the data processing device 12. For example, the reception unit receives natural language instructions from a user using the microphone 238 of the smart glasses 214. The analysis unit is realized by the identification processing unit 290 of the data processing device 12 and analyzes the user's instructions using a generation AI to identify appropriate settings and operations. The execution unit is realized by the control unit 46A of the smart glasses 214 and executes the settings and operations identified by the analysis unit. === Hard Collateral 1-3 === Each of the multiple elements including the above-mentioned reception unit, analysis unit, and execution unit is realized, for example, by at least one of the headset type terminal 314 and the data processing device 12. For example, the reception unit receives instructions in natural language from the user using the microphone 238 of the headset type terminal 314. The analysis unit is realized by the specification processing unit 290 of the data processing device 12, and analyzes the user's instructions using a generation AI to identify appropriate settings and operations. The execution unit is realized by the control unit 46A of the headset type terminal 314, and executes the settings and operations identified by the analysis unit. === Hard Collateral 1-4 === Each of the multiple elements including the above-mentioned reception unit, analysis unit, and execution unit is realized, for example, by at least one of the robot 414 and the data processing device 12. For example, the reception unit receives instructions in natural language from a user using the microphone 238 of the robot 414. The analysis unit is realized by the specification processing unit 290 of the data processing device 12, analyzes the user's instructions using a generation AI, and identifies appropriate settings and operations. The execution unit is realized by the control unit 46A of the robot 414, and executes the settings and operations identified by the analysis unit.

[0092] The system according to the embodiment is not limited to the above-described example, and various modifications are possible, for example, as follows.

[0093] The reception unit can analyze the user's past instruction history and select the optimal reception method. For example, it prioritizes reception of instruction methods (such as voice and text) that the user has frequently used in the past. For example, the reception unit analyzes the user's past instruction history and, if the user has frequently used voice instructions, prioritizes reception of voice instructions. The reception unit can also predict and suggest an instruction method to be used in a specific time period based on the user's past instruction history. For example, if the user frequently uses text instructions in the morning, the reception unit prioritizes reception of text instructions in the morning. The reception unit can also analyze patterns of instructions given by the user in the past and select the optimal reception method. For example, if the user uses a specific instruction method when performing a specific operation, the reception unit prioritizes reception of that instruction method. In this way, the optimal reception method can be selected by analyzing the user's past instruction history. Some or all of the above-described processing in the reception unit may be performed using, for example, a generation AI, or may be performed without using a generation AI. For example, the reception unit can select a reception method using a generation AI model that inputs the user's past instruction history and outputs the optimal reception method.

[0094] During analysis, the analysis unit can adjust the level of detail of the analysis based on the importance of the instruction. For example, the generation AI performs a detailed analysis for an instruction with high importance. For example, the analysis unit analyzes the content of the user's instruction and performs a detailed analysis if it determines that the instruction is highly important. The analysis unit can also perform a concise analysis for an instruction with low importance. For example, the analysis unit analyzes the content of the user's instruction and performs a concise analysis if it determines that the instruction is low important. The analysis unit can also perform an analysis with an appropriate level of detail for an instruction with medium importance. For example, the analysis unit analyzes the content of the user's instruction and performs an analysis with an appropriate level of detail if it determines that the instruction is medium important. This allows for adjusting the level of detail of the analysis based on the importance of the instruction, thereby providing a more appropriate analysis result. Some or all of the above-described processing in the analysis unit may be performed using, for example, the generation AI, or may be performed without using the generation AI. For example, the analysis unit can adjust the level of detail of the analysis using a generation AI model that inputs the content of the user's instruction and outputs the level of detail of the analysis.

[0095] During execution, the execution unit can analyze the user's past operation history and select an appropriate execution method. For example, the execution unit can prioritize execution of operation methods that the user frequently used in the past. For example, the execution unit can analyze the user's past operation history and prioritize execution of operation methods that the user frequently used. The execution unit can also predict and suggest operation methods to be performed in a specific time period based on the user's past operation history. For example, the execution unit can analyze the user's past operation history and predict operation methods to be performed in a specific time period. The execution unit can also analyze patterns of operations performed by the user in the past and select an optimal execution method. For example, the execution unit can analyze the user's past operation history, identify operation patterns, and select an optimal execution method. In this way, the optimal execution method can be selected by analyzing the user's past operation history. Some or all of the above-described processing in the execution unit can be performed using, for example, a generation AI, or can be performed without using a generation AI. For example, the execution unit can select an execution method using a generation AI model that inputs the user's past operation history and outputs an optimal execution method.

[0096] The feedback unit can adjust the level of detail of the feedback based on the importance of the analysis result. For example, detailed feedback is provided for an analysis result with high importance. For example, the feedback unit evaluates the importance of the analysis result and provides detailed feedback if it is determined to be high. The feedback unit can also provide brief feedback for an analysis result with low importance. For example, the feedback unit evaluates the importance of the analysis result and provides brief feedback if it is determined to be low. The feedback unit can also provide feedback with an appropriate level of detail for an analysis result with medium importance. For example, the feedback unit evaluates the importance of the analysis result and provides feedback with an appropriate level of detail if it is determined to be medium. In this way, by adjusting the level of detail of the feedback based on the importance of the analysis result, more appropriate feedback can be provided. Some or all of the above-described processing in the feedback unit may be performed, for example, using a generation AI, or may be performed without using a generation AI. For example, the feedback unit can adjust the level of detail of the feedback using a generation AI model that inputs the importance of the analysis result and outputs the level of detail of the feedback.

[0097] When recording history, the history unit can select the optimal recording method by referring to past history data. For example, if a user has preferred detailed history in the past, the history unit records detailed history. For example, the history unit can refer to past history data and determine that the user has preferred detailed history. Furthermore, if a user has preferred concise history in the past, the history unit can also record concise history. For example, the history unit can refer to past history data and determine that the user has preferred concise history. Furthermore, the history unit can select the optimal recording method from the user's past history data. For example, the history unit can analyze past history data and select the optimal recording method. In this way, the optimal recording method can be selected by referring to the past history data. Some or all of the above-described processing in the history unit may be performed, for example, using a generation AI, or may be performed without using a generation AI. For example, the history unit can select the recording method using a generation AI model that inputs past history data and outputs the optimal recording method.

[0098] The reception unit can estimate the user's emotions and adjust the timing of instruction acceptance based on the estimated user emotions. For example, if the user is feeling stressed, the generation AI can delay accepting instructions and wait until the user is relaxed. For example, the reception unit can capture the user's facial expression with a camera and estimate their emotions using an emotion estimation algorithm. Furthermore, if the user is relaxed, the reception unit can have the generation AI immediately accept instructions and quickly start processing them. For example, the reception unit can record the user's voice and estimate their emotions using voice analysis technology. Furthermore, if the user is in a hurry, the reception unit can have the generation AI prioritize instructions and quickly process them. For example, the reception unit can collect the user's biometric data (heart rate and electrodermal activity) using a sensor and estimate their emotions using an emotion estimation algorithm. This allows the timing of instruction acceptance to be adjusted according to the user's emotions, thereby allowing instructions to be accepted at a more appropriate time. Emotion estimation is realized using an emotion estimation function, such as an emotion engine or a generation AI. The generation AI may be, but is not limited to, a text generation AI (e.g., GPT-4) or a multimodal generation AI. Some or all of the above-described processing in the reception unit may be performed using, for example, the generation AI. For example, the reception unit may adjust the timing of receiving instructions using a generation AI model that receives user emotion data as input and outputs the timing of receiving instructions.

[0099] The analysis unit can estimate the user's emotions and adjust the way the analysis is presented based on the estimated user emotions. For example, if the user is relaxed, the generation AI can provide detailed analysis results. For example, the analysis unit can capture the user's facial expressions with a camera and estimate their emotions using an emotion estimation algorithm. Alternatively, if the user is in a hurry, the generation AI can provide concise analysis results. For example, the analysis unit can record the user's voice and estimate their emotions using voice analysis technology. Alternatively, if the user is feeling stressed, the generation AI can provide visually easy-to-understand analysis results. For example, the analysis unit can collect the user's biometric data (heart rate and electrodermal activity) using a sensor and estimate their emotions using an emotion estimation algorithm. This allows the system to adjust the way the analysis is presented based on the user's emotions, thereby providing more appropriate analysis results. Emotion estimation is achieved using an emotion estimation function, such as an emotion engine or generation AI. The generation AI can be, but is not limited to, a text generation AI (e.g., GPT-4) or a multimodal generation AI. Some or all of the above-described processing in the analysis unit may be performed using, for example, a generative AI, or may be performed without using a generative AI. For example, the analysis unit may adjust the expression method of the analysis using a generative AI model that receives user emotion data as input and outputs the expression method of the analysis.

[0100] The execution unit can estimate the user's emotions and adjust the execution method based on the estimated user emotions. For example, if the user is relaxed, the generation AI provides detailed instructions and executes the task. For example, the execution unit captures the user's facial expressions with a camera and estimates the user's emotions using an emotion estimation algorithm. Furthermore, if the user is in a hurry, the execution unit can also execute the task quickly. For example, the execution unit records the user's voice and estimates the user's emotions using voice analysis technology. Furthermore, if the user is feeling stressed, the generation AI can provide easy-to-understand visual instructions and execute the task. For example, the execution unit collects the user's biometric data (heart rate and electrodermal activity) using a sensor and estimates the user's emotions using an emotion estimation algorithm. This allows the execution method to be adjusted according to the user's emotions, thereby providing a more appropriate execution method. Emotion estimation is achieved using an emotion estimation function, for example, using an emotion engine or generation AI. The generation AI can be, but is not limited to, a text generation AI (e.g., GPT-4) or a multimodal generation AI. Some or all of the above-described processing in the execution unit may be performed using, for example, a generative AI, or may be performed without using a generative AI. For example, the execution unit may adjust the execution method using a generative AI model that receives user emotion data as input and outputs an execution method.

[0101] The feedback unit can estimate the user's emotions and adjust the way the feedback is expressed based on the estimated user emotions. For example, if the user is relaxed, detailed feedback can be provided. For example, the feedback unit can capture the user's facial expressions with a camera and estimate the user's emotions using an emotion estimation algorithm. Furthermore, if the user is in a hurry, the feedback unit can provide concise feedback. For example, the feedback unit can record the user's voice and estimate the user's emotions using voice analysis technology. Furthermore, if the user is feeling stressed, the feedback unit can provide visually easy-to-understand feedback. For example, the feedback unit can collect the user's biometric data (heart rate and electrodermal activity) using a sensor and estimate the user's emotions using an emotion estimation algorithm. This allows the system to adjust the way the feedback is expressed based on the user's emotions, thereby providing more appropriate feedback. The emotion estimation is achieved using an emotion estimation function, such as an emotion engine or a generative AI. The generative AI can be, but is not limited to, a text generation AI (e.g., GPT-4) or a multimodal generation AI. Some or all of the above-described processing in the feedback unit may be performed using, for example, a generation AI, or may be performed without using a generation AI. For example, the feedback unit may adjust the feedback expression method using a generation AI model that receives user emotion data as input and outputs a feedback expression method.

[0102] The history unit can estimate the user's emotions and adjust the history recording method based on the estimated user emotions. For example, if the user is relaxed, a detailed history is recorded. For example, the history unit captures the user's facial expressions with a camera and estimates the user's emotions using an emotion estimation algorithm. Furthermore, if the user is in a hurry, the history unit can record a concise history. For example, the history unit records the user's voice and estimates the user's emotions using voice analysis technology. Furthermore, if the user is feeling stressed, the history unit can record a visually easy-to-understand history. For example, the history unit collects the user's biometric data (heart rate and electrodermal activity) using a sensor and estimates the user's emotions using an emotion estimation algorithm. This allows for more appropriate history recording by adjusting the history recording method according to the user's emotions. Emotion estimation is achieved using an emotion estimation function, such as an emotion engine or generative AI. Generative AI can be, but is not limited to, a text generation AI (e.g., GPT-4) or a multimodal generation AI. Some or all of the above-described processing in the history unit may be performed using, for example, a generative AI, or may be performed without using a generative AI. For example, the history unit may adjust the history recording method using a generative AI model that receives user emotion data as input and outputs a history recording method.

[0103] The processing flow of the second embodiment will be briefly explained below.

[0104] Step 1: The reception unit receives instructions in natural language from the user. The natural language instructions from the user include voice instructions and text instructions. The reception unit can convert the voice instructions into text data using voice recognition technology, and the user can also directly input text instructions using a text input interface. Step 2: The analysis unit analyzes the natural language instructions received by the reception unit and identifies appropriate settings and operations. The analysis unit analyzes the instructions using natural language processing techniques such as morphological analysis, grammatical analysis, and semantic analysis, and analyzes the user's instructions using a generative AI (e.g., GPT-4). This allows the system to understand the intent of the user's instructions and identify appropriate settings and operations. Step 3: The execution unit executes the settings and operations identified by the analysis unit. The execution unit can access the phone's settings menu and execute operations such as turning on Wi-Fi or maximizing screen brightness. The processing in the execution unit can be performed with or without the aid of the generation AI.

[0105] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0106] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> Examples of the generative AI include a neural network (NN) and a neural network (NN). The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 receives a prompt containing an instruction, as well as inference data such as voice data representing speech, text data representing text, and image data representing an image (e.g., still image data or video data). The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in one or more data formats of voice data, text data, image data, etc. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specification processing unit 290 performs the above-mentioned specification processing using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction. In this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. The data processing device 12 and the like include multiple types of data generation models 58, and the data generation model 58 includes AIs other than the generative AI. The AI ​​other than the generative AI may be, for example, linear regression, logistic regression, decision tree, random forest, support vector machine (SVM), k-means clustering, convolutional neural network (CNN), recurrent neural network (RNN), generative adversarial network (GAN), or naive Bayes, and may perform various processes, but is not limited to these examples. The AI ​​may also be an AI agent. When the processing of each of the above-mentioned parts is performed by an AI, the processing may be performed in part or entirely by the AI, but is not limited to these examples. The processing performed by an AI including the generative AI may be replaced with rule-based processing, and rule-based processing may be replaced with processing performed by an AI including the generative AI.

[0107] Furthermore, the processing by the data processing system 10 described above is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart device 14, but may also be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart device 14. Furthermore, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the smart device 14 or an external device, and the smart device 14 acquires or collects information necessary for processing from the data processing device 12 or an external device.

[0108] The correspondence between each part and the device or control part is not limited to the example described above, and various modifications are possible.

[0109] [Second embodiment] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0110] 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0111] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN and / or a LAN.

[0112] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0113] The microphone 238 receives instructions and the like from the user by receiving voice uttered by the user. The microphone 238 captures the voice uttered by the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to instructions from the processor 46.

[0114] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the user's surroundings (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0115] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0116] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0117] The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0118] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate a user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotion, including estimation and prediction of the user's emotion, but is not limited to these examples. Furthermore, the estimation and prediction of emotion also includes, for example, emotion analysis.

[0119] In the smart glasses 214, the specific processing is performed by the processor 46. A specific processing program 60 is stored in the storage 50. The processor 46 reads the specific processing program 60 from the storage 50 and executes the read specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as the control unit 46A in accordance with the specific processing program 60 executed on the RAM 48. The smart glasses 214 also have a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59, and can perform processing similar to that of the specific processing unit 290 using these models.

[0120] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 communicates with the server device having the data generation model 58 to obtain a processing result (such as a prediction result) using the data generation model 58. Furthermore, the data processing device 12 may be a server device, or may be a terminal device (for example, a mobile phone, a robot, a home appliance, etc.) owned by a user.

[0121] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0122] The data generation model 58 is a so-called generative AI. An example of the data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 receives a prompt including an instruction, as well as inference data such as audio data indicating speech, text data indicating text, and image data indicating an image (e.g., still image data or video data). The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in one or more data formats, such as audio data, text data, and image data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The identification processing unit 290 performs the above-mentioned identification processing using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction. In this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. The data processing device 12 and the like include multiple types of data generation models 58, and the data generation models 58 include AIs other than the generation AI. Examples of AIs other than the generation AI include, but are not limited to, linear regression, logistic regression, decision trees, random forests, support vector machines (SVMs), k-means clustering, convolutional neural networks (CNNs), recurrent neural networks (RNNs), generative adversarial networks (GANs), and naive Bayes. These AIs can perform various types of processing, but are not limited to these examples. The AI ​​may also be an AI agent. When the processing of each of the above-described parts is performed by an AI, the processing may be performed in part or entirely by the AI, but is not limited to these examples. Processing performed by an AI, including the generation AI, may be replaced with rule-based processing, and rule-based processing may be replaced with processing performed by an AI, including the generation AI.

[0123] The data processing system 210 according to the second embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 210 is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart glasses 214, but may also be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart glasses 214. Furthermore, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the smart glasses 214 or an external device, etc., and the smart glasses 214 acquires or collects information required for processing from the data processing device 12 or an external device, etc.

[0124] The correspondence between each part and the device or control part is not limited to the example described above, and various modifications are possible.

[0125] [Third embodiment] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0126] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0127] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN and / or a LAN.

[0128] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0129] The microphone 238 receives instructions and the like from the user by receiving voice uttered by the user. The microphone 238 captures the voice uttered by the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to instructions from the processor 46.

[0130] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the user's surroundings (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0131] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0132] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0133] The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0134] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate a user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotion, including estimation and prediction of the user's emotion, but is not limited to these examples. Furthermore, the estimation and prediction of emotion also includes, for example, emotion analysis.

[0135] In the headset type terminal 314, the identification process is performed by the processor 46. A identification program 60 is stored in the storage 50. The processor 46 reads the identification program 60 from the storage 50 and executes the read identification program 60 on the RAM 48. The identification process is realized by the processor 46 operating as a control unit 46A in accordance with the identification program 60 executed on the RAM 48. Note that the headset type terminal 314 has a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59, and can also perform processing similar to that of the identification processing unit 290 using these models.

[0136] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 communicates with the server device having the data generation model 58 to obtain a processing result (such as a prediction result) using the data generation model 58. Furthermore, the data processing device 12 may be a server device, or may be a terminal device (for example, a mobile phone, a robot, a home appliance, etc.) owned by a user.

[0137] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0138] The data generation model 58 is a so-called generative AI. An example of the data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 receives a prompt including an instruction, as well as inference data such as audio data indicating speech, text data indicating text, and image data indicating an image (e.g., still image data or video data). The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in one or more data formats, such as audio data, text data, and image data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The identification processing unit 290 performs the above-mentioned identification processing using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction. In this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. The data processing device 12 and the like include multiple types of data generation models 58, and the data generation models 58 include AIs other than the generation AI. Examples of AIs other than the generation AI include, but are not limited to, linear regression, logistic regression, decision trees, random forests, support vector machines (SVMs), k-means clustering, convolutional neural networks (CNNs), recurrent neural networks (RNNs), generative adversarial networks (GANs), and naive Bayes. These AIs can perform various types of processing, but are not limited to these examples. The AI ​​may also be an AI agent. When the processing of each of the above-described parts is performed by an AI, the processing may be performed in part or entirely by the AI, but is not limited to these examples. Processing performed by an AI, including the generation AI, may be replaced with rule-based processing, and rule-based processing may be replaced with processing performed by an AI, including the generation AI.

[0139] The data processing system 310 according to the third embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 310 is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the headset type terminal 314, but may also be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the headset type terminal 314. Furthermore, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the headset type terminal 314 or an external device, etc., and the headset type terminal 314 acquires or collects information required for processing from the data processing device 12 or an external device, etc.

[0140] The correspondence between each part and the device or control part is not limited to the example described above, and various modifications are possible.

[0141] [Fourth embodiment] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[0142] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0143] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN and / or a LAN.

[0144] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[0145] The microphone 238 receives instructions and the like from the user by receiving voice uttered by the user. The microphone 238 captures the voice uttered by the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to instructions from the processor 46.

[0146] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS image sensor or a CCD image sensor, and captures images of the user's surroundings (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0147] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0148] The control object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[0149] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0150] The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0151] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate a user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotion, including estimation and prediction of the user's emotion, but is not limited to these examples. Furthermore, the estimation and prediction of emotion also includes, for example, emotion analysis.

[0152] In the robot 414, the processor 46 performs the identification process. The storage 50 stores the identification program 60. The processor 46 reads the identification program 60 from the storage 50 and executes the read identification program 60 on the RAM 48. The identification process is realized by the processor 46 operating as the control unit 46A in accordance with the identification program 60 executed on the RAM 48. The robot 414 also has a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59, and can perform the same process as the identification processing unit 290 using these models.

[0153] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 communicates with the server device having the data generation model 58 to obtain a processing result (such as a prediction result) using the data generation model 58. Furthermore, the data processing device 12 may be a server device, or may be a terminal device (for example, a mobile phone, a robot, a home appliance, etc.) owned by a user.

[0154] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0155] The data generation model 58 is a so-called generative AI. An example of the data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 receives a prompt including an instruction, as well as inference data such as audio data indicating speech, text data indicating text, and image data indicating an image (e.g., still image data or video data). The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in one or more data formats, such as audio data, text data, and image data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The identification processing unit 290 performs the above-mentioned identification processing using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction. In this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. The data processing device 12 and the like include multiple types of data generation models 58, and the data generation models 58 include AIs other than the generation AI. Examples of AIs other than the generation AI include, but are not limited to, linear regression, logistic regression, decision trees, random forests, support vector machines (SVMs), k-means clustering, convolutional neural networks (CNNs), recurrent neural networks (RNNs), generative adversarial networks (GANs), and naive Bayes. These AIs can perform various types of processing, but are not limited to these examples. The AI ​​may also be an AI agent. When the processing of each of the above-described parts is performed by an AI, the processing may be performed in part or entirely by the AI, but is not limited to these examples. Processing performed by an AI, including the generation AI, may be replaced with rule-based processing, and rule-based processing may be replaced with processing performed by an AI, including the generation AI.

[0156] The data processing system 410 according to the fourth embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 410 is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the robot 414, but may also be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the robot 414. Furthermore, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the robot 414 or an external device, etc., and the robot 414 acquires or collects information required for processing from the data processing device 12 or an external device, etc.

[0157] The correspondence between each part and the device or control part is not limited to the example described above, and various modifications are possible.

[0158] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0159] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion encompasses both emotions and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[0160] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[0161] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[0162] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, cars, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on speech emotion recognition and brain physiological signal analysis systems for emotions, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the area called "reaction," where sensation is dominant. The right half of the emotion map lists emotions belonging to the area called "situation," where situational awareness is dominant.

[0163] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[0164] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[0165] In the above embodiment, an example was given in which a specific process is performed by one computer 22, but the technology disclosed herein is not limited to this, and distributed processing of the specific process may be performed by multiple computers including computer 22.

[0166] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[0167] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0168] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[0169] The hardware resource for executing a specific process can be any of the following types of processors: A processor, for example, is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. A processor also includes a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[0170] The hardware resource that executes the specific process may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific process may be a single processor.

[0171] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[0172] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[0173] In the above example, the first to fourth embodiments have been described separately, but some or all of these embodiments may be combined. The smart device 14, smart glasses 214, headset terminal 314, and robot 414 are merely examples, and they may be combined, or other devices may be used. In the above example, the first and second embodiments have been described separately, but they may be combined.

[0174] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[0175] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[0176] [Explanation of symbols]

[0177] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot

Claims

1. a reception unit that receives instructions in natural language from a user; an analysis unit that analyzes the natural language instruction received by the reception unit and identifies appropriate settings and operations; an execution unit that executes the settings and operations identified by the analysis unit; A system characterized by:

2. The analysis unit Generative AI analyzes natural language instructions to identify appropriate settings and operations 2. The system of claim 1.

3. The execution unit: Turn Wi-Fi on or off or adjust screen brightness 2. The system of claim 1.

4. The execution unit: Install or uninstall apps and perform notification settings operations 2. The system of claim 1.

5. The analysis unit A feedback unit is provided that analyzes the user's instructions and feeds back the analysis results to the user.

2. The system of claim 1.

6. The execution unit: Equipped with a history section that records operation history 2. The system of claim 1.

7. The reception unit Estimates the user's emotions and adjusts the timing of accepting instructions based on the estimated user emotions.

2. The system of claim 1.

8. The reception unit Analyze the user's past instruction history and select the appropriate reception method 2. The system of claim 1.

Images