system

Abstract

The system according to this embodiment aims to improve work efficiency and support learning by learning the user's PC operations and automating tasks. [Solution] The system according to the embodiment comprises an operation recording unit, a reproduction unit, an optimization unit, a voice command unit, a multi-device support unit, and a learning support unit. The operation recording unit records the user's PC operations. The reproduction unit reproduces the operations recorded by the operation recording unit. The optimization unit optimizes the operations reproduced by the reproduction unit. The voice command unit accepts voice commands. The multi-device support unit synchronizes operations between different devices. The learning support unit generates explanations of operation procedures and conducts operation tests in a quiz format.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, and includes steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the conventional technology, it is difficult to efficiently learn a user's PC operations and automate tasks, and there is room for improvement in business efficiency and learning support.

[0005] The system according to the embodiment aims to achieve business efficiency improvement and learning support by learning a user's PC operations and automating tasks.

Means for Solving the Problems

[0006] The system according to this embodiment comprises an operation recording unit, a reproduction unit, an optimization unit, a voice command unit, a multi-device support unit, and a learning support unit. The operation recording unit records the user's PC operations. The reproduction unit reproduces the operations recorded by the operation recording unit. The optimization unit optimizes the operations reproduced by the reproduction unit. The voice command unit accepts voice commands. The multi-device support unit synchronizes operations between different devices. The learning support unit generates explanations of operation procedures and conducts operation tests in a quiz format. [Effects of the Invention]

[0007] The system according to this embodiment can improve work efficiency and support learning by learning the user's PC operations and automating tasks. [Brief explanation of the drawing]

[0008] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Modes for carrying out the invention]

[0009] Hereinafter, an example of an embodiment of the system relating to the technology of this disclosure will be described with reference to the attached drawings.

[0010] First, let's explain the terminology used in the following explanation.

[0011] In the following embodiments, the signed processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Furthermore, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), APU (Accelerated Processing Unit), or TPU (Tensor Processing Unit).

[0012] In the following embodiments, signed RAM (Random Access Memory) is a memory that temporarily stores information and is used as work memory by the processor.

[0013] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0014] In the following embodiments, the labeled communication I / F (Interface) is an interface including a communication processor, an antenna, and the like. The communication I / F manages communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0015] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B". That is, "A and / or B" means that it may be only A, only B, or a combination of A and B. Also, in this specification, when expressing three or more matters connected by "and / or", the same concept as "A and / or B" is applied.

[0016] [First Embodiment] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0017] As shown in FIG. 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0018] The data processing device 12 includes a computer​​​The smart device 14 comprises a computer 36, a receiving device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The receiving device 38, output device 40, and camera 42 are also connected to the bus 52.

[0020] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, and accepts user input. The touch panel 38A accepts user input via touch by detecting contact with an object (e.g., a pen or finger). The microphone 38B accepts user input via voice by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 (see Figure 2) acquires the data indicating the user input.

[0021] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user by outputting the data in a form perceptible to the user (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0022] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0023] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0024] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0025] Storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the user's emotions using the emotion identification model 59 and perform identification processing using the user's emotions. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotions, including but not limited to these examples. Furthermore, emotion estimation and prediction also include, for example, emotion analysis.

[0026] In the smart device 14, specific processing is performed by the processor 46. The storage 50 stores a specific processing program 60. The specific processing program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the specific processing program 60 from the storage 50 and executes the read specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as a control unit 46A according to the specific processing program 60 executed on the RAM 48. The smart device 14 also has a data generation model 58 and an emotion identification model 59, similar to the data generation model and emotion identification model 59, and can perform processing similar to that of the specific processing unit 290 using these models.

[0027] Furthermore, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device (e.g., a generation server) may have the data generation model 58. In this case, the data processing device 12 obtains processing results (such as prediction results) using the data generation model 58 by communicating with the server device having the data generation model 58. The data processing device 12 may also be a server device or a terminal device owned by a user (e.g., a mobile phone, robot, home appliance, etc.). Next, an example of processing by the data processing system 10 according to the first embodiment will be described.

[0028] (Example of form 1) The assistant software according to an embodiment of the present invention is a generation AI-equipped assistant software that learns the user's PC operations and automates tasks. This assistant software improves work efficiency and provides learning support by learning the user's PC operations and automating tasks. For example, the assistant software's operation recording unit records the user's PC operations unit by unit and reproduces those operations with high accuracy. Next, the assistant software's optimization unit analyzes the recorded operations and proposes more efficient procedures. Furthermore, the assistant software's voice command unit gives voice instructions for operations, and the multi-device support unit synchronizes operations between different devices. Finally, the assistant software's learning support unit generates explanations of the operation procedures and conducts operation tests in a quiz format. As a result, the user can proceed with work efficiently and learn the operation procedures. Thus, the assistant software can efficiently record, reproduce, optimize, accept voice commands, synchronize between devices, and provide learning support for the user's PC operations.

[0029] The assistant software according to this embodiment comprises an operation recording unit, a reproduction unit, an optimization unit, a voice command unit, a multi-device compatibility unit, and a learning support unit. The operation recording unit records the user's PC operations. The operation recording unit can record operations such as file operations, software usage, and web browsing performed by the user. The operation recording unit records based on the frequency and type of operations. For example, the operation recording unit can prioritize recording frequently performed operations. The operation recording unit can also adjust the recording method according to the type of operation. For example, the operation recording unit can record file operations in detail and software usage concisely. The reproduction unit reproduces the operations recorded by the operation recording unit. The reproduction unit can reproduce recorded operations with high accuracy. The reproduction unit can adjust the accuracy and timing of reproduction. For example, the reproduction unit can minimize the error range of operations and perform accurate reproduction. The reproduction unit can also adjust the timing of operation reproduction to match the user's operating environment. The optimization unit optimizes the operations reproduced by the reproduction unit. The optimization unit can, for example, analyze recorded operations and propose more efficient procedures. The optimization unit performs optimization based on efficiency criteria and algorithms for operations. For example, the optimization unit can optimize operations with the aim of reducing operation time and the number of operations. The optimization unit can also learn the user's operation patterns and propose the optimal procedure. The voice command unit gives instructions for operations by voice. The voice command unit can, for example, accept the user's voice commands using speech recognition technology. The voice command unit can adjust the types of voice commands and how they are accepted. For example, the voice command unit can prioritize the acceptance of specific voice commands. The voice command unit can also learn the user's voice characteristics to improve the accuracy of speech recognition. The multi-device support unit synchronizes operations between different devices. For example, the multi-device support unit can synchronize operations between devices such as PCs, smartphones, and tablets. The multi-device support unit can adjust the synchronization methods and criteria between devices.For example, the multi-device support unit can prioritize the synchronization of operations between specific devices. The multi-device support unit can also learn the usage history of devices to improve the synchronization accuracy between devices. The learning support unit generates explanations of operating procedures and conducts operation tests in a quiz format. For example, the learning support unit can generate and provide explanations of operating procedures to the user. The learning support unit can adjust the format and content of the explanations. For example, the learning support unit can provide detailed explanations. Furthermore, the learning support unit can conduct operation tests in a quiz format to improve the user's learning effectiveness. As a result, the assistant software according to this embodiment can efficiently record, reproduce, optimize, accept voice commands, synchronize between devices, and provide learning support for the user's PC operations.

[0030] The operation log unit records the user's PC operations. For example, it can record operations such as file operations, software usage, and web browsing. Specifically, the operation log unit records in detail operations such as opening, saving, and moving files. This includes file paths, operation timestamps, and operation types. Regarding software usage, it records which software was launched, when, and what operations were performed. For web browsing, it records information such as accessed URLs, browsing time, and clicked links. The operation log unit records based on the frequency and type of operations. For example, it can prioritize recording frequently performed operations. This allows for focused recording of operations frequently performed by the user, which can be useful for later analysis and reproduction. Furthermore, the operation log unit can adjust its recording method depending on the type of operation. For example, it can record file operations in detail and software usage concisely. This allows for efficient management of recorded data by recording important operations in detail and concisely when necessary. The operation recording unit can also learn the user's operation patterns and improve the accuracy and efficiency of recording. For example, the operation recording unit can learn the frequency and timing of specific operations performed by the user and automatically select the optimal recording method. This allows the operation recording unit to efficiently and accurately record the user's operations, which can then be used for later reproduction and optimization.

[0031] The reproduction unit reproduces the operations recorded by the operation recording unit. The reproduction unit can reproduce recorded operations with high accuracy. Specifically, the reproduction unit accurately reproduces operations such as file operations, software usage, and web browsing recorded by the operation recording unit. The reproduction unit can adjust the accuracy and timing of the reproduction. For example, the reproduction unit can minimize the error range of the operation and perform accurate reproduction. The reproduction unit can also adjust the timing of the operation reproduction to match the user's operating environment. The reproduction unit can optimize the accuracy and timing of the reproduction according to the user's operating environment and usage. For example, the reproduction unit can adjust the reproduction method according to the device and software version used by the user. This allows the reproduction unit to perform optimal reproduction for the user's operating environment, improving the accuracy and efficiency of the operation. The reproduction unit can also detect errors and problems that occur during operation reproduction and take appropriate countermeasures. For example, the reproduction unit can detect error messages and warnings that occur during operation reproduction and notify the user. This allows the reproduction unit to quickly resolve problems that occur during operation reproduction and maintain the accuracy and efficiency of the operation. Furthermore, the reproduction unit can record the results of the operation reproduction, which can then be used for later analysis and optimization. For example, the reproduction unit can record the results of the operation reproduction in a log file and evaluate the accuracy and efficiency of the reproduction. Based on these results, the reproduction unit can improve the reproduction method and algorithm, thereby improving the accuracy and efficiency of the operation.

[0032] The optimization unit optimizes the operations reproduced by the reproduction unit. For example, the optimization unit can analyze recorded operations and propose more efficient procedures. Specifically, the optimization unit performs optimization based on efficiency criteria and algorithms for operations. For example, the optimization unit can optimize operations with the aim of reducing operation time and the number of operations. The optimization unit can also learn user operation patterns and propose optimal procedures. For example, the optimization unit can analyze operations frequently performed by the user and propose more efficient procedures. This allows the optimization unit to streamline user operations, reduce work time, and simplify operations. Furthermore, the optimization unit can record the optimization results of operations and use them for later analysis and improvement. For example, the optimization unit can record the optimization results of operations in a log file and evaluate the effectiveness and accuracy of the optimization. This allows the optimization unit to improve the optimization method and algorithm based on the optimization results, thereby improving the efficiency and accuracy of operations. In addition, the optimization unit can collect user feedback and continuously improve the accuracy and effectiveness of the optimization. For example, based on user feedback, the optimization unit can review the optimization procedures and proposals to achieve more effective optimization. This allows the optimization unit to streamline user operations and improve work productivity.

[0033] The voice command unit provides voice commands. For example, it can receive user voice commands using speech recognition technology. Specifically, the voice command unit analyzes user voice commands in real time and executes the corresponding operations. The voice command unit can adjust the types and methods of receiving voice commands. For example, it can prioritize certain voice commands. Furthermore, the voice command unit can learn the user's voice characteristics to improve the accuracy of speech recognition. This allows the voice command unit to accurately recognize user voice commands and quickly execute the corresponding operations. The voice command unit can also analyze user voice commands and suggest appropriate operations. For example, it can analyze user voice commands and suggest the optimal operation procedure. This allows the voice command unit to streamline user operations and improve work productivity. Additionally, the voice command unit can record the history of received voice commands for later analysis and improvement. For example, it can record the history of received voice commands in a log file to evaluate the accuracy and effectiveness of speech recognition. This allows the voice command unit to improve its speech recognition technology and algorithms based on the history of received voice commands, thereby enhancing the accuracy and efficiency of voice commands. Furthermore, the voice command unit can collect user feedback and continuously improve the accuracy and effectiveness of voice commands. For example, based on user feedback, the voice command unit can review how voice commands are received and what suggestions are made, resulting in more effective voice commands. This allows the voice command unit to streamline user operations and improve work productivity.

[0034] The multi-device support unit synchronizes operations across different devices. For example, it can synchronize operations between devices such as PCs, smartphones, and tablets. Specifically, it can reflect operations performed by a user on a PC on their smartphone or tablet. This allows users to operate seamlessly across different devices. The multi-device support unit can adjust the synchronization methods and criteria between devices. For example, it can prioritize the synchronization of operations between specific devices. Furthermore, the multi-device support unit can learn device usage history to improve the accuracy of synchronization between devices. This allows the multi-device support unit to accurately synchronize user operations and maintain consistency across different devices. The multi-device support unit can record user operation history for later analysis and improvement. For example, it can record the operation history between devices in a log file to evaluate the accuracy and effectiveness of synchronization. Based on this operation history, the multi-device support unit can improve the synchronization methods and algorithms, thereby increasing the accuracy and efficiency of synchronization between devices. Furthermore, the multi-device support unit can collect user feedback and continuously improve the accuracy and effectiveness of synchronization. For example, based on user feedback, the multi-device support unit can review the synchronization method and suggestions to achieve more effective synchronization. This allows the multi-device support unit to streamline user operations and maintain consistency across different devices.

[0035] The learning support unit generates explanations of operating procedures and conducts operational tests in a quiz format. For example, the learning support unit can generate and provide explanations of operating procedures to the user. Specifically, the learning support unit provides detailed explanations of the operating procedures the user must perform, explaining them step by step. This allows the user to accurately understand the operating procedures and perform them efficiently. The learning support unit can adjust the format and content of the explanations. For example, the learning support unit can provide detailed explanations. The learning support unit can also improve the user's learning effectiveness by conducting operational tests in a quiz format. Specifically, the learning support unit generates quizzes based on the operations performed by the user and presents them to the user. This allows the user to review the operating procedures and deepen their understanding. The learning support unit can adjust the content and format of the quizzes. For example, the learning support unit can adjust the difficulty level and frequency of quizzes according to the user's operation history and learning status. This allows the learning support unit to maximize the user's learning effectiveness and improve their proficiency in operations. Furthermore, the learning support unit can record users' learning history and use it for later analysis and improvement. For example, the learning support unit can record users' learning history in a log file and evaluate the effectiveness and accuracy of their learning. Based on this, the learning support unit can improve the content of explanations and the way quizzes are presented, thereby enhancing learning effectiveness. In addition, the learning support unit can collect user feedback and continuously improve the accuracy and effectiveness of explanations and quizzes. For example, based on user feedback, the learning support unit can review the content of explanations and the way quizzes are presented, thereby achieving more effective learning support. This allows the learning support unit to streamline user operations and improve their proficiency in using the system.

[0036] The operation recording unit can record the user's PC operations unit by unit. For example, the operation recording unit can record the user's PC operations by type of operation. For example, the operation recording unit can record operations such as file operations, software usage, and web browsing as separate units. The operation recording unit can also record operations by time. For example, the operation recording unit can record the user's operations every hour. Furthermore, the operation recording unit can record operations by task. For example, the operation recording unit can record operations related to a specific project as a unit. This improves the reproducibility of operations by recording the user's PC operations unit by unit.

[0037] The reproduction unit can reproduce recorded operations with high accuracy. For example, the reproduction unit can accurately reproduce recorded operations. The reproduction unit can adjust the accuracy of the reproduction. For example, the reproduction unit can minimize the error range of operations and perform accurate reproduction. The reproduction unit can also adjust the timing of the reproduction. For example, the reproduction unit can adjust the timing of operation reproduction to match the user's operating environment. Furthermore, the reproduction unit can adjust the format of the reproduction. For example, the reproduction unit can reproduce operations in a visual format. This improves the accuracy of operations by reproducing recorded operations with high accuracy.

[0038] The optimization unit can analyze recorded operations and propose more efficient procedures. For example, the optimization unit can analyze recorded operations in detail and propose efficient procedures. The optimization unit performs optimization based on criteria for operational efficiency. For example, the optimization unit can perform optimization with the aim of reducing operation time or the number of operations. The optimization unit can also propose efficient procedures using optimization algorithms. For example, the optimization unit can learn operation patterns and propose the optimal procedure. Furthermore, the optimization unit can perform optimization based on the user's operation history. For example, the optimization unit can analyze past operation history and propose the most efficient procedure. As a result, by analyzing recorded operations and proposing efficient procedures, business efficiency is improved.

[0039] The voice command unit can be controlled by voice. For example, it can receive user voice commands using speech recognition technology. The voice command unit can adjust the types and methods of receiving voice commands. For example, it can prioritize certain voice commands. Furthermore, the voice command unit can learn the user's voice characteristics to improve the accuracy of speech recognition. In addition, the voice command unit can adjust how voice commands are interpreted. For example, it can adjust how voice commands are interpreted based on the user's emotions. This enables hands-free operation by controlling the device with voice commands.

[0040] The multi-device support unit can synchronize operations across different devices. For example, it can synchronize operations between devices such as PCs, smartphones, and tablets. The multi-device support unit can adjust the synchronization method and criteria between devices. For example, it can prioritize the synchronization of operations between specific devices. It can also learn the usage history of devices to improve the accuracy of synchronization between devices. Furthermore, the multi-device support unit can adjust the synchronization method between devices based on the user's emotions. For example, it can perform detailed synchronization when the user is relaxed. This improves inter-device coordination by synchronizing operations across different devices.

[0041] The operation recording unit can analyze the user's past operation history and select the optimal recording method. For example, the operation recording unit can prioritize recording operations that the user has frequently performed in the past. The operation recording unit can extract specific operation patterns from the user's operation history and adjust the recording method based on those patterns. The operation recording unit can analyze the user's operation history and propose the most efficient recording method. In this way, the optimal recording method can be selected by analyzing the user's past operation history.

[0042] The operation recording unit can filter operations based on the user's current work status and areas of interest during the recording process. For example, if a user is focused on a specific project, the operation recording unit can record only operations related to that project. The operation recording unit can prioritize recording relevant operations based on the user's areas of interest. The operation recording unit can analyze the user's work status in real time and select the optimal operation recording method. This allows for the priority recording of important operations by filtering based on the user's work status and areas of interest.

[0043] The operation recording unit can prioritize recording operations that are highly relevant, taking into account the user's geographical location information. For example, the operation recording unit can prioritize recording operations performed by the user at a specific location. The operation recording unit can filter relevant operations based on the user's geographical location information. The operation recording unit can acquire the user's location information in real time and select the optimal operation recording method. This allows for the priority recording of highly relevant operations by considering the user's geographical location information.

[0044] The operation recording unit can analyze the user's social media activity and record relevant operations during operation recording. For example, the operation recording unit can prioritize recording operations performed by the user on social media. The operation recording unit can filter relevant operations based on the user's social media activity. The operation recording unit can analyze the user's social media activity in real time and select the optimal operation recording method. This allows for the recording of relevant operations by analyzing the user's social media activity.

[0045] The reproduction unit can adjust the level of detail of the reproduction based on the importance of the operation during reproduction. For example, the reproduction unit can reproduce highly important operations in detail, while reproducing less important operations concisely. The reproduction unit can dynamically adjust the level of detail of the reproduction according to the importance of the operation. This allows important operations to be reproduced in detail by adjusting the level of detail of the reproduction based on the importance of the operation.

[0046] The reproduction unit can apply different reproduction algorithms depending on the category of operation during reproduction. For example, the reproduction unit can apply a dedicated reproduction algorithm to project management operations, a dedicated reproduction algorithm to design operations, and a dedicated reproduction algorithm to coding operations. By applying different reproduction algorithms depending on the category of operation, the accuracy of reproduction is improved.

[0047] The reproduction unit can determine the reproduction priority based on when the operations were executed. For example, the reproduction unit can prioritize the reproduction of recently executed operations. The reproduction unit can postpone the reproduction of operations that were executed in the past. The reproduction unit can dynamically adjust the reproduction priority according to when the operations were executed. This allows important operations to be reproduced preferentially by determining the reproduction priority based on when the operations were executed.

[0048] The reproduction unit can adjust the order of reproduction based on the relevance of the operations during reproduction. For example, the reproduction unit can reproduce highly relevant operations consecutively. The reproduction unit can postpone less relevant operations. The reproduction unit can dynamically adjust the order of reproduction according to the relevance of the operations. This allows related operations to be reproduced consecutively by adjusting the order of reproduction based on the relevance of the operations.

[0049] The optimization unit can improve the accuracy of optimization by considering the interrelationships between operations during the optimization process. For example, the optimization unit can analyze the interrelationships between operations and propose the optimal procedure. Based on the interrelationships between operations, the optimization unit can perform efficient optimization. The optimization unit can improve the accuracy of optimization by considering the interrelationships between operations. Thus, considering the interrelationships between operations improves the accuracy of optimization.

[0050] The optimization unit can perform optimization while considering the attribute information of the person performing the operation. For example, the optimization unit can propose the optimal procedure according to the skill level of the person performing the operation. The optimization unit can perform efficient optimization according to the job title of the person performing the operation. The optimization unit can improve the accuracy of optimization by considering the attribute information of the person performing the operation. As a result, the accuracy of optimization is improved by considering the attribute information of the person performing the operation.

[0051] The optimization unit can perform optimization while considering the geographical distribution of operations. For example, the optimization unit can analyze the geographical distribution of operations and propose the optimal procedure. The optimization unit can perform efficient optimization based on the geographical distribution of operations. The optimization unit can improve the accuracy of optimization by considering the geographical distribution of operations. As a result, the accuracy of optimization is improved by considering the geographical distribution of operations.

[0052] The optimization unit can improve the accuracy of optimization by referring to relevant documentation for the operation during the optimization process. For example, the optimization unit can refer to relevant documentation for the operation and propose the optimal procedure. The optimization unit can perform efficient optimization based on relevant documentation for the operation. The optimization unit can improve the accuracy of optimization by considering relevant documentation for the operation. As a result, the accuracy of optimization is improved by referring to relevant documentation for the operation.

[0053] The voice command unit can interpret a voice command optimally by referring to the user's past voice command history when it receives a voice command. For example, the voice command unit can prioritize interpreting voice commands that the user has used in the past. The voice command unit can extract specific patterns from the user's past voice command history and interpret the command based on those patterns. The voice command unit can analyze the user's voice command history and propose the most efficient interpretation method. This makes it possible to perform optimal interpretation by referring to the user's past voice command history.

[0054] The voice command unit can determine the priority of voice commands based on the user's current work status when it receives a voice command. For example, if the user is concentrating on a particular task, the voice command unit can prioritize receiving voice commands related to that task. The voice command unit can filter relevant voice commands based on the user's work status. The voice command unit can analyze the user's work status in real time and determine the optimal priority of voice commands. This enables efficient operation by prioritizing voice commands based on the user's current work status.

[0055] The voice command unit can provide the optimal response when receiving a voice command, taking into account the user's geographical location. For example, if the user is in a specific location, the voice command unit can prioritize providing information related to that location. The voice command unit can filter relevant responses based on the user's geographical location. The voice command unit can acquire the user's location information in real time and select the optimal response method. This enables the optimal response by considering the user's geographical location.

[0056] The voice command unit can analyze the user's social media activity and provide relevant responses upon receiving a voice command. For example, it can provide relevant responses based on the user's activities on social media. The voice command unit can filter relevant information based on the user's social media activity. The voice command unit can analyze the user's social media activity in real time and select the optimal response method. This enables relevant responses by analyzing the user's social media activity.

[0057] The multi-device support unit can select the optimal synchronization method by referring to the user's past device usage history during device synchronization. For example, the multi-device support unit can prioritize synchronization of devices the user has used in the past. The multi-device support unit can extract specific patterns from the user's device usage history and adjust the synchronization method based on those patterns. The multi-device support unit can analyze the user's device usage history and propose the most efficient synchronization method. This allows for the selection of the optimal synchronization method by referring to the user's past device usage history.

[0058] The multi-device support unit can determine synchronization priorities based on the user's current work status during device synchronization. For example, if a user is concentrating on a particular task, the multi-device support unit can prioritize the synchronization of devices related to that task. The multi-device support unit can filter related devices based on the user's work status. The multi-device support unit can analyze the user's work status in real time and determine the optimal synchronization priority. This enables efficient synchronization by determining synchronization priorities based on the user's current work status.

[0059] The multi-device support unit can select the optimal synchronization method when synchronizing devices, taking into account the user's geographical location information. For example, if the user is in a specific location, the multi-device support unit can prioritize synchronizing devices associated with that location. The multi-device support unit can filter related devices based on the user's geographical location information. The multi-device support unit can acquire the user's location information in real time and select the optimal synchronization method. This allows for the selection of the optimal synchronization method by considering the user's geographical location information.

[0060] The multi-device support unit can analyze the user's social media activity and perform relevant synchronization when synchronizing devices. For example, the multi-device support unit can synchronize related devices based on the user's social media activities. The multi-device support unit can filter relevant information based on the user's social media activities. The multi-device support unit can analyze the user's social media activity in real time and select the optimal synchronization method. This enables relevant synchronization by analyzing the user's social media activity.

[0061] The learning support unit can provide optimal explanations by referring to the user's past learning history when explaining operating procedures. For example, the learning support unit can provide optimal explanations based on what the user has learned in the past. The learning support unit can extract specific patterns from the user's learning history and provide explanations based on those patterns. The learning support unit can analyze the user's learning history and propose the most efficient explanation method. In this way, the optimal explanation can be provided by referring to the user's past learning history.

[0062] The learning support unit can prioritize explanations of operating procedures based on the user's current learning status. For example, if the user is focused on a specific operation, the learning support unit can prioritize providing explanations related to that operation. The learning support unit can filter relevant explanations based on the user's learning status. The learning support unit can analyze the user's learning status in real time and determine the optimal explanation priority. This enables efficient learning by prioritizing explanations based on the user's current learning status.

[0063] The learning support unit can provide optimal explanations of operating procedures by considering the user's geographical location. For example, if the user is in a specific location, the learning support unit can prioritize explaining operating procedures related to that location. The learning support unit can filter relevant explanations based on the user's geographical location. The learning support unit can acquire the user's location information in real time and select the optimal explanation method. This allows for the provision of optimal explanations by considering the user's geographical location.

[0064] The learning support unit can analyze the user's social media activity and provide relevant explanations when explaining operating procedures. For example, the learning support unit can explain relevant operating procedures based on the user's activities on social media. The learning support unit can filter relevant information based on the user's social media activity. The learning support unit can analyze the user's social media activity in real time and select the optimal explanation method. As a result, relevant explanations can be provided by analyzing the user's social media activity.

[0065] The system according to the embodiment is not limited to the example described above, and various modifications are possible, for example, as follows.

[0066] Assistant software can analyze a user's past operation history and suggest the optimal procedure. For example, it can prioritize suggesting operations that the user has frequently performed in the past, supporting efficient work. It can also extract specific patterns from the user's operation history and suggest the optimal procedure based on those patterns. Furthermore, it can analyze the user's operation history and suggest the most efficient procedure. In this way, by analyzing the user's past operation history, it can suggest the optimal procedure and support efficient work.

[0067] Assistant software can adjust operating procedures based on the user's current work status. For example, if a user is focused on a specific project, it can prioritize providing operating procedures related to that project. It can also filter relevant operating procedures based on the user's work status. Furthermore, it can analyze the user's work status in real time and suggest the optimal operating procedures. In this way, by adjusting operating procedures based on the user's current work status, it can support efficient work.

[0068] Assistant software can adjust operating procedures based on the user's geographical location. For example, if the user is in a specific location, it can prioritize providing operating procedures relevant to that location. It can also filter relevant operating procedures based on the user's geographical location. Furthermore, it can acquire the user's location information in real time and suggest the most suitable operating procedures. In this way, by considering the user's geographical location, it can provide optimal operating procedures and support efficient work.

[0069] Assistant software can analyze a user's social media activity and provide relevant instructions. For example, it can prioritize providing relevant instructions based on the user's social media activities. It can also filter relevant information based on the user's social media activity. Furthermore, it can analyze the user's social media activity in real time and suggest the optimal instructions. In this way, by analyzing the user's social media activity, it can provide relevant instructions and support efficient work.

[0070] The following briefly describes the processing flow for example form 1.

[0071] Step 1: The operation recording unit records the user's PC operations. The operation recording unit can record operations such as file operations, software usage, and web browsing performed by the user. The operation recording unit records based on the frequency and type of operations. For example, the operation recording unit can prioritize recording frequently performed operations. The operation recording unit can also adjust the recording method according to the type of operation. For example, the operation recording unit can record file operations in detail and software usage concisely. Step 2: The reproduction unit reproduces the operations recorded by the operation recording unit. The reproduction unit can reproduce the recorded operations with high accuracy, for example. The reproduction unit can adjust the accuracy and timing of the reproduction. For example, the reproduction unit can minimize the error range of the operations and perform accurate reproduction. In addition, the reproduction unit can adjust the timing of the operation reproduction to match the user's operating environment. Step 3: The optimization unit optimizes the operations reproduced by the reproduction unit. For example, the optimization unit can analyze the recorded operations and propose a more efficient procedure. The optimization unit performs optimization based on efficiency criteria and algorithms. For example, the optimization unit can optimize to reduce the time or number of operations performed. The optimization unit can also learn the user's operation patterns and propose the optimal procedure. Step 4: The voice command unit provides voice commands. The voice command unit can receive user voice commands, for example, using speech recognition technology. The voice command unit can adjust the types of voice commands it receives and how it receives them. For example, it can prioritize certain voice commands. The voice command unit can also learn the user's voice characteristics to improve the accuracy of speech recognition. Step 5: The multi-device support unit synchronizes operations between different devices. The multi-device support unit can synchronize operations between devices such as a PC, smartphone, and tablet. The multi-device support unit can adjust the synchronization method and criteria between devices. For example, the multi-device support unit can prioritize the synchronization of operations between specific devices. In addition, the multi-device support unit can learn the usage history of devices to improve the accuracy of synchronization between devices. Step 6: The learning support unit generates explanations of the operating procedures and conducts operational tests in a quiz format. The learning support unit can, for example, generate explanations of the operating procedures and provide them to the user. The learning support unit can adjust the format and content of the explanations. For example, the learning support unit can provide detailed explanations. The learning support unit can also conduct operational tests in a quiz format to improve the user's learning effectiveness.

[0072] (Example of form 2) The assistant software according to an embodiment of the present invention is a generation AI-equipped assistant software that learns the user's PC operations and automates tasks. This assistant software improves work efficiency and provides learning support by learning the user's PC operations and automating tasks. For example, the assistant software's operation recording unit records the user's PC operations unit by unit and reproduces those operations with high accuracy. Next, the assistant software's optimization unit analyzes the recorded operations and proposes more efficient procedures. Furthermore, the assistant software's voice command unit gives voice instructions for operations, and the multi-device support unit synchronizes operations between different devices. Finally, the assistant software's learning support unit generates explanations of the operation procedures and conducts operation tests in a quiz format. As a result, the user can proceed with work efficiently and learn the operation procedures. Thus, the assistant software can efficiently record, reproduce, optimize, accept voice commands, synchronize between devices, and provide learning support for the user's PC operations.

[0073] The assistant software according to this embodiment comprises an operation recording unit, a reproduction unit, an optimization unit, a voice command unit, a multi-device compatibility unit, and a learning support unit. The operation recording unit records the user's PC operations. The operation recording unit can record operations such as file operations, software usage, and web browsing performed by the user. The operation recording unit records based on the frequency and type of operations. For example, the operation recording unit can prioritize recording frequently performed operations. The operation recording unit can also adjust the recording method according to the type of operation. For example, the operation recording unit can record file operations in detail and software usage concisely. The reproduction unit reproduces the operations recorded by the operation recording unit. The reproduction unit can reproduce recorded operations with high accuracy. The reproduction unit can adjust the accuracy and timing of reproduction. For example, the reproduction unit can minimize the error range of operations and perform accurate reproduction. The reproduction unit can also adjust the timing of operation reproduction to match the user's operating environment. The optimization unit optimizes the operations reproduced by the reproduction unit. The optimization unit can, for example, analyze recorded operations and propose more efficient procedures. The optimization unit performs optimization based on efficiency criteria and algorithms for operations. For example, the optimization unit can optimize operations with the aim of reducing operation time and the number of operations. The optimization unit can also learn the user's operation patterns and propose the optimal procedure. The voice command unit gives instructions for operations by voice. The voice command unit can, for example, accept the user's voice commands using speech recognition technology. The voice command unit can adjust the types of voice commands and how they are accepted. For example, the voice command unit can prioritize the acceptance of specific voice commands. The voice command unit can also learn the user's voice characteristics to improve the accuracy of speech recognition. The multi-device support unit synchronizes operations between different devices. For example, the multi-device support unit can synchronize operations between devices such as PCs, smartphones, and tablets. The multi-device support unit can adjust the synchronization methods and criteria between devices.For example, the multi-device support unit can prioritize the synchronization of operations between specific devices. The multi-device support unit can also learn the usage history of devices to improve the synchronization accuracy between devices. The learning support unit generates explanations of operating procedures and conducts operation tests in a quiz format. For example, the learning support unit can generate and provide explanations of operating procedures to the user. The learning support unit can adjust the format and content of the explanations. For example, the learning support unit can provide detailed explanations. Furthermore, the learning support unit can conduct operation tests in a quiz format to improve the user's learning effectiveness. As a result, the assistant software according to this embodiment can efficiently record, reproduce, optimize, accept voice commands, synchronize between devices, and provide learning support for the user's PC operations.

[0074] The operation log unit records the user's PC operations. For example, it can record operations such as file operations, software usage, and web browsing. Specifically, the operation log unit records in detail operations such as opening, saving, and moving files. This includes file paths, operation timestamps, and operation types. Regarding software usage, it records which software was launched, when, and what operations were performed. For web browsing, it records information such as accessed URLs, browsing time, and clicked links. The operation log unit records based on the frequency and type of operations. For example, it can prioritize recording frequently performed operations. This allows for focused recording of operations frequently performed by the user, which can be useful for later analysis and reproduction. Furthermore, the operation log unit can adjust its recording method depending on the type of operation. For example, it can record file operations in detail and software usage concisely. This allows for efficient management of recorded data by recording important operations in detail and concisely when necessary. The operation recording unit can also learn the user's operation patterns and improve the accuracy and efficiency of recording. For example, the operation recording unit can learn the frequency and timing of specific operations performed by the user and automatically select the optimal recording method. This allows the operation recording unit to efficiently and accurately record the user's operations, which can then be used for later reproduction and optimization.

[0075] The reproduction unit reproduces the operations recorded by the operation recording unit. The reproduction unit can reproduce recorded operations with high accuracy. Specifically, the reproduction unit accurately reproduces operations such as file operations, software usage, and web browsing recorded by the operation recording unit. The reproduction unit can adjust the accuracy and timing of the reproduction. For example, the reproduction unit can minimize the error range of the operation and perform accurate reproduction. The reproduction unit can also adjust the timing of the operation reproduction to match the user's operating environment. The reproduction unit can optimize the accuracy and timing of the reproduction according to the user's operating environment and usage. For example, the reproduction unit can adjust the reproduction method according to the device and software version used by the user. This allows the reproduction unit to perform optimal reproduction for the user's operating environment, improving the accuracy and efficiency of the operation. The reproduction unit can also detect errors and problems that occur during operation reproduction and take appropriate countermeasures. For example, the reproduction unit can detect error messages and warnings that occur during operation reproduction and notify the user. This allows the reproduction unit to quickly resolve problems that occur during operation reproduction and maintain the accuracy and efficiency of the operation. Furthermore, the reproduction unit can record the results of the operation reproduction, which can then be used for later analysis and optimization. For example, the reproduction unit can record the results of the operation reproduction in a log file and evaluate the accuracy and efficiency of the reproduction. Based on these results, the reproduction unit can improve the reproduction method and algorithm, thereby improving the accuracy and efficiency of the operation.

[0076] The optimization unit optimizes the operations reproduced by the reproduction unit. For example, the optimization unit can analyze recorded operations and propose more efficient procedures. Specifically, the optimization unit performs optimization based on efficiency criteria and algorithms for operations. For example, the optimization unit can optimize operations with the aim of reducing operation time and the number of operations. The optimization unit can also learn user operation patterns and propose optimal procedures. For example, the optimization unit can analyze operations frequently performed by the user and propose more efficient procedures. This allows the optimization unit to streamline user operations, reduce work time, and simplify operations. Furthermore, the optimization unit can record the optimization results of operations and use them for later analysis and improvement. For example, the optimization unit can record the optimization results of operations in a log file and evaluate the effectiveness and accuracy of the optimization. This allows the optimization unit to improve the optimization method and algorithm based on the optimization results, thereby improving the efficiency and accuracy of operations. In addition, the optimization unit can collect user feedback and continuously improve the accuracy and effectiveness of the optimization. For example, based on user feedback, the optimization unit can review the optimization procedures and proposals to achieve more effective optimization. This allows the optimization unit to streamline user operations and improve work productivity.

[0077] The voice command unit provides voice commands. For example, it can receive user voice commands using speech recognition technology. Specifically, the voice command unit analyzes user voice commands in real time and executes the corresponding operations. The voice command unit can adjust the types and methods of receiving voice commands. For example, it can prioritize certain voice commands. Furthermore, the voice command unit can learn the user's voice characteristics to improve the accuracy of speech recognition. This allows the voice command unit to accurately recognize user voice commands and quickly execute the corresponding operations. The voice command unit can also analyze user voice commands and suggest appropriate operations. For example, it can analyze user voice commands and suggest the optimal operation procedure. This allows the voice command unit to streamline user operations and improve work productivity. Additionally, the voice command unit can record the history of received voice commands for later analysis and improvement. For example, it can record the history of received voice commands in a log file to evaluate the accuracy and effectiveness of speech recognition. This allows the voice command unit to improve its speech recognition technology and algorithms based on the history of received voice commands, thereby enhancing the accuracy and efficiency of voice commands. Furthermore, the voice command unit can collect user feedback and continuously improve the accuracy and effectiveness of voice commands. For example, based on user feedback, the voice command unit can review how voice commands are received and what suggestions are made, resulting in more effective voice commands. This allows the voice command unit to streamline user operations and improve work productivity.

[0078] The multi-device support unit synchronizes operations across different devices. For example, it can synchronize operations between devices such as PCs, smartphones, and tablets. Specifically, it can reflect operations performed by a user on a PC on their smartphone or tablet. This allows users to operate seamlessly across different devices. The multi-device support unit can adjust the synchronization methods and criteria between devices. For example, it can prioritize the synchronization of operations between specific devices. Furthermore, the multi-device support unit can learn device usage history to improve the accuracy of synchronization between devices. This allows the multi-device support unit to accurately synchronize user operations and maintain consistency across different devices. The multi-device support unit can record user operation history for later analysis and improvement. For example, it can record the operation history between devices in a log file to evaluate the accuracy and effectiveness of synchronization. Based on this operation history, the multi-device support unit can improve the synchronization methods and algorithms, thereby increasing the accuracy and efficiency of synchronization between devices. Furthermore, the multi-device support unit can collect user feedback and continuously improve the accuracy and effectiveness of synchronization. For example, based on user feedback, the multi-device support unit can review the synchronization method and suggestions to achieve more effective synchronization. This allows the multi-device support unit to streamline user operations and maintain consistency across different devices.

[0079] The learning support unit generates explanations of operating procedures and conducts operational tests in a quiz format. For example, the learning support unit can generate and provide explanations of operating procedures to the user. Specifically, the learning support unit provides detailed explanations of the operating procedures the user must perform, explaining them step by step. This allows the user to accurately understand the operating procedures and perform them efficiently. The learning support unit can adjust the format and content of the explanations. For example, the learning support unit can provide detailed explanations. The learning support unit can also improve the user's learning effectiveness by conducting operational tests in a quiz format. Specifically, the learning support unit generates quizzes based on the operations performed by the user and presents them to the user. This allows the user to review the operating procedures and deepen their understanding. The learning support unit can adjust the content and format of the quizzes. For example, the learning support unit can adjust the difficulty level and frequency of quizzes according to the user's operation history and learning status. This allows the learning support unit to maximize the user's learning effectiveness and improve their proficiency in operations. Furthermore, the learning support unit can record users' learning history and use it for later analysis and improvement. For example, the learning support unit can record users' learning history in a log file and evaluate the effectiveness and accuracy of their learning. Based on this, the learning support unit can improve the content of explanations and the way quizzes are presented, thereby enhancing learning effectiveness. In addition, the learning support unit can collect user feedback and continuously improve the accuracy and effectiveness of explanations and quizzes. For example, based on user feedback, the learning support unit can review the content of explanations and the way quizzes are presented, thereby achieving more effective learning support. This allows the learning support unit to streamline user operations and improve their proficiency in using the system.

[0080] The operation recording unit can record the user's PC operations unit by unit. For example, the operation recording unit can record the user's PC operations by type of operation. For example, the operation recording unit can record operations such as file operations, software usage, and web browsing as separate units. The operation recording unit can also record operations by time. For example, the operation recording unit can record the user's operations every hour. Furthermore, the operation recording unit can record operations by task. For example, the operation recording unit can record operations related to a specific project as a unit. This improves the reproducibility of operations by recording the user's PC operations unit by unit.

[0081] The reproduction unit can reproduce recorded operations with high accuracy. For example, the reproduction unit can accurately reproduce recorded operations. The reproduction unit can adjust the accuracy of the reproduction. For example, the reproduction unit can minimize the error range of operations and perform accurate reproduction. The reproduction unit can also adjust the timing of the reproduction. For example, the reproduction unit can adjust the timing of operation reproduction to match the user's operating environment. Furthermore, the reproduction unit can adjust the format of the reproduction. For example, the reproduction unit can reproduce operations in a visual format. This improves the accuracy of operations by reproducing recorded operations with high accuracy.

[0082] The optimization unit can analyze recorded operations and propose more efficient procedures. For example, the optimization unit can analyze recorded operations in detail and propose efficient procedures. The optimization unit performs optimization based on criteria for operational efficiency. For example, the optimization unit can perform optimization with the aim of reducing operation time or the number of operations. The optimization unit can also propose efficient procedures using optimization algorithms. For example, the optimization unit can learn operation patterns and propose the optimal procedure. Furthermore, the optimization unit can perform optimization based on the user's operation history. For example, the optimization unit can analyze past operation history and propose the most efficient procedure. As a result, by analyzing recorded operations and proposing efficient procedures, business efficiency is improved.

[0083] The voice command unit can be controlled by voice. For example, it can receive user voice commands using speech recognition technology. The voice command unit can adjust the types and methods of receiving voice commands. For example, it can prioritize certain voice commands. Furthermore, the voice command unit can learn the user's voice characteristics to improve the accuracy of speech recognition. In addition, the voice command unit can adjust how voice commands are interpreted. For example, it can adjust how voice commands are interpreted based on the user's emotions. This enables hands-free operation by controlling the device with voice commands.

[0084] The multi-device support unit can synchronize operations across different devices. For example, it can synchronize operations between devices such as PCs, smartphones, and tablets. The multi-device support unit can adjust the synchronization method and criteria between devices. For example, it can prioritize the synchronization of operations between specific devices. It can also learn the usage history of devices to improve the accuracy of synchronization between devices. Furthermore, the multi-device support unit can adjust the synchronization method between devices based on the user's emotions. For example, it can perform detailed synchronization when the user is relaxed. This improves inter-device coordination by synchronizing operations across different devices.

[0085] The learning support unit can generate explanations of operating procedures and conduct operational tests in a quiz format. For example, the learning support unit can generate and provide explanations of operating procedures to the user. The learning support unit can adjust the format and content of the explanations. For example, the learning support unit can provide detailed explanations. The learning support unit can also conduct operational tests in a quiz format to improve the user's learning effectiveness. Furthermore, the learning support unit can adjust the method of explaining operating procedures based on the user's emotions. For example, if the user is relaxed, the learning support unit can provide detailed explanations. If the user is in a hurry, the learning support unit can provide concise explanations. If the user is stressed, the learning support unit can adjust the explanation method to reduce the user's burden. As a result, generating explanations of operating procedures and conducting operational tests in a quiz format improves learning effectiveness.

[0086] The operation recording unit can estimate the user's emotions and adjust the timing of operation recording based on the estimated emotions. For example, if the user is concentrating, the operation recording unit can increase the frequency of operation recording and record more detailed information. If the user is tired, the operation recording unit can decrease the frequency of operation recording and record only important operations. If the user is stressed, the operation recording unit can adjust the timing of operation recording to reduce the user's burden. In this way, the user's burden is reduced by adjusting the timing of operation recording based on the user's emotions. Emotion estimation is achieved using an emotion estimation function, for example, using an emotion engine or generative AI. Generative AI is, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI.

[0087] The operation recording unit can analyze the user's past operation history and select the optimal recording method. For example, the operation recording unit can prioritize recording operations that the user has frequently performed in the past. The operation recording unit can extract specific operation patterns from the user's operation history and adjust the recording method based on those patterns. The operation recording unit can analyze the user's operation history and propose the most efficient recording method. In this way, the optimal recording method can be selected by analyzing the user's past operation history.

[0088] The operation recording unit can filter operations based on the user's current work status and areas of interest during the recording process. For example, if a user is focused on a specific project, the operation recording unit can record only operations related to that project. The operation recording unit can prioritize recording relevant operations based on the user's areas of interest. The operation recording unit can analyze the user's work status in real time and select the optimal operation recording method. This allows for the priority recording of important operations by filtering based on the user's work status and areas of interest.

[0089] The operation recording unit can estimate the user's emotions and determine the priority of operations to record based on the estimated emotions. For example, if the user is relaxed, the operation recording unit can record operations of low importance. If the user is in a hurry, the operation recording unit can prioritize recording only high-importance operations. If the user is stressed, the operation recording unit can adjust the priority of operations to reduce the user's burden. In this way, the user's burden is reduced by determining the priority of operations based on the user's emotions. Emotion estimation is achieved using an emotion estimation function, for example, using an emotion engine or generative AI. Generative AI is, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI.

[0090] The operation recording unit can prioritize recording operations that are highly relevant, taking into account the user's geographical location information. For example, the operation recording unit can prioritize recording operations performed by the user at a specific location. The operation recording unit can filter relevant operations based on the user's geographical location information. The operation recording unit can acquire the user's location information in real time and select the optimal operation recording method. This allows for the priority recording of highly relevant operations by considering the user's geographical location information.

[0091] The operation recording unit can analyze the user's social media activity and record relevant operations during operation recording. For example, the operation recording unit can prioritize recording operations performed by the user on social media. The operation recording unit can filter relevant operations based on the user's social media activity. The operation recording unit can analyze the user's social media activity in real time and select the optimal operation recording method. This allows for the recording of relevant operations by analyzing the user's social media activity.

[0092] The reenactment unit can estimate the user's emotions and adjust the reenactment's presentation based on the estimated emotions. For example, if the user is relaxed, the reenactment unit can provide a detailed reenactment. If the user is in a hurry, the reenactment unit can provide a concise reenactment that gets straight to the point. If the user is stressed, the reenactment unit can adjust the presentation of the reenactment to reduce the user's burden. In this way, the user's burden is reduced by adjusting the presentation of the reenactment based on the user's emotions. Emotion estimation is achieved using an emotion estimation function, for example, using an emotion engine or generative AI. Generative AI is, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI.

[0093] The reproduction unit can adjust the level of detail of the reproduction based on the importance of the operation during reproduction. For example, the reproduction unit can reproduce highly important operations in detail, while reproducing less important operations concisely. The reproduction unit can dynamically adjust the level of detail of the reproduction according to the importance of the operation. This allows important operations to be reproduced in detail by adjusting the level of detail of the reproduction based on the importance of the operation.

[0094] The reproduction unit can apply different reproduction algorithms depending on the category of operation during reproduction. For example, the reproduction unit can apply a dedicated reproduction algorithm to project management operations, a dedicated reproduction algorithm to design operations, and a dedicated reproduction algorithm to coding operations. By applying different reproduction algorithms depending on the category of operation, the accuracy of reproduction is improved.

[0095] The replay unit can estimate the user's emotions and adjust the length of the replay based on the estimated emotions. For example, if the user is relaxed, the replay unit can perform a longer replay. If the user is in a hurry, the replay unit can perform a shorter replay. If the user is stressed, the replay unit can adjust the length of the replay to reduce the user's burden. In this way, the user's burden is reduced by adjusting the length of the replay based on the user's emotions. Emotion estimation is achieved using an emotion estimation function, for example, using an emotion engine or generative AI. Generative AI is, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI.

[0096] The reproduction unit can determine the reproduction priority based on when the operations were executed. For example, the reproduction unit can prioritize the reproduction of recently executed operations. The reproduction unit can postpone the reproduction of operations that were executed in the past. The reproduction unit can dynamically adjust the reproduction priority according to when the operations were executed. This allows important operations to be reproduced preferentially by determining the reproduction priority based on when the operations were executed.

[0097] The reproduction unit can adjust the order of reproduction based on the relevance of the operations during reproduction. For example, the reproduction unit can reproduce highly relevant operations consecutively. The reproduction unit can postpone less relevant operations. The reproduction unit can dynamically adjust the order of reproduction according to the relevance of the operations. This allows related operations to be reproduced consecutively by adjusting the order of reproduction based on the relevance of the operations.

[0098] The optimization unit can estimate the user's emotions and adjust the optimization criteria based on the estimated emotions. For example, if the user is relaxed, the optimization unit can perform detailed optimization. If the user is in a hurry, the optimization unit can perform concise optimization. If the user is stressed, the optimization unit can adjust the optimization criteria to reduce the user's burden. In this way, the user's burden is reduced by adjusting the optimization criteria based on the user's emotions. Emotion estimation is achieved using an emotion estimation function, for example, using an emotion engine or generative AI. Generative AI is, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI.

[0099] The optimization unit can improve the accuracy of optimization by considering the interrelationships between operations during the optimization process. For example, the optimization unit can analyze the interrelationships between operations and propose the optimal procedure. Based on the interrelationships between operations, the optimization unit can perform efficient optimization. The optimization unit can improve the accuracy of optimization by considering the interrelationships between operations. Thus, considering the interrelationships between operations improves the accuracy of optimization.

[0100] The optimization unit can perform optimization while considering the attribute information of the person performing the operation. For example, the optimization unit can propose the optimal procedure according to the skill level of the person performing the operation. The optimization unit can perform efficient optimization according to the job title of the person performing the operation. The optimization unit can improve the accuracy of optimization by considering the attribute information of the person performing the operation. As a result, the accuracy of optimization is improved by considering the attribute information of the person performing the operation.

[0101] The optimization unit can estimate the user's emotions and adjust the order in which the optimization results are displayed based on the estimated emotions. For example, if the user is relaxed, the optimization unit can prioritize displaying detailed results. If the user is in a hurry, the optimization unit can prioritize displaying concise results. If the user is stressed, the optimization unit can adjust the optimization results to reduce the user's burden. In this way, the user's burden is reduced by adjusting the order in which the optimization results are displayed based on the user's emotions. Emotion estimation is achieved using an emotion estimation function, for example, using an emotion engine or generative AI. Generative AI is, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI.

[0102] The optimization unit can perform optimization while considering the geographical distribution of operations. For example, the optimization unit can analyze the geographical distribution of operations and propose the optimal procedure. The optimization unit can perform efficient optimization based on the geographical distribution of operations. The optimization unit can improve the accuracy of optimization by considering the geographical distribution of operations. As a result, the accuracy of optimization is improved by considering the geographical distribution of operations.

[0103] The optimization unit can improve the accuracy of optimization by referring to relevant documentation for the operation during the optimization process. For example, the optimization unit can refer to relevant documentation for the operation and propose the optimal procedure. The optimization unit can perform efficient optimization based on relevant documentation for the operation. The optimization unit can improve the accuracy of optimization by considering relevant documentation for the operation. As a result, the accuracy of optimization is improved by referring to relevant documentation for the operation.

[0104] The voice command unit can estimate the user's emotions and adjust the interpretation of voice commands based on the estimated emotions. For example, if the user is relaxed, the voice command unit can provide a detailed interpretation. If the user is in a hurry, the voice command unit can provide a concise interpretation. If the user is stressed, the voice command unit can adjust the interpretation of voice commands to reduce the user's burden. In this way, the user's burden is reduced by adjusting the interpretation of voice commands based on the user's emotions. Emotion estimation is achieved using an emotion estimation function, for example, using an emotion engine or generative AI. Generative AI is, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI.

[0105] The voice command unit can interpret a voice command optimally by referring to the user's past voice command history when it receives a voice command. For example, the voice command unit can prioritize interpreting voice commands that the user has used in the past. The voice command unit can extract specific patterns from the user's past voice command history and interpret the command based on those patterns. The voice command unit can analyze the user's voice command history and propose the most efficient interpretation method. This makes it possible to perform optimal interpretation by referring to the user's past voice command history.

[0106] The voice command unit can determine the priority of voice commands based on the user's current work status when it receives a voice command. For example, if the user is concentrating on a particular task, the voice command unit can prioritize receiving voice commands related to that task. The voice command unit can filter relevant voice commands based on the user's work status. The voice command unit can analyze the user's work status in real time and determine the optimal priority of voice commands. This enables efficient operation by prioritizing voice commands based on the user's current work status.

[0107] The voice command unit can estimate the user's emotions and adjust the response method of voice commands based on the estimated emotions. For example, if the user is relaxed, the voice command unit can provide a detailed response. If the user is in a hurry, the voice command unit can provide a concise response. If the user is stressed, the voice command unit can adjust the response method of voice commands to reduce the user's burden. In this way, the user's burden is reduced by adjusting the response method of voice commands based on the user's emotions. Emotion estimation is achieved using an emotion estimation function, for example, using an emotion engine or generative AI. Generative AI is, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI.

[0108] The voice command unit can provide the optimal response when receiving a voice command, taking into account the user's geographical location. For example, if the user is in a specific location, the voice command unit can prioritize providing information related to that location. The voice command unit can filter relevant responses based on the user's geographical location. The voice command unit can acquire the user's location information in real time and select the optimal response method. This enables the optimal response by considering the user's geographical location.

[0109] The voice command unit can analyze the user's social media activity and provide relevant responses upon receiving a voice command. For example, it can provide relevant responses based on the user's activities on social media. The voice command unit can filter relevant information based on the user's social media activity. The voice command unit can analyze the user's social media activity in real time and select the optimal response method. This enables relevant responses by analyzing the user's social media activity.

[0110] The multi-device support unit can estimate the user's emotions and adjust the synchronization method between devices based on the estimated emotions. For example, if the user is relaxed, the multi-device support unit can perform detailed synchronization. If the user is in a hurry, the multi-device support unit can perform concise synchronization. If the user is stressed, the multi-device support unit can adjust the synchronization method between devices to reduce the user's burden. In this way, the user's burden is reduced by adjusting the synchronization method between devices based on the user's emotions. Emotion estimation is achieved using an emotion estimation function, for example, using an emotion engine or generative AI. Generative AI is, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI.

[0111] The multi-device support unit can select the optimal synchronization method by referring to the user's past device usage history during device synchronization. For example, the multi-device support unit can prioritize synchronization of devices the user has used in the past. The multi-device support unit can extract specific patterns from the user's device usage history and adjust the synchronization method based on those patterns. The multi-device support unit can analyze the user's device usage history and propose the most efficient synchronization method. This allows for the selection of the optimal synchronization method by referring to the user's past device usage history.

[0112] The multi-device support unit can determine synchronization priorities based on the user's current work status during device synchronization. For example, if a user is concentrating on a particular task, the multi-device support unit can prioritize the synchronization of devices related to that task. The multi-device support unit can filter related devices based on the user's work status. The multi-device support unit can analyze the user's work status in real time and determine the optimal synchronization priority. This enables efficient synchronization by determining synchronization priorities based on the user's current work status.

[0113] The multi-device support unit can estimate the user's emotions and adjust the frequency of synchronization between devices based on the estimated emotions. For example, if the user is relaxed, the multi-device support unit can synchronize frequently. If the user is in a hurry, the multi-device support unit can reduce the frequency of synchronization. If the user is stressed, the multi-device support unit can adjust the frequency of synchronization to reduce the user's burden. In this way, the user's burden is reduced by adjusting the frequency of synchronization between devices based on the user's emotions. Emotion estimation is achieved using an emotion estimation function, for example, using an emotion engine or generative AI. Generative AI is, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI.

[0114] The multi-device support unit can select the optimal synchronization method when synchronizing devices, taking into account the user's geographical location information. For example, if the user is in a specific location, the multi-device support unit can prioritize synchronizing devices associated with that location. The multi-device support unit can filter related devices based on the user's geographical location information. The multi-device support unit can acquire the user's location information in real time and select the optimal synchronization method. This allows for the selection of the optimal synchronization method by considering the user's geographical location information.

[0115] The multi-device support unit can analyze the user's social media activity and perform relevant synchronization when synchronizing devices. For example, the multi-device support unit can synchronize related devices based on the user's social media activities. The multi-device support unit can filter relevant information based on the user's social media activities. The multi-device support unit can analyze the user's social media activity in real time and select the optimal synchronization method. This enables relevant synchronization by analyzing the user's social media activity.

[0116] The learning support unit can estimate the user's emotions and adjust the explanation method of the operating procedures based on the estimated emotions. For example, if the user is relaxed, the learning support unit can provide detailed explanations. If the user is in a hurry, the learning support unit can provide concise explanations. If the user is stressed, the learning support unit can adjust the explanation method to reduce the user's burden. In this way, the user's burden is reduced by adjusting the explanation method of the operating procedures based on the user's emotions. Emotion estimation is achieved using an emotion estimation function, for example, using an emotion engine or generative AI. Generative AI is, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI.

[0117] The learning support unit can provide optimal explanations by referring to the user's past learning history when explaining operating procedures. For example, the learning support unit can provide optimal explanations based on what the user has learned in the past. The learning support unit can extract specific patterns from the user's learning history and provide explanations based on those patterns. The learning support unit can analyze the user's learning history and propose the most efficient explanation method. In this way, the optimal explanation can be provided by referring to the user's past learning history.

[0118] The learning support unit can prioritize explanations of operating procedures based on the user's current learning status. For example, if the user is focused on a specific operation, the learning support unit can prioritize providing explanations related to that operation. The learning support unit can filter relevant explanations based on the user's learning status. The learning support unit can analyze the user's learning status in real time and determine the optimal explanation priority. This enables efficient learning by prioritizing explanations based on the user's current learning status.

[0119] The learning support unit can estimate the user's emotions and adjust the difficulty level of quiz-style operational tests based on the estimated emotions. For example, if the user is relaxed, the learning support unit can provide a difficult quiz. If the user is in a hurry, the learning support unit can provide an easy quiz. If the user is stressed, the learning support unit can adjust the difficulty level of the quiz to reduce the user's burden. In this way, the user's burden is reduced by adjusting the difficulty level of quiz-style operational tests based on the user's emotions. Emotion estimation is achieved using an emotion estimation function, for example, using an emotion engine or generative AI. Generative AI is, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI.

[0120] The learning support unit can provide optimal explanations of operating procedures by considering the user's geographical location. For example, if the user is in a specific location, the learning support unit can prioritize explaining operating procedures related to that location. The learning support unit can filter relevant explanations based on the user's geographical location. The learning support unit can acquire the user's location information in real time and select the optimal explanation method. This allows for the provision of optimal explanations by considering the user's geographical location.

[0121] The learning support unit can analyze the user's social media activity and provide relevant explanations when explaining operating procedures. For example, the learning support unit can explain relevant operating procedures based on the user's activities on social media. The learning support unit can filter relevant information based on the user's social media activity. The learning support unit can analyze the user's social media activity in real time and select the optimal explanation method. As a result, relevant explanations can be provided by analyzing the user's social media activity.

[0122] The system according to the embodiment is not limited to the example described above, and various modifications are possible, for example, as follows.

[0123] Assistant software can estimate a user's emotions and adjust the priority of operations based on those emotions. For example, if a user is stressed, it can prioritize high-priority operations to reduce the user's burden. If a user is relaxed, it can provide detailed instructions to enhance learning. Furthermore, if a user is in a hurry, it can provide concise instructions to support efficient work. In this way, by adjusting the priority of operations based on the user's emotions, it can reduce the user's burden and support efficient work.

[0124] Assistant software can analyze a user's past operation history and suggest the optimal procedure. For example, it can prioritize suggesting operations that the user has frequently performed in the past, supporting efficient work. It can also extract specific patterns from the user's operation history and suggest the optimal procedure based on those patterns. Furthermore, it can analyze the user's operation history and suggest the most efficient procedure. In this way, by analyzing the user's past operation history, it can suggest the optimal procedure and support efficient work.

[0125] Assistant software can adjust operating procedures based on the user's current work status. For example, if a user is focused on a specific project, it can prioritize providing operating procedures related to that project. It can also filter relevant operating procedures based on the user's work status. Furthermore, it can analyze the user's work status in real time and suggest the optimal operating procedures. In this way, by adjusting operating procedures based on the user's current work status, it can support efficient work.

[0126] Assistant software can adjust operating procedures based on the user's geographical location. For example, if the user is in a specific location, it can prioritize providing operating procedures relevant to that location. It can also filter relevant operating procedures based on the user's geographical location. Furthermore, it can acquire the user's location information in real time and suggest the most suitable operating procedures. In this way, by considering the user's geographical location, it can provide optimal operating procedures and support efficient work.

[0127] Assistant software can analyze a user's social media activity and provide relevant instructions. For example, it can prioritize providing relevant instructions based on the user's social media activities. It can also filter relevant information based on the user's social media activity. Furthermore, it can analyze the user's social media activity in real time and suggest the optimal instructions. In this way, by analyzing the user's social media activity, it can provide relevant instructions and support efficient work.

[0128] Assistant software can estimate the user's emotions and adjust the way it explains operating procedures based on those emotions. For example, if the user is relaxed, it can provide detailed explanations. If the user is in a hurry, it can provide concise explanations. Furthermore, if the user is stressed, it can adjust the explanation method to reduce the user's burden. In this way, by adjusting the explanation method of operating procedures based on the user's emotions, it can reduce the user's burden and support efficient learning.

[0129] The assistant software can estimate the user's emotions and adjust the difficulty of quiz-style operational tests based on those emotions. For example, if the user is relaxed, it can provide a more difficult quiz. If the user is in a hurry, it can provide an easier quiz. Furthermore, if the user is stressed, the difficulty of the quiz can be adjusted to reduce the user's burden. In this way, by adjusting the difficulty of quiz-style operational tests based on the user's emotions, the burden on the user can be reduced, and efficient learning can be supported.

[0130] Assistant software can estimate the user's emotions and adjust how voice commands are interpreted based on those emotions. For example, if the user is relaxed, it can provide a detailed interpretation. If the user is in a hurry, it can provide a concise interpretation. Furthermore, if the user is stressed, it can adjust how voice commands are interpreted to reduce the user's burden. In this way, by adjusting how voice commands are interpreted based on the user's emotions, it can reduce the user's burden and support efficient operation.

[0131] Assistant software can estimate the user's emotions and adjust how it responds to voice commands based on those emotions. For example, if the user is relaxed, it can provide detailed responses. If the user is in a hurry, it can provide concise responses. Furthermore, if the user is stressed, it can adjust how it responds to voice commands to reduce the user's burden. In this way, by adjusting how it responds to voice commands based on the user's emotions, it can reduce the user's burden and support efficient operation.

[0132] The assistant software can estimate the user's emotions and adjust the synchronization method between devices based on those emotions. For example, if the user is relaxed, it can perform detailed synchronization. If the user is in a hurry, it can perform concise synchronization. Furthermore, if the user is stressed, it can adjust the synchronization method between devices to reduce the user's burden. In this way, by adjusting the synchronization method between devices based on the user's emotions, it can reduce the user's burden and support efficient operation.

[0133] The following briefly describes the processing flow for example form 2.

[0134] Step 1: The operation recording unit records the user's PC operations. The operation recording unit can record operations such as file operations, software usage, and web browsing performed by the user. The operation recording unit records based on the frequency and type of operations. For example, the operation recording unit can prioritize recording frequently performed operations. The operation recording unit can also adjust the recording method according to the type of operation. For example, the operation recording unit can record file operations in detail and software usage concisely. Step 2: The reproduction unit reproduces the operations recorded by the operation recording unit. The reproduction unit can reproduce the recorded operations with high accuracy, for example. The reproduction unit can adjust the accuracy and timing of the reproduction. For example, the reproduction unit can minimize the error range of the operations and perform accurate reproduction. In addition, the reproduction unit can adjust the timing of the operation reproduction to match the user's operating environment. Step 3: The optimization unit optimizes the operations reproduced by the reproduction unit. For example, the optimization unit can analyze the recorded operations and propose a more efficient procedure. The optimization unit performs optimization based on efficiency criteria and algorithms. For example, the optimization unit can optimize to reduce the time or number of operations performed. The optimization unit can also learn the user's operation patterns and propose the optimal procedure. Step 4: The voice command unit provides voice commands. The voice command unit can receive user voice commands, for example, using speech recognition technology. The voice command unit can adjust the types of voice commands it receives and how it receives them. For example, it can prioritize certain voice commands. The voice command unit can also learn the user's voice characteristics to improve the accuracy of speech recognition. Step 5: The multi-device support unit synchronizes operations between different devices. The multi-device support unit can synchronize operations between devices such as a PC, smartphone, and tablet. The multi-device support unit can adjust the synchronization method and criteria between devices. For example, the multi-device support unit can prioritize the synchronization of operations between specific devices. In addition, the multi-device support unit can learn the usage history of devices to improve the accuracy of synchronization between devices. Step 6: The learning support unit generates explanations of the operating procedures and conducts operational tests in a quiz format. The learning support unit can, for example, generate explanations of the operating procedures and provide them to the user. The learning support unit can adjust the format and content of the explanations. For example, the learning support unit can provide detailed explanations. The learning support unit can also conduct operational tests in a quiz format to improve the user's learning effectiveness.

[0135] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0136] Data generation model 58 is a form of so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> Examples of generative AI include text generation AI, image generation AI, and multimodal generation AI. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images (e.g., still image data or video data). The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference result in one or more data formats from audio data, text data, and image data. The data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specific processing unit 290 performs the specific processing described above using the data generation model 58. The data generation model 58 may be a fine-tuned model that outputs inference results from prompts that do not contain instructions, in which case the data generation model 58 can output inference results from prompts that do not contain instructions. In the data processing device 12, etc., there are multiple types of data generation models 58, and the data generation model 58 includes AI other than generative AI. AI other than generative AI includes, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVMs), k-means clustering, convolutional neural networks (CNNs), recurrent neural networks (RNNs), generative adversarial networks (GANs), or naive Bayes, and can perform various processes, but is not limited to these examples. Also, the AI ​​may be an AI agent. Furthermore, when the processing of each of the above parts is performed by the AI, the processing may be performed by the AI ​​in part or in whole, but is not limited to this example.Furthermore, processing performed by AI, including generative AI, may be replaced with rule-based processing, and rule-based processing may be replaced with processing performed by AI, including generative AI.

[0137] Furthermore, the processing performed by the data processing system 10 described above is carried out by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart device 14, but it may also be carried out by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart device 14. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the smart device 14 or an external device, and the smart device 14 acquires or collects information necessary for processing from the data processing device 12 or an external device.

[0138] Each of the multiple elements described above, including the operation recording unit, reproduction unit, optimization unit, voice command unit, multi-device support unit, and learning support unit, is implemented in at least one of the smart device 14 and the data processing unit 12. For example, the operation recording unit is implemented by the control unit 46A of the smart device 14 and records the user's PC operations. The reproduction unit is implemented by the specific processing unit 290 of the data processing unit 12 and reproduces the recorded operations with high accuracy. The optimization unit is implemented by the specific processing unit 290 of the data processing unit 12 and analyzes the recorded operations and proposes a more efficient procedure. The voice command unit is implemented by the control unit 46A of the smart device 14 and gives voice commands for operations. The multi-device support unit is implemented by the control unit 46A of the smart device 14 and synchronizes operations between different devices. The learning support unit is implemented by the specific processing unit 290 of the data processing unit 12 and generates explanations of operation procedures and conducts operation tests in a quiz format. The correspondence between each part and the device or control unit is not limited to the examples described above, and various modifications are possible.

[0139] [Second Embodiment] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0140] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0141] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN and / or LAN.

[0142] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0143] The microphone 238 receives voice signals from the user and accepts instructions from the user. The microphone 238 captures the voice signals from the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0144] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, which captures images of the area around the user (for example, an imaging range defined by a field of view equivalent to the field of vision of a typical healthy person).

[0145] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0146] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing by the processor 28. The storage 32 stores the specific processing program 56.

[0147] The processor 28 reads a specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 acting as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0148] Storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the user's emotions using the emotion identification model 59 and perform identification processing using the user's emotions. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotions, including but not limited to these examples. Furthermore, emotion estimation and prediction also include, for example, emotion analysis.

[0149] In the smart glasses 214, specific processing is performed by the processor 46. The storage 50 stores a specific processing program 60. The processor 46 reads the specific processing program 60 from the storage 50 and executes the read specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 acting as a control unit 46A according to the specific processing program 60 executed on the RAM 48. The smart glasses 214 also have a data generation model 58 and an emotion identification model 59, similar to the data generation model and emotion identification model 59, and can perform processing similar to that of the specific processing unit 290 using these models.

[0150] Furthermore, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 obtains processing results (such as prediction results) using the data generation model 58 by communicating with the server device that has the data generation model 58. Also, the data processing device 12 may be a server device or a terminal device owned by the user (for example, a mobile phone, robot, home appliance, etc.).

[0151] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0152] The data generation model 58 is a so-called generative AI. An example of a data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and inference data such as audio data representing speech, text data representing text, and image data representing images (e.g., still image data or video data). The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference result in one or more data formats such as audio data, text data, and image data. The data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specific processing unit 290 performs the specific processing described above using the data generation model 58. The data generation model 58 may be a fine-tuned model that outputs inference results from prompts that do not contain instructions, in which case the data generation model 58 can output inference results from prompts that do not contain instructions. In the data processing device 12, etc., there are multiple types of data generation models 58, and the data generation model 58 includes AI other than generative AI. AI other than generative AI includes, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. Also, the AI ​​may be an AI agent. Furthermore, when the processing of each part described above is performed by the AI, the processing may be performed by the AI ​​in part or in whole, but is not limited to this example. Also, processing performed by an AI including a generative AI may be replaced by rule-based processing, and rule-based processing may be replaced by processing performed by an AI including a generative AI.

[0153] The data processing system 210 according to the second embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 210 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart glasses 214, but it may also be performed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart glasses 214. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the smart glasses 214 or an external device, and the smart glasses 214 acquires or collects information necessary for processing from the data processing device 12 or an external device.

[0154] Each of the multiple elements described above, including the operation recording unit, reproduction unit, optimization unit, voice command unit, multi-device compatibility unit, and learning support unit, is implemented in at least one of the smart glasses 214 and the data processing unit 12. For example, the operation recording unit is implemented by the control unit 46A of the smart glasses 214 and records the user's PC operations. The reproduction unit is implemented by the specific processing unit 290 of the data processing unit 12 and reproduces the recorded operations with high accuracy. The optimization unit is implemented by the specific processing unit 290 of the data processing unit 12 and analyzes the recorded operations and proposes a more efficient procedure. The voice command unit is implemented by the control unit 46A of the smart glasses 214 and gives voice commands for operations. The multi-device compatibility unit is implemented by the control unit 46A of the smart glasses 214 and synchronizes operations between different devices. The learning support unit is implemented by the specific processing unit 290 of the data processing unit 12 and generates explanations of operation procedures and conducts operation tests in a quiz format. The correspondence between each part and the device or control unit is not limited to the examples described above, and various modifications are possible.

[0155] [Third Embodiment] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0156] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0157] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN and / or LAN.

[0158] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0159] The microphone 238 receives voice signals from the user and accepts instructions from the user. The microphone 238 captures the voice signals from the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0160] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, which captures images of the area around the user (for example, an imaging range defined by a field of view equivalent to the field of vision of a typical healthy person).

[0161] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0162] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0163] The processor 28 reads a specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 acting as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0164] Storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the user's emotions using the emotion identification model 59 and perform identification processing using the user's emotions. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotions, including but not limited to these examples. Furthermore, emotion estimation and prediction also include, for example, emotion analysis.

[0165] In the headset terminal 314, specific processing is performed by the processor 46. The storage 50 stores a specific program 60. The processor 46 reads the specific program 60 from the storage 50 and executes the read specific program 60 on the RAM 48. The specific processing is realized by the processor 46 acting as a control unit 46A according to the specific program 60 executed on the RAM 48. The headset terminal 314 also has a data generation model 58 and an emotion identification model 59, similar to the data generation model and emotion identification model 59, and can perform processing similar to that of the specific processing unit 290 using these models.

[0166] Furthermore, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 obtains processing results (such as prediction results) using the data generation model 58 by communicating with the server device that has the data generation model 58. Also, the data processing device 12 may be a server device or a terminal device owned by the user (for example, a mobile phone, robot, home appliance, etc.).

[0167] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0168] The data generation model 58 is a so-called generative AI. An example of a data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and inference data such as audio data representing speech, text data representing text, and image data representing images (e.g., still image data or video data). The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference result in one or more data formats such as audio data, text data, and image data. The data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specific processing unit 290 performs the specific processing described above using the data generation model 58. The data generation model 58 may be a fine-tuned model that outputs inference results from prompts that do not contain instructions, in which case the data generation model 58 can output inference results from prompts that do not contain instructions. In the data processing device 12, etc., there are multiple types of data generation models 58, and the data generation model 58 includes AI other than generative AI. AI other than generative AI includes, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. Also, the AI ​​may be an AI agent. Furthermore, when the processing of each part described above is performed by the AI, the processing may be performed by the AI ​​in part or in whole, but is not limited to this example. Also, processing performed by an AI including a generative AI may be replaced by rule-based processing, and rule-based processing may be replaced by processing performed by an AI including a generative AI.

[0169] The data processing system 310 according to the third embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 310 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the headset terminal 314, but may also be performed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the headset terminal 314. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the headset terminal 314 or an external device, and the headset terminal 314 acquires or collects information necessary for processing from the data processing device 12 or an external device.

[0170] Each of the multiple elements described above, including the operation recording unit, reproduction unit, optimization unit, voice command unit, multi-device support unit, and learning support unit, is implemented in at least one of the headset terminal 314 and the data processing unit 12. For example, the operation recording unit is implemented by the control unit 46A of the headset terminal 314 and records the user's PC operations. The reproduction unit is implemented by the specific processing unit 290 of the data processing unit 12 and reproduces the recorded operations with high accuracy. The optimization unit is implemented by the specific processing unit 290 of the data processing unit 12 and analyzes the recorded operations and proposes a more efficient procedure. The voice command unit is implemented by the control unit 46A of the headset terminal 314 and gives voice instructions for operations. The multi-device support unit is implemented by the control unit 46A of the headset terminal 314 and synchronizes operations between different devices. The learning support unit is implemented by the specific processing unit 290 of the data processing unit 12 and generates explanations of operation procedures and conducts operation tests in a quiz format. The correspondence between each part and the device or control unit is not limited to the examples described above, and various modifications are possible.

[0171] [Fourth Embodiment] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0172] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0173] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN and / or LAN.

[0174] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0175] The microphone 238 receives voice signals from the user and accepts instructions from the user. The microphone 238 captures the voice signals from the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0176] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS image sensor or CCD image sensor, which captures images of the area around the user (for example, an imaging range defined by a field of view equivalent to the field of vision of a typical healthy person).

[0177] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0178] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. The robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0179] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0180] The processor 28 reads a specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 acting as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0181] Storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the user's emotions using the emotion identification model 59 and perform identification processing using the user's emotions. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotions, including but not limited to these examples. Furthermore, emotion estimation and prediction also include, for example, emotion analysis.

[0182] In robot 414, specific processing is performed by processor 46. A specific program 60 is stored in storage 50. Processor 46 reads the specific program 60 from storage 50 and executes it on RAM 48. The specific processing is achieved by processor 46 acting as a control unit 46A according to the specific program 60 executed on RAM 48. Robot 414 also has data generation model 58 and emotion identification model 59, similar to those of the robot, and can perform processing similar to that of the specific processing unit 290 using these models.

[0183] Furthermore, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 obtains processing results (such as prediction results) using the data generation model 58 by communicating with the server device that has the data generation model 58. Also, the data processing device 12 may be a server device or a terminal device owned by the user (for example, a mobile phone, robot, home appliance, etc.).

[0184] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0185] The data generation model 58 is a so-called generative AI. An example of a data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and inference data such as audio data representing speech, text data representing text, and image data representing images (e.g., still image data or video data). The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference result in one or more data formats such as audio data, text data, and image data. The data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specific processing unit 290 performs the specific processing described above using the data generation model 58. The data generation model 58 may be a fine-tuned model that outputs inference results from prompts that do not contain instructions, in which case the data generation model 58 can output inference results from prompts that do not contain instructions. In the data processing device 12, etc., there are multiple types of data generation models 58, and the data generation model 58 includes AI other than generative AI. AI other than generative AI includes, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. Also, the AI ​​may be an AI agent. Furthermore, when the processing of each part described above is performed by the AI, the processing may be performed by the AI ​​in part or in whole, but is not limited to this example. Also, processing performed by an AI including a generative AI may be replaced by rule-based processing, and rule-based processing may be replaced by processing performed by an AI including a generative AI.

[0186] The data processing system 410 according to the fourth embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 410 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the robot 414, but it may also be performed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the robot 414. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the robot 414 or an external device, and the robot 414 acquires or collects information necessary for processing from the data processing device 12 or an external device.

[0187] Each of the multiple elements described above, including the operation recording unit, reproduction unit, optimization unit, voice command unit, multi-device compatibility unit, and learning support unit, is implemented in at least one of the robot 414 and the data processing unit 12. For example, the operation recording unit is implemented by the control unit 46A of the robot 414 and records the user's PC operations. The reproduction unit is implemented by the specific processing unit 290 of the data processing unit 12 and reproduces the recorded operations with high accuracy. The optimization unit is implemented by the specific processing unit 290 of the data processing unit 12 and analyzes the recorded operations and proposes a more efficient procedure. The voice command unit is implemented by the control unit 46A of the robot 414 and gives voice commands for operations. The multi-device compatibility unit is implemented by the control unit 46A of the robot 414 and synchronizes operations between different devices. The learning support unit is implemented by the specific processing unit 290 of the data processing unit 12 and generates explanations of operation procedures and conducts operation tests in a quiz format. The correspondence between each part and the device or control unit is not limited to the examples described above, and various modifications are possible.

[0188] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0189] Figure 9 shows the emotion map 400, in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0190] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0191] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0192] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, and motorcycles, emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated based, for example, on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0193] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0194] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0195] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing method for the specific process may be used, which includes computer 22 and multiple other computers.

[0196] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0197] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0198] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0199] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0200] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0201] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0202] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0203] Furthermore, although the above-described examples were divided into four embodiments, some or all of these embodiments may be combined. Also, the smart device 14, smart glasses 214, headset terminal 314, and robot 414 are just examples, and they may be combined, or other devices may be used. Also, although the above-described examples were divided into two embodiments, Embodiment 1 and Embodiment 2, these may be combined.

[0204] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and other things that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0205] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0206] (Note 1) An operation recording unit that records the user's PC operations, A reproduction unit that reproduces the operations recorded by the operation recording unit, An optimization unit that optimizes the operations reproduced by the reproduction unit, A voice command unit that accepts voice commands, A multi-device compatible unit that synchronizes operations across different devices, It includes a learning support unit that generates explanations of operating procedures and conducts operation tests in a quiz format. A system characterized by the following features. (Note 2) The operation recording unit is, Record the user's PC operations unit by unit. The system described in Appendix 1, characterized by the features described herein. (Note 3) The reproduction unit is, Reproduce recorded operations with high precision. The system described in Appendix 1, characterized by the features described herein. (Note 4) The optimization unit, Analyze recorded operations and suggest more efficient procedures. The system described in Appendix 1, characterized by the features described herein. (Note 5) The aforementioned voice command unit is Control the operation using voice commands. The system described in Appendix 1, characterized by the features described herein. (Note 6) The multi-device compatible unit is, Synchronize operations across different devices. The system described in Appendix 1, characterized by the features described herein. (Note 7) The aforementioned Learning Support Department Generate instructions for operation and conduct an operation test in a quiz format. The system described in Appendix 1, characterized by the features described herein. (Note 8) The operation recording unit is, It estimates the user's emotions and adjusts the timing of operation recordings based on the estimated user emotions. The system described in Appendix 1, characterized by the features described herein. (Note 9) The operation recording unit is, Analyze the user's past operation history and select the optimal recording method. The system described in Appendix 1, characterized by the features described herein. (Note 10) The operation recording unit is, When recording operations, filtering is performed based on the user's current work status and areas of interest. The system described in Appendix 1, characterized by the features described herein. (Note 11) The operation recording unit is, It estimates the user's emotions and determines the priority of actions to record based on the estimated user emotions. The system described in Appendix 1, characterized by the features described herein. (Note 12) The operation recording unit is, When recording operations, the system prioritizes recording operations that are highly relevant, taking into account the user's geographical location. The system described in Appendix 1, characterized by the features described herein. (Note 13) The operation recording unit is, During operation recording, the system analyzes the user's social media activity and records related actions. The system described in Appendix 1, characterized by the features described herein. (Note 14) The reproduction unit is, It estimates the user's emotions and adjusts the representation based on the estimated emotions. The system described in Appendix 1, characterized by the features described herein. (Note 15) The reproduction unit is, During reproduction, adjust the level of detail based on the importance of the operation. The system described in Appendix 1, characterized by the features described herein. (Note 16) The reproduction unit is, During reproduction, different reproduction algorithms are applied depending on the category of the operation. The system described in Appendix 1, characterized by the features described herein. (Note 17) The reproduction unit is, It estimates the user's emotions and adjusts the length of the retrieval based on the estimated user emotions. The system described in Appendix 1, characterized by the features described herein. (Note 18) The reproduction unit is, During reproduction, the reproduction priority is determined based on when the operations were performed. The system described in Appendix 1, characterized by the features described herein. (Note 19) The reproduction unit is, During reproduction, adjust the order of reproduction based on the relevance of the operations. The system described in Appendix 1, characterized by the features described herein. (Note 20) The optimization unit, It estimates user sentiment and adjusts optimization criteria based on the estimated user sentiment. The system described in Appendix 1, characterized by the features described herein. (Note 21) The optimization unit, During optimization, consider the interrelationships between operations to improve the accuracy of the optimization. The system described in Appendix 1, characterized by the features described herein. (Note 22) The optimization unit, During optimization, the attribute information of the person performing the operation is taken into consideration. The system described in Appendix 1, characterized by the features described herein. (Note 23) The optimization unit, It estimates the user's emotions and adjusts the order in which the optimization results are displayed based on the estimated user emotions. The system described in Appendix 1, characterized by the features described herein. (Note 24) The optimization unit, During optimization, the geographical distribution of operations is taken into consideration. The system described in Appendix 1, characterized by the features described herein. (Note 25) The optimization unit, During optimization, refer to relevant documentation to improve the accuracy of the optimization process. The system described in Appendix 1, characterized by the features described herein. (Note 26) The aforementioned voice command unit is It estimates the user's emotions and adjusts how voice commands are interpreted based on those estimated emotions. The system described in Appendix 1, characterized by the features described herein. (Note 27) The aforementioned voice command unit is When receiving a voice command, the system refers to the user's past voice command history to perform the most appropriate interpretation. The system described in Appendix 1, characterized by the features described herein. (Note 28) The aforementioned voice command unit is When a voice command is received, the system prioritizes the voice command based on the user's current work status. The system described in Appendix 1, characterized by the features described herein. (Note 29) The aforementioned voice command unit is It estimates the user's emotions and adjusts how voice commands are responded to based on those estimated emotions. The system described in Appendix 1, characterized by the features described herein. (Note 30) The aforementioned voice command unit is When receiving voice commands, the system takes the user's geographical location into consideration to provide the most appropriate response. The system described in Appendix 1, characterized by the features described herein. (Note 31) The aforementioned voice command unit is When a voice command is received, the system analyzes the user's social media activity and provides a relevant response. The system described in Appendix 1, characterized by the features described herein. (Note 32) The multi-device compatible unit is, It estimates the user's emotions and adjusts how devices synchronize based on those estimated emotions. The system described in Appendix 1, characterized by the features described herein. (Note 33) The multi-device compatible unit is, When syncing devices, the system selects the optimal syncing method by referring to the user's past device usage history. The system described in Appendix 1, characterized by the features described herein. (Note 34) The multi-device compatible unit is, When syncing between devices, the sync priority is determined based on the user's current work status. The system described in Appendix 1, characterized by the features described herein. (Note 35) The multi-device compatible unit is, It estimates the user's emotions and adjusts the frequency of synchronization between devices based on the estimated user emotions. The system described in Appendix 1, characterized by the features described herein. (Note 36) The multi-device compatible unit is, When syncing between devices, the system selects the optimal syncing method by considering the user's geographical location. The system described in Appendix 1, characterized by the features described herein. (Note 37) The multi-device compatible unit is, When syncing between devices, the system analyzes the user's social media activity and performs relevant syncing. The system described in Appendix 1, characterized by the features described herein. (Note 38) The aforementioned Learning Support Department The system estimates the user's emotions and adjusts the explanation of the operating procedures based on those estimated emotions. The system described in Appendix 1, characterized by the features described herein. (Note 39) The aforementioned Learning Support Department When explaining operating procedures, the system provides the most appropriate explanation by referring to the user's past learning history. The system described in Appendix 1, characterized by the features described herein. (Note 40) The aforementioned Learning Support Department When explaining operating procedures, the priority of the explanation is determined based on the user's current learning level. The system described in Appendix 1, characterized by the features described herein. (Note 41) The aforementioned Learning Support Department The system estimates the user's emotions and adjusts the difficulty level of the quiz-style operational test based on those emotions. The system described in Appendix 1, characterized by the features described herein. (Note 42) The aforementioned Learning Support Department When explaining operating procedures, we provide optimal explanations that take into account the user's geographical location. The system described in Appendix 1, characterized by the features described herein. (Note 43) The aforementioned Learning Support Department When explaining operating procedures, analyze the user's social media activity and provide relevant explanations. The system described in Appendix 1, characterized by the features described herein. [Explanation of symbols]

[0207] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots

Claims

1. An operation recording unit that records the user's PC operations, A reproduction unit that reproduces the operations recorded by the operation recording unit, An optimization unit that optimizes the operation reproduced by the reproduction unit, A voice command unit that accepts voice commands, A multi-device compatible unit that synchronizes operations across different devices, It includes a learning support unit that generates explanations of operating procedures and conducts operation tests in a quiz format. A system characterized by the following features.

2. The operation recording unit is, Record the user's PC operations unit by unit. The system according to feature 1.

3. The reproduction unit is, Reproduce recorded operations with high precision. The system according to feature 1.

4. The optimization unit, Analyze recorded operations and suggest more efficient procedures. The system according to feature 1.

5. The aforementioned voice command unit is Control the operation using voice commands. The system according to feature 1.

6. The multi-device compatible unit is, Synchronize operations across different devices. The system according to feature 1.

7. The aforementioned Learning Support Department, Generate instructions for operation and conduct an operation test in a quiz format. The system according to feature 1.

8. The operation recording unit is, It estimates the user's emotions and adjusts the timing of operation recordings based on the estimated user emotions. The system according to feature 1.

Images