system

Abstract

An object of a system according to an embodiment is to realize automation of a task without expert knowledge.SOLUTION: A system according to an embodiment includes a generation unit, a learning unit, a speech recognition unit, a text conversion unit, and an automation unit. The generation unit understands the PC operation of the user and creates an automation program. The learning unit learns a work pattern of the user based on the program generated by the generation unit. The voice recognition unit understands the voice command based on the work pattern learned by the learning unit and automates the work. The text generator generates a procedure manual by texting the automated work by the voice recognizer. Automation uses specific techniques to learn and automate complex operations.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional technologies have had the problem that automating business processes requires specialized knowledge and that automating complex operations is difficult.

[0005] The system according to the embodiment aims to realize automation of business processes without requiring specialized knowledge. [Means for solving the problem]

[0006] The system according to the embodiment includes a generation unit, a learning unit, a voice recognition unit, a text conversion unit, and an automation unit. The generation unit understands a user's PC operations and creates an automation program. The learning unit learns the user's work patterns based on the program generated by the generation unit. The voice recognition unit understands voice commands based on the work patterns learned by the learning unit and automates the work. The text conversion unit converts the work automated by the voice recognition unit into text and creates a procedure manual. The automation unit uses specific technology to learn and automate complex operations. [Effects of the Invention]

[0007] The system according to the embodiment can automate business processes without requiring specialized knowledge. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. DETAILED DESCRIPTION OF THE INVENTION

[0009] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0010] First, the terms used in the following description will be explained.

[0011] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, the processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (Tensor Processing Unit).

[0012] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0013] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0014] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), and Bluetooth (registered trademark).

[0015] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0016] [First embodiment] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0017] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0018] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0019] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0020] The reception device 38 includes a touch panel 38A and a microphone 38B, and receives user input. The touch panel 38A detects contact with a pointer (for example, a pen or a finger) to receive user input by the touch of the pointer. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 (see FIG. 2) acquires the data indicating the user input.

[0021] Output device 40 includes a display 40A and a speaker 40B, and presents data to a user by outputting the data in a form of expression that the user can perceive (e.g., audio and / or text). Display 40A displays visible information such as text and images in accordance with instructions from processor 46. Speaker 40B outputs audio in accordance with instructions from processor 46. Camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0022] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0023] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0024] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0025] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate a user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotion, including estimation and prediction of the user's emotion, but is not limited to these examples. Furthermore, the estimation and prediction of emotion also includes, for example, emotion analysis.

[0026] In the smart device 14, the specific processing is performed by the processor 46. The storage 50 stores a specific processing program 60. The specific processing program 60 is used together with the specific processing program 56 by the data processing system 10. The processor 46 reads the specific processing program 60 from the storage 50 and executes the read specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as the control unit 46A in accordance with the specific processing program 60 executed on the RAM 48. Note that the smart device 14 has a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59, and can also perform processing similar to that of the specific processing unit 290 using these models.

[0027] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device (e.g., a generation server) may have the data generation model 58. In this case, the data processing device 12 obtains a processing result (prediction result, etc.) using the data generation model 58 by communicating with the server device having the data generation model 58. Furthermore, the data processing device 12 may be a server device, or may be a terminal device owned by a user (e.g., a mobile phone, a robot, a home appliance, etc.). Next, an example of processing by the data processing system 10 according to the first embodiment will be described.

[0028] (Example 1) A business automation system according to an embodiment of the present invention understands a user's PC operations, creates an automation program, learns their work patterns, understands their voice commands, automates tasks, and converts them into text. The business automation system uses a generation AI to understand the user's PC operations, create an automation program, and convert it into text. Next, it uses a machine learning model to learn the user's work patterns. Furthermore, it uses a voice recognition AI to understand the user's voice commands and automate the tasks. For example, the business automation system observes the user's PC operations and learns their procedures. Next, it generates an automation program based on the learned procedures, and when the user issues a voice command, the AI ​​automatically performs the task. This frees the user from repetitive tasks and allows them to focus on more creative tasks. The business automation system also converts user operations into text and automatically generates procedure manuals. This facilitates training new employees and promotes business standardization. Furthermore, by using a self-operating computer framework, even complex operations can be learned and automated. For example, it can learn business processes that use multiple tools and automatically execute the optimal procedure. This reduces the burden on users and improves business efficiency. For example, automating the data entry tasks that users perform on a daily basis eliminates the hassle of simple tasks and reduces man-hours. Even without programming knowledge, users can automate their own computer operations and create textual instructions. Furthermore, by learning and automating complex operations, AI can independently determine and execute optimal processes.

[0029] A task automation system according to an embodiment includes a generation unit, a learning unit, a speech recognition unit, a text conversion unit, and an automation unit. The generation unit understands a user's PC operations and creates an automation program. The generation unit, for example, uses a generation AI to observe the user's operations and learn the procedures. The generation unit can also receive a prompt such as "Please automate this operation" and convert the operation procedures into an automation program. The generation unit can also monitor the user's operations in real time and generate an optimal automation program using the generation AI. The learning unit learns the user's work patterns based on the program generated by the generation unit. The learning unit, for example, analyzes the user's operation history using a machine learning model to learn frequently occurring operations and combinations of consecutive operations. The learning unit can also suggest optimal work procedures based on the user's operation patterns. The learning unit can also continuously learn the user's operation patterns and improve the accuracy of the program. The speech recognition unit understands voice commands based on the operation patterns learned by the learning unit and automates tasks. The voice recognition unit, for example, uses voice recognition AI to analyze a user's voice commands and automate the corresponding tasks. The voice recognition unit can also recognize a user's voice commands in real time and instantly execute tasks. Furthermore, the voice recognition unit can learn the user's voice commands and improve accuracy. The text conversion unit converts the tasks automated by the voice recognition unit into text and creates a procedure manual. The text conversion unit, for example, automatically generates a procedure manual based on text data generated by the voice recognition AI. The text conversion unit can also record the user's operation procedures in detail and reflect them in the procedure manual. Furthermore, the text conversion unit can customize the format of the procedure manual to create a procedure manual tailored to the user's needs. The automation unit learns and automates complex operations using a Self-Operating Computer Framework. For example, the automation unit can learn a business process using multiple tools and automatically execute the optimal procedure. The automation unit can also observe user operations and consider the optimal process. Furthermore, the automation unit can generate a program to automate complex operations based on the user's operation history.As a result, the business automation system according to the embodiment can reduce the burden on users and improve business efficiency. For example, by automating the data entry work that users perform on a daily basis, the effort required for simple tasks can be eliminated, reducing the number of man-hours required. Furthermore, even if you do not have programming knowledge, you can automate your own computer operations, convert them into text, and create procedure manuals. Furthermore, by learning and automating even complex operations, the AI ​​can independently determine and execute optimal processes.

[0030] The generation unit can analyze the user's past operation history and generate an efficient automation program. The generation unit can analyze the user's past operation history using, for example, a generation AI. The generation unit can identify operations that the user frequently performed in the past through, for example, analysis of operation logs. The generation unit can also predict operations that will be performed in a specific time period through frequency analysis. The generation unit can also propose efficient operation procedures based on the operation history and generate an automation program based on the proposed procedures. For example, the generation unit can generate an optimal automation program based on operations that the user frequently performed in the past. The generation unit can also predict operations that will be performed in a specific time period from the user's operation history and generate a program that automates the operation. The generation unit can also analyze the user's operation history, propose efficient operation procedures, and generate an automation program based on the proposed procedures. This enables efficient automation by generating an optimal automation program based on the user's past operation history. Some or all of the above-described processing in the generation unit can be performed using, for example, AI, or without AI.

[0031] When generating an automation program, the generation unit can customize the program based on the user's current task or project. The generation unit, for example, uses a generation AI to identify the user's current task or project. The generation unit, for example, acquires data from a task management tool to identify the user's current task. The generation unit can also identify the user's current project based on calendar information. The generation unit can also customize the automation program based on the user's current task or project. For example, the generation unit can generate an optimal automation program based on the content of the project the user is currently working on. The generation unit can also acquire information from the user's task management tool to generate an automation program optimal for that task. Furthermore, the generation unit can generate an automation program tailored to the task, taking into account the progress of the user's current task. This improves work efficiency by generating an automation program tailored to the user's current task or project. Some or all of the above-mentioned processing in the generation unit may be performed using AI, for example, or without AI.

[0032] When generating an automation program, the generation unit can select an appropriate generation means according to the user's input method. The generation unit, for example, uses a generation AI to identify the user's input method. The generation unit can identify a specific type, such as voice input, text input, or image input. The generation unit can also select the optimal generation means according to the user's input method. For example, if the user uses voice input, the generation unit can generate an automation program optimized for voice input. Also, if the user uses text input, the generation unit can generate an automation program optimized for text input. Furthermore, if the user uses image input, the generation unit can generate an optimal automation program using image recognition technology. This improves user convenience by generating an optimal automation program according to the user's input method. Some or all of the above-described processing in the generation unit may be performed using, for example, AI, or may be performed without using AI.

[0033] When generating an automation program, the generation unit can prioritize generating a specific program by taking into consideration the user's geographical location information. The generation unit, for example, acquires the user's geographical location information using a generation AI. The generation unit identifies the user's location information, for example, through analysis of GPS data or an IP address. The generation unit can also prioritize generating a program that is highly relevant based on the user's geographical location information. For example, if the user is working in a specific area, the generation unit can generate an automation program based on information related to that area. If the user is on a business trip, the generation unit can also generate an optimal automation program based on information about the business trip destination. Furthermore, if the user is working at home, the generation unit can generate an automation program optimized for the home environment. This improves user convenience by generating a highly relevant program based on the user's geographical location information. Some or all of the above-described processing in the generation unit may be performed, for example, using AI or without using AI.

[0034] The generation unit can analyze the user's social media activities and generate a related program when generating an automation program. The generation unit can analyze the user's social media activities using, for example, a generation AI. The generation unit can identify the user's interests through, for example, analyzing the content of posts. The generation unit can also understand the user's social network through an analysis of followers. The generation unit can also generate a related program based on the user's social media activities. For example, the generation unit can generate a related automation program based on the content frequently posted by the user on social media. The generation unit can also generate a related program based on the activity of the user's friends on social media. The generation unit can also generate a related automation program based on the user's social media check-in information. This improves user convenience by generating a related program based on the user's social media activities. Some or all of the above-described processing in the generation unit can be performed using, for example, AI, or without AI.

[0035] When generating an automation program, the generation unit can customize the generation method by reflecting the user's past feedback. The generation unit, for example, analyzes the user's past feedback using a generation AI. The generation unit, for example, identifies the user's requests and areas for improvement through analyzing the feedback. The generation unit can also adjust the generation method of the automation program based on the feedback. For example, the generation unit adjusts the generation method of the automation program based on feedback provided by the user in the past. The generation unit can also generate an automation program with enhanced specific functions based on the user's feedback. Furthermore, the generation unit can analyze the user's feedback, propose an optimal generation method, and generate the automation program based on that. This improves user convenience by customizing the generation method by reflecting the user's past feedback. Some or all of the above-described processing in the generation unit may be performed, for example, using AI, or may be performed without using AI.

[0036] During learning, the learning unit can optimize the learning algorithm by referring to past learning data. The learning unit, for example, uses a generation AI to refer to past learning data. The learning unit, for example, analyzes past operation logs and extracts effective learning patterns. The learning unit can also adjust parameters of the learning algorithm by referring to a database of past learning results. Furthermore, the learning unit can improve the learning algorithm based on the past learning data to increase accuracy. For example, the learning unit adjusts parameters of the learning algorithm based on the past learning data. The learning unit can also extract effective learning patterns from the past learning data and reflect them in the learning algorithm. Furthermore, the learning unit can analyze the past learning data and make improvements to improve the accuracy of the learning algorithm. As a result, optimizing the learning algorithm by referring to the past learning data improves the accuracy of learning. Some or all of the above-described processing in the learning unit may be performed, for example, using AI or without using AI.

[0037] During learning, the learning unit can update the learning data to reflect user feedback. The learning unit, for example, analyzes the user feedback using a generation AI. The learning unit, for example, identifies the user's requests and areas for improvement through analyzing the feedback. The learning unit can also update the learning data based on the feedback. For example, the learning unit updates the learning data based on feedback provided by the user. The learning unit can also add or delete specific data from the user feedback. Furthermore, the learning unit can analyze the user feedback and make improvements to improve the quality of the learning data. In this way, updating the learning data to reflect user feedback improves the accuracy of learning. Some or all of the above-mentioned processing in the learning unit may be performed, for example, using AI, or may be performed without using AI.

[0038] During learning, the learning unit can weight the learning based on the frequency of user operations. The learning unit, for example, uses a generation AI to analyze the frequency of user operations. The learning unit, for example, identifies operations frequently performed by the user through analysis of an operation log. The learning unit can also set a high weighting in the learning algorithm for operations that are frequently performed. The learning unit can also set a low weighting in the learning algorithm for operations that are infrequent. For example, the learning unit sets a high weighting in the learning algorithm for operations that are frequently performed by the user. The learning unit can also set a low weighting in the learning algorithm for operations that are rarely performed by the user. The learning unit can also analyze the frequency of user operations and reflect optimal weighting in the learning algorithm. As a result, weighting the learning based on the frequency of user operations improves learning efficiency. Some or all of the above-described processing in the learning unit may be performed using, for example, AI, or may be performed without using AI.

[0039] During learning, the learning unit can integrate information from different data sources to enrich the learning data. The learning unit, for example, uses a generative AI to acquire information from different data sources. The learning unit, for example, acquires information from a database via an external API and integrates it into the learning data. The learning unit can also collect information from online resources and add it to the learning data. The learning unit can also integrate data provided by the user with external data to enrich the learning data. For example, the learning unit acquires information from different databases and integrates it into the learning data. The learning unit can also collect information from online resources and add it to the learning data. The learning unit can also integrate data provided by the user with external data to enrich the learning data. In this way, by integrating information from different data sources, the learning data is enriched and the accuracy of learning is improved. Some or all of the above-described processing in the learning unit may be performed, for example, using AI or without AI.

[0040] The learning unit can customize the learning content based on the user's occupation and lifestyle during learning. The learning unit, for example, uses a generation AI to identify the user's occupation and lifestyle. The learning unit can identify the user's occupation and lifestyle, for example, through an analysis of survey results and behavioral patterns. The learning unit can also customize the learning content based on the user's occupation and lifestyle. For example, the learning unit prioritizes learning data related to the user's occupation. The learning unit can also select learning data that matches the user's lifestyle. Furthermore, the learning unit can customize the learning content taking into account the user's occupation and lifestyle. This improves learning efficiency by providing learning content that matches the user's occupation and lifestyle. Some or all of the above-described processing in the learning unit may be performed, for example, using AI, or may be performed without using AI.

[0041] During learning, the learning unit can select an optimal learning method based on the user's device information. The learning unit, for example, acquires the user's device information using a generation AI. The learning unit, for example, identifies the device type and OS version. The learning unit can also select an optimal learning method based on the user's device information. For example, if the user is using a smartphone, the learning unit can select a learning method optimized for mobile devices. If the user is using a tablet, the learning unit can also select a learning method optimized for large screens. Furthermore, if the user is using a desktop computer, the learning unit can also select a learning method optimized for high resolution. This improves learning efficiency by providing an optimal learning method based on the user's device information. Some or all of the above-described processing in the learning unit may be performed using AI, for example, or without AI.

[0042] During speech recognition, the speech recognition unit can optimize the recognition algorithm according to the user's pronunciation and accent. The speech recognition unit, for example, uses generative AI to identify the user's pronunciation and accent. The speech recognition unit, for example, learns the user's pronunciation pattern through analysis of voice samples. The speech recognition unit can also adjust parameters of the recognition algorithm according to the user's accent. Furthermore, the speech recognition unit can make improvements to improve recognition accuracy by taking the user's pronunciation and accent into consideration. For example, the speech recognition unit learns the user's pronunciation pattern and optimizes the recognition algorithm. The speech recognition unit can also adjust parameters of the recognition algorithm according to the user's accent. Furthermore, the speech recognition unit can make improvements to improve recognition accuracy by taking the user's pronunciation and accent into consideration. As a result, optimizing the recognition algorithm according to the user's pronunciation and accent improves the accuracy of speech recognition. Some or all of the above-described processing in the speech recognition unit may be performed using AI, for example, or without using AI.

[0043] During voice recognition, the voice recognition unit can prioritize voice commands based on the user's current task. The voice recognition unit, for example, uses a generative AI to identify the user's current task. The voice recognition unit, for example, acquires data from a task management tool to identify the user's current task. The voice recognition unit can also identify the user's current project based on calendar information. The voice recognition unit can also prioritize voice commands based on the user's current task. For example, the voice recognition unit prioritizes recognition of voice commands related to the user's current task. The voice recognition unit can also acquire information from the user's task management tool and prioritize recognition of related voice commands. The voice recognition unit can also prioritize recognition of optimal voice commands by taking into account the progress of the user's current task. This improves the efficiency of voice recognition by prioritizing voice commands based on the user's current task. Some or all of the above-mentioned processing in the voice recognition unit may be performed using AI, for example, or without AI.

[0044] During voice recognition, the voice recognition unit can prioritize recognition of specific commands by taking into account the user's geographical location information. The voice recognition unit, for example, acquires the user's geographical location information using a generation AI. The voice recognition unit identifies the user's location information, for example, through analysis of GPS data or an IP address. The voice recognition unit can also prioritize recognition of highly relevant commands based on the user's geographical location information. For example, if the user is working in a specific area, the voice recognition unit can prioritize recognition of voice commands related to that area. Furthermore, if the user is on a business trip, the voice recognition unit can prioritize recognition of relevant voice commands based on information about the business trip destination. Furthermore, if the user is working at home, the voice recognition unit can prioritize recognition of voice commands related to the home environment. This improves the efficiency of voice recognition by recognizing highly relevant commands based on the user's geographical location information. Some or all of the above-described processing in the voice recognition unit may be performed, for example, using AI or without AI.

[0045] The voice recognition unit can analyze the user's social media activities and recognize related commands during voice recognition. The voice recognition unit can analyze the user's social media activities using, for example, generative AI. The voice recognition unit can identify the user's interests through, for example, analyzing the content of posts. The voice recognition unit can also understand the user's social network through an analysis of followers. The voice recognition unit can also recognize related commands based on the user's social media activities. For example, the voice recognition unit can recognize related voice commands based on the content frequently posted by the user on social media. The voice recognition unit can also recognize related commands based on the activities of the user's friends on social media. The voice recognition unit can also recognize related voice commands based on the user's social media check-in information. This improves the efficiency of voice recognition by recognizing related commands based on the user's social media activities. Some or all of the above-described processing in the voice recognition unit can be performed using, for example, AI, or without AI.

[0046] The voice recognition unit can customize the recognition method by reflecting the user's past feedback during voice recognition. The voice recognition unit, for example, analyzes the user's past feedback using a generative AI. The voice recognition unit, for example, identifies the user's requests and areas for improvement through analyzing the feedback. The voice recognition unit can also adjust the voice recognition algorithm based on the feedback. For example, the voice recognition unit adjusts the voice recognition algorithm based on feedback provided by the user in the past. The voice recognition unit can also preferentially recognize specific voice commands based on the user's feedback. Furthermore, the voice recognition unit can analyze the user's feedback and make improvements to improve the accuracy of voice recognition. As a result, the accuracy of voice recognition is improved by customizing the recognition method by reflecting the user's past feedback. Some or all of the above-mentioned processing in the voice recognition unit may be performed using AI, for example, or without using AI.

[0047] The text conversion unit can adjust the level of detail of the text based on the importance of the task when converting the text to text. The text conversion unit, for example, uses a generation AI to identify the importance of the task. The text conversion unit evaluates the importance of the task based on, for example, task priority or project importance. The text conversion unit can also adjust the level of detail of the text based on the importance of the task. For example, the text conversion unit generates text including a detailed explanation for an important task. The text conversion unit can also generate concise text for a low-importance task. Furthermore, the text conversion unit can adjust the level of detail of the text according to the importance of the task. This improves the efficiency of the text conversion by adjusting the level of detail of the text according to the importance of the task. Some or all of the above-described processing in the text conversion unit may be performed using, for example, AI, or may be performed without using AI.

[0048] The text conversion unit can apply different text conversion algorithms depending on the task category when converting the data into text. The text conversion unit, for example, uses a generative AI to identify the task category. The text conversion unit can identify the task category based on, for example, task classification or project category. The text conversion unit can also apply an optimal text conversion algorithm depending on the task category. For example, the text conversion unit can generate text including technical terms for technical tasks. The text conversion unit can also generate text using easy-to-understand expressions for general tasks. The text conversion unit can also apply an optimal text conversion algorithm depending on the task category. In this way, applying a text conversion algorithm depending on the task category improves the accuracy of the text conversion. Some or all of the above-mentioned processing in the text conversion unit may be performed using, for example, AI, or may be performed without using AI.

[0049] The text conversion unit can improve the accuracy of the text by referring to the user's past text conversion results during text conversion. The text conversion unit, for example, uses a generation AI to refer to the user's past text conversion results. The text conversion unit can identify the user's past text conversion results, for example, based on past procedure manuals or text conversion logs. The text conversion unit can also optimize the text conversion algorithm based on the past text conversion results. For example, the text conversion unit optimizes the text conversion algorithm based on text generated by the user in the past. The text conversion unit can also learn specific expression methods from the user's past text conversion results to improve accuracy. Furthermore, the text conversion unit can analyze the user's past text conversion results and suggest an optimal text conversion method. This improves the accuracy of the text conversion by referring to the user's past text conversion results. Some or all of the above-mentioned processing in the text conversion unit may be performed, for example, using AI or without AI.

[0050] The text conversion unit can determine the priority of text based on the submission date of work during text conversion. The text conversion unit, for example, uses a generation AI to identify the submission date of work. The text conversion unit can also determine the priority of text based on the submission date of work. For example, the text conversion unit prioritizes text conversion for work with an approaching deadline. The text conversion unit can also postpone text conversion for work with a distant submission date. Furthermore, the text conversion unit can adjust the priority of text depending on the submission date of work. Thus, by determining the priority of text based on the submission date of work, the efficiency of text conversion is improved. Some or all of the above-mentioned processing in the text conversion unit may be performed using, for example, AI, or may be performed without using AI.

[0051] The text conversion unit can adjust the order of text based on the relevance of tasks during text conversion. The text conversion unit, for example, uses generative AI to identify the relevance of tasks. The text conversion unit can evaluate the relevance of tasks based on, for example, task dependencies or project relevance. The text conversion unit can also adjust the order of text based on the relevance of tasks. For example, the text conversion unit can prioritize the conversion of highly relevant tasks into text and adjust the order. The text conversion unit can also postpone the conversion of less relevant tasks. Furthermore, the text conversion unit can optimize the order of text according to the relevance of tasks. As a result, adjusting the order of text based on the relevance of tasks improves the efficiency of text conversion. Some or all of the above-described processing in the text conversion unit may be performed using, for example, AI, or may be performed without using AI.

[0052] The text conversion unit can adjust the use of technical terms in the text during text conversion according to the user's level of expertise. The text conversion unit, for example, uses a generation AI to identify the user's level of expertise. The text conversion unit can identify the user's level of expertise based on, for example, survey results or past operation history. The text conversion unit can also adjust the use of technical terms in the text according to the user's level of expertise. For example, if the user has technical expertise, the text conversion unit can generate text that uses a lot of technical terms. Also, if the user is a beginner, the text conversion unit can generate text that uses easy-to-understand expressions. Furthermore, the text conversion unit can select an optimal text conversion method according to the user's level of expertise. This improves the accuracy of the text conversion by adjusting the use of technical terms in the text according to the user's level of expertise. Some or all of the above-mentioned processing in the text conversion unit may be performed using, for example, AI, or may be performed without using AI.

[0053] During automation, the automation unit can analyze the user's past operation history and select the optimal automation method. The automation unit can analyze the user's past operation history using, for example, a generation AI. The automation unit can identify operations that the user frequently performed in the past through, for example, analysis of operation logs. The automation unit can also predict operations that will be performed in a specific time period through frequency analysis. The automation unit can also propose efficient operation procedures based on the operation history and select an automation method based on the procedures. For example, the automation unit can select the optimal automation method based on operations that the user frequently performed in the past. The automation unit can also predict operations that will be performed in a specific time period from the user's operation history and select the optimal automation method. The automation unit can also analyze the user's operation history, propose efficient operation procedures, and select an automation method based on the procedures. This improves the efficiency of automation by analyzing the user's past operation history and selecting the optimal automation method. Some or all of the above-described processing in the automation unit can be performed using, for example, AI, or without AI.

[0054] During automation, the automation unit can customize the means of automation based on the user's current living situation. The automation unit, for example, uses a generative AI to identify the user's current living situation. The automation unit can identify the user's living situation, for example, through an analysis of survey results or behavioral patterns. The automation unit can also customize the means of automation based on the user's current living situation. For example, if the user is working from home, the automation unit can provide an automation means optimized for the home environment. Also, if the user is on a business trip, the automation unit can provide an automation means optimized for the environment of the business trip destination. Furthermore, the automation unit can customize the optimal automation means by taking the user's living situation into consideration. This improves the efficiency of automation by customizing the means of automation based on the user's current living situation. Some or all of the above-described processing in the automation unit may be performed, for example, using AI or without using AI.

[0055] During automation, the automation unit can improve the automation method by reflecting user feedback. The automation unit, for example, analyzes user feedback using generative AI. The automation unit, for example, identifies user requests and areas for improvement through analyzing the feedback. The automation unit can also improve the automation method based on the feedback. For example, the automation unit improves the automation method based on feedback provided by the user. The automation unit can also provide an automation method with enhanced specific functions based on the user feedback. Furthermore, the automation unit can analyze user feedback and make improvements to improve the accuracy of the automation. As a result, improving the automation method by reflecting user feedback improves the accuracy of the automation. Some or all of the above-mentioned processing in the automation unit may be performed, for example, using AI or without using AI.

[0056] During automation, the automation unit can select the optimal automation method taking into account the user's geographic location information. The automation unit, for example, acquires the user's geographic location information using a generation AI. The automation unit identifies the user's location information, for example, through analysis of GPS data or an IP address. The automation unit can also select the optimal automation method based on the user's geographic location information. For example, if the user is working in a specific area, the automation unit selects an automation method related to that area. If the user is on a business trip, the automation unit can select the optimal automation method based on information about the business trip destination. Furthermore, if the user is working at home, the automation unit can select an automation method optimized for the home environment. This improves the efficiency of automation by selecting the optimal automation method based on the user's geographic location information. Some or all of the above-described processing in the automation unit may be performed using AI, for example, or without AI.

[0057] During automation, the automation unit can analyze the user's social media activity and suggest automation measures. The automation unit can analyze the user's social media activity using, for example, generative AI. The automation unit can identify the user's interests through analysis of posted content, for example. The automation unit can also understand the user's social network through analysis of followers. The automation unit can also suggest automation measures based on the user's social media activity. For example, the automation unit can suggest relevant automation measures based on the content the user frequently posts on social media. The automation unit can also suggest relevant automation measures based on the activity of the user's friends on social media. The automation unit can also suggest relevant automation measures based on the user's social media check-in information. This improves the efficiency of automation by suggesting automation measures based on the user's social media activity. Some or all of the above-mentioned processing in the automation unit can be performed using, for example, AI, or without AI.

[0058] During automation, the automation unit can customize the automation method by reflecting the user's past feedback. The automation unit, for example, analyzes the user's past feedback using a generative AI. The automation unit, for example, identifies the user's requests and areas for improvement through analyzing the feedback. The automation unit can also customize the automation method based on the feedback. For example, the automation unit customizes the automation method based on feedback provided by the user in the past. The automation unit can also provide an automation method with enhanced specific functions based on the user's feedback. Furthermore, the automation unit can analyze the user's feedback and suggest an optimal automation method. In this way, customizing the automation method by reflecting the user's past feedback improves the accuracy of automation. Some or all of the above-described processing in the automation unit may be performed, for example, using AI or without using AI.

[0059] The system according to the embodiment is not limited to the above-described example, and various modifications are possible, for example, as follows.

[0060] The generation unit can also acquire biometric information about the user and adjust the generation of the automation program based on the user's health condition. For example, the generation unit can acquire data such as heart rate and blood pressure from the wearable device and delay the generation of the automation program if the user is tired. Furthermore, the generation unit can generate a more complex automation program if the user is in good health. Furthermore, the generation unit can adjust the priority of tasks based on the user's health condition. This allows the generation of an optimal automation program tailored to the user's health condition.

[0061] The learning unit can also analyze the user's operation history and make suggestions to improve operation efficiency. For example, the learning unit can identify operations that the user frequently performs and make suggestions to automate those operations. The learning unit can also make suggestions to optimize the order of operations performed by the user. Furthermore, the learning unit can also suggest new procedures to improve operation efficiency based on the user's operation history. This makes it possible to provide efficient operation procedures based on the user's operation history.

[0062] The speech recognition unit can also analyze the user's speech and use context information to understand the user's intent. For example, the speech recognition unit can refer to the user's past speech history to understand the current speech. The speech recognition unit can also interpret the speech based on the user's current task or project. Furthermore, the speech recognition unit can obtain external information related to the user's speech and use it as auxiliary information to understand the speech. This can result in a more accurate understanding of the user's speech.

[0063] The text conversion unit can also analyze the user's input content and select appropriate expressions according to the context. For example, the text conversion unit can select appropriate technical terms and expressions based on the user's input content. The text conversion unit can also obtain external information related to the user's input content and select expressions according to the context. Furthermore, the text conversion unit can analyze the user's input content and select an appropriate format according to the context. This makes it possible to provide appropriate expressions according to the user's input content.

[0064] The automation unit can also propose new automation means for improving the efficiency of operations based on the user's operation history. For example, the automation unit can identify operations that the user frequently performs and propose automating those operations. The automation unit can also propose optimizing the order of operations performed by the user. Furthermore, the automation unit can propose new procedures for improving the efficiency of operations based on the user's operation history. This makes it possible to provide efficient automation means based on the user's operation history.

[0065] The processing flow of the first embodiment will be briefly explained below.

[0066] Step 1: The generation unit understands the user's PC operations and creates an automation program. The generation unit uses the generation AI to observe the user's operations and learn the procedures. The generation AI can also receive prompts such as "Please automate this operation" and convert the operation procedures into an automation program. Furthermore, the generation AI can monitor the user's operations in real time and generate the optimal automation program. Step 2: The learning unit learns the user's work patterns based on the program generated by the generation unit. The learning unit uses a machine learning model to analyze the user's operation history and learns frequently performed operations and combinations of consecutive operations. It can also suggest optimal work procedures based on the user's operation patterns. Furthermore, it can continuously learn the user's operation patterns and improve the accuracy of the program. Step 3: The voice recognition unit understands voice commands and automates tasks based on the task patterns learned by the learning unit. The voice recognition unit uses voice recognition AI to analyze the user's voice commands and automate the corresponding tasks. It can also recognize the user's voice commands in real time and execute tasks instantly. It can also learn the user's voice commands and improve accuracy. Step 4: The text conversion unit converts the automated tasks created by the voice recognition unit into text and creates a procedure manual. The procedure manual is automatically generated based on the text data generated by the voice recognition AI. It is also possible to record the user's operation procedures in detail and reflect them in the procedure manual. Furthermore, it is possible to customize the format of the procedure manual and create a procedure manual tailored to the user's needs. Step 5: The automation unit uses the Self-Operating Computer Framework to learn and automate complex operations. It learns business processes that use multiple tools and automatically executes the optimal procedure. It can also observe user operations and consider the optimal process. It can also generate programs to automate complex operations based on the user's operation history.

[0067] (Example 2) A business automation system according to an embodiment of the present invention understands a user's PC operations, creates an automation program, learns their work patterns, understands their voice commands, automates tasks, and converts them into text. The business automation system uses a generation AI to understand the user's PC operations, create an automation program, and convert it into text. Next, it uses a machine learning model to learn the user's work patterns. Furthermore, it uses a voice recognition AI to understand the user's voice commands and automate the tasks. For example, the business automation system observes the user's PC operations and learns their procedures. Next, it generates an automation program based on the learned procedures, and when the user issues a voice command, the AI ​​automatically performs the task. This frees the user from repetitive tasks and allows them to focus on more creative tasks. The business automation system also converts user operations into text and automatically generates procedure manuals. This facilitates training new employees and promotes business standardization. Furthermore, by using a self-operating computer framework, even complex operations can be learned and automated. For example, it can learn business processes that use multiple tools and automatically execute the optimal procedure. This reduces the burden on users and improves business efficiency. For example, automating the data entry tasks that users perform on a daily basis eliminates the hassle of simple tasks and reduces man-hours. Even without programming knowledge, users can automate their own computer operations and create textual instructions. Furthermore, by learning and automating complex operations, AI can independently determine and execute optimal processes.

[0068] A task automation system according to an embodiment includes a generation unit, a learning unit, a speech recognition unit, a text conversion unit, and an automation unit. The generation unit understands a user's PC operations and creates an automation program. The generation unit, for example, uses a generation AI to observe the user's operations and learn the procedures. The generation unit can also receive a prompt such as "Please automate this operation" and convert the operation procedures into an automation program. The generation unit can also monitor the user's operations in real time and generate an optimal automation program using the generation AI. The learning unit learns the user's work patterns based on the program generated by the generation unit. The learning unit, for example, analyzes the user's operation history using a machine learning model to learn frequently occurring operations and combinations of consecutive operations. The learning unit can also suggest optimal work procedures based on the user's operation patterns. The learning unit can also continuously learn the user's operation patterns and improve the accuracy of the program. The speech recognition unit understands voice commands based on the operation patterns learned by the learning unit and automates tasks. The voice recognition unit, for example, uses voice recognition AI to analyze a user's voice commands and automate the corresponding tasks. The voice recognition unit can also recognize a user's voice commands in real time and instantly execute tasks. Furthermore, the voice recognition unit can learn the user's voice commands and improve accuracy. The text conversion unit converts the tasks automated by the voice recognition unit into text and creates a procedure manual. The text conversion unit, for example, automatically generates a procedure manual based on text data generated by the voice recognition AI. The text conversion unit can also record the user's operation procedures in detail and reflect them in the procedure manual. Furthermore, the text conversion unit can customize the format of the procedure manual to create a procedure manual tailored to the user's needs. The automation unit learns and automates complex operations using a Self-Operating Computer Framework. For example, the automation unit can learn a business process using multiple tools and automatically execute the optimal procedure. The automation unit can also observe user operations and consider the optimal process. Furthermore, the automation unit can generate a program to automate complex operations based on the user's operation history.As a result, the business automation system according to the embodiment can reduce the burden on users and improve business efficiency. For example, by automating the data entry work that users perform on a daily basis, the effort required for simple tasks can be eliminated, reducing the number of man-hours required. Furthermore, even if you do not have programming knowledge, you can automate your own computer operations, convert them into text, and create procedure manuals. Furthermore, by learning and automating even complex operations, the AI ​​can independently determine and execute optimal processes.

[0069] The generation unit estimates the user's emotions and adjusts the timing of generating the automation program based on the estimated user emotions. The generation unit estimates the user's emotions using, for example, a generation AI. The generation unit may analyze the user's facial expressions using, for example, facial expression recognition technology to estimate emotions. The generation unit may also use voice analysis technology to analyze the tone and speed of the user's voice to estimate emotions. The generation unit may also estimate the user's emotions through behavioral pattern analysis. The generation unit adjusts the timing of generating the automation program based on the estimated user emotions. For example, if the user is stressed, the generation AI may delay generating the automation program and wait until the user is relaxed. Also, if the user is concentrating, the generation AI may immediately generate the automation program to prevent interruptions to the workflow. Furthermore, if the user is tired, the generation AI may speed up the generation of the automation program to reduce the user's burden. This reduces the user's burden by adjusting the timing of generating the automation program according to the user's emotions. Emotion estimation is achieved using, for example, an emotion estimation function using an emotion engine or generation AI. The generation AI may be a text generation AI (e.g., LLM) or a multimodal generation AI, but is not limited to such examples.

[0070] The generation unit can analyze the user's past operation history and generate an efficient automation program. The generation unit can analyze the user's past operation history using, for example, a generation AI. The generation unit can identify operations that the user frequently performed in the past through, for example, analysis of operation logs. The generation unit can also predict operations that will be performed in a specific time period through frequency analysis. The generation unit can also propose efficient operation procedures based on the operation history and generate an automation program based on the proposed procedures. For example, the generation unit can generate an optimal automation program based on operations that the user frequently performed in the past. The generation unit can also predict operations that will be performed in a specific time period from the user's operation history and generate a program that automates the operation. The generation unit can also analyze the user's operation history, propose efficient operation procedures, and generate an automation program based on the proposed procedures. This enables efficient automation by generating an optimal automation program based on the user's past operation history. Some or all of the above-described processing in the generation unit can be performed using, for example, AI, or without AI.

[0071] When generating an automation program, the generation unit can customize the program based on the user's current task or project. The generation unit, for example, uses a generation AI to identify the user's current task or project. The generation unit, for example, acquires data from a task management tool to identify the user's current task. The generation unit can also identify the user's current project based on calendar information. The generation unit can also customize the automation program based on the user's current task or project. For example, the generation unit can generate an optimal automation program based on the content of the project the user is currently working on. The generation unit can also acquire information from the user's task management tool to generate an automation program optimal for that task. Furthermore, the generation unit can generate an automation program tailored to the task, taking into account the progress of the user's current task. This improves work efficiency by generating an automation program tailored to the user's current task or project. Some or all of the above-mentioned processing in the generation unit may be performed using AI, for example, or without AI.

[0072] When generating an automation program, the generation unit can select an appropriate generation means according to the user's input method. The generation unit, for example, uses a generation AI to identify the user's input method. The generation unit can identify a specific type, such as voice input, text input, or image input. The generation unit can also select the optimal generation means according to the user's input method. For example, if the user uses voice input, the generation unit can generate an automation program optimized for voice input. Also, if the user uses text input, the generation unit can generate an automation program optimized for text input. Furthermore, if the user uses image input, the generation unit can generate an optimal automation program using image recognition technology. This improves user convenience by generating an optimal automation program according to the user's input method. Some or all of the above-described processing in the generation unit may be performed using, for example, AI, or may be performed without using AI.

[0073] The generation unit can estimate the user's emotions and prioritize the programs to be generated based on the estimated user emotions. The generation unit, for example, uses a generation AI to estimate the user's emotions. The generation unit can also estimate the user's emotions by analyzing the user's facial expressions using facial expression recognition technology. The generation unit can also estimate the user's emotions by analyzing the tone and speed of the user's voice using voice analysis technology. The generation unit can also estimate the user's emotions through behavioral pattern analysis. The generation unit prioritizes the programs to be generated based on the estimated user emotions. For example, if the user is feeling stressed, the generation AI can prioritize generating programs to reduce stress. Also, if the user is relaxing, the generation AI can prioritize generating programs that support long-term work. Furthermore, if the user is in a hurry, the generation AI can prioritize generating programs that can be completed quickly. This reduces the user's burden by prioritizing the programs to be generated based on the user's emotions. Emotion estimation is achieved using an emotion estimation function, for example, an emotion engine or a generation AI. The generation AI may be a text generation AI (e.g., LLM) or a multimodal generation AI, but is not limited to such examples.

[0074] When generating an automation program, the generation unit can prioritize generating a specific program by taking into consideration the user's geographical location information. The generation unit, for example, acquires the user's geographical location information using a generation AI. The generation unit identifies the user's location information, for example, through analysis of GPS data or an IP address. The generation unit can also prioritize generating a program that is highly relevant based on the user's geographical location information. For example, if the user is working in a specific area, the generation unit can generate an automation program based on information related to that area. If the user is on a business trip, the generation unit can also generate an optimal automation program based on information about the business trip destination. Furthermore, if the user is working at home, the generation unit can generate an automation program optimized for the home environment. This improves user convenience by generating a highly relevant program based on the user's geographical location information. Some or all of the above-described processing in the generation unit may be performed, for example, using AI or without using AI.

[0075] The generation unit can analyze the user's social media activities and generate a related program when generating an automation program. The generation unit can analyze the user's social media activities using, for example, a generation AI. The generation unit can identify the user's interests through, for example, analyzing the content of posts. The generation unit can also understand the user's social network through an analysis of followers. The generation unit can also generate a related program based on the user's social media activities. For example, the generation unit can generate a related automation program based on the content frequently posted by the user on social media. The generation unit can also generate a related program based on the activity of the user's friends on social media. The generation unit can also generate a related automation program based on the user's social media check-in information. This improves user convenience by generating a related program based on the user's social media activities. Some or all of the above-described processing in the generation unit can be performed using, for example, AI, or without AI.

[0076] When generating an automation program, the generation unit can customize the generation method by reflecting the user's past feedback. The generation unit, for example, analyzes the user's past feedback using a generation AI. The generation unit, for example, identifies the user's requests and areas for improvement through analyzing the feedback. The generation unit can also adjust the generation method of the automation program based on the feedback. For example, the generation unit adjusts the generation method of the automation program based on feedback provided by the user in the past. The generation unit can also generate an automation program with enhanced specific functions based on the user's feedback. Furthermore, the generation unit can analyze the user's feedback, propose an optimal generation method, and generate the automation program based on that. This improves user convenience by customizing the generation method by reflecting the user's past feedback. Some or all of the above-described processing in the generation unit may be performed, for example, using AI, or may be performed without using AI.

[0077] The learning unit can estimate a user's emotions and select training data based on the estimated user emotions. The learning unit estimates the user's emotions using, for example, a generative AI. The learning unit can analyze the user's facial expressions using, for example, facial expression recognition technology to estimate emotions. The learning unit can also estimate emotions by analyzing the tone and speed of the user's voice using voice analysis technology. The learning unit can also estimate the user's emotions through behavioral pattern analysis. The learning unit selects training data based on the estimated user emotions. For example, if the user is stressed, the learning unit prioritizes selecting training data with relaxing content. If the user is concentrating, the learning unit prioritizes selecting training data with high difficulty. Furthermore, if the user is tired, the learning unit can prioritize selecting training data with easy content. This improves learning efficiency by selecting training data according to the user's emotions. Emotion estimation is achieved using, for example, an emotion engine or generative AI with an emotion estimation function. The generative AI can be, but is not limited to, a text generation AI (e.g., LLM) or a multimodal generation AI.

[0078] During learning, the learning unit can optimize the learning algorithm by referring to past learning data. The learning unit, for example, uses a generation AI to refer to past learning data. The learning unit, for example, analyzes past operation logs and extracts effective learning patterns. The learning unit can also adjust parameters of the learning algorithm by referring to a database of past learning results. Furthermore, the learning unit can improve the learning algorithm based on the past learning data to increase accuracy. For example, the learning unit adjusts parameters of the learning algorithm based on the past learning data. The learning unit can also extract effective learning patterns from the past learning data and reflect them in the learning algorithm. Furthermore, the learning unit can analyze the past learning data and make improvements to improve the accuracy of the learning algorithm. As a result, optimizing the learning algorithm by referring to the past learning data improves the accuracy of learning. Some or all of the above-described processing in the learning unit may be performed, for example, using AI or without using AI.

[0079] During learning, the learning unit can update the learning data to reflect user feedback. The learning unit, for example, analyzes the user feedback using a generation AI. The learning unit, for example, identifies the user's requests and areas for improvement through analyzing the feedback. The learning unit can also update the learning data based on the feedback. For example, the learning unit updates the learning data based on feedback provided by the user. The learning unit can also add or delete specific data from the user feedback. Furthermore, the learning unit can analyze the user feedback and make improvements to improve the quality of the learning data. In this way, updating the learning data to reflect user feedback improves the accuracy of learning. Some or all of the above-mentioned processing in the learning unit may be performed, for example, using AI, or may be performed without using AI.

[0080] During learning, the learning unit can weight the learning based on the frequency of user operations. The learning unit, for example, uses a generation AI to analyze the frequency of user operations. The learning unit, for example, identifies operations frequently performed by the user through analysis of an operation log. The learning unit can also set a high weighting in the learning algorithm for operations that are frequently performed. The learning unit can also set a low weighting in the learning algorithm for operations that are infrequent. For example, the learning unit sets a high weighting in the learning algorithm for operations that are frequently performed by the user. The learning unit can also set a low weighting in the learning algorithm for operations that are rarely performed by the user. The learning unit can also analyze the frequency of user operations and reflect optimal weighting in the learning algorithm. As a result, weighting the learning based on the frequency of user operations improves learning efficiency. Some or all of the above-described processing in the learning unit may be performed using, for example, AI, or may be performed without using AI.

[0081] The learning unit can estimate the user's emotions and adjust the frequency of learning based on the estimated user emotions. The learning unit estimates the user's emotions using, for example, a generative AI. The learning unit can analyze the user's facial expressions using, for example, facial expression recognition technology to estimate emotions. The learning unit can also estimate emotions by analyzing the tone and speed of the user's voice using voice analysis technology. The learning unit can also estimate the user's emotions through behavioral pattern analysis. The learning unit adjusts the frequency of learning based on the estimated user emotions. For example, if the user is stressed, the learning frequency can be reduced to reduce the user's burden. If the user is relaxed, the learning frequency can be increased to promote efficient learning. Furthermore, if the user is focused, the learning frequency can be optimized to achieve effective learning. Thus, adjusting the learning frequency according to the user's emotions improves learning efficiency. Emotion estimation is achieved using an emotion estimation function using, for example, an emotion engine or generative AI. The generative AI can be, but is not limited to, a text generation AI (e.g., LLM) or a multimodal generation AI.

[0082] During learning, the learning unit can integrate information from different data sources to enrich the learning data. The learning unit, for example, uses a generative AI to acquire information from different data sources. The learning unit, for example, acquires information from a database via an external API and integrates it into the learning data. The learning unit can also collect information from online resources and add it to the learning data. The learning unit can also integrate data provided by the user with external data to enrich the learning data. For example, the learning unit acquires information from different databases and integrates it into the learning data. The learning unit can also collect information from online resources and add it to the learning data. The learning unit can also integrate data provided by the user with external data to enrich the learning data. In this way, by integrating information from different data sources, the learning data is enriched and the accuracy of learning is improved. Some or all of the above-described processing in the learning unit may be performed, for example, using AI or without AI.

[0083] The learning unit can customize the learning content based on the user's occupation and lifestyle during learning. The learning unit, for example, uses a generation AI to identify the user's occupation and lifestyle. The learning unit can identify the user's occupation and lifestyle, for example, through an analysis of survey results and behavioral patterns. The learning unit can also customize the learning content based on the user's occupation and lifestyle. For example, the learning unit prioritizes learning data related to the user's occupation. The learning unit can also select learning data that matches the user's lifestyle. Furthermore, the learning unit can customize the learning content taking into account the user's occupation and lifestyle. This improves learning efficiency by providing learning content that matches the user's occupation and lifestyle. Some or all of the above-described processing in the learning unit may be performed, for example, using AI, or may be performed without using AI.

[0084] During learning, the learning unit can select an optimal learning method based on the user's device information. The learning unit, for example, acquires the user's device information using a generation AI. The learning unit, for example, identifies the device type and OS version. The learning unit can also select an optimal learning method based on the user's device information. For example, if the user is using a smartphone, the learning unit can select a learning method optimized for mobile devices. If the user is using a tablet, the learning unit can also select a learning method optimized for large screens. Furthermore, if the user is using a desktop computer, the learning unit can also select a learning method optimized for high resolution. This improves learning efficiency by providing an optimal learning method based on the user's device information. Some or all of the above-described processing in the learning unit may be performed using AI, for example, or without AI.

[0085] The voice recognition unit can estimate a user's emotions and adjust the interpretation method of voice commands based on the estimated user emotions. The voice recognition unit estimates the user's emotions using, for example, generative AI. The voice recognition unit can analyze the user's facial expressions and estimate emotions using, for example, facial expression recognition technology. The voice recognition unit can also estimate emotions by analyzing the tone and speed of the user's voice using voice analysis technology. Furthermore, the voice recognition unit can estimate the user's emotions through behavioral pattern analysis. The voice recognition unit adjusts the interpretation method of voice commands based on the estimated user emotions. For example, if the user is nervous, the voice recognition unit can simplify the interpretation of voice commands to reduce misunderstandings. If the user is relaxed, the voice recognition unit can interpret detailed voice commands to increase accuracy. Furthermore, if the user is in a hurry, the voice recognition unit can adopt a voice command interpretation method that allows for quick interpretation. This improves the accuracy of voice recognition by adjusting the interpretation method of voice commands according to the user's emotions. Emotion estimation is achieved using an emotion estimation function, for example, an emotion engine or generative AI. The generation AI may be a text generation AI (e.g., LLM) or a multimodal generation AI, but is not limited to such examples.

[0086] During speech recognition, the speech recognition unit can optimize the recognition algorithm according to the user's pronunciation and accent. The speech recognition unit, for example, uses generative AI to identify the user's pronunciation and accent. The speech recognition unit, for example, learns the user's pronunciation pattern through analysis of voice samples. The speech recognition unit can also adjust parameters of the recognition algorithm according to the user's accent. Furthermore, the speech recognition unit can make improvements to improve recognition accuracy by taking the user's pronunciation and accent into consideration. For example, the speech recognition unit learns the user's pronunciation pattern and optimizes the recognition algorithm. The speech recognition unit can also adjust parameters of the recognition algorithm according to the user's accent. Furthermore, the speech recognition unit can make improvements to improve recognition accuracy by taking the user's pronunciation and accent into consideration. As a result, optimizing the recognition algorithm according to the user's pronunciation and accent improves the accuracy of speech recognition. Some or all of the above-described processing in the speech recognition unit may be performed using AI, for example, or without using AI.

[0087] During voice recognition, the voice recognition unit can prioritize voice commands based on the user's current task. The voice recognition unit, for example, uses a generative AI to identify the user's current task. The voice recognition unit, for example, acquires data from a task management tool to identify the user's current task. The voice recognition unit can also identify the user's current project based on calendar information. The voice recognition unit can also prioritize voice commands based on the user's current task. For example, the voice recognition unit prioritizes recognition of voice commands related to the user's current task. The voice recognition unit can also acquire information from the user's task management tool and prioritize recognition of related voice commands. The voice recognition unit can also prioritize recognition of optimal voice commands by taking into account the progress of the user's current task. This improves the efficiency of voice recognition by prioritizing voice commands based on the user's current task. Some or all of the above-mentioned processing in the voice recognition unit may be performed using AI, for example, or without AI.

[0088] The voice recognition unit can estimate a user's emotions and prioritize voice commands based on the estimated user emotions. The voice recognition unit estimates the user's emotions using, for example, generative AI. The voice recognition unit can estimate emotions by analyzing the user's facial expressions using, for example, facial expression recognition technology. The voice recognition unit can also estimate emotions by analyzing the tone and speed of the user's voice using voice analysis technology. Furthermore, the voice recognition unit can estimate the user's emotions through behavioral pattern analysis. The voice recognition unit prioritizes voice commands based on the estimated user emotions. For example, if the user is stressed, voice commands for stress reduction can be prioritized. Also, if the user is relaxed, detailed voice commands can be prioritized. Furthermore, if the user is in a hurry, voice commands that can be responded to quickly can be prioritized. This improves the efficiency of voice recognition by prioritizing voice commands according to the user's emotions. Emotion estimation is achieved using an emotion estimation function, for example, an emotion engine or generative AI. The generation AI may be a text generation AI (e.g., LLM) or a multimodal generation AI, but is not limited to such examples.

[0089] During voice recognition, the voice recognition unit can prioritize recognition of specific commands by taking into account the user's geographical location information. The voice recognition unit, for example, acquires the user's geographical location information using a generation AI. The voice recognition unit identifies the user's location information, for example, through analysis of GPS data or an IP address. The voice recognition unit can also prioritize recognition of highly relevant commands based on the user's geographical location information. For example, if the user is working in a specific area, the voice recognition unit can prioritize recognition of voice commands related to that area. Furthermore, if the user is on a business trip, the voice recognition unit can prioritize recognition of relevant voice commands based on information about the business trip destination. Furthermore, if the user is working at home, the voice recognition unit can prioritize recognition of voice commands related to the home environment. This improves the efficiency of voice recognition by recognizing highly relevant commands based on the user's geographical location information. Some or all of the above-described processing in the voice recognition unit may be performed, for example, using AI or without AI.

[0090] The voice recognition unit can analyze the user's social media activities and recognize related commands during voice recognition. The voice recognition unit can analyze the user's social media activities using, for example, generative AI. The voice recognition unit can identify the user's interests through, for example, analyzing the content of posts. The voice recognition unit can also understand the user's social network through an analysis of followers. The voice recognition unit can also recognize related commands based on the user's social media activities. For example, the voice recognition unit can recognize related voice commands based on the content frequently posted by the user on social media. The voice recognition unit can also recognize related commands based on the activities of the user's friends on social media. The voice recognition unit can also recognize related voice commands based on the user's social media check-in information. This improves the efficiency of voice recognition by recognizing related commands based on the user's social media activities. Some or all of the above-described processing in the voice recognition unit can be performed using, for example, AI, or without AI.

[0091] The voice recognition unit can customize the recognition method by reflecting the user's past feedback during voice recognition. The voice recognition unit, for example, analyzes the user's past feedback using a generative AI. The voice recognition unit, for example, identifies the user's requests and areas for improvement through analyzing the feedback. The voice recognition unit can also adjust the voice recognition algorithm based on the feedback. For example, the voice recognition unit adjusts the voice recognition algorithm based on feedback provided by the user in the past. The voice recognition unit can also preferentially recognize specific voice commands based on the user's feedback. Furthermore, the voice recognition unit can analyze the user's feedback and make improvements to improve the accuracy of voice recognition. As a result, the accuracy of voice recognition is improved by customizing the recognition method by reflecting the user's past feedback. Some or all of the above-mentioned processing in the voice recognition unit may be performed using AI, for example, or without using AI.

[0092] The text conversion unit can estimate the user's emotions and adjust the text expression method based on the estimated user's emotions. The text conversion unit, for example, uses a generative AI to estimate the user's emotions. The text conversion unit can also estimate the user's emotions by analyzing the user's facial expressions using facial expression recognition technology. The text conversion unit can also estimate the user's emotions by analyzing the tone and speed of the user's voice using voice analysis technology. The text conversion unit can also estimate the user's emotions through behavioral pattern analysis. The text conversion unit adjusts the text expression method based on the estimated user's emotions. For example, if the user is stressed, a simple and easy-to-understand expression method can be used. If the user is relaxed, a detailed expression method can be used. If the user is in a hurry, a concise expression method that focuses on the main points can be used. This improves the accuracy of the text conversion by adjusting the text expression method according to the user's emotions. Emotion estimation is achieved using an emotion estimation function, for example, an emotion engine or generative AI. The generation AI may be a text generation AI (e.g., LLM) or a multimodal generation AI, but is not limited to such examples.

[0093] The text conversion unit can adjust the level of detail of the text based on the importance of the task when converting the text to text. The text conversion unit, for example, uses a generation AI to identify the importance of the task. The text conversion unit evaluates the importance of the task based on, for example, task priority or project importance. The text conversion unit can also adjust the level of detail of the text based on the importance of the task. For example, the text conversion unit generates text including a detailed explanation for an important task. The text conversion unit can also generate concise text for a low-importance task. Furthermore, the text conversion unit can adjust the level of detail of the text according to the importance of the task. This improves the efficiency of the text conversion by adjusting the level of detail of the text according to the importance of the task. Some or all of the above-described processing in the text conversion unit may be performed using, for example, AI, or may be performed without using AI.

[0094] The text conversion unit can apply different text conversion algorithms depending on the task category when converting the data into text. The text conversion unit, for example, uses a generative AI to identify the task category. The text conversion unit can identify the task category based on, for example, task classification or project category. The text conversion unit can also apply an optimal text conversion algorithm depending on the task category. For example, the text conversion unit can generate text including technical terms for technical tasks. The text conversion unit can also generate text using easy-to-understand expressions for general tasks. The text conversion unit can also apply an optimal text conversion algorithm depending on the task category. In this way, applying a text conversion algorithm depending on the task category improves the accuracy of the text conversion. Some or all of the above-mentioned processing in the text conversion unit may be performed using, for example, AI, or may be performed without using AI.

[0095] The text conversion unit can improve the accuracy of the text by referring to the user's past text conversion results during text conversion. The text conversion unit, for example, uses a generation AI to refer to the user's past text conversion results. The text conversion unit can identify the user's past text conversion results, for example, based on past procedure manuals or text conversion logs. The text conversion unit can also optimize the text conversion algorithm based on the past text conversion results. For example, the text conversion unit optimizes the text conversion algorithm based on text generated by the user in the past. The text conversion unit can also learn specific expression methods from the user's past text conversion results to improve accuracy. Furthermore, the text conversion unit can analyze the user's past text conversion results and suggest an optimal text conversion method. This improves the accuracy of the text conversion by referring to the user's past text conversion results. Some or all of the above-mentioned processing in the text conversion unit may be performed, for example, using AI or without AI.

[0096] The text conversion unit can estimate the user's emotion and adjust the length of the text based on the estimated user's emotion. The text conversion unit estimates the user's emotion using, for example, a generation AI. The text conversion unit can analyze the user's facial expression using, for example, facial expression recognition technology to estimate the emotion. The text conversion unit can also analyze the user's tone and speed of voice using voice analysis technology to estimate the emotion. The text conversion unit can also estimate the user's emotion through behavioral pattern analysis. The text conversion unit adjusts the length of the text based on the estimated user's emotion. For example, if the user is stressed, the text conversion unit generates short, to-the-point text. If the user is relaxed, the text conversion unit generates longer text with detailed explanations. If the user is in a hurry, the text conversion unit can generate concise, easy-to-understand text. This improves the efficiency of text conversion by adjusting the length of the text according to the user's emotion. Emotion estimation is achieved using, for example, an emotion engine or a generation AI with an emotion estimation function. The generation AI can be, but is not limited to, a text generation AI (e.g., LLM) or a multimodal generation AI.

[0097] The text conversion unit can determine the priority of text based on the submission date of work during text conversion. The text conversion unit, for example, uses a generation AI to identify the submission date of work. The text conversion unit can also determine the priority of text based on the submission date of work. For example, the text conversion unit prioritizes text conversion for work with an approaching deadline. The text conversion unit can also postpone text conversion for work with a distant submission date. Furthermore, the text conversion unit can adjust the priority of text depending on the submission date of work. Thus, by determining the priority of text based on the submission date of work, the efficiency of text conversion is improved. Some or all of the above-mentioned processing in the text conversion unit may be performed using, for example, AI, or may be performed without using AI.

[0098] The text conversion unit can adjust the order of text based on the relevance of tasks during text conversion. The text conversion unit, for example, uses generative AI to identify the relevance of tasks. The text conversion unit can evaluate the relevance of tasks based on, for example, task dependencies or project relevance. The text conversion unit can also adjust the order of text based on the relevance of tasks. For example, the text conversion unit can prioritize the conversion of highly relevant tasks into text and adjust the order. The text conversion unit can also postpone the conversion of less relevant tasks. Furthermore, the text conversion unit can optimize the order of text according to the relevance of tasks. As a result, adjusting the order of text based on the relevance of tasks improves the efficiency of text conversion. Some or all of the above-described processing in the text conversion unit may be performed using, for example, AI, or may be performed without using AI.

[0099] The text conversion unit can adjust the use of technical terms in the text during text conversion according to the user's level of expertise. The text conversion unit, for example, uses a generation AI to identify the user's level of expertise. The text conversion unit can identify the user's level of expertise based on, for example, survey results or past operation history. The text conversion unit can also adjust the use of technical terms in the text according to the user's level of expertise. For example, if the user has technical expertise, the text conversion unit can generate text that uses a lot of technical terms. Also, if the user is a beginner, the text conversion unit can generate text that uses easy-to-understand expressions. Furthermore, the text conversion unit can select an optimal text conversion method according to the user's level of expertise. This improves the accuracy of the text conversion by adjusting the use of technical terms in the text according to the user's level of expertise. Some or all of the above-mentioned processing in the text conversion unit may be performed using, for example, AI, or may be performed without using AI.

[0100] The automation unit can estimate the user's emotions and adjust the automation method based on the estimated user's emotions. The automation unit estimates the user's emotions using, for example, a generative AI. The automation unit can analyze the user's facial expressions using, for example, facial expression recognition technology to estimate emotions. The automation unit can also use voice analysis technology to analyze the tone and speed of the user's voice to estimate emotions. The automation unit can also estimate the user's emotions through behavioral pattern analysis. The automation unit adjusts the automation method based on the estimated user's emotions. For example, if the user is stressed, a simple and easy-to-understand automation method is adopted. If the user is relaxed, a detailed automation method is adopted. Furthermore, if the user is in a hurry, an automation method that can respond quickly can be adopted. This improves the accuracy of automation by adjusting the automation method according to the user's emotions. Emotion estimation is achieved using an emotion estimation function using, for example, an emotion engine or generative AI. The generative AI can be, but is not limited to, a text generation AI (e.g., LLM) or a multimodal generation AI.

[0101] During automation, the automation unit can analyze the user's past operation history and select the optimal automation method. The automation unit can analyze the user's past operation history using, for example, a generation AI. The automation unit can identify operations that the user frequently performed in the past through, for example, analysis of operation logs. The automation unit can also predict operations that will be performed in a specific time period through frequency analysis. The automation unit can also propose efficient operation procedures based on the operation history and select an automation method based on the procedures. For example, the automation unit can select the optimal automation method based on operations that the user frequently performed in the past. The automation unit can also predict operations that will be performed in a specific time period from the user's operation history and select the optimal automation method. The automation unit can also analyze the user's operation history, propose efficient operation procedures, and select an automation method based on the procedures. This improves the efficiency of automation by analyzing the user's past operation history and selecting the optimal automation method. Some or all of the above-described processing in the automation unit can be performed using, for example, AI, or without AI.

[0102] During automation, the automation unit can customize the means of automation based on the user's current living situation. The automation unit, for example, uses a generative AI to identify the user's current living situation. The automation unit can identify the user's living situation, for example, through an analysis of survey results or behavioral patterns. The automation unit can also customize the means of automation based on the user's current living situation. For example, if the user is working from home, the automation unit can provide an automation means optimized for the home environment. Also, if the user is on a business trip, the automation unit can provide an automation means optimized for the environment of the business trip destination. Furthermore, the automation unit can customize the optimal automation means by taking the user's living situation into consideration. This improves the efficiency of automation by customizing the means of automation based on the user's current living situation. Some or all of the above-described processing in the automation unit may be performed, for example, using AI or without using AI.

[0103] During automation, the automation unit can improve the automation method by reflecting user feedback. The automation unit, for example, analyzes user feedback using generative AI. The automation unit, for example, identifies user requests and areas for improvement through analyzing the feedback. The automation unit can also improve the automation method based on the feedback. For example, the automation unit improves the automation method based on feedback provided by the user. The automation unit can also provide an automation method with enhanced specific functions based on the user feedback. Furthermore, the automation unit can analyze user feedback and make improvements to improve the accuracy of the automation. As a result, improving the automation method by reflecting user feedback improves the accuracy of the automation. Some or all of the above-mentioned processing in the automation unit may be performed, for example, using AI or without using AI.

[0104] During automation, the automation unit can select the optimal automation method taking into account the user's geographic location information. The automation unit, for example, acquires the user's geographic location information using a generation AI. The automation unit identifies the user's location information, for example, through analysis of GPS data or an IP address. The automation unit can also select the optimal automation method based on the user's geographic location information. For example, if the user is working in a specific area, the automation unit selects an automation method related to that area. If the user is on a business trip, the automation unit can select the optimal automation method based on information about the business trip destination. Furthermore, if the user is working at home, the automation unit can select an automation method optimized for the home environment. This improves the efficiency of automation by selecting the optimal automation method based on the user's geographic location information. Some or all of the above-described processing in the automation unit may be performed using AI, for example, or without AI.

[0105] During automation, the automation unit can analyze the user's social media activity and suggest automation measures. The automation unit can analyze the user's social media activity using, for example, generative AI. The automation unit can identify the user's interests through analysis of posted content, for example. The automation unit can also understand the user's social network through analysis of followers. The automation unit can also suggest automation measures based on the user's social media activity. For example, the automation unit can suggest relevant automation measures based on the content the user frequently posts on social media. The automation unit can also suggest relevant automation measures based on the activity of the user's friends on social media. The automation unit can also suggest relevant automation measures based on the user's social media check-in information. This improves the efficiency of automation by suggesting automation measures based on the user's social media activity. Some or all of the above-mentioned processing in the automation unit can be performed using, for example, AI, or without AI.

[0106] During automation, the automation unit can customize the automation method by reflecting the user's past feedback. The automation unit, for example, analyzes the user's past feedback using a generative AI. The automation unit, for example, identifies the user's requests and areas for improvement through analyzing the feedback. The automation unit can also customize the automation method based on the feedback. For example, the automation unit customizes the automation method based on feedback provided by the user in the past. The automation unit can also provide an automation method with enhanced specific functions based on the user's feedback. Furthermore, the automation unit can analyze the user's feedback and suggest an optimal automation method. In this way, customizing the automation method by reflecting the user's past feedback improves the accuracy of automation. Some or all of the above-described processing in the automation unit may be performed, for example, using AI or without using AI. === Hard Collateral 1-1 === Each of the multiple elements including the above-mentioned generation unit, learning unit, voice recognition unit, text conversion unit, and automation unit is realized, for example, by at least one of the smart device 14 and the data processing device 12. For example, the generation unit is realized by the control unit 46A of the smart device 14 or the specific processing unit 290 of the data processing device 12. The learning unit is realized, for example, by the specific processing unit 290 of the data processing device 12. The voice recognition unit is realized, for example, by the control unit 46A of the smart device 14 or the specific processing unit 290 of the data processing device 12. The text conversion unit is realized, for example, by the control unit 46A of the smart device 14 or the specific processing unit 290 of the data processing device 12. The automation unit is realized, for example, by the control unit 46A of the smart device 14 or the specific processing unit 290 of the data processing device 12. === Hard Collateral 1-2 === Each of the multiple elements including the above-mentioned generation unit, learning unit, voice recognition unit, text conversion unit, and automation unit is realized, for example, by at least one of the smart glasses 214 and the data processing device 12. For example, the generation unit is realized by the control unit 46A of the smart glasses 214 or the specific processing unit 290 of the data processing device 12. The learning unit is realized, for example, by the specific processing unit 290 of the data processing device 12. The voice recognition unit is realized, for example, by the control unit 46A of the smart glasses 214 or the specific processing unit 290 of the data processing device 12. The text conversion unit is realized, for example, by the control unit 46A of the smart glasses 214 or the specific processing unit 290 of the data processing device 12. The automation unit is realized, for example, by the control unit 46A of the smart glasses 214 or the specific processing unit 290 of the data processing device 12. === Hard Collateral 1-3 === Each of the multiple elements including the above-mentioned generation unit, learning unit, speech recognition unit, text conversion unit, and automation unit is realized, for example, by at least one of the headset type terminal 314 and the data processing device 12. For example, the generation unit is realized by the control unit 46A of the headset type terminal 314 or the specific processing unit 290 of the data processing device 12. The learning unit is realized, for example, by the specific processing unit 290 of the data processing device 12. The speech recognition unit is realized, for example, by the control unit 46A of the headset type terminal 314 or the specific processing unit 290 of the data processing device 12. The text conversion unit is realized, for example, by the control unit 46A of the headset type terminal 314 or the specific processing unit 290 of the data processing device 12. The automation unit is realized, for example, by the control unit 46A of the headset type terminal 314 or the specific processing unit 290 of the data processing device 12. === Hard Collateral 1-4 === Each of the multiple elements including the above-mentioned generation unit, learning unit, voice recognition unit, text conversion unit, and automation unit is realized, for example, by at least one of the robot 414 and the data processing device 12. For example, the generation unit is realized by the control unit 46A of the robot 414 or the specific processing unit 290 of the data processing device 12. The learning unit is realized, for example, by the specific processing unit 290 of the data processing device 12. The voice recognition unit is realized, for example, by the control unit 46A of the robot 414 or the specific processing unit 290 of the data processing device 12. The text conversion unit is realized, for example, by the control unit 46A of the robot 414 or the specific processing unit 290 of the data processing device 12. The automation unit is realized, for example, by the control unit 46A of the robot 414 or the specific processing unit 290 of the data processing device 12.

[0107] The system according to the embodiment is not limited to the above-described example, and various modifications are possible, for example, as follows.

[0108] The generation unit can also acquire biometric information about the user and adjust the generation of the automation program based on the user's health condition. For example, the generation unit can acquire data such as heart rate and blood pressure from the wearable device and delay the generation of the automation program if the user is tired. Furthermore, the generation unit can generate a more complex automation program if the user is in good health. Furthermore, the generation unit can adjust the priority of tasks based on the user's health condition. This allows the generation of an optimal automation program tailored to the user's health condition.

[0109] The generation unit can also estimate the user's emotions and adjust the complexity of the automation program based on the estimated user emotions. For example, if the user is feeling stressed, the generation unit generates a simple automation program, and if the user is relaxed, the generation unit generates a more complex automation program. Also, if the user is in a hurry, the generation unit can generate an automation program that can be completed quickly. Furthermore, if the user is concentrating, the generation unit can generate an automation program that includes detailed steps. This makes it possible to provide an optimal automation program according to the user's emotions.

[0110] The learning unit can also analyze the user's operation history and make suggestions to improve operation efficiency. For example, the learning unit can identify operations that the user frequently performs and make suggestions to automate those operations. The learning unit can also make suggestions to optimize the order of operations performed by the user. Furthermore, the learning unit can also suggest new procedures to improve operation efficiency based on the user's operation history. This makes it possible to provide efficient operation procedures based on the user's operation history.

[0111] The learning unit can also estimate the user's emotions and adjust the learning progress speed based on the estimated user emotions. For example, if the user is feeling stressed, the learning unit slows down the learning progress speed, and if the user is relaxed, the learning progress speed can be increased. Also, if the user is concentrating, the learning progress speed can be optimized. Furthermore, if the user is tired, the learning progress speed can be adjusted to reduce the user's burden. In this way, it is possible to provide an optimal learning progress speed according to the user's emotions.

[0112] The speech recognition unit can also analyze the user's speech and use context information to understand the user's intent. For example, the speech recognition unit can refer to the user's past speech history to understand the current speech. The speech recognition unit can also interpret the speech based on the user's current task or project. Furthermore, the speech recognition unit can obtain external information related to the user's speech and use it as auxiliary information to understand the speech. This can result in a more accurate understanding of the user's speech.

[0113] The voice recognition unit can also estimate the user's emotions and adjust the feedback method of voice commands based on the estimated user's emotions. For example, if the user is stressed, the voice recognition unit can provide simple and easy-to-understand feedback, and if the user is relaxed, it can provide detailed feedback. If the user is in a hurry, it can provide feedback that allows for a quick response. Furthermore, if the user is concentrating, it can provide feedback that includes detailed instructions. In this way, it is possible to provide optimal feedback according to the user's emotions.

[0114] The text conversion unit can also analyze the user's input content and select appropriate expressions according to the context. For example, the text conversion unit can select appropriate technical terms and expressions based on the user's input content. The text conversion unit can also obtain external information related to the user's input content and select expressions according to the context. Furthermore, the text conversion unit can analyze the user's input content and select an appropriate format according to the context. This makes it possible to provide appropriate expressions according to the user's input content.

[0115] The text generator can also estimate the user's emotions and adjust the tone of the text based on the estimated user's emotions. For example, if the user is stressed, the text generator can generate text with a gentle tone, and if the user is relaxed, the text generator can generate text with a friendly tone. If the user is in a hurry, the text generator can generate text with a concise and direct tone. Furthermore, if the user is focused, the text generator can generate text with a detailed and professional tone. This makes it possible to provide text with an optimal tone according to the user's emotions.

[0116] The automation unit can also propose new automation means for improving the efficiency of operations based on the user's operation history. For example, the automation unit can identify operations that the user frequently performs and propose automating those operations. The automation unit can also propose optimizing the order of operations performed by the user. Furthermore, the automation unit can propose new procedures for improving the efficiency of operations based on the user's operation history. This makes it possible to provide efficient automation means based on the user's operation history.

[0117] The automation unit can also estimate the user's emotions and adjust the timing of automation based on the estimated user emotions. For example, if the user is feeling stressed, the automation unit delays the timing of automation, and if the user is relaxed, the automation unit speeds up the timing of automation. Also, if the user is in a hurry, the automation unit can set the timing of automation to allow for a quick response. Furthermore, if the user is concentrating, the automation unit can perform the automation at the optimal timing. This makes it possible to provide the optimal automation timing according to the user's emotions.

[0118] The processing flow of the second embodiment will be briefly explained below.

[0119] Step 1: The generation unit understands the user's PC operations and creates an automation program. The generation unit uses the generation AI to observe the user's operations and learn the procedures. The generation AI can also receive prompts such as "Please automate this operation" and convert the operation procedures into an automation program. Furthermore, the generation AI can monitor the user's operations in real time and generate the optimal automation program. Step 2: The learning unit learns the user's work patterns based on the program generated by the generation unit. The learning unit uses a machine learning model to analyze the user's operation history and learns frequently performed operations and combinations of consecutive operations. It can also suggest optimal work procedures based on the user's operation patterns. Furthermore, it can continuously learn the user's operation patterns and improve the accuracy of the program. Step 3: The voice recognition unit understands voice commands and automates tasks based on the task patterns learned by the learning unit. The voice recognition unit uses voice recognition AI to analyze the user's voice commands and automate the corresponding tasks. It can also recognize the user's voice commands in real time and execute tasks instantly. It can also learn the user's voice commands and improve accuracy. Step 4: The text conversion unit converts the automated tasks created by the voice recognition unit into text and creates a procedure manual. The procedure manual is automatically generated based on the text data generated by the voice recognition AI. It is also possible to record the user's operation procedures in detail and reflect them in the procedure manual. Furthermore, it is possible to customize the format of the procedure manual and create a procedure manual tailored to the user's needs. Step 5: The automation unit uses the Self-Operating Computer Framework to learn and automate complex operations. It learns business processes that use multiple tools and automatically executes the optimal procedure. It can also observe user operations and consider the optimal process. It can also generate programs to automate complex operations based on the user's operation history.

[0120] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0121] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> Examples of generative AIs include the data generation model 58, such as a neural network model (e.g., a neural network model), and a neural network model (e.g., a neural network model). The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating speech, text data indicating text, and image data indicating an image is also input to the data generation model 58. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specification processing unit 290 performs the above-mentioned specification processing using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction. In this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. The data processing device 12 and the like include multiple types of data generation models 58, and the data generation model 58 includes AIs other than the generative AI. The AI ​​other than the generative AI may be, for example, linear regression, logistic regression, decision tree, random forest, support vector machine (SVM), k-means clustering, convolutional neural network (CNN), recurrent neural network (RNN), generative adversarial network (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. The AI ​​may also be an AI agent. When the processes of each of the above-mentioned parts are performed by AI, the processes may be performed in part or entirely by AI, but are not limited to these examples. The processes performed by AI, including the generative AI, may be replaced with rule-based processes.

[0122] Furthermore, the processing by the data processing system 10 described above is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart device 14, but may also be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart device 14. Furthermore, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the smart device 14 or an external device, and the smart device 14 acquires or collects information necessary for processing from the data processing device 12 or an external device.

[0123] The correspondence between each part and the device or control part is not limited to the example described above, and various modifications are possible.

[0124] [Second embodiment] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0125] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0126] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN and / or a LAN.

[0127] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0128] The microphone 238 receives instructions and the like from the user by receiving voice uttered by the user. The microphone 238 captures the voice uttered by the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to instructions from the processor 46.

[0129] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the user's surroundings (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0130] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0131] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0132] The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0133] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate a user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotion, including estimation and prediction of the user's emotion, but is not limited to these examples. Furthermore, the estimation and prediction of emotion also includes, for example, emotion analysis.

[0134] In the smart glasses 214, the specific processing is performed by the processor 46. A specific processing program 60 is stored in the storage 50. The processor 46 reads the specific processing program 60 from the storage 50 and executes the read specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as the control unit 46A in accordance with the specific processing program 60 executed on the RAM 48. The smart glasses 214 also have a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59, and can perform processing similar to that of the specific processing unit 290 using these models.

[0135] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 communicates with the server device having the data generation model 58 to obtain a processing result (such as a prediction result) using the data generation model 58. Furthermore, the data processing device 12 may be a server device, or may be a terminal device (for example, a mobile phone, a robot, a home appliance, etc.) owned by a user.

[0136] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0137] The data generation model 58 is a so-called generative AI. An example of the data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 receives a prompt containing an instruction, as well as inference data such as voice data representing speech, text data representing text, and image data representing an image. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The identification processing unit 290 performs the above-mentioned identification processing using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction. In this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. The data processing device 12 and the like include multiple types of data generation models 58, and the data generation model 58 includes AIs other than the generative AI. The AI ​​other than the generative AI may be, for example, linear regression, logistic regression, decision tree, random forest, support vector machine (SVM), k-means clustering, convolutional neural network (CNN), recurrent neural network (RNN), generative adversarial network (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. The AI ​​may also be an AI agent. When the processes of each of the above-mentioned parts are performed by AI, the processes may be performed in part or entirely by AI, but are not limited to these examples. The processes performed by AI, including the generative AI, may be replaced with rule-based processes.

[0138] The data processing system 210 according to the second embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 210 is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart glasses 214, but may also be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart glasses 214. Furthermore, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the smart glasses 214 or an external device, etc., and the smart glasses 214 acquires or collects information required for processing from the data processing device 12 or an external device, etc.

[0139] The correspondence between each part and the device or control part is not limited to the example described above, and various modifications are possible.

[0140] [Third embodiment] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0141] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0142] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN and / or a LAN.

[0143] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0144] The microphone 238 receives instructions and the like from the user by receiving voice uttered by the user. The microphone 238 captures the voice uttered by the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to instructions from the processor 46.

[0145] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the user's surroundings (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0146] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0147] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0148] The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0149] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate a user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotion, including estimation and prediction of the user's emotion, but is not limited to these examples. Furthermore, the estimation and prediction of emotion also includes, for example, emotion analysis.

[0150] In the headset type terminal 314, the identification process is performed by the processor 46. A identification program 60 is stored in the storage 50. The processor 46 reads the identification program 60 from the storage 50 and executes the read identification program 60 on the RAM 48. The identification process is realized by the processor 46 operating as a control unit 46A in accordance with the identification program 60 executed on the RAM 48. Note that the headset type terminal 314 has a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59, and can also perform processing similar to that of the identification processing unit 290 using these models.

[0151] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 communicates with the server device having the data generation model 58 to obtain a processing result (such as a prediction result) using the data generation model 58. Furthermore, the data processing device 12 may be a server device, or may be a terminal device (for example, a mobile phone, a robot, a home appliance, etc.) owned by a user.

[0152] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0153] The data generation model 58 is a so-called generative AI. An example of the data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 receives a prompt containing an instruction, as well as inference data such as voice data representing speech, text data representing text, and image data representing an image. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The identification processing unit 290 performs the above-mentioned identification processing using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction. In this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. The data processing device 12 and the like include multiple types of data generation models 58, and the data generation model 58 includes AIs other than the generative AI. The AI ​​other than the generative AI may be, for example, linear regression, logistic regression, decision tree, random forest, support vector machine (SVM), k-means clustering, convolutional neural network (CNN), recurrent neural network (RNN), generative adversarial network (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. The AI ​​may also be an AI agent. When the processes of each of the above-mentioned parts are performed by AI, the processes may be performed in part or entirely by AI, but are not limited to these examples. The processes performed by AI, including the generative AI, may be replaced with rule-based processes.

[0154] The data processing system 310 according to the third embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 310 is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the headset type terminal 314, but may also be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the headset type terminal 314. Furthermore, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the headset type terminal 314 or an external device, etc., and the headset type terminal 314 acquires or collects information required for processing from the data processing device 12 or an external device, etc.

[0155] The correspondence between each part and the device or control part is not limited to the example described above, and various modifications are possible.

[0156] [Fourth embodiment] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[0157] 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0158] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN and / or a LAN.

[0159] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[0160] The microphone 238 receives instructions and the like from the user by receiving voice uttered by the user. The microphone 238 captures the voice uttered by the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to instructions from the processor 46.

[0161] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS image sensor or a CCD image sensor, and captures images of the user's surroundings (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0162] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0163] The control object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[0164] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0165] The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0166] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate a user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotion, including estimation and prediction of the user's emotion, but is not limited to these examples. Furthermore, the estimation and prediction of emotion also includes, for example, emotion analysis.

[0167] In the robot 414, the processor 46 performs the identification process. The storage 50 stores the identification program 60. The processor 46 reads the identification program 60 from the storage 50 and executes the read identification program 60 on the RAM 48. The identification process is realized by the processor 46 operating as the control unit 46A in accordance with the identification program 60 executed on the RAM 48. The robot 414 also has a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59, and can perform the same process as the identification processing unit 290 using these models.

[0168] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 communicates with the server device having the data generation model 58 to obtain a processing result (such as a prediction result) using the data generation model 58. Furthermore, the data processing device 12 may be a server device, or may be a terminal device (for example, a mobile phone, a robot, a home appliance, etc.) owned by a user.

[0169] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0170] The data generation model 58 is a so-called generative AI. An example of the data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 receives a prompt containing an instruction, as well as inference data such as voice data representing speech, text data representing text, and image data representing an image. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The identification processing unit 290 performs the above-mentioned identification processing using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction. In this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. The data processing device 12 and the like include multiple types of data generation models 58, and the data generation model 58 includes AIs other than the generative AI. The AI ​​other than the generative AI may be, for example, linear regression, logistic regression, decision tree, random forest, support vector machine (SVM), k-means clustering, convolutional neural network (CNN), recurrent neural network (RNN), generative adversarial network (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. The AI ​​may also be an AI agent. When the processes of each of the above-mentioned parts are performed by AI, the processes may be performed in part or entirely by AI, but are not limited to these examples. The processes performed by AI, including the generative AI, may be replaced with rule-based processes.

[0171] The data processing system 410 according to the fourth embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 410 is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the robot 414, but may also be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the robot 414. Furthermore, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the robot 414 or an external device, etc., and the robot 414 acquires or collects information required for processing from the data processing device 12 or an external device, etc.

[0172] The correspondence between each part and the device or control part is not limited to the example described above, and various modifications are possible.

[0173] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0174] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion encompasses both emotions and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[0175] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[0176] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[0177] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is expressed, and when they approach the ideal, a state of pleasure is expressed. Emotions can also be created for robots, cars, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is expressed, and when they approach the ideal, a state of pleasure is expressed. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on speech emotion recognition and brain physiological signal analysis systems for emotions, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the area called "reaction," where sensation is dominant. The right half of the emotion map lists emotions belonging to the area called "situation," where situational awareness is dominant.

[0178] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[0179] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[0180] In the above embodiment, an example was given in which a specific process is performed by one computer 22, but the technology disclosed herein is not limited to this, and distributed processing of the specific process may be performed by multiple computers including computer 22.

[0181] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[0182] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0183] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[0184] The hardware resource for executing a specific process can be any of the following types of processors: A processor, for example, is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. A processor also includes a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[0185] The hardware resource that executes the specific process may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific process may be a single processor.

[0186] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[0187] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[0188] In the above example, the first to fourth embodiments have been described separately, but some or all of these embodiments may be combined. The smart device 14, smart glasses 214, headset terminal 314, and robot 414 are merely examples, and they may be combined, or other devices may be used. In the above example, the first and second embodiments have been described separately, but they may be combined.

[0189] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[0190] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[0191] [Explanation of symbols]

[0192] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot

Claims

1. A generation part that understands the user's PC operations and creates automation programs; a learning unit that learns a user's work pattern based on the program generated by the generation unit; a voice recognition unit that understands voice commands based on the work patterns learned by the learning unit and automates the work; a text generation unit that generates a procedure manual by converting the automated work performed by the voice recognition unit into text; An automation unit that uses specific technology to learn and automate complex operations. A system characterized by:

2. The generation unit Estimate the user's emotions and adjust the timing of the automated program generation based on the estimated user emotions.

2. The system of claim 1.

3. The generation unit Analyze user operation history and generate efficient automation programs 2. The system of claim 1.

4. The generation unit When generating an automation program, customize it based on the user's current task or project.

2. The system of claim 1.

5. The generation unit When generating an automated program, select the appropriate generation method according to the user's input method.

2. The system of claim 1.

6. The generation unit Estimate the user's emotions and determine the priority of the programs to be generated based on the estimated user emotions.

2. The system of claim 1.

7. The generation unit When generating automated programs, the geographical location information of the user is taken into consideration to generate specific programs with priority.

2. The system of claim 1.

8. The generation unit When generating automated programs, analyze users' social media activity and generate relevant programs.

2. The system of claim 1.

Images