system

Abstract

We provide the system. [Solution] A means of monitoring user actions and collecting them as data, A means for receiving the voice input of the user and analyzing the voice data, A system device that analyzes the collected operation data and voice data and generates appropriate support information, A means for providing the generated support information to the user via screen display and audio output, Means for performing an operation specified based on the response from the user, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the current business environment where remote work has increased, there is a problem that the opportunity for face - to - face communication has decreased and the follow - up of work is insufficient. At the same time, the variation in employees' work skills may prevent efficient work performance and may have an adverse impact on productivity. To solve these problems, a mechanism is needed that allows employees to receive appropriate support in their PC operations.

Means for Solving the Problems

[0005] This invention solves the above problems by providing a system for monitoring and analyzing user operations. This system monitors the user's PC operations in real time and also receives and analyzes voice input to generate useful support information for the user. This support information is then presented on the screen and audibly, and the system can automatically execute necessary operations based on the user's response. As a result, employees can improve their work skills and efficiency, and achieve increased productivity in their work.

[0006] "Means for monitoring user operations" refers to a device that has the function of detecting and recording actions such as keyboard input and mouse clicks on a PC in real time.

[0007] A "means for receiving voice input" refers to a device that uses a microphone to collect the user's verbal instructions and questions as digital data.

[0008] "Means for analyzing data" refers to a system or program that has the function of performing analysis to generate appropriate support information based on collected operational data and voice data.

[0009] A "system device for generating support information" refers to a device or program that has the function of automatically creating useful advice and suggestions based on the user's operations and requests.

[0010] "Means of providing information to users through screen display and audio output" refers to a device that has functions using a display and speakers, etc., to present generated support information to the user visually or audibly.

[0011] "Means for performing specified operations" refers to a system and related equipment that have the function of automatically performing the determined operations on a PC in response to instructions from the user.

[0012] "Providing suggestions based on past operation history" refers to the act of using data from the user's previous operations to provide advice on how to streamline future tasks.

[0013] "Sending to a server and communicating in an encrypted state" refers to the process of using encryption technology when a user's confidential information is transferred over the internet in order to maintain data security. [Brief explanation of the drawing]

[0014] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when the emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when the emotion engine is combined.

Mode for Carrying Out the Invention

[0015] Hereinafter, an example of an embodiment of the system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0016] First, the language used in the following description will be explained.

[0017] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0018] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0019] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0020] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0022] [First Embodiment]

[0023] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0024] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0027] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0030] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0034] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0035] This invention relates to a system that comprehensively supports user operations on a PC, enabling effective work performance even in a teleworking environment.

[0036] Program Processing Overview

[0037] This system operates by monitoring and analyzing various operations and inputs that occur on the user's PC in real time. First, the terminal constantly monitors a series of user operations on the PC and collects data. This includes keyboard input, mouse movements, and application usage. The terminal also accepts voice input, using a microphone to recognize user instructions and questions and record them as audio data.

[0038] Next, the collected data is transmitted to a server via the internet, and secure communication is ensured using encryption technology. The server analyzes this data and uses a generative AI model to generate support information that the user needs. This support information includes suggestions for streamlining the user's current tasks and specific advice on operations.

[0039] Upon receiving the analysis results, the terminal displays support information as a pop-up on the screen and, if necessary, provides audio output through the speaker. The user responds by voice or click and selects an action provided by the system. Based on the selection, the terminal automatically performs the specified action. For example, this can automate operations such as updating a spreadsheet or retrieving documents.

[0040] As a concrete example, suppose a user is preparing for a regular meeting. At this time, the user needs to retrieve the latest data for the meeting. The device detects the user's editing operation in the spreadsheet and makes a voice suggestion, "Do you want to retrieve the latest data?" If the user voice-approves with "Yes," the device automatically retrieves the necessary information from the database and updates the specified spreadsheet.

[0041] By implementing this system, users can perform their tasks without worrying about a lack of information or operational errors, enabling efficient and accurate work. Therefore, variations in work skills are reduced, contributing to improved overall productivity.

[0042] The following describes the processing flow.

[0043] Step 1:

[0044] The device monitors the user's keyboard input, mouse movements, and application usage in real time, collecting this data locally. A microphone is also constantly on standby for voice input, recording user voice commands as digital audio data.

[0045] Step 2:

[0046] The device transmits the collected operation data and voice data to the server via the internet. During this transmission, the data is encrypted to ensure its security.

[0047] Step 3:

[0048] The server starts analysis using a generative AI model based on the received operation data and voice data. This determines the context and intent of the user's actions and generates appropriate support information.

[0049] Step 4:

[0050] The server generates support information derived from the analysis results and sends it back to the terminal. This support information includes suggestions for the user's current operations and advice on tasks that can be made more efficient.

[0051] Step 5:

[0052] The terminal provides the user with support information received from the server. This information is displayed as a pop-up on the screen or output as audio through the speaker. The user can respond to this information by voice or by clicking.

[0053] Step 6:

[0054] Based on the support information displayed on the terminal, users make selections according to their work needs. For example, they decide whether to perform an operation according to the suggested actions.

[0055] Step 7:

[0056] The terminal automatically performs specific operations based on user selections. This includes operations such as updating spreadsheets and searching for and retrieving data, automating the user's work.

[0057] (Example 1)

[0058] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0059] Modern information processing systems require efficient support for user operations and improved productivity. However, current systems struggle to accurately understand user operations and provide appropriate support in real time. In particular, it is difficult to integrate multiple functions such as operation monitoring, voice analysis, secure data communication, and the provision of support information.

[0060] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0061] In this invention, the server includes means for monitoring user operations in an information processing device and collecting them as data, means for receiving and analyzing user voice, and means for generating appropriate support using a generative model. This makes it possible to monitor user operations and voice in real time, provide appropriate support information based on them, and automatically execute actions based on user responses.

[0062] An "information processing device" is a device that monitors user operations and voice, and collects and analyzes them as data.

[0063] A "user" refers to a person who uses an information processing device, and whose operations and voice are monitored by the system.

[0064] "Operation" refers to actions performed by the user, such as keyboard input, mouse movement, and launching and closing applications.

[0065] "Voice" refers to the words or instructions that a user speaks to an information processing device through a microphone.

[0066] "Data" refers to information about user actions and voice recordings collected by information processing devices.

[0067] A "generative model" refers to algorithms and technologies used to analyze user actions and voice to generate appropriate support information.

[0068] "Support" refers to information and suggestions aimed at improving the user's current operations and work efficiency, and is provided to the user as support information.

[0069] A "prompt" is when an information processing device presents the user with an actionable choice or instruction.

[0070] The "Internet" is a global computer network used to transmit data collected from users to external devices.

[0071] "Encryption" refers to information transformation technology used to ensure the security of data being transmitted.

[0072] The system for implementing this invention mainly consists of an information processing device that monitors user operations and voice and provides support information based on them. The terminal monitors user operations in real time and collects data on keyboard input, mouse movements, and application usage. The terminal is also connected to a microphone that receives voices emitted by the user and records them as voice data.

[0073] Subsequently, the collected operation data and voice data are transmitted to a server via the internet. The server receives this data and analyzes it using a generative AI model. It then generates appropriate support information to improve the user's work efficiency. This analysis is based on the user's patterns and operation history.

[0074] The generated support information is sent to the device and displayed as a pop-up notification on the screen or the user is guided by voice. For example, if the user is editing a spreadsheet, the device will display a prompt asking, "Do you want to retrieve the latest data?" If the user responds with "Yes," the device will automatically retrieve the necessary data from the database and update the spreadsheet.

[0075] As a concrete example, consider a scenario where a user is preparing for a meeting. If the materials needed for the meeting need to be updated, the device can automatically retrieve the materials and keep them up-to-date. Examples of prompts include, "Please prepare the materials needed for the next meeting," or "Please create a new spreadsheet and retrieve the latest sales data."

[0076] This system allows users to perform their tasks while feeling that their actions are always supported, leading to increased efficiency and improved accuracy in their work.

[0077] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0078] Step 1:

[0079] The device monitors user actions and collects data in real time. Specifically, it records keyboard input, mouse movements, and application usage. It also uses a microphone to receive user voice and saves it as audio data. Inputs are user actions and voice, while outputs are action data and audio data.

[0080] Step 2:

[0081] The terminal transmits collected operation data and voice data to the server via the internet. During this process, the data is protected using encryption technology to prevent unauthorized access by third parties. The input consists of encrypted operation data and voice data, while the output is securely protected data on the server.

[0082] Step 3:

[0083] The server analyzes the received data. Using a generative AI model, it analyzes the user's operation patterns and generates necessary support information. This analysis determines the proposal of an optimized workflow and the efficient use of the tools to be used. The input is the received operator data and voice data, and the output is the generated support information.

[0084] Step 4:

[0085] The terminal receives analysis results and support information from the server and provides it to the user. Specifically, it displays a pop-up on the screen or provides audio notifications through the speaker. For example, it might prompt, "Do you want to use the slide template needed for your next meeting?" The input is the generated support information, and the output is the information presented to the user.

[0086] Step 5:

[0087] The user responds to prompts from the device. Based on these responses, the device automatically performs the necessary actions. For example, if the user responds with "yes" by voice, a new spreadsheet is automatically created and the necessary data is entered. The input is the user's response, and the output is the user-specified action that was performed.

[0088] (Application Example 1)

[0089] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0090] In today's remote work environment, users often face challenges in managing vast amounts of information and efficiently performing their tasks. In particular, when working with multiple digital tools, operational errors and inefficiencies can become problematic. Furthermore, there is a need for efficient support tailored to the user's skills and experience. In addition, there is a need for means of receiving information without disrupting concentration on work, by providing support information through visual and auditory methods.

[0091] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0092] In this invention, the server includes means for monitoring user operations and collecting them as data, means for receiving user voice input and analyzing the voice data, and an information processing device that analyzes the collected operation data and voice data and generates appropriate support information. As a result, users can receive necessary support information in real time and improve work efficiency through automated support.

[0093] "Monitoring user operations" refers to the process of observing a series of operations performed by a user on a personal computer in real time and collecting that data.

[0094] "Analyzing audio data" refers to the process of processing audio input provided by the user and converting it into information that allows for understanding its content.

[0095] An "information processing device" is a device that includes hardware and software with computing power to analyze collected data and generate necessary support information.

[0096] "Visualization and audio" refers to means of communicating generated support information to users visually and aurally.

[0097] "Augmented reality display through a virtual reality display" refers to a method of displaying information generated using virtual reality technology, merging the real world with digital information and presenting it to the user.

[0098] A "remote processing device" is a centrally managed computing device designed to receive data via a network and perform specific processing on it.

[0099] The system that realizes this invention supports users in efficiently performing their work in a teleworking environment. It promotes work efficiency by utilizing the user's personal computer and connected smart devices.

[0100] The system monitors user operations and collects operation data in real time. In one embodiment of the present invention, keyboard input, mouse operations, and application usage on a personal computer are acquired and recorded. The user provides voice input via a smart device, and the system converts this voice into text data using speech recognition technology.

[0101] The server analyzes the collected operational and audio data and uses a generative AI model to generate appropriate support information to assist the user's work. This process includes text analysis of the data and the generation of predictions and suggestions by the AI ​​model. Based on the analysis results, the information processing device generates the support information and provides it to the user as visualizations and audio.

[0102] Support information is displayed in augmented reality through a virtual reality display and communicated directly to the user via the smart device's display or audio output. Users can confirm the information provided through visual displays and audio guidance, which helps improve their work efficiency. For example, if a user gives a voice command such as "I want to check the latest meeting materials," the server uses a generative AI model to retrieve the appropriate information from the database and displays it on the user's display.

[0103] The generation AI model used in this system includes generation technology that makes suggestions based on the user's work content, and prompts are entered in the format of "Please retrieve the following data: Meeting updates." This system allows users to reduce errors in their work and improve work efficiency.

[0104] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0105] Step 1:

[0106] The terminal monitors user activity on a personal computer, collecting data such as keyboard input, mouse movements, and application usage. Input includes real-time user activity, and output is log data of these activities. This ensures the collection of necessary data.

[0107] Step 2:

[0108] The device receives user instructions via voice input and converts them into text data using speech recognition technology. The input is the user's voice, and the output is the text converted from the voice. Accurate text conversion of voice data enables subsequent data processing.

[0109] Step 3:

[0110] The terminal transmits collected operational and text data to the server using a secure protocol. The input is all digital data collected on the terminal side, and the output is the data transferred to the server. Once the data is securely transferred to the server, it is ready for analysis.

[0111] Step 4:

[0112] The server analyzes the received data and generates appropriate support information using a generative AI model. Input data includes operation logs and text data, while output is the generated support information. The server inputs prompt messages into the generative AI model, performing data analysis and prediction to generate useful suggestions for the user.

[0113] Step 5:

[0114] The server sends generated support information to the terminal, and the terminal provides the information to the user. The input is the support information sent from the server, and the output is the display or audio output of the information by the terminal. This allows the user to obtain the necessary support information in real time.

[0115] Step 6:

[0116] The user responds based on the information presented and provides instructions as needed. Input is the instructions given by the user based on the display or voice, and output is the feedback received by the device. Based on the user's instructions, subsequent operations and information retrieval proceed efficiently.

[0117] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0118] This invention provides comprehensive support that takes into account the user's emotional state by combining an emotion engine with a system that assists user operations on a PC.

[0119] Program Processing Overview

[0120] This system has the capability to monitor and analyze user actions and voice input in real time. By combining this with an emotion engine, it recognizes the user's current emotional state and takes that information into account when generating support information. Specifically, the terminal collects data on user actions and peripheral devices, and records voice input. It also uses a camera to capture the user's facial expressions and analyze their emotional state.

[0121] This data is encrypted for security purposes before being sent to the server, where it is analyzed using a generative AI model and an emotion engine. The server generates support information by combining the user's emotional state with the context of their actions, and returns this support information to the terminal.

[0122] The device uses the received support information to suggest appropriate actions to the user. This generated support information is provided to the user through screen display and audio output. For example, if the emotion engine recognizes that the user is experiencing stress, it can suggest ways to reduce workload or encourage rest.

[0123] As a concrete example, imagine a situation where a user is busy preparing a presentation. If the system detects fatigue from the user's facial expression, it will display a notification saying, "It's time for a break. The latest slide has been saved." If the user expresses gratitude for this, the system will continue to learn so that it can quickly perform the same action in similar situations in the future.

[0124] The implementation of this system will allow employees to perform their duties with psychological support, leading to improved work quality and increased productivity. By providing support information that takes emotional states into account, it will be possible to provide support that is tailored to the individual needs of each employee.

[0125] The following describes the processing flow.

[0126] Step 1:

[0127] The terminal monitors the user's PC operations in real time, collecting information on keyboard and mouse inputs, as well as the applications being used. Simultaneously, it uses a microphone to collect user voice commands as data.

[0128] Step 2:

[0129] The device uses its camera to capture the user's face and collect facial expression data. This facial expression data is processed as information necessary to analyze the user's emotional state.

[0130] Step 3:

[0131] The terminal encrypts the collected operation data, voice data, and facial expression data and transmits them to the server using a secure protocol. This makes it possible to transmit data while guaranteeing the security of the information.

[0132] Step 4:

[0133] The server analyzes the received data using a generative AI model to interpret the context of the operation and the content of the voice. Simultaneously, it uses an emotion engine to recognize the user's emotional state from facial expression data.

[0134] Step 5:

[0135] Based on the analyzed information, the server generates support information optimized for the user's current emotional state and operational status. This support information includes suggestions, advice, and warnings for the user.

[0136] Step 6:

[0137] The terminal receives support information sent from the server and provides it to the user through screen display and audio output. For example, if the user is feeling stressed, a message such as "Please pause your work and take a break" will be displayed.

[0138] Step 7:

[0139] Based on the support information provided by the device, the user selects the necessary actions for their task. Once the user makes a selection, the corresponding actions are automatically executed on the device.

[0140] Step 8:

[0141] The device records user responses and adjusts the system based on those responses to better optimize future support information suggestions. This enables personalized support based on the user's long-term usage history.

[0142] (Example 2)

[0143] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0144] Conventional user support systems generated support information using only the user's operation history and voice input, making it difficult to provide appropriate support that took into account the user's psychological and emotional state. Furthermore, there were security challenges regarding the information, requiring the secure processing and transmission of collected data.

[0145] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0146] In this invention, the server includes means for analyzing the user's emotional state, means for generating appropriate support information using a generative AI model, and means for sending and receiving encrypted data via a communication network. This makes it possible to provide comprehensive support information that takes the user's emotions into consideration.

[0147] "User" refers to an individual or organization that operates this system and receives support from it.

[0148] "Operation data" refers to input information generated when a user uses a computing device, including, for example, keyboard input and mouse operations.

[0149] "Voice input" refers to voice information emitted by the user through a microphone or similar device, and this information serves as the basis for the system's analysis.

[0150] "Facial expression information" refers to data based on the user's facial expressions captured by a camera device, and is used to analyze the user's emotional state.

[0151] "Emotional state" refers to the psychological or emotional state exhibited by the user, and is inferred from facial expressions, voice, etc.

[0152] A "generative AI model" is a system that uses machine learning algorithms to generate useful information from data, and is used to generate appropriate support information for users.

[0153] A "prompt statement" is a sentence containing commands or instructions that are input to a generative AI model to obtain a specific output.

[0154] "Support information" refers to information generated based on the analyzed user's actions and emotional state, and includes advice and suggestions to assist the user's work.

[0155] "Encryption" is a technology that converts data into a format that cannot be easily understood by third parties, thereby guaranteeing the secure transmission and reception of information.

[0156] A "communication network" refers to a network infrastructure used to exchange data between different devices, and includes the internet and local networks.

[0157] This invention is a system that assists user operations and provides advanced, personalized support that takes into account the user's emotional state. Specific embodiments of this system are described below.

[0158] The terminal is equipped with input devices (e.g., keyboard, mouse) and a microphone to collect user operation data. This allows for the capture of operation data and voice input, and also captures the user's facial expressions using a camera. This input is acquired as data and used for analyzing emotional states.

[0159] The terminal encrypts the collected data in real time and sends it to the server. HTTPS is used as the security protocol to ensure the safe transfer of data. Encryption methods such as AES are applied.

[0160] The server decodes the received data and processes the information using a generative AI model. In particular, the emotion analysis engine identifies the user's emotional state from their facial expressions and voice tone. Based on this analysis result and user interaction data, the generative AI model creates optimal support information using prompt messages. For example, a prompt message such as "The user is showing signs of fatigue. Please create suggestions to reduce their workload" might be used.

[0161] The server returns the generated support information to the terminal. The terminal notifies the user of this information through screen display or audio output and provides specific suggestions. For example, if the user is preparing a presentation and shows signs of fatigue, the terminal will provide a message such as, "We recommend you take a break. Your latest slide has been automatically saved."

[0162] This system receives feedback from users and continuously trains its generating AI model based on that feedback, thereby improving the quality of support information for subsequent uses. This makes it possible to continuously provide comprehensive support, including emotional support tailored to each individual user.

[0163] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0164] Step 1:

[0165] The device monitors user actions and collects input data from the keyboard and mouse. Furthermore, it captures voice input using a microphone and facial expressions with a camera. The collected data (action data, voice data, and facial expression data) is encrypted in real time for processing. Input is raw data of actions, voice, and facial expressions, while output is encrypted data. Specifically, encryption technologies such as AES are used to protect the data.

[0166] Step 2:

[0167] The terminal sends encrypted data to the server via the communication network. The input is encrypted data, and the output is a secure transmission to the server. HTTPS is used as the security protocol for this transmission. Specifically, the data is transmitted via the network interface.

[0168] Step 3:

[0169] The server decodes the received data and analyzes operation data, voice data, and emotion data individually. The input is the decoded data from the terminal, and the output is information indicating the analyzed emotion state and operation status. Specifically, it uses a generative AI model and an emotion analysis engine to extract data features and identify the emotion state.

[0170] Step 4:

[0171] The server uses an AI model based on the analysis results to create prompts and generate optimal support information. For example, a prompt might be "The user is showing signs of fatigue. Please create suggestions to reduce their workload." The input is the analysis results and the prompt, and the output is specific support information. In terms of operation, the model utilizes deep learning techniques to understand the context and generate appropriate suggestions.

[0172] Step 5:

[0173] The server sends the generated support information to the terminal, and the terminal receives it. The input is the generated support information, and the output is the secure reception to the terminal. Specifically, the information is encrypted before transmission and decrypted after reception on the terminal side.

[0174] Step 6:

[0175] The terminal provides support information to the user through screen display and audio output. Input is support information received from the server, and output is specific action suggestions for the user. For example, it might display a suggestion such as, "We recommend you take a break." Specific actions include using display control and audio output devices.

[0176] Step 7:

[0177] When the user accepts a suggestion, the device records the response. The input is user feedback, and the output is recorded training data. Specifically, the user's responses are saved to a database and used for subsequent support.

[0178] Step 8:

[0179] The server continuously trains the generated AI model based on feedback, improving the accuracy of subsequent assistance. The input is the recorded training data, and the output is the updated AI model. Specifically, it adjusts the model's parameters to improve accuracy.

[0180] (Application Example 2)

[0181] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0182] While systems that provide support for daily life require detailed support tailored to the user's actions and circumstances, there is a problem in that support that takes into account the user's emotional state is lacking. Therefore, a mechanism is needed that understands the user's emotions and provides appropriate support based on those emotions.

[0183] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0184] In this invention, the server includes means for monitoring the user's operations and surrounding environment and collecting this information; means for recognizing the user's voice input and facial expressions and analyzing the voice and facial expression data; and means for integrating and analyzing the collected operation data, voice data, and facial expression data to generate support information that takes into account the user's emotional state. This enables detailed support that responds to the user's emotions.

[0185] "User" refers to a person who performs operations or voice input on the system.

[0186] "Operation" refers to the interaction or instructions that a user gives to the system.

[0187] "Situation" refers to the environment or conditions in which the user finds themselves.

[0188] "Voice input" refers to voice information that a user sends to the system through a device such as a microphone.

[0189] "Facial expression" refers to the movements and expressions of a user's face, and is used to judge their emotional state.

[0190] "Data" refers to information that a system collects or receives from users.

[0191] "Analysis" refers to the act of processing collected data and extracting intended information.

[0192] "Emotional state" refers to the user's psychological condition, specifically the emotions inferred from their facial expressions and voice.

[0193] "Support information" refers to helpful information and suggestions generated by the system based on the user's actions and emotional state.

[0194] "Visual display" refers to information that is provided to the user visually through a screen, display, or other means.

[0195] "Audio output" refers to information provided to the user audibly through devices such as speakers.

[0196] "Processing" refers to a series of operations, calculations, and decisions performed by a system.

[0197] A "data center" refers to a facility or location where collected data is stored and processed.

[0198] "Encryption" refers to the process of securely converting data so that it cannot be read by third parties.

[0199] This invention is a system for providing support information that takes into account the user's emotional state. This system is primarily intended to support users in their homes and living environments using consumer robots.

[0200] The server uses a generative AI model to analyze the user's voice input and facial expression data to infer the user's emotions. This analysis includes a process of collecting data in real time using hardware such as cameras and microphones. Voice input is converted to text by a speech recognition engine, and facial expression data is processed by an emotion analysis algorithm.

[0201] The collected data is encrypted and sent to a data center. In this data center, generative AI models and emotion engines operate to analyze the data and generate supportive information. The generated supportive information is returned to the device through visual displays and audio output.

[0202] As a concrete example, if a user is experiencing stress in their daily life, the system will suggest relaxing music or rest. For instance, a notification might appear saying, "You seem tired today. Would you like to play some relaxing music?" This suggestion is generated by inputting the prompt "What is the best way to encourage relaxation when a user appears tired?" into a generating AI model.

[0203] This allows users to receive support tailored to their emotional state at any given time, thereby improving their quality of life. This system can be flexibly applied to various situations within the home, providing support that enriches the lives of users.

[0204] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0205] Step 1:

[0206] The device monitors the user's actions and surrounding environment, and collects input data using a camera and microphone. This input data includes the user's voice input and facial expressions. The device acquires this data in real time and temporarily stores it for subsequent processing.

[0207] Step 2:

[0208] The server receives encrypted audio data from the terminal and converts it to text using a speech recognition engine. This process visualizes the content of the audio and converts it into an analyzable format. The output is the converted text data.

[0209] Step 3:

[0210] The server receives encrypted facial expression data from the terminal. It applies an emotion analysis algorithm to infer the user's emotional state from the facial expression data. The input is a facial expression image, and the output is data indicating the emotional state.

[0211] Step 4:

[0212] The server integrates text and emotional state data from the audio data and generates support information using a generative AI model. During this process, prompts are used to determine the specific support content. The input consists of text and emotional state data, and the output is appropriate support information.

[0213] Step 5:

[0214] The terminal provides support information received from the server to the user through visual display and audio output. The terminal conveys support information to the user by displaying messages on the monitor and outputting audio through the speaker.

[0215] Step 6:

[0216] The device monitors the user's response again, and if there are any additional actions or new voice inputs, it sends them back to the server for further analysis. Through this process, the system adaptively learns from the user's feedback and improves the accuracy of assistance in subsequent interactions.

[0217] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0218] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0219] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0220] [Second Embodiment]

[0221] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0222] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0223] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0224] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0225] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0226] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0227] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0228] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0229] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0230] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0231] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0232] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0233] This invention relates to a system that comprehensively supports user operations on a PC, enabling effective work performance even in a teleworking environment.

[0234] Program Processing Overview

[0235] This system operates by monitoring and analyzing various operations and inputs that occur on the user's PC in real time. First, the terminal constantly monitors a series of user operations on the PC and collects data. This includes keyboard input, mouse movements, and application usage. The terminal also accepts voice input, using a microphone to recognize user instructions and questions and record them as audio data.

[0236] Next, the collected data is transmitted to a server via the internet, and secure communication is ensured using encryption technology. The server analyzes this data and uses a generative AI model to generate support information that the user needs. This support information includes suggestions for streamlining the user's current tasks and specific advice on operations.

[0237] Upon receiving the analysis results, the terminal displays support information as a pop-up on the screen and, if necessary, provides audio output through the speaker. The user responds by voice or click and selects an action provided by the system. Based on the selection, the terminal automatically performs the specified action. For example, this can automate operations such as updating a spreadsheet or retrieving documents.

[0238] As a concrete example, suppose a user is preparing for a regular meeting. At this time, the user needs to retrieve the latest data for the meeting. The device detects the user's editing operation in the spreadsheet and makes a voice suggestion, "Do you want to retrieve the latest data?" If the user voice-approves with "Yes," the device automatically retrieves the necessary information from the database and updates the specified spreadsheet.

[0239] By implementing this system, users can perform their tasks without worrying about a lack of information or operational errors, enabling efficient and accurate work. Therefore, variations in work skills are reduced, contributing to improved overall productivity.

[0240] The following describes the processing flow.

[0241] Step 1:

[0242] The device monitors the user's keyboard input, mouse movements, and application usage in real time, collecting this data locally. A microphone is also constantly on standby for voice input, recording user voice commands as digital audio data.

[0243] Step 2:

[0244] The device transmits the collected operation data and voice data to the server via the internet. During this transmission, the data is encrypted to ensure its security.

[0245] Step 3:

[0246] The server starts analysis using a generative AI model based on the received operation data and voice data. This determines the context and intent of the user's actions and generates appropriate support information.

[0247] Step 4:

[0248] The server generates support information derived from the analysis results and sends it back to the terminal. This support information includes suggestions for the user's current operations and advice on tasks that can be made more efficient.

[0249] Step 5:

[0250] The terminal provides the user with support information received from the server. This information is displayed as a pop-up on the screen or output as audio through the speaker. The user can respond to this information by voice or by clicking.

[0251] Step 6:

[0252] Based on the support information displayed on the terminal, users make selections according to their work needs. For example, they decide whether to perform an operation according to the suggested actions.

[0253] Step 7:

[0254] The terminal automatically performs specific operations based on user selections. This includes operations such as updating spreadsheets and searching for and retrieving data, automating the user's work.

[0255] (Example 1)

[0256] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0257] Modern information processing systems require efficient support for user operations and improved productivity. However, current systems struggle to accurately understand user operations and provide appropriate support in real time. In particular, it is difficult to integrate multiple functions such as operation monitoring, voice analysis, secure data communication, and the provision of support information.

[0258] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0259] In this invention, the server includes means for monitoring user operations in an information processing device and collecting them as data, means for receiving and analyzing user voice, and means for generating appropriate support using a generative model. This makes it possible to monitor user operations and voice in real time, provide appropriate support information based on them, and automatically execute actions based on user responses.

[0260] An "information processing device" is a device that monitors user operations and voice, and collects and analyzes them as data.

[0261] A "user" refers to a person who uses an information processing device, and whose operations and voice are monitored by the system.

[0262] "Operation" refers to actions performed by the user, such as keyboard input, mouse movement, and launching and closing applications.

[0263] "Voice" refers to the words or instructions that a user speaks to an information processing device through a microphone.

[0264] "Data" refers to information about user actions and voice recordings collected by information processing devices.

[0265] A "generative model" refers to algorithms and technologies used to analyze user actions and voice to generate appropriate support information.

[0266] "Support" refers to information and suggestions aimed at improving the user's current operations and work efficiency, and is provided to the user as support information.

[0267] A "prompt" is when an information processing device presents the user with an actionable choice or instruction.

[0268] The "Internet" is a global computer network used to transmit data collected from users to external devices.

[0269] "Encryption" refers to information transformation technology used to ensure the security of data being transmitted.

[0270] The system for implementing this invention mainly consists of an information processing device that monitors user operations and voice and provides support information based on them. The terminal monitors user operations in real time and collects data on keyboard input, mouse movements, and application usage. The terminal is also connected to a microphone that receives voices emitted by the user and records them as voice data.

[0271] Subsequently, the collected operation data and voice data are transmitted to a server via the internet. The server receives this data and analyzes it using a generative AI model. It then generates appropriate support information to improve the user's work efficiency. This analysis is based on the user's patterns and operation history.

[0272] The generated support information is sent to the device and displayed as a pop-up notification on the screen or the user is guided by voice. For example, if the user is editing a spreadsheet, the device will display a prompt asking, "Do you want to retrieve the latest data?" If the user responds with "Yes," the device will automatically retrieve the necessary data from the database and update the spreadsheet.

[0273] As a concrete example, consider a scenario where a user is preparing for a meeting. If the materials needed for the meeting need to be updated, the device can automatically retrieve the materials and keep them up-to-date. Examples of prompts include, "Please prepare the materials needed for the next meeting," or "Please create a new spreadsheet and retrieve the latest sales data."

[0274] This system allows users to perform their tasks while feeling that their actions are always supported, leading to increased efficiency and improved accuracy in their work.

[0275] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0276] Step 1:

[0277] The device monitors user actions and collects data in real time. Specifically, it records keyboard input, mouse movements, and application usage. It also uses a microphone to receive user voice and saves it as audio data. Inputs are user actions and voice, while outputs are action data and audio data.

[0278] Step 2:

[0279] The terminal sends the collected operation data and voice data to the server via the Internet. At this time, the data is protected using encryption technology to prevent unauthorized access by third parties. The input is the encrypted operation data and voice data, and the output is the data securely protected on the server.

[0280] Step 3:

[0281] The server analyzes the received data. Using a generated AI model, it analyzes the user's operation pattern and generates necessary support information. Based on this analysis, it determines proposals for optimized workflows, efficient usage methods of tools to be used, etc. The input is the received operator data and voice data, and the output is the generated support information.

[0282] Step 4:

[0283] The terminal receives the analysis results and support information from the server and provides them to the user. Specifically, it displays a pop-up on the screen or makes a voice notification through the speaker. For example, a prompt such as "Do you want to use the template for the slides needed for the next meeting?" is given. The input is the generated support information, and the output is the presentation of information to the user.

[0284] Step 5:

[0285] The user responds to the prompt from the terminal. Based on that response, the terminal automatically executes the necessary actions. For example, when the user responds with "Yes" by voice, a new spreadsheet is automatically created and the necessary data is input. The input is the user's response, and the output is the executed user-specified action.

[0286] (Application Example 1)

[0287] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0288] In today's remote work environment, users often face challenges in managing vast amounts of information and efficiently performing their tasks. In particular, when working with multiple digital tools, operational errors and inefficiencies can become problematic. Furthermore, there is a need for efficient support tailored to the user's skills and experience. In addition, there is a need for means of receiving information without disrupting concentration on work, by providing support information through visual and auditory methods.

[0289] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0290] In this invention, the server includes means for monitoring user operations and collecting them as data, means for receiving user voice input and analyzing the voice data, and an information processing device that analyzes the collected operation data and voice data and generates appropriate support information. As a result, users can receive necessary support information in real time and improve work efficiency through automated support.

[0291] "Monitoring user operations" refers to the process of observing a series of operations performed by a user on a personal computer in real time and collecting that data.

[0292] "Analyzing audio data" refers to the process of processing audio input provided by the user and converting it into information that allows for understanding its content.

[0293] An "information processing device" is a device that includes hardware and software with computing power to analyze collected data and generate necessary support information.

[0294] "Visualization and audio" refers to means of communicating generated support information to users visually and aurally.

[0295] "Augmented reality display through a virtual reality display" refers to a method of displaying information generated using virtual reality technology, merging the real world with digital information and presenting it to the user.

[0296] A "remote processing device" is a centrally managed computing device designed to receive data via a network and perform specific processing on it.

[0297] The system that realizes this invention supports users in efficiently performing their work in a teleworking environment. It promotes work efficiency by utilizing the user's personal computer and connected smart devices.

[0298] The system monitors user operations and collects operation data in real time. In one embodiment of the present invention, keyboard input, mouse operations, and application usage on a personal computer are acquired and recorded. The user provides voice input via a smart device, and the system converts this voice into text data using speech recognition technology.

[0299] The server analyzes the collected operational and audio data and uses a generative AI model to generate appropriate support information to assist the user's work. This process includes text analysis of the data and the generation of predictions and suggestions by the AI ​​model. Based on the analysis results, the information processing device generates the support information and provides it to the user as visualizations and audio.

[0300] Support information is augmented reality displayed through a virtual reality display and directly transmitted to the user via the display or acoustic output of a smart device. The user can confirm the information provided through visual displays and voice guidance and utilize it to streamline operations. As a specific example, when the user issues a voice instruction such as "I want to check the latest materials for the meeting," the server uses a generative AI model to retrieve appropriate information from the database and display it on the user's display.

[0301] The generative AI model used at this time includes generative technologies that make proposals based on the user's work content and are input in the form of prompt sentences such as "Please retrieve the following data: latest information on the meeting." With this system, the user can reduce work errors and improve work efficiency.

[0302] The flow of specific processing in Application Example 1 will be described using FIG. 12.

[0303] Step 1:

[0304] The terminal monitors the user's operations on a personal computer and collects data such as keyboard input, mouse operations, and usage status of applications. The input is the user's real-time operations, and the output is the operation log data generated therefrom. This collects the necessary data.

[0305] Step 2:

[0306] The terminal receives the user's instructions through voice input and converts them into text data using voice recognition technology. The input is the user's voice, and the output is the text converted from the voice. The accurate conversion of voice data into text enables subsequent data processing.

[0307] Step 3:

[0308] The terminal transmits collected operational and text data to the server using a secure protocol. The input is all digital data collected on the terminal side, and the output is the data transferred to the server. Once the data is securely transferred to the server, it is ready for analysis.

[0309] Step 4:

[0310] The server analyzes the received data and generates appropriate support information using a generative AI model. Input data includes operation logs and text data, while output is the generated support information. The server inputs prompt messages into the generative AI model, performing data analysis and prediction to generate useful suggestions for the user.

[0311] Step 5:

[0312] The server sends generated support information to the terminal, and the terminal provides the information to the user. The input is the support information sent from the server, and the output is the display or audio output of the information by the terminal. This allows the user to obtain the necessary support information in real time.

[0313] Step 6:

[0314] The user responds based on the information presented and provides instructions as needed. Input is the instructions given by the user based on the display or voice, and output is the feedback received by the device. Based on the user's instructions, subsequent operations and information retrieval proceed efficiently.

[0315] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0316] This invention provides comprehensive support that takes into account the user's emotional state by combining an emotion engine with a system that assists user operations on a PC.

[0317] Program Processing Overview

[0318] This system has the capability to monitor and analyze user actions and voice input in real time. By combining this with an emotion engine, it recognizes the user's current emotional state and takes that information into account when generating support information. Specifically, the terminal collects data on user actions and peripheral devices, and records voice input. It also uses a camera to capture the user's facial expressions and analyze their emotional state.

[0319] This data is encrypted for security purposes before being sent to the server, where it is analyzed using a generative AI model and an emotion engine. The server generates support information by combining the user's emotional state with the context of their actions, and returns this support information to the terminal.

[0320] The device uses the received support information to suggest appropriate actions to the user. This generated support information is provided to the user through screen display and audio output. For example, if the emotion engine recognizes that the user is experiencing stress, it can suggest ways to reduce workload or encourage rest.

[0321] As a concrete example, imagine a situation where a user is busy preparing a presentation. If the system detects fatigue from the user's facial expression, it will display a notification saying, "It's time for a break. The latest slide has been saved." If the user expresses gratitude for this, the system will continue to learn so that it can quickly perform the same action in similar situations in the future.

[0322] The implementation of this system will allow employees to perform their duties with psychological support, leading to improved work quality and increased productivity. By providing support information that takes emotional states into account, it will be possible to provide support that is tailored to the individual needs of each employee.

[0323] The following describes the processing flow.

[0324] Step 1:

[0325] The terminal monitors the user's PC operations in real time, collecting information on keyboard and mouse inputs, as well as the applications being used. Simultaneously, it uses a microphone to collect user voice commands as data.

[0326] Step 2:

[0327] The device uses its camera to capture the user's face and collect facial expression data. This facial expression data is processed as information necessary to analyze the user's emotional state.

[0328] Step 3:

[0329] The terminal encrypts the collected operation data, voice data, and facial expression data and transmits them to the server using a secure protocol. This makes it possible to transmit data while guaranteeing the security of the information.

[0330] Step 4:

[0331] The server analyzes the received data using a generative AI model to interpret the context of the operation and the content of the voice. Simultaneously, it uses an emotion engine to recognize the user's emotional state from facial expression data.

[0332] Step 5:

[0333] Based on the analyzed information, the server generates support information optimized for the user's current emotional state and operational status. This support information includes suggestions, advice, and warnings for the user.

[0334] Step 6:

[0335] The terminal receives support information sent from the server and provides it to the user through screen display and audio output. For example, if the user is feeling stressed, a message such as "Please pause your work and take a break" will be displayed.

[0336] Step 7:

[0337] Based on the support information provided by the device, the user selects the necessary actions for their task. Once the user makes a selection, the corresponding actions are automatically executed on the device.

[0338] Step 8:

[0339] The device records user responses and adjusts the system based on those responses to better optimize future support information suggestions. This enables personalized support based on the user's long-term usage history.

[0340] (Example 2)

[0341] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0342] Conventional user support systems generated support information using only the user's operation history and voice input, making it difficult to provide appropriate support that took into account the user's psychological and emotional state. Furthermore, there were security challenges regarding the information, requiring the secure processing and transmission of collected data.

[0343] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0344] In this invention, the server includes means for analyzing the user's emotional state, means for generating appropriate support information using a generative AI model, and means for sending and receiving encrypted data via a communication network. This makes it possible to provide comprehensive support information that takes the user's emotions into consideration.

[0345] "User" refers to an individual or organization that operates this system and receives support from it.

[0346] "Operation data" refers to input information generated when a user uses a computing device, including, for example, keyboard input and mouse operations.

[0347] "Voice input" refers to voice information emitted by the user through a microphone or similar device, and this information serves as the basis for the system's analysis.

[0348] "Facial expression information" refers to data based on the user's facial expressions captured by a camera device, and is used to analyze the user's emotional state.

[0349] "Emotional state" refers to the psychological or emotional state exhibited by the user, and is inferred from facial expressions, voice, etc.

[0350] A "generative AI model" is a system that uses machine learning algorithms to generate useful information from data, and is used to generate appropriate support information for users.

[0351] A "prompt statement" is a sentence containing commands or instructions that are input to a generative AI model to obtain a specific output.

[0352] "Support information" refers to information generated based on the analyzed user's actions and emotional state, and includes advice and suggestions to assist the user's work.

[0353] "Encryption" is a technology that converts data into a format that cannot be easily understood by third parties, thereby guaranteeing the secure transmission and reception of information.

[0354] A "communication network" refers to a network infrastructure used to exchange data between different devices, and includes the internet and local networks.

[0355] This invention is a system that assists user operations and provides advanced, personalized support that takes into account the user's emotional state. Specific embodiments of this system are described below.

[0356] The terminal is equipped with input devices (e.g., keyboard, mouse) and a microphone to collect user operation data. This allows for the capture of operation data and voice input, and also captures the user's facial expressions using a camera. This input is acquired as data and used for analyzing emotional states.

[0357] The terminal encrypts the collected data in real time and sends it to the server. HTTPS is used as the security protocol to ensure the safe transfer of data. Encryption methods such as AES are applied.

[0358] The server decodes the received data and processes the information using a generative AI model. In particular, the emotion analysis engine identifies the user's emotional state from their facial expressions and voice tone. Based on this analysis result and user interaction data, the generative AI model creates optimal support information using prompt messages. For example, a prompt message such as "The user is showing signs of fatigue. Please create suggestions to reduce their workload" might be used.

[0359] The server returns the generated support information to the terminal. The terminal notifies the user of this information through screen display or audio output and provides specific suggestions. For example, if the user is preparing a presentation and shows signs of fatigue, the terminal will provide a message such as, "We recommend you take a break. Your latest slide has been automatically saved."

[0360] This system receives feedback from users and continuously trains its generating AI model based on that feedback, thereby improving the quality of support information for subsequent uses. This makes it possible to continuously provide comprehensive support, including emotional support tailored to each individual user.

[0361] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0362] Step 1:

[0363] The device monitors user actions and collects input data from the keyboard and mouse. Furthermore, it captures voice input using a microphone and facial expressions with a camera. The collected data (action data, voice data, and facial expression data) is encrypted in real time for processing. Input is raw data of actions, voice, and facial expressions, while output is encrypted data. Specifically, encryption technologies such as AES are used to protect the data.

[0364] Step 2:

[0365] The terminal sends encrypted data to the server via the communication network. The input is encrypted data, and the output is a secure transmission to the server. HTTPS is used as the security protocol for this transmission. Specifically, the data is transmitted via the network interface.

[0366] Step 3:

[0367] The server decodes the received data and analyzes operation data, voice data, and emotion data individually. The input is the decoded data from the terminal, and the output is information indicating the analyzed emotion state and operation status. Specifically, it uses a generative AI model and an emotion analysis engine to extract data features and identify the emotion state.

[0368] Step 4:

[0369] The server uses an AI model based on the analysis results to create prompts and generate optimal support information. For example, a prompt might be "The user is showing signs of fatigue. Please create suggestions to reduce their workload." The input is the analysis results and the prompt, and the output is specific support information. In terms of operation, the model utilizes deep learning techniques to understand the context and generate appropriate suggestions.

[0370] Step 5:

[0371] The server sends the generated support information to the terminal, and the terminal receives it. The input is the generated support information, and the output is the secure reception to the terminal. Specifically, the information is encrypted before transmission and decrypted after reception on the terminal side.

[0372] Step 6:

[0373] The terminal provides support information to the user through screen display and audio output. Input is support information received from the server, and output is specific action suggestions for the user. For example, it might display a suggestion such as, "We recommend you take a break." Specific actions include using display control and audio output devices.

[0374] Step 7:

[0375] When the user accepts a suggestion, the device records the response. The input is user feedback, and the output is recorded training data. Specifically, the user's responses are saved to a database and used for subsequent support.

[0376] Step 8:

[0377] The server continuously trains the generated AI model based on feedback, improving the accuracy of subsequent assistance. The input is the recorded training data, and the output is the updated AI model. Specifically, it adjusts the model's parameters to improve accuracy.

[0378] (Application Example 2)

[0379] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0380] While systems that provide support for daily life require detailed support tailored to the user's actions and circumstances, there is a problem in that support that takes into account the user's emotional state is lacking. Therefore, a mechanism is needed that understands the user's emotions and provides appropriate support based on those emotions.

[0381] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0382] In this invention, the server includes means for monitoring the user's operations and surrounding environment and collecting this information; means for recognizing the user's voice input and facial expressions and analyzing the voice and facial expression data; and means for integrating and analyzing the collected operation data, voice data, and facial expression data to generate support information that takes into account the user's emotional state. This enables detailed support that responds to the user's emotions.

[0383] "User" refers to a person who performs operations or voice input on the system.

[0384] "Operation" refers to the interaction or instructions that a user gives to the system.

[0385] "Situation" refers to the environment or conditions in which the user finds themselves.

[0386] "Voice input" refers to voice information that a user sends to the system through a device such as a microphone.

[0387] "Facial expression" refers to the movements and expressions of a user's face, and is used to judge their emotional state.

[0388] "Data" refers to information that a system collects or receives from users.

[0389] "Analysis" refers to the act of processing collected data and extracting intended information.

[0390] "Emotional state" refers to the user's psychological condition, specifically the emotions inferred from their facial expressions and voice.

[0391] "Support information" refers to helpful information and suggestions generated by the system based on the user's actions and emotional state.

[0392] "Visual display" refers to information that is provided to the user visually through a screen, display, or other means.

[0393] "Audio output" refers to information provided to the user audibly through devices such as speakers.

[0394] "Processing" refers to a series of operations, calculations, and decisions performed by a system.

[0395] A "data center" refers to a facility or location where collected data is stored and processed.

[0396] "Encryption" refers to the process of securely converting data so that it cannot be read by third parties.

[0397] This invention is a system for providing support information that takes into account the user's emotional state. This system is primarily intended to support users in their homes and living environments using consumer robots.

[0398] The server uses a generative AI model to analyze the user's voice input and facial expression data to infer the user's emotions. This analysis includes a process of collecting data in real time using hardware such as cameras and microphones. Voice input is converted to text by a speech recognition engine, and facial expression data is processed by an emotion analysis algorithm.

[0399] The collected data is encrypted and sent to a data center. In this data center, generative AI models and emotion engines operate to analyze the data and generate supportive information. The generated supportive information is returned to the device through visual displays and audio output.

[0400] As a concrete example, if a user is experiencing stress in their daily life, the system will suggest relaxing music or rest. For instance, a notification might appear saying, "You seem tired today. Would you like to play some relaxing music?" This suggestion is generated by inputting the prompt "What is the best way to encourage relaxation when a user appears tired?" into a generating AI model.

[0401] This allows users to receive support tailored to their emotional state at any given time, thereby improving their quality of life. This system can be flexibly applied to various situations within the home, providing support that enriches the lives of users.

[0402] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0403] Step 1:

[0404] The device monitors the user's actions and surrounding environment, and collects input data using a camera and microphone. This input data includes the user's voice input and facial expressions. The device acquires this data in real time and temporarily stores it for subsequent processing.

[0405] Step 2:

[0406] The server receives encrypted audio data from the terminal and converts it to text using a speech recognition engine. This process visualizes the content of the audio and converts it into an analyzable format. The output is the converted text data.

[0407] Step 3:

[0408] The server receives encrypted facial expression data from the terminal. It applies an emotion analysis algorithm to infer the user's emotional state from the facial expression data. The input is a facial expression image, and the output is data indicating the emotional state.

[0409] Step 4:

[0410] The server integrates text and emotional state data from the audio data and generates support information using a generative AI model. During this process, prompts are used to determine the specific support content. The input consists of text and emotional state data, and the output is appropriate support information.

[0411] Step 5:

[0412] The terminal provides support information received from the server to the user through visual display and audio output. The terminal conveys support information to the user by displaying messages on the monitor and outputting audio through the speaker.

[0413] Step 6:

[0414] The device monitors the user's response again, and if there are any additional actions or new voice inputs, it sends them back to the server for further analysis. Through this process, the system adaptively learns from the user's feedback and improves the accuracy of assistance in subsequent interactions.

[0415] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0416] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0417] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0418] [Third Embodiment]

[0419] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0420] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0421] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0422] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0423] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0424] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0425] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0426] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0427] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0428] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0429] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0430] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0431] This invention relates to a system that comprehensively supports user operations on a PC, enabling effective work performance even in a teleworking environment.

[0432] Program Processing Overview

[0433] This system operates by monitoring and analyzing various operations and inputs that occur on the user's PC in real time. First, the terminal constantly monitors a series of user operations on the PC and collects data. This includes keyboard input, mouse movements, and application usage. The terminal also accepts voice input, using a microphone to recognize user instructions and questions and record them as audio data.

[0434] Next, the collected data is transmitted to a server via the internet, and secure communication is ensured using encryption technology. The server analyzes this data and uses a generative AI model to generate support information that the user needs. This support information includes suggestions for streamlining the user's current tasks and specific advice on operations.

[0435] Upon receiving the analysis results, the terminal displays support information as a pop-up on the screen and, if necessary, provides audio output through the speaker. The user responds by voice or click and selects an action provided by the system. Based on the selection, the terminal automatically performs the specified action. For example, this can automate operations such as updating a spreadsheet or retrieving documents.

[0436] As a concrete example, suppose a user is preparing for a regular meeting. At this time, the user needs to retrieve the latest data for the meeting. The device detects the user's editing operation in the spreadsheet and makes a voice suggestion, "Do you want to retrieve the latest data?" If the user voice-approves with "Yes," the device automatically retrieves the necessary information from the database and updates the specified spreadsheet.

[0437] By implementing this system, users can perform their tasks without worrying about a lack of information or operational errors, enabling efficient and accurate work. Therefore, variations in work skills are reduced, contributing to improved overall productivity.

[0438] The following describes the processing flow.

[0439] Step 1:

[0440] The device monitors the user's keyboard input, mouse movements, and application usage in real time, collecting this data locally. A microphone is also constantly on standby for voice input, recording user voice commands as digital audio data.

[0441] Step 2:

[0442] The device transmits the collected operation data and voice data to the server via the internet. During this transmission, the data is encrypted to ensure its security.

[0443] Step 3:

[0444] The server starts analysis using a generative AI model based on the received operation data and voice data. This determines the context and intent of the user's actions and generates appropriate support information.

[0445] Step 4:

[0446] The server generates support information derived from the analysis results and sends it back to the terminal. This support information includes suggestions for the user's current operations and advice on tasks that can be made more efficient.

[0447] Step 5:

[0448] The terminal provides the user with support information received from the server. This information is displayed as a pop-up on the screen or output as audio through the speaker. The user can respond to this information by voice or by clicking.

[0449] Step 6:

[0450] Based on the support information displayed on the terminal, users make selections according to their work needs. For example, they decide whether to perform an operation according to the suggested actions.

[0451] Step 7:

[0452] The terminal automatically performs specific operations based on user selections. This includes operations such as updating spreadsheets and searching for and retrieving data, automating the user's work.

[0453] (Example 1)

[0454] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0455] Modern information processing systems require efficient support for user operations and improved productivity. However, current systems struggle to accurately understand user operations and provide appropriate support in real time. In particular, it is difficult to integrate multiple functions such as operation monitoring, voice analysis, secure data communication, and the provision of support information.

[0456] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0457] In this invention, the server includes means for monitoring user operations in an information processing device and collecting them as data, means for receiving and analyzing user voice, and means for generating appropriate support using a generative model. This makes it possible to monitor user operations and voice in real time, provide appropriate support information based on them, and automatically execute actions based on user responses.

[0458] An "information processing device" is a device that monitors user operations and voice, and collects and analyzes them as data.

[0459] A "user" refers to a person who uses an information processing device, and whose operations and voice are monitored by the system.

[0460] "Operation" refers to actions performed by the user, such as keyboard input, mouse movement, and launching and closing applications.

[0461] "Voice" refers to the words or instructions that a user speaks to an information processing device through a microphone.

[0462] "Data" refers to information about user actions and voice recordings collected by information processing devices.

[0463] A "generative model" refers to algorithms and technologies used to analyze user actions and voice to generate appropriate support information.

[0464] "Support" refers to information and suggestions aimed at improving the user's current operations and work efficiency, and is provided to the user as support information.

[0465] A "prompt" is when an information processing device presents the user with an actionable choice or instruction.

[0466] The "Internet" is a global computer network used to transmit data collected from users to external devices.

[0467] "Encryption" refers to information transformation technology used to ensure the security of data being transmitted.

[0468] The system for implementing this invention mainly consists of an information processing device that monitors user operations and voice and provides support information based on them. The terminal monitors user operations in real time and collects data on keyboard input, mouse movements, and application usage. The terminal is also connected to a microphone that receives voices emitted by the user and records them as voice data.

[0469] Subsequently, the collected operation data and voice data are transmitted to a server via the internet. The server receives this data and analyzes it using a generative AI model. It then generates appropriate support information to improve the user's work efficiency. This analysis is based on the user's patterns and operation history.

[0470] The generated support information is sent to the device and displayed as a pop-up notification on the screen or the user is guided by voice. For example, if the user is editing a spreadsheet, the device will display a prompt asking, "Do you want to retrieve the latest data?" If the user responds with "Yes," the device will automatically retrieve the necessary data from the database and update the spreadsheet.

[0471] As a concrete example, consider a scenario where a user is preparing for a meeting. If the materials needed for the meeting need to be updated, the device can automatically retrieve the materials and keep them up-to-date. Examples of prompts include, "Please prepare the materials needed for the next meeting," or "Please create a new spreadsheet and retrieve the latest sales data."

[0472] This system allows users to perform their tasks while feeling that their actions are always supported, leading to increased efficiency and improved accuracy in their work.

[0473] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0474] Step 1:

[0475] The device monitors user actions and collects data in real time. Specifically, it records keyboard input, mouse movements, and application usage. It also uses a microphone to receive user voice and saves it as audio data. Inputs are user actions and voice, while outputs are action data and audio data.

[0476] Step 2:

[0477] The terminal transmits collected operation data and voice data to the server via the internet. During this process, the data is protected using encryption technology to prevent unauthorized access by third parties. The input consists of encrypted operation data and voice data, while the output is securely protected data on the server.

[0478] Step 3:

[0479] The server analyzes the received data. Using a generative AI model, it analyzes the user's operation patterns and generates necessary support information. This analysis determines the proposal of an optimized workflow and the efficient use of the tools to be used. The input is the received operator data and voice data, and the output is the generated support information.

[0480] Step 4:

[0481] The terminal receives analysis results and support information from the server and provides it to the user. Specifically, it displays a pop-up on the screen or provides audio notifications through the speaker. For example, it might prompt, "Do you want to use the slide template needed for your next meeting?" The input is the generated support information, and the output is the information presented to the user.

[0482] Step 5:

[0483] The user responds to prompts from the device. Based on these responses, the device automatically performs the necessary actions. For example, if the user responds with "yes" by voice, a new spreadsheet is automatically created and the necessary data is entered. The input is the user's response, and the output is the user-specified action that was performed.

[0484] (Application Example 1)

[0485] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0486] In today's remote work environment, users often face challenges in managing vast amounts of information and efficiently performing their tasks. In particular, when working with multiple digital tools, operational errors and inefficiencies can become problematic. Furthermore, there is a need for efficient support tailored to the user's skills and experience. In addition, there is a need for means of receiving information without disrupting concentration on work, by providing support information through visual and auditory methods.

[0487] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0488] In this invention, the server includes means for monitoring user operations and collecting them as data, means for receiving user voice input and analyzing the voice data, and an information processing device that analyzes the collected operation data and voice data and generates appropriate support information. As a result, users can receive necessary support information in real time and improve work efficiency through automated support.

[0489] "Monitoring user operations" refers to the process of observing a series of operations performed by a user on a personal computer in real time and collecting that data.

[0490] "Analyzing audio data" refers to the process of processing audio input provided by the user and converting it into information that allows for understanding its content.

[0491] An "information processing device" is a device that includes hardware and software with computing power to analyze collected data and generate necessary support information.

[0492] "Visualization and audio" refers to means of communicating generated support information to users visually and aurally.

[0493] "Augmented reality display through a virtual reality display" refers to a method of displaying information generated using virtual reality technology, merging the real world with digital information and presenting it to the user.

[0494] A "remote processing device" is a centrally managed computing device designed to receive data via a network and perform specific processing on it.

[0495] The system that realizes this invention supports users in efficiently performing their work in a teleworking environment. It promotes work efficiency by utilizing the user's personal computer and connected smart devices.

[0496] The system monitors user operations and collects operation data in real time. In one embodiment of the present invention, keyboard input, mouse operations, and application usage on a personal computer are acquired and recorded. The user provides voice input via a smart device, and the system converts this voice into text data using speech recognition technology.

[0497] The server analyzes the collected operational and audio data and uses a generative AI model to generate appropriate support information to assist the user's work. This process includes text analysis of the data and the generation of predictions and suggestions by the AI ​​model. Based on the analysis results, the information processing device generates the support information and provides it to the user as visualizations and audio.

[0498] Support information is displayed in augmented reality through a virtual reality display and communicated directly to the user via the smart device's display or audio output. Users can confirm the information provided through visual displays and audio guidance, which helps improve their work efficiency. For example, if a user gives a voice command such as "I want to check the latest meeting materials," the server uses a generative AI model to retrieve the appropriate information from the database and displays it on the user's display.

[0499] The generation AI model used in this system includes generation technology that makes suggestions based on the user's work content, and prompts are entered in the format of "Please retrieve the following data: Meeting updates." This system allows users to reduce errors in their work and improve work efficiency.

[0500] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0501] Step 1:

[0502] The terminal monitors user activity on a personal computer, collecting data such as keyboard input, mouse movements, and application usage. Input includes real-time user activity, and output is log data of these activities. This ensures the collection of necessary data.

[0503] Step 2:

[0504] The device receives user instructions via voice input and converts them into text data using speech recognition technology. The input is the user's voice, and the output is the text converted from the voice. Accurate text conversion of voice data enables subsequent data processing.

[0505] Step 3:

[0506] The terminal transmits collected operational and text data to the server using a secure protocol. The input is all digital data collected on the terminal side, and the output is the data transferred to the server. Once the data is securely transferred to the server, it is ready for analysis.

[0507] Step 4:

[0508] The server analyzes the received data and generates appropriate support information using a generative AI model. Input data includes operation logs and text data, while output is the generated support information. The server inputs prompt messages into the generative AI model, performing data analysis and prediction to generate useful suggestions for the user.

[0509] Step 5:

[0510] The server sends generated support information to the terminal, and the terminal provides the information to the user. The input is the support information sent from the server, and the output is the display or audio output of the information by the terminal. This allows the user to obtain the necessary support information in real time.

[0511] Step 6:

[0512] The user responds based on the information presented and provides instructions as needed. Input is the instructions given by the user based on the display or voice, and output is the feedback received by the device. Based on the user's instructions, subsequent operations and information retrieval proceed efficiently.

[0513] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0514] This invention provides comprehensive support that takes into account the user's emotional state by combining an emotion engine with a system that assists user operations on a PC.

[0515] Program Processing Overview

[0516] This system has the capability to monitor and analyze user actions and voice input in real time. By combining this with an emotion engine, it recognizes the user's current emotional state and takes that information into account when generating support information. Specifically, the terminal collects data on user actions and peripheral devices, and records voice input. It also uses a camera to capture the user's facial expressions and analyze their emotional state.

[0517] This data is encrypted for security purposes before being sent to the server, where it is analyzed using a generative AI model and an emotion engine. The server generates support information by combining the user's emotional state with the context of their actions, and returns this support information to the terminal.

[0518] The device uses the received support information to suggest appropriate actions to the user. This generated support information is provided to the user through screen display and audio output. For example, if the emotion engine recognizes that the user is experiencing stress, it can suggest ways to reduce workload or encourage rest.

[0519] As a concrete example, imagine a situation where a user is busy preparing a presentation. If the system detects fatigue from the user's facial expression, it will display a notification saying, "It's time for a break. The latest slide has been saved." If the user expresses gratitude for this, the system will continue to learn so that it can quickly perform the same action in similar situations in the future.

[0520] The implementation of this system will allow employees to perform their duties with psychological support, leading to improved work quality and increased productivity. By providing support information that takes emotional states into account, it will be possible to provide support that is tailored to the individual needs of each employee.

[0521] The following describes the processing flow.

[0522] Step 1:

[0523] The terminal monitors the user's PC operations in real time, collecting information on keyboard and mouse inputs, as well as the applications being used. Simultaneously, it uses a microphone to collect user voice commands as data.

[0524] Step 2:

[0525] The device uses its camera to capture the user's face and collect facial expression data. This facial expression data is processed as information necessary to analyze the user's emotional state.

[0526] Step 3:

[0527] The terminal encrypts the collected operation data, voice data, and facial expression data and transmits them to the server using a secure protocol. This makes it possible to transmit data while guaranteeing the security of the information.

[0528] Step 4:

[0529] The server analyzes the received data using a generative AI model to interpret the context of the operation and the content of the voice. Simultaneously, it uses an emotion engine to recognize the user's emotional state from facial expression data.

[0530] Step 5:

[0531] Based on the analyzed information, the server generates support information optimized for the user's current emotional state and operational status. This support information includes suggestions, advice, and warnings for the user.

[0532] Step 6:

[0533] The terminal receives support information sent from the server and provides it to the user through screen display and audio output. For example, if the user is feeling stressed, a message such as "Please pause your work and take a break" will be displayed.

[0534] Step 7:

[0535] Based on the support information provided by the device, the user selects the necessary actions for their task. Once the user makes a selection, the corresponding actions are automatically executed on the device.

[0536] Step 8:

[0537] The device records user responses and adjusts the system based on those responses to better optimize future support information suggestions. This enables personalized support based on the user's long-term usage history.

[0538] (Example 2)

[0539] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0540] Conventional user support systems generated support information using only the user's operation history and voice input, making it difficult to provide appropriate support that took into account the user's psychological and emotional state. Furthermore, there were security challenges regarding the information, requiring the secure processing and transmission of collected data.

[0541] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0542] In this invention, the server includes means for analyzing the user's emotional state, means for generating appropriate support information using a generative AI model, and means for sending and receiving encrypted data via a communication network. This makes it possible to provide comprehensive support information that takes the user's emotions into consideration.

[0543] "User" refers to an individual or organization that operates this system and receives support from it.

[0544] "Operation data" refers to input information generated when a user uses a computing device, including, for example, keyboard input and mouse operations.

[0545] "Voice input" refers to voice information emitted by the user through a microphone or similar device, and this information serves as the basis for the system's analysis.

[0546] "Facial expression information" refers to data based on the user's facial expressions captured by a camera device, and is used to analyze the user's emotional state.

[0547] "Emotional state" refers to the psychological or emotional state exhibited by the user, and is inferred from facial expressions, voice, etc.

[0548] A "generative AI model" is a system that uses machine learning algorithms to generate useful information from data, and is used to generate appropriate support information for users.

[0549] A "prompt statement" is a sentence containing commands or instructions that are input to a generative AI model to obtain a specific output.

[0550] "Support information" refers to information generated based on the analyzed user's actions and emotional state, and includes advice and suggestions to assist the user's work.

[0551] "Encryption" is a technology that converts data into a format that cannot be easily understood by third parties, thereby guaranteeing the secure transmission and reception of information.

[0552] A "communication network" refers to a network infrastructure used to exchange data between different devices, and includes the internet and local networks.

[0553] This invention is a system that assists user operations and provides advanced, personalized support that takes into account the user's emotional state. Specific embodiments of this system are described below.

[0554] The terminal is equipped with input devices (e.g., keyboard, mouse) and a microphone to collect user operation data. This allows for the capture of operation data and voice input, and also captures the user's facial expressions using a camera. This input is acquired as data and used for analyzing emotional states.

[0555] The terminal encrypts the collected data in real time and sends it to the server. HTTPS is used as the security protocol to ensure the safe transfer of data. Encryption methods such as AES are applied.

[0556] The server decodes the received data and processes the information using a generative AI model. In particular, the emotion analysis engine identifies the user's emotional state from their facial expressions and voice tone. Based on this analysis result and user interaction data, the generative AI model creates optimal support information using prompt messages. For example, a prompt message such as "The user is showing signs of fatigue. Please create suggestions to reduce their workload" might be used.

[0557] The server returns the generated support information to the terminal. The terminal notifies the user of this information through screen display or audio output and provides specific suggestions. For example, if the user is preparing a presentation and shows signs of fatigue, the terminal will provide a message such as, "We recommend you take a break. Your latest slide has been automatically saved."

[0558] This system receives feedback from users and continuously trains its generating AI model based on that feedback, thereby improving the quality of support information for subsequent uses. This makes it possible to continuously provide comprehensive support, including emotional support tailored to each individual user.

[0559] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0560] Step 1:

[0561] The device monitors user actions and collects input data from the keyboard and mouse. Furthermore, it captures voice input using a microphone and facial expressions with a camera. The collected data (action data, voice data, and facial expression data) is encrypted in real time for processing. Input is raw data of actions, voice, and facial expressions, while output is encrypted data. Specifically, encryption technologies such as AES are used to protect the data.

[0562] Step 2:

[0563] The terminal sends encrypted data to the server via the communication network. The input is encrypted data, and the output is a secure transmission to the server. HTTPS is used as the security protocol for this transmission. Specifically, the data is transmitted via the network interface.

[0564] Step 3:

[0565] The server decodes the received data and analyzes operation data, voice data, and emotion data individually. The input is the decoded data from the terminal, and the output is information indicating the analyzed emotion state and operation status. Specifically, it uses a generative AI model and an emotion analysis engine to extract data features and identify the emotion state.

[0566] Step 4:

[0567] The server uses an AI model based on the analysis results to create prompts and generate optimal support information. For example, a prompt might be "The user is showing signs of fatigue. Please create suggestions to reduce their workload." The input is the analysis results and the prompt, and the output is specific support information. In terms of operation, the model utilizes deep learning techniques to understand the context and generate appropriate suggestions.

[0568] Step 5:

[0569] The server sends the generated support information to the terminal, and the terminal receives it. The input is the generated support information, and the output is the secure reception to the terminal. Specifically, the information is encrypted before transmission and decrypted after reception on the terminal side.

[0570] Step 6:

[0571] The terminal provides support information to the user through screen display and audio output. Input is support information received from the server, and output is specific action suggestions for the user. For example, it might display a suggestion such as, "We recommend you take a break." Specific actions include using display control and audio output devices.

[0572] Step 7:

[0573] When the user accepts a suggestion, the device records the response. The input is user feedback, and the output is recorded training data. Specifically, the user's responses are saved to a database and used for subsequent support.

[0574] Step 8:

[0575] The server continuously trains the generated AI model based on feedback, improving the accuracy of subsequent assistance. The input is the recorded training data, and the output is the updated AI model. Specifically, it adjusts the model's parameters to improve accuracy.

[0576] (Application Example 2)

[0577] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0578] While systems that provide support for daily life require detailed support tailored to the user's actions and circumstances, there is a problem in that support that takes into account the user's emotional state is lacking. Therefore, a mechanism is needed that understands the user's emotions and provides appropriate support based on those emotions.

[0579] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0580] In this invention, the server includes means for monitoring the user's operations and surrounding environment and collecting this information; means for recognizing the user's voice input and facial expressions and analyzing the voice and facial expression data; and means for integrating and analyzing the collected operation data, voice data, and facial expression data to generate support information that takes into account the user's emotional state. This enables detailed support that responds to the user's emotions.

[0581] "User" refers to a person who performs operations or voice input on the system.

[0582] "Operation" refers to the interaction or instructions that a user gives to the system.

[0583] "Situation" refers to the environment or conditions in which the user finds themselves.

[0584] "Voice input" refers to voice information that a user sends to the system through a device such as a microphone.

[0585] "Facial expression" refers to the movements and expressions of a user's face, and is used to judge their emotional state.

[0586] "Data" refers to information that a system collects or receives from users.

[0587] "Analysis" refers to the act of processing collected data and extracting intended information.

[0588] "Emotional state" refers to the user's psychological condition, specifically the emotions inferred from their facial expressions and voice.

[0589] "Support information" refers to helpful information and suggestions generated by the system based on the user's actions and emotional state.

[0590] "Visual display" refers to information that is provided to the user visually through a screen, display, or other means.

[0591] "Audio output" refers to information provided to the user audibly through devices such as speakers.

[0592] "Processing" refers to a series of operations, calculations, and decisions performed by a system.

[0593] A "data center" refers to a facility or location where collected data is stored and processed.

[0594] "Encryption" refers to the process of securely converting data so that it cannot be read by third parties.

[0595] This invention is a system for providing support information that takes into account the user's emotional state. This system is primarily intended to support users in their homes and living environments using consumer robots.

[0596] The server uses a generative AI model to analyze the user's voice input and facial expression data to infer the user's emotions. This analysis includes a process of collecting data in real time using hardware such as cameras and microphones. Voice input is converted to text by a speech recognition engine, and facial expression data is processed by an emotion analysis algorithm.

[0597] The collected data is encrypted and sent to a data center. In this data center, generative AI models and emotion engines operate to analyze the data and generate supportive information. The generated supportive information is returned to the device through visual displays and audio output.

[0598] As a concrete example, if a user is experiencing stress in their daily life, the system will suggest relaxing music or rest. For instance, a notification might appear saying, "You seem tired today. Would you like to play some relaxing music?" This suggestion is generated by inputting the prompt "What is the best way to encourage relaxation when a user appears tired?" into a generating AI model.

[0599] This allows users to receive support tailored to their emotional state at any given time, thereby improving their quality of life. This system can be flexibly applied to various situations within the home, providing support that enriches the lives of users.

[0600] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0601] Step 1:

[0602] The device monitors the user's actions and surrounding environment, and collects input data using a camera and microphone. This input data includes the user's voice input and facial expressions. The device acquires this data in real time and temporarily stores it for subsequent processing.

[0603] Step 2:

[0604] The server receives encrypted audio data from the terminal and converts it to text using a speech recognition engine. This process visualizes the content of the audio and converts it into an analyzable format. The output is the converted text data.

[0605] Step 3:

[0606] The server receives encrypted facial expression data from the terminal. It applies an emotion analysis algorithm to infer the user's emotional state from the facial expression data. The input is a facial expression image, and the output is data indicating the emotional state.

[0607] Step 4:

[0608] The server integrates text and emotional state data from the audio data and generates support information using a generative AI model. During this process, prompts are used to determine the specific support content. The input consists of text and emotional state data, and the output is appropriate support information.

[0609] Step 5:

[0610] The terminal provides support information received from the server to the user through visual display and audio output. The terminal conveys support information to the user by displaying messages on the monitor and outputting audio through the speaker.

[0611] Step 6:

[0612] The device monitors the user's response again, and if there are any additional actions or new voice inputs, it sends them back to the server for further analysis. Through this process, the system adaptively learns from the user's feedback and improves the accuracy of assistance in subsequent interactions.

[0613] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0614] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0615] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0616] [Fourth Embodiment]

[0617] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0618] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0619] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0620] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0621] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0622] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0623] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0624] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0625] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0626] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0627] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0628] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0629] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0630] This invention relates to a system that comprehensively supports user operations on a PC, enabling effective work performance even in a teleworking environment.

[0631] Program Processing Overview

[0632] This system operates by monitoring and analyzing various operations and inputs that occur on the user's PC in real time. First, the terminal constantly monitors a series of user operations on the PC and collects data. This includes keyboard input, mouse movements, and application usage. The terminal also accepts voice input, using a microphone to recognize user instructions and questions and record them as audio data.

[0633] Next, the collected data is transmitted to a server via the internet, and secure communication is ensured using encryption technology. The server analyzes this data and uses a generative AI model to generate support information that the user needs. This support information includes suggestions for streamlining the user's current tasks and specific advice on operations.

[0634] Upon receiving the analysis results, the terminal displays support information as a pop-up on the screen and, if necessary, provides audio output through the speaker. The user responds by voice or click and selects an action provided by the system. Based on the selection, the terminal automatically performs the specified action. For example, this can automate operations such as updating a spreadsheet or retrieving documents.

[0635] As a concrete example, suppose a user is preparing for a regular meeting. At this time, the user needs to retrieve the latest data for the meeting. The device detects the user's editing operation in the spreadsheet and makes a voice suggestion, "Do you want to retrieve the latest data?" If the user voice-approves with "Yes," the device automatically retrieves the necessary information from the database and updates the specified spreadsheet.

[0636] By implementing this system, users can perform their tasks without worrying about a lack of information or operational errors, enabling efficient and accurate work. Therefore, variations in work skills are reduced, contributing to improved overall productivity.

[0637] The following describes the processing flow.

[0638] Step 1:

[0639] The device monitors the user's keyboard input, mouse movements, and application usage in real time, collecting this data locally. A microphone is also constantly on standby for voice input, recording user voice commands as digital audio data.

[0640] Step 2:

[0641] The device transmits the collected operation data and voice data to the server via the internet. During this transmission, the data is encrypted to ensure its security.

[0642] Step 3:

[0643] The server starts analysis using a generative AI model based on the received operation data and voice data. This determines the context and intent of the user's actions and generates appropriate support information.

[0644] Step 4:

[0645] The server generates support information derived from the analysis results and sends it back to the terminal. This support information includes suggestions for the user's current operations and advice on tasks that can be made more efficient.

[0646] Step 5:

[0647] The terminal provides the user with support information received from the server. This information is displayed as a pop-up on the screen or output as audio through the speaker. The user can respond to this information by voice or by clicking.

[0648] Step 6:

[0649] Based on the support information displayed on the terminal, users make selections according to their work needs. For example, they decide whether to perform an operation according to the suggested actions.

[0650] Step 7:

[0651] The terminal automatically performs specific operations based on user selections. This includes operations such as updating spreadsheets and searching for and retrieving data, automating the user's work.

[0652] (Example 1)

[0653] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0654] Modern information processing systems require efficient support for user operations and improved productivity. However, current systems struggle to accurately understand user operations and provide appropriate support in real time. In particular, it is difficult to integrate multiple functions such as operation monitoring, voice analysis, secure data communication, and the provision of support information.

[0655] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0656] In this invention, the server includes means for monitoring user operations in an information processing device and collecting them as data, means for receiving and analyzing user voice, and means for generating appropriate support using a generative model. This makes it possible to monitor user operations and voice in real time, provide appropriate support information based on them, and automatically execute actions based on user responses.

[0657] An "information processing device" is a device that monitors user operations and voice, and collects and analyzes them as data.

[0658] A "user" refers to a person who uses an information processing device, and whose operations and voice are monitored by the system.

[0659] "Operation" refers to actions performed by the user, such as keyboard input, mouse movement, and launching and closing applications.

[0660] "Voice" refers to the words or instructions that a user speaks to an information processing device through a microphone.

[0661] "Data" refers to information about user actions and voice recordings collected by information processing devices.

[0662] A "generative model" refers to algorithms and technologies used to analyze user actions and voice to generate appropriate support information.

[0663] "Support" refers to information and suggestions aimed at improving the user's current operations and work efficiency, and is provided to the user as support information.

[0664] A "prompt" is when an information processing device presents the user with an actionable choice or instruction.

[0665] The "Internet" is a global computer network used to transmit data collected from users to external devices.

[0666] "Encryption" refers to information transformation technology used to ensure the security of data being transmitted.

[0667] The system for implementing this invention mainly consists of an information processing device that monitors user operations and voice and provides support information based on them. The terminal monitors user operations in real time and collects data on keyboard input, mouse movements, and application usage. The terminal is also connected to a microphone that receives voices emitted by the user and records them as voice data.

[0668] Subsequently, the collected operation data and voice data are transmitted to a server via the internet. The server receives this data and analyzes it using a generative AI model. It then generates appropriate support information to improve the user's work efficiency. This analysis is based on the user's patterns and operation history.

[0669] The generated support information is sent to the device and displayed as a pop-up notification on the screen or the user is guided by voice. For example, if the user is editing a spreadsheet, the device will display a prompt asking, "Do you want to retrieve the latest data?" If the user responds with "Yes," the device will automatically retrieve the necessary data from the database and update the spreadsheet.

[0670] As a concrete example, consider a scenario where a user is preparing for a meeting. If the materials needed for the meeting need to be updated, the device can automatically retrieve the materials and keep them up-to-date. Examples of prompts include, "Please prepare the materials needed for the next meeting," or "Please create a new spreadsheet and retrieve the latest sales data."

[0671] This system allows users to perform their tasks while feeling that their actions are always supported, leading to increased efficiency and improved accuracy in their work.

[0672] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0673] Step 1:

[0674] The device monitors user actions and collects data in real time. Specifically, it records keyboard input, mouse movements, and application usage. It also uses a microphone to receive user voice and saves it as audio data. Inputs are user actions and voice, while outputs are action data and audio data.

[0675] Step 2:

[0676] The terminal transmits collected operation data and voice data to the server via the internet. During this process, the data is protected using encryption technology to prevent unauthorized access by third parties. The input consists of encrypted operation data and voice data, while the output is securely protected data on the server.

[0677] Step 3:

[0678] The server analyzes the received data. Using a generative AI model, it analyzes the user's operation patterns and generates necessary support information. This analysis determines the proposal of an optimized workflow and the efficient use of the tools to be used. The input is the received operator data and voice data, and the output is the generated support information.

[0679] Step 4:

[0680] The terminal receives analysis results and support information from the server and provides it to the user. Specifically, it displays a pop-up on the screen or provides audio notifications through the speaker. For example, it might prompt, "Do you want to use the slide template needed for your next meeting?" The input is the generated support information, and the output is the information presented to the user.

[0681] Step 5:

[0682] The user responds to prompts from the device. Based on these responses, the device automatically performs the necessary actions. For example, if the user responds with "yes" by voice, a new spreadsheet is automatically created and the necessary data is entered. The input is the user's response, and the output is the user-specified action that was performed.

[0683] (Application Example 1)

[0684] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0685] In today's remote work environment, users often face challenges in managing vast amounts of information and efficiently performing their tasks. In particular, when working with multiple digital tools, operational errors and inefficiencies can become problematic. Furthermore, there is a need for efficient support tailored to the user's skills and experience. In addition, there is a need for means of receiving information without disrupting concentration on work, by providing support information through visual and auditory methods.

[0686] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0687] In this invention, the server includes means for monitoring user operations and collecting them as data, means for receiving user voice input and analyzing the voice data, and an information processing device that analyzes the collected operation data and voice data and generates appropriate support information. As a result, users can receive necessary support information in real time and improve work efficiency through automated support.

[0688] "Monitoring user operations" refers to the process of observing a series of operations performed by a user on a personal computer in real time and collecting that data.

[0689] "Analyzing audio data" refers to the process of processing audio input provided by the user and converting it into information that allows for understanding its content.

[0690] An "information processing device" is a device that includes hardware and software with computing power to analyze collected data and generate necessary support information.

[0691] "Visualization and audio" refers to means of communicating generated support information to users visually and aurally.

[0692] "Augmented reality display through a virtual reality display" refers to a method of displaying information generated using virtual reality technology, merging the real world with digital information and presenting it to the user.

[0693] A "remote processing device" is a centrally managed computing device designed to receive data via a network and perform specific processing on it.

[0694] The system that realizes this invention supports users in efficiently performing their work in a teleworking environment. It promotes work efficiency by utilizing the user's personal computer and connected smart devices.

[0695] The system monitors user operations and collects operation data in real time. In one embodiment of the present invention, keyboard input, mouse operations, and application usage on a personal computer are acquired and recorded. The user provides voice input via a smart device, and the system converts this voice into text data using speech recognition technology.

[0696] The server analyzes the collected operational and audio data and uses a generative AI model to generate appropriate support information to assist the user's work. This process includes text analysis of the data and the generation of predictions and suggestions by the AI ​​model. Based on the analysis results, the information processing device generates the support information and provides it to the user as visualizations and audio.

[0697] Support information is displayed in augmented reality through a virtual reality display and communicated directly to the user via the smart device's display or audio output. Users can confirm the information provided through visual displays and audio guidance, which helps improve their work efficiency. For example, if a user gives a voice command such as "I want to check the latest meeting materials," the server uses a generative AI model to retrieve the appropriate information from the database and displays it on the user's display.

[0698] The generation AI model used in this system includes generation technology that makes suggestions based on the user's work content, and prompts are entered in the format of "Please retrieve the following data: Meeting updates." This system allows users to reduce errors in their work and improve work efficiency.

[0699] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0700] Step 1:

[0701] The terminal monitors user activity on a personal computer, collecting data such as keyboard input, mouse movements, and application usage. Input includes real-time user activity, and output is log data of these activities. This ensures the collection of necessary data.

[0702] Step 2:

[0703] The device receives user instructions via voice input and converts them into text data using speech recognition technology. The input is the user's voice, and the output is the text converted from the voice. Accurate text conversion of voice data enables subsequent data processing.

[0704] Step 3:

[0705] The terminal transmits collected operational and text data to the server using a secure protocol. The input is all digital data collected on the terminal side, and the output is the data transferred to the server. Once the data is securely transferred to the server, it is ready for analysis.

[0706] Step 4:

[0707] The server analyzes the received data and generates appropriate support information using a generative AI model. Input data includes operation logs and text data, while output is the generated support information. The server inputs prompt messages into the generative AI model, performing data analysis and prediction to generate useful suggestions for the user.

[0708] Step 5:

[0709] The server sends generated support information to the terminal, and the terminal provides the information to the user. The input is the support information sent from the server, and the output is the display or audio output of the information by the terminal. This allows the user to obtain the necessary support information in real time.

[0710] Step 6:

[0711] The user responds based on the information presented and provides instructions as needed. Input is the instructions given by the user based on the display or voice, and output is the feedback received by the device. Based on the user's instructions, subsequent operations and information retrieval proceed efficiently.

[0712] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0713] This invention provides comprehensive support that takes into account the user's emotional state by combining an emotion engine with a system that assists user operations on a PC.

[0714] Program Processing Overview

[0715] This system has the capability to monitor and analyze user actions and voice input in real time. By combining this with an emotion engine, it recognizes the user's current emotional state and takes that information into account when generating support information. Specifically, the terminal collects data on user actions and peripheral devices, and records voice input. It also uses a camera to capture the user's facial expressions and analyze their emotional state.

[0716] This data is encrypted for security purposes before being sent to the server, where it is analyzed using a generative AI model and an emotion engine. The server generates support information by combining the user's emotional state with the context of their actions, and returns this support information to the terminal.

[0717] The device uses the received support information to suggest appropriate actions to the user. This generated support information is provided to the user through screen display and audio output. For example, if the emotion engine recognizes that the user is experiencing stress, it can suggest ways to reduce workload or encourage rest.

[0718] As a concrete example, imagine a situation where a user is busy preparing a presentation. If the system detects fatigue from the user's facial expression, it will display a notification saying, "It's time for a break. The latest slide has been saved." If the user expresses gratitude for this, the system will continue to learn so that it can quickly perform the same action in similar situations in the future.

[0719] The implementation of this system will allow employees to perform their duties with psychological support, leading to improved work quality and increased productivity. By providing support information that takes emotional states into account, it will be possible to provide support that is tailored to the individual needs of each employee.

[0720] The following describes the processing flow.

[0721] Step 1:

[0722] The terminal monitors the user's PC operations in real time, collecting information on keyboard and mouse inputs, as well as the applications being used. Simultaneously, it uses a microphone to collect user voice commands as data.

[0723] Step 2:

[0724] The device uses its camera to capture the user's face and collect facial expression data. This facial expression data is processed as information necessary to analyze the user's emotional state.

[0725] Step 3:

[0726] The terminal encrypts the collected operation data, voice data, and facial expression data and transmits them to the server using a secure protocol. This makes it possible to transmit data while guaranteeing the security of the information.

[0727] Step 4:

[0728] The server analyzes the received data using a generative AI model to interpret the context of the operation and the content of the voice. Simultaneously, it uses an emotion engine to recognize the user's emotional state from facial expression data.

[0729] Step 5:

[0730] Based on the analyzed information, the server generates support information optimized for the user's current emotional state and operational status. This support information includes suggestions, advice, and warnings for the user.

[0731] Step 6:

[0732] The terminal receives support information sent from the server and provides it to the user through screen display and audio output. For example, if the user is feeling stressed, a message such as "Please pause your work and take a break" will be displayed.

[0733] Step 7:

[0734] Based on the support information provided by the device, the user selects the necessary actions for their task. Once the user makes a selection, the corresponding actions are automatically executed on the device.

[0735] Step 8:

[0736] The device records user responses and adjusts the system based on those responses to better optimize future support information suggestions. This enables personalized support based on the user's long-term usage history.

[0737] (Example 2)

[0738] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0739] Conventional user support systems generated support information using only the user's operation history and voice input, making it difficult to provide appropriate support that took into account the user's psychological and emotional state. Furthermore, there were security challenges regarding the information, requiring the secure processing and transmission of collected data.

[0740] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0741] In this invention, the server includes means for analyzing the user's emotional state, means for generating appropriate support information using a generative AI model, and means for sending and receiving encrypted data via a communication network. This makes it possible to provide comprehensive support information that takes the user's emotions into consideration.

[0742] "User" refers to an individual or organization that operates this system and receives support from it.

[0743] "Operation data" refers to input information generated when a user uses a computing device, including, for example, keyboard input and mouse operations.

[0744] "Voice input" refers to voice information emitted by the user through a microphone or similar device, and this information serves as the basis for the system's analysis.

[0745] "Facial expression information" refers to data based on the user's facial expressions captured by a camera device, and is used to analyze the user's emotional state.

[0746] "Emotional state" refers to the psychological or emotional state exhibited by the user, and is inferred from facial expressions, voice, etc.

[0747] A "generative AI model" is a system that uses machine learning algorithms to generate useful information from data, and is used to generate appropriate support information for users.

[0748] A "prompt statement" is a sentence containing commands or instructions that are input to a generative AI model to obtain a specific output.

[0749] "Support information" refers to information generated based on the analyzed user's actions and emotional state, and includes advice and suggestions to assist the user's work.

[0750] "Encryption" is a technology that converts data into a format that cannot be easily understood by third parties, thereby guaranteeing the secure transmission and reception of information.

[0751] A "communication network" refers to a network infrastructure used to exchange data between different devices, and includes the internet and local networks.

[0752] This invention is a system that assists user operations and provides advanced, personalized support that takes into account the user's emotional state. Specific embodiments of this system are described below.

[0753] The terminal is equipped with input devices (e.g., keyboard, mouse) and a microphone to collect user operation data. This allows for the capture of operation data and voice input, and also captures the user's facial expressions using a camera. This input is acquired as data and used for analyzing emotional states.

[0754] The terminal encrypts the collected data in real time and sends it to the server. HTTPS is used as the security protocol to ensure the safe transfer of data. Encryption methods such as AES are applied.

[0755] The server decodes the received data and processes the information using a generative AI model. In particular, the emotion analysis engine identifies the user's emotional state from their facial expressions and voice tone. Based on this analysis result and user interaction data, the generative AI model creates optimal support information using prompt messages. For example, a prompt message such as "The user is showing signs of fatigue. Please create suggestions to reduce their workload" might be used.

[0756] The server returns the generated support information to the terminal. The terminal notifies the user of this information through screen display or audio output and provides specific suggestions. For example, if the user is preparing a presentation and shows signs of fatigue, the terminal will provide a message such as, "We recommend you take a break. Your latest slide has been automatically saved."

[0757] This system receives feedback from users and continuously trains its generating AI model based on that feedback, thereby improving the quality of support information for subsequent uses. This makes it possible to continuously provide comprehensive support, including emotional support tailored to each individual user.

[0758] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0759] Step 1:

[0760] The device monitors user actions and collects input data from the keyboard and mouse. Furthermore, it captures voice input using a microphone and facial expressions with a camera. The collected data (action data, voice data, and facial expression data) is encrypted in real time for processing. Input is raw data of actions, voice, and facial expressions, while output is encrypted data. Specifically, encryption technologies such as AES are used to protect the data.

[0761] Step 2:

[0762] The terminal sends encrypted data to the server via the communication network. The input is encrypted data, and the output is a secure transmission to the server. HTTPS is used as the security protocol for this transmission. Specifically, the data is transmitted via the network interface.

[0763] Step 3:

[0764] The server decodes the received data and analyzes operation data, voice data, and emotion data individually. The input is the decoded data from the terminal, and the output is information indicating the analyzed emotion state and operation status. Specifically, it uses a generative AI model and an emotion analysis engine to extract data features and identify the emotion state.

[0765] Step 4:

[0766] The server uses an AI model based on the analysis results to create prompts and generate optimal support information. For example, a prompt might be "The user is showing signs of fatigue. Please create suggestions to reduce their workload." The input is the analysis results and the prompt, and the output is specific support information. In terms of operation, the model utilizes deep learning techniques to understand the context and generate appropriate suggestions.

[0767] Step 5:

[0768] The server sends the generated support information to the terminal, and the terminal receives it. The input is the generated support information, and the output is the secure reception to the terminal. Specifically, the information is encrypted before transmission and decrypted after reception on the terminal side.

[0769] Step 6:

[0770] The terminal provides support information to the user through screen display and audio output. Input is support information received from the server, and output is specific action suggestions for the user. For example, it might display a suggestion such as, "We recommend you take a break." Specific actions include using display control and audio output devices.

[0771] Step 7:

[0772] When the user accepts a suggestion, the device records the response. The input is user feedback, and the output is recorded training data. Specifically, the user's responses are saved to a database and used for subsequent support.

[0773] Step 8:

[0774] The server continuously trains the generated AI model based on feedback, improving the accuracy of subsequent assistance. The input is the recorded training data, and the output is the updated AI model. Specifically, it adjusts the model's parameters to improve accuracy.

[0775] (Application Example 2)

[0776] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0777] While systems that provide support for daily life require detailed support tailored to the user's actions and circumstances, there is a problem in that support that takes into account the user's emotional state is lacking. Therefore, a mechanism is needed that understands the user's emotions and provides appropriate support based on those emotions.

[0778] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0779] In this invention, the server includes means for monitoring the user's operations and surrounding environment and collecting this information; means for recognizing the user's voice input and facial expressions and analyzing the voice and facial expression data; and means for integrating and analyzing the collected operation data, voice data, and facial expression data to generate support information that takes into account the user's emotional state. This enables detailed support that responds to the user's emotions.

[0780] "User" refers to a person who performs operations or voice input on the system.

[0781] "Operation" refers to the interaction or instructions that a user gives to the system.

[0782] "Situation" refers to the environment or conditions in which the user finds themselves.

[0783] "Voice input" refers to voice information that a user sends to the system through a device such as a microphone.

[0784] "Facial expression" refers to the movements and expressions of a user's face, and is used to judge their emotional state.

[0785] "Data" refers to information that a system collects or receives from users.

[0786] "Analysis" refers to the act of processing collected data and extracting intended information.

[0787] "Emotional state" refers to the user's psychological condition, specifically the emotions inferred from their facial expressions and voice.

[0788] "Support information" refers to helpful information and suggestions generated by the system based on the user's actions and emotional state.

[0789] "Visual display" refers to information that is provided to the user visually through a screen, display, or other means.

[0790] "Audio output" refers to information provided to the user audibly through devices such as speakers.

[0791] "Processing" refers to a series of operations, calculations, and decisions performed by a system.

[0792] A "data center" refers to a facility or location where collected data is stored and processed.

[0793] "Encryption" refers to the process of securely converting data so that it cannot be read by third parties.

[0794] This invention is a system for providing support information that takes into account the user's emotional state. This system is primarily intended to support users in their homes and living environments using consumer robots.

[0795] The server uses a generative AI model to analyze the user's voice input and facial expression data to infer the user's emotions. This analysis includes a process of collecting data in real time using hardware such as cameras and microphones. Voice input is converted to text by a speech recognition engine, and facial expression data is processed by an emotion analysis algorithm.

[0796] The collected data is encrypted and sent to a data center. In this data center, generative AI models and emotion engines operate to analyze the data and generate supportive information. The generated supportive information is returned to the device through visual displays and audio output.

[0797] As a concrete example, if a user is experiencing stress in their daily life, the system will suggest relaxing music or rest. For instance, a notification might appear saying, "You seem tired today. Would you like to play some relaxing music?" This suggestion is generated by inputting the prompt "What is the best way to encourage relaxation when a user appears tired?" into a generating AI model.

[0798] This allows users to receive support tailored to their emotional state at any given time, thereby improving their quality of life. This system can be flexibly applied to various situations within the home, providing support that enriches the lives of users.

[0799] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0800] Step 1:

[0801] The device monitors the user's actions and surrounding environment, and collects input data using a camera and microphone. This input data includes the user's voice input and facial expressions. The device acquires this data in real time and temporarily stores it for subsequent processing.

[0802] Step 2:

[0803] The server receives encrypted audio data from the terminal and converts it to text using a speech recognition engine. This process visualizes the content of the audio and converts it into an analyzable format. The output is the converted text data.

[0804] Step 3:

[0805] The server receives encrypted facial expression data from the terminal. It applies an emotion analysis algorithm to infer the user's emotional state from the facial expression data. The input is a facial expression image, and the output is data indicating the emotional state.

[0806] Step 4:

[0807] The server integrates text and emotional state data from the audio data and generates support information using a generative AI model. During this process, prompts are used to determine the specific support content. The input consists of text and emotional state data, and the output is appropriate support information.

[0808] Step 5:

[0809] The terminal provides support information received from the server to the user through visual display and audio output. The terminal conveys support information to the user by displaying messages on the monitor and outputting audio through the speaker.

[0810] Step 6:

[0811] The device monitors the user's response again, and if there are any additional actions or new voice inputs, it sends them back to the server for further analysis. Through this process, the system adaptively learns from the user's feedback and improves the accuracy of assistance in subsequent interactions.

[0812] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0813] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0814] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0815] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0816] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0817] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0818] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0819] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0820] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0821] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0822] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0823] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0824] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0825] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0826] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0827] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0828] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0829] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0830] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0831] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0832] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0833] The following is further disclosed regarding the embodiments described above.

[0834] (Claim 1)

[0835] A means of monitoring user actions and collecting them as data,

[0836] A means for receiving the voice input of the user and analyzing the voice data,

[0837] A system device that analyzes the collected operation data and voice data and generates appropriate support information,

[0838] A means for providing the generated support information to the user via screen display and audio output,

[0839] Means for performing an operation specified based on the response from the user,

[0840] A system that includes this.

[0841] (Claim 2)

[0842] The system according to claim 1, characterized in that it makes suggestions for improving work efficiency based on the user's past operation history.

[0843] (Claim 3)

[0844] The system according to claim 1, characterized in that it transmits the collected data and generated support information to a server and communicates in an encrypted state.

[0845] "Example 1"

[0846] (Claim 1)

[0847] In an information processing device, means for monitoring user operations and collecting them as data,

[0848] A means for receiving the user's voice and analyzing that voice,

[0849] A means for generating appropriate support using a model that analyzes the collected operations and voices,

[0850] The means for providing the generated support to the user through information display and audio output,

[0851] Means for performing an action specified based on the response from the user,

[0852] A means for transmitting the collected data and generated support to an external device via the internet, and for communicating in an encrypted state,

[0853] A system that includes this.

[0854] (Claim 2)

[0855] The system according to claim 1, characterized in that it makes suggestions for improving work efficiency based on the user's past operation history.

[0856] (Claim 3)

[0857] The system according to claim 1, characterized in that it provides the aforementioned support to the user as a prompt and automatically performs the relevant tasks based on the user's response.

[0858] "Application Example 1"

[0859] (Claim 1)

[0860] A means of monitoring user actions and collecting them as data,

[0861] A means for receiving the voice input of the user and analyzing the voice data,

[0862] An information processing device that analyzes the collected operation data and voice data and generates appropriate support information,

[0863] The means of providing the generated support information to the user in the form of visualization and audio,

[0864] Means for performing an operation specified based on the response from the user,

[0865] Furthermore, the means for displaying the aforementioned support information in augmented reality through a virtual reality display,

[0866] A system that includes this.

[0867] (Claim 2)

[0868] The system according to claim 1, characterized in that it makes suggestions for improving work efficiency based on the user's past operation history.

[0869] (Claim 3)

[0870] The system according to claim 1, characterized in that it transmits the collected data and generated support information to a remote processing technology device and communicates in an encrypted state.

[0871] "Example 2 of combining an emotion engine"

[0872] (Claim 1)

[0873] A means of monitoring user actions and collecting them as data,

[0874] A means for analyzing the emotional state using the user's voice input and facial expression information,

[0875] An information processing device that analyzes the collected operation data, voice data, and emotion data, and generates appropriate support information based on prompt sentences using a generation AI model,

[0876] A means for providing the generated support information to the user via screen display and audio output,

[0877] A means by which the system learns based on the user's response,

[0878] A system that includes this.

[0879] (Claim 2)

[0880] The system according to claim 1, characterized in that it adjusts the workload and suggests breaks based on the emotional state of the user.

[0881] (Claim 3)

[0882] The system according to claim 1, characterized in that the collected data and generated support information are transmitted to an information processing server via a communication network, and the information is exchanged in an encrypted state.

[0883] "Application example 2 when combining with an emotional engine"

[0884] (Claim 1)

[0885] A means of monitoring the user's actions and surrounding circumstances, and collecting that information,

[0886] The means for recognizing the user's voice input and facial expressions, and for analyzing the voice and facial expression data,

[0887] A system device that integrates and analyzes the collected operation data, voice data, and facial expression data to generate support information that takes into account the emotional state,

[0888] A means for providing the generated support information to the user through visual display and audio output,

[0889] Means for performing a specified process based on the user's emotional response,

[0890] A system that includes this.

[0891] (Claim 2)

[0892] The system according to claim 1, characterized in that it makes suggestions for improving the efficiency of daily activities based on the user's past operation history and emotional state.

[0893] (Claim 3)

[0894] The system according to claim 1, characterized in that it transmits the collected data and generated support information to a data center and communicates in an encrypted state. [Explanation of symbols]

[0895] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of monitoring user actions and collecting them as data, A means for receiving the voice input of the user and analyzing the voice data, A system device that analyzes the collected operation data and voice data and generates appropriate support information, A means for providing the generated support information to the user via screen display and audio output, Means for performing an operation specified based on the response from the user, A system that includes this.

2. The system according to claim 1, characterized in that it makes suggestions for improving work efficiency based on the user's past operation history.

3. The system according to claim 1, characterized in that it transmits the collected data and generated support information to a server and communicates in an encrypted state.

Images