System

Abstract

A system is provided.SOLUTION: A system comprising: means for recording an operation performed by a user on an information processing apparatus in real time; means for analyzing recorded operation data using generative artificial intelligence to extract important points; means for integrating the extracted important points and additional related data to generate a manual; and means for providing the generated manual on the information processing apparatus.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] When creating and sharing manuals for business processes and operational procedures, it is difficult to explain specific operational procedures and background information using text alone. Furthermore, integrating information from different formats into a single manual requires significant technical and time-consuming effort. This invention aims to effectively integrate information from multiple data sources to more clearly explain operational procedures and their background and promote understanding of the information. [Means for solving the problem]

[0005] This invention provides the following means. First, it includes means for recording operations performed by a user on an information processing device in real time. Next, it includes means for analyzing the recorded operation data using generative artificial intelligence and automatically extracting important points from the results. It further includes means for integrating the extracted important points with additional related data and generating a manual that clearly explains the background and context of the operations based on this information. Finally, it includes means for providing the generated manual to the user on the information processing device. This makes it easier for the user to understand the intention and background of the operations, thereby improving work efficiency.

[0006] A "user" is an individual or group that operates an information processing device.

[0007] An "information processing device" is a computer or other digital device for recording and analyzing user operations.

[0008] "Real-time" refers to recording and processing data simultaneously with user actions.

[0009] "Recording" means saving a user's operations in the form of video, audio, text, etc.

[0010] "Operation data" is data indicating a series of operations performed by a user on an information processing device.

[0011] "Generative artificial intelligence" is a computer program or system that has the ability to analyze and learn from collected data.

[0012] "Analysis" is the process of examining the recorded data in detail to understand the user's operations and their intentions.

[0013] An "important point" is a part of a user's operation that is particularly noteworthy or essential.

[0014] "Related data" is additional information related to a user's operation, and is provided in the form of text, images, audio, or the like.

[0015] "Integration" means bringing together information collected from multiple data sources into a single document or system.

[0016] A "manual" is a document or piece of paper created to explain the user's operating procedures and the background to them.

[0017] "Providing" means making the generated manual available in a form that can be accessed by users. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0019] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0022] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0023] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0024] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0026] [First embodiment]

[0027] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0028] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0031] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0034] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0038] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0039] This invention relates to a system that records operations performed by a user on an information processing device in real time, analyzes and integrates the data, and generates a manual. Specific embodiments of this system will be described below.

[0040] Recording user actions

[0041] First, when a user starts a specific task on an information processing device, the device records the user's actions in real time. The recording is done in three forms:

[0042] 1. Video recording: The device captures the user's screen and saves it as video data, allowing you to visually confirm the operations that the user actually performed.

[0043] 2. Audio recording: The device will record the user's voice and environmental sounds through the microphone input, so that what the user says and the background sounds while operating the device will be recorded.

[0044] 3. Text recording: The terminal stores keyboard and mouse inputs as text data, thereby recording the commands and text entered by the user.

[0045] Sending data

[0046] Once recording begins, the device periodically generates and transmits data packets containing video, audio, and text data to the server, each accompanied by metadata such as a timestamp and user ID.

[0047] Data analysis

[0048] The server analyzes the received data packet and performs the following actions:

[0049] 1. Video and audio analysis using generative AI: The server uses generative AI to extract important points from video and audio data, such as clicking on important menu items or operating settings screens.

[0050] 2. Natural language processing: The server analyzes the text data using a natural language processing engine to identify operational procedures and related background information, thereby clarifying the user's intent and purpose.

[0051] Context Integration

[0052] Based on the analysis results, the server collects and integrates additional data related to the operation procedure, including other text, images, and audio, such as help documents related to the operation and screenshots of the settings screen.

[0053] Generate manual

[0054] The server uses the integrated data to create a manual that explains the operation procedures in an easy-to-understand format. The manual contains the following elements:

[0055] 1. Video Clips: Short video clips are inserted to show user actions.

[0056] 2. Text Instructions: Contains detailed text instructions related to the operating procedure.

[0057] 3. Additional Related Information: Provides additional background information and notes related to the operation.

[0058] Providing manuals

[0059] Finally, the generated manual is sent to the terminal and made accessible to the user. By viewing this manual, the user can gain a deeper understanding of the operating procedures and their background, allowing them to work more efficiently.

[0060] Specific examples

[0061] For example, if a user records the process of installing new software, the device will record the entire installation process as video, audio, and text. The server will analyze this data and extract important installation steps. It will then integrate additional help information and system requirements related to the installation process and generate a manual that includes them. Finally, this manual will be sent to the user's device, where the user can follow the installation process while checking the instructions.

[0062] In this way, by effectively recording, analyzing, and integrating user operations and their background information, a detailed and easy-to-understand manual is provided, allowing users to understand the intentions and concepts behind the operations and work efficiently.

[0063] The processing flow will be explained below.

[0064] Step 1:

[0065] The user starts a specific task on the information processing device and clicks the record button, thereby starting the recording process.

[0066] Step 2:

[0067] The device detects the user's clicks and begins recording the user's actions in real time, including by:

[0068] Video recording: Using screen capture software, the entire user's PC screen is saved as video data.

[0069] Audio recording: Environmental sounds and the user's voice are saved as audio data via microphone input.

[0070] Text recording: Using a keylogger or mouse event capture function, keyboard input and mouse operations are saved as text data.

[0071] Step 3:

[0072] The terminal divides the recorded data into packets of a certain size and adds metadata such as a timestamp and a user ID to each packet.

[0073] Step 4:

[0074] The terminal sends data packets sequentially to the server, using the SSL / TLS protocol to ensure data security.

[0075] Step 5:

[0076] The server takes the received data packets sequentially and reassembles them into one continuous data set, allowing the entire user interaction to be analyzed as a continuous data stream.

[0077] Step 6:

[0078] The server uses generative AI to analyze the received video and audio data. The AI ​​model does the following:

[0079] It detects and tags important actions from video data, for example identifying the moment a specific button is clicked or a settings menu is opened.

[0080] The voice data is converted into text by a speech recognition engine, and specific instructions or explanations are extracted.

[0081] Step 7:

[0082] The server analyzes the text data with a natural language processing (NLP) engine to identify operational steps and related context, such as when a particular command is entered and what its meaning is.

[0083] Step 8:

[0084] Based on the analysis, the server collects additional data related to the operation procedure from internal databases and external APIs, including:

[0085] The URL and content of the website accessed during the user's operation.

[0086] Links and contents of manuals and help documents related to the operation.

[0087] System requirements and configuration information required for operation.

[0088] Step 9:

[0089] The server consolidates the collected data and assembles it into a document structure that provides operational instructions, highlighting background information and important points related to each step.

[0090] Step 10:

[0091] The server applies the generated content to the presentation layer, producing the final manual in multiple formats (PDF, HTML, video files).

[0092] Step 11:

[0093] The server sends the generated manual to the terminal and uploads it to a repository accessible to the user. The user receives a notification and can view the manual on their PC.

[0094] Step 12:

[0095] Users can refer to the generated manual to check the necessary operating procedures and related background information, which will deepen their understanding of the operations and improve work efficiency.

[0096] Example 1

[0097] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0098] In the past, recording the operations performed by users on information processing devices and creating manuals to communicate those operating procedures to other users was a time-consuming and labor-intensive process. Furthermore, manual recording and editing is prone to human error and omissions, making it difficult to create efficient procedure manuals. For this reason, there was a demand for a system that could accurately and efficiently record user operations and automatically generate manuals based on that information.

[0099] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0100] In this invention, the server includes means for recording in real time operations performed by a user on an information processing device, means for saving the recorded operation data in video, audio, and text formats, means for analyzing the recorded operation data using a generative AI model and extracting important points, means for generating a manual by integrating the extracted important points with additional related information, and means for providing the generated manual on the information processing device. This makes it possible to automatically and accurately record user operations and quickly generate a manual based on the operation procedures.

[0101] The term "user" refers to a person who performs an operation on an information processing device.

[0102] "Information processing device" refers to a computer or its peripheral devices for inputting, processing, storing, and outputting data.

[0103] "Real-time recording" refers to recording user operations immediately.

[0104] "Operation data" refers to data indicating the history and content of operations performed by a user on an information processing device.

[0105] "Video format" refers to video data captured from the user's screen.

[0106] "Audio format" refers to audio data that records the user's voice or environmental sounds.

[0107] "Text format" refers to character data that records keyboard input and mouse operations.

[0108] A "generative AI model" is a model that uses artificial intelligence to analyze data and extract important information and patterns.

[0109] "Analysis" refers to the act of analyzing recorded data to extract meaning and patterns.

[0110] The "important points" refer to particularly noteworthy or critical parts of the operations performed by the user.

[0111] "Related information" refers to information such as additional text, images, and audio that is related to the user's operating procedure.

[0112] "Integration" refers to the act of bringing together data or information in different formats.

[0113] A "manual" refers to a document or multimedia content that clearly shows and explains user operating procedures.

[0114] "Providing" refers to the act of displaying or saving the generated manual on an information processing device in a form that can be used by the user.

[0115] This invention relates to a system that records operations performed by a user on an information processing device in real time, analyzes and integrates the data, and generates a manual. Specific embodiments of this system will be described below.

[0116] Recording user actions

[0117] First, when a user starts a specific task on an information processing device, the device records the user's actions in real time. The recording is done in three forms:

[0118] 1. Video recording: The device captures the user's screen and saves it as video data using video capture software such as OBS Studio, allowing you to visually confirm the operations that the user actually performed.

[0119] 2. Audio recording: The device records the user's voice and environmental sounds through microphone input. For example, the device uses a built-in microphone or an external microphone to collect audio data, recording what the user says and the background sounds while operating the device.

[0120] 3. Text recording: The terminal stores keyboard and mouse inputs as text data. Logging software is used to record the commands and text entered by the user.

[0121] Sending data

[0122] Once recording begins, the device periodically generates data packets and sends them to the server. The data packets contain video, audio, and text data, and each packet is assigned metadata such as a timestamp and user ID. For example, a packet could be generated every minute and sent to the server via the network.

[0123] Data analysis

[0124] The server analyzes the received data packet and performs the following actions:

[0125] 1. Video and audio analysis using generative AI models: The server uses generative AI models (such as OpenAI's GPT model) to extract important points from video and audio data, such as clicking on important menu items or operating settings screens.

[0126] 2. Natural Language Processing: The server analyzes the text data using a natural language processing engine (e.g., spaCy) to identify operational steps and related background information, thereby clarifying the user's intent and purpose.

[0127] Context Integration

[0128] Based on the analysis results, the server collects and integrates additional data related to the operation procedure, including other text, images, and audio, such as help documents related to the operation and screenshots of the settings screen.

[0129] Generate manual

[0130] The server uses the integrated data to create a manual that explains the operation procedures in an easy-to-understand format. The manual contains the following elements:

[0131] 1. Video Clips: Short video clips are inserted to show user actions.

[0132] 2. Text Instructions: Contains detailed text instructions related to the operating procedure.

[0133] 3. Additional Related Information: Provides additional background information and notes related to the operation.

[0134] Providing manuals

[0135] Finally, the generated manual is sent to the terminal and made accessible to the user. By viewing this manual, the user can gain a deeper understanding of the operating procedures and their background, allowing them to work more efficiently.

[0136] Specific examples

[0137] For example, if a user records the process of installing new software, the device will record the entire installation process as video, audio, and text. The server will analyze this data and extract important installation steps. It will then integrate additional help information and system requirements related to the installation process and generate a manual that includes them. Finally, this manual will be sent to the user's device, allowing the user to easily follow the installation process.

[0138] Prompt Sentence Examples

[0139] "This system records the steps a user takes to install new software, then analyzes and integrates the data to generate a detailed operating manual. Please provide a concrete example of how the generative AI model employed in this process extracts key installation steps and integrates relevant information."

[0140] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0141] Step 1:

[0142] When a user starts a specific task on the information processing device, the device records the user's actions in real time. For example, when starting to install new software, the device launches OBS Studio and starts capturing the screen. It also records the user's voice via microphone and logs keyboard and mouse input data.

[0143] Input: User operation begins

[0144] Output: Video files, audio files, text logs

[0145] Step 2:

[0146] The device packets the collected data at regular intervals and sends it to the server. For example, video, audio, and text data are packetized every minute, and metadata such as timestamps and user IDs are added. A library for packet generation is used to send the data over the network to the server.

[0147] Input: Video files, audio files, text logs

[0148] Output: Data packets

[0149] Step 3:

[0150] The server analyzes the received data packets and extracts key points using generative AI models, such as OpenAI's GPT model, to detect click events from video, commands from audio, and settings from text.

[0151] Input: Data packet

[0152] Output: List of key points

[0153] Step 4:

[0154] The server then collects and integrates additional relevant information based on the extracted key points, using a natural language processing engine (e.g., spaCy) to analyze the text data and collect relevant help documentation and screenshots of configuration screens.

[0155] Input: List of key points

[0156] Output: Consolidated related information

[0157] Step 5:

[0158] The server then uses the integrated data to create easy-to-understand operation procedures in a manual, generating documents including video clips, detailed text explanations, and background information.

[0159] Input: Integrated related information

[0160] Output: Manual document

[0161] Step 6:

[0162] The server sends the generated manual to the terminal, where the user can view the manual and check the operating procedures as they work.

[0163] Input: Manual document

[0164] Output: A user-readable manual

[0165] (Application example 1)

[0166] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0167] Conventional methods for creating manuals for factory robot operation procedures required a great deal of time and effort for manual recording and analysis, making them inefficient. Furthermore, operation procedures were rarely provided in a format that was easy for workers to understand, leading to operational errors and reduced efficiency. Furthermore, it was difficult to check operation procedures on different devices, creating a need for an integrated solution to improve on-site work efficiency.

[0168] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0169] In this invention, the server includes means for recording operations performed by a user on an information processing device in real time, means for analyzing the recorded operation data using artificial intelligence and extracting important points, means for generating a manual by integrating the extracted important points with additional related data, means for providing the generated manual on the information processing device, and means for recording and viewing operation procedures for robot operations in factories via wearable devices such as smart glasses and head-mounted displays. This allows for real-time recording and analysis of operation procedures, enabling workers to instantly check easy-to-understand manuals via wearable devices, thereby enabling efficient and accurate operation.

[0170] A "user" is a person or entity that performs an operation on an information processing device.

[0171] An "information processing device" is a device that inputs, processes, and outputs data, and includes personal computers, tablets, smartphones, etc.

[0172] "Real-time" refers to operations and processing that are recorded and analyzed at the same time as they are performed.

[0173] "Recording" refers to the process of saving user operations as data.

[0174] "Analysis" refers to the process of evaluating recorded data and extracting important information.

[0175] "Generative artificial intelligence" refers to algorithms and software that have the ability to extract meaning and patterns from input data.

[0176] "Important points" refer to parts or steps that require special attention in the user's operations.

[0177] "Extraction" refers to the process of extracting specific information from the overall data.

[0178] "Integration" refers to the process of combining multiple pieces of data or information into one whole.

[0179] A "manual" refers to a document or material that describes operating procedures and guidelines.

[0180] "Providing" refers to the process of displaying the generated manual in a form that is accessible to users.

[0181] "Robot operation in a factory" refers to operating a robot in a factory to perform specific tasks.

[0182] "Smart glasses" refers to a wearable device in the shape of glasses that has the function of displaying information.

[0183] A "head-mounted display" refers to a device worn on the head that displays images and information.

[0184] A "wearable device" refers to an information processing device that can be worn.

[0185] This invention is a system that records and analyzes operations performed by a user on an information processing device in real time to efficiently operate robots in a factory, and provides the results as a manual.

[0186] System Overview

[0187] The system consists of the following elements:

[0188] 1. Information processing devices used by users (PCs, tablets, smartphones, etc.)

[0189] 2. Recording equipment (camera and microphone)

[0190] 3. Generative Artificial Intelligence (AI Model)

[0191] 4. Server (performs data analysis and manual generation)

[0192] 5. Wearable devices (smart glasses, head-mounted displays)

[0193] How it works

[0194] Recording user actions

[0195] When the user starts operating the robot, the camera records the operation video in real time. At the same time, the microphone records the audio, and keyboard input and mouse operations are recorded as text data. This recorded data is periodically sent to the server as data packets.

[0196] Data transmission and analysis

[0197] The server analyzes the received data packets and uses generative artificial intelligence to extract key points from the video and audio data. Specifically, it detects important operation steps, commands, setting changes, etc. At the same time, it also analyzes the text data using a natural language processing engine to clarify operation procedures and intentions.

[0198] Context integration and manual generation

[0199] Based on the analysis results, the server collects and integrates additional related data (help documents, screenshots, configuration details, etc.). Based on the integrated data, an easy-to-understand operation manual is created. The manual includes video clips, text explanations, and additional related information.

[0200] Providing manuals

[0201] The generated manual is sent to the user's information processing device and wearable device, where the user can check the operation procedures in real time via smart glasses or a head-mounted display.

[0202] Hardware and software used

[0203] The system uses the following hardware and software:

[0204] Recording device: Camera (e.g., Logitech C920) and microphone (e.g., Blue Yeti)

[0205] Generative Artificial Intelligence: Azure Cognitive Services Text Analytics API

[0206] Information processing devices: Windows PCs, Macs, tablets, smartphones

[0207] Wearable devices: smart glasses (e.g., Google Glass), head-mounted displays (e.g., Microsoft HoloLens)

[0208] Specific examples

[0209] For example, when a user assembles parts using a robot in a factory, the entire operation is recorded using a camera and microphone. The server analyzes this recorded data, identifies important steps (such as the correct position of some parts or how to tighten screws), and generates a detailed manual. The user can then proceed with the work while viewing the generated manual in real time through smart glasses or a head-mounted display.

[0210] Prompt Sentence Examples

[0211] "Record the steps the robot takes to assemble the parts, extract the important steps, and generate a manual based on them."

[0212] In this way, operating procedures can be recorded and analyzed in real time, and workers can instantly check easy-to-understand manuals via wearable devices, enabling efficient and accurate operations.

[0213] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0214] Step 1:

[0215] User operation initiation and recording

[0216] The user begins operating the factory robot. The device uses a camera and microphone to record video and audio of the operation in real time. At the same time, keyboard and mouse operations are also recorded as text data.

[0217] Input: User operation video, voice, keyboard input, mouse operation

[0218] Data processing: Capture video, record audio, record text input

[0219] Output: Data packets collected in real time

[0220] Specific behavior: Capture video with a camera (e.g. Logitech C920), record audio with a microphone (e.g. Blue Yeti), and save the text to a local file.

[0221] Step 2:

[0222] Creating and sending data packets

[0223] The terminal generates the collected data as data packets at regular intervals and transmits them to the server.

[0224] Input: Data collected in real time (video, audio, text)

[0225] Data processing: Packetizing data (adding timestamps and user IDs)

[0226] Output: Data packet with timestamp and user ID added

[0227] Specific operation: Data packets are generated at regular intervals and sent to the server via the network.

[0228] Step 3:

[0229] Data reception and storage by the server

[0230] The server stores the received data packets, organizing them based on user ID and timestamp.

[0231] Input: Data packets (video, audio, text)

[0232] Data processing: packets are decoded, stored, and organized by metadata

[0233] Output: Organized data (stored in a database or file system)

[0234] Specific operation: Save to a database system (e.g. MySQL), record to the file system.

[0235] Step 4:

[0236] Data analysis

[0237] The server analyzes the stored data using artificial intelligence to extract key points, while voice and text data is analyzed using a natural language processing engine.

[0238] Input: stored video, audio, and text data

[0239] Data calculation: AI model-based video analysis (e.g., object detection), audio analysis (e.g., sound recognition), and text analysis (e.g., key phrase extraction)

[0240] Output: Analysis results (important operation points, key phrases, etc.)

[0241] Specific operation: Uses Azure Cognitive Services' Text Analytics API and object detection models (e.g., YOLO).

[0242] Step 5:

[0243] Context Integration

[0244] Based on the analyzed results, the server collects and integrates additional relevant data (help documentation, screenshots, configuration details, etc.).

[0245] Input: Analysis results (important operation points, key phrases, etc.)

[0246] Data processing: Collecting and integrating additional data (e.g., web crawling, database searching)

[0247] Output: Integrated data set (full operating procedures and related information)

[0248] Specific actions: API calls, database queries.

[0249] Step 6:

[0250] Generate manual

[0251] The server uses the integrated dataset to create easy-to-understand instruction manuals, including video clips, text instructions, and additional relevant information.

[0252] Input: Integrated data set (full operating procedures and related information)

[0253] Data processing: Data formatting and manual generation (using document generation tools)

[0254] Output: Generated manual (PDF, HTML, etc.)

[0255] Specific behavior: Uses a documentation generation tool (e.g., LaTeX, Markdown).

[0256] Step 7:

[0257] Providing manuals

[0258] The server then sends the generated manual to the user's information processing device or wearable device, where it can be displayed. The user can view the operating procedures via smart glasses or a head-mounted display.

[0259] Input: Generated manual (PDF, HTML, etc.)

[0260] Data processing: Sending data (over a network connection)

[0261] Output: Manual displayed on terminal and wearable devices

[0262] Specific operations: Send data to the device, use a display application (e.g., a dedicated viewer).

[0263] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0264] This invention combines a system that records user operations on an information processing device in real time, analyzes and integrates the data, and generates a manual, with an emotion engine. Specific embodiments of this system will be described below.

[0265] Recording user actions

[0266] First, when a user starts a specific task on an information processing device, the device records the user's actions in real time. The recordings are made in the following four forms:

[0267] 1. Video recording: The device captures the user's screen and saves it as video data, allowing you to visually confirm the operations that the user actually performed.

[0268] 2. Audio recording: The device will record the user's voice and environmental sounds through the microphone input, so that what the user says and the background sounds while operating the device will be recorded.

[0269] 3. Text recording: The terminal stores keyboard and mouse inputs as text data, thereby recording the commands and text entered by the user.

[0270] 4. Emotion recording: The device uses a camera and microphone to capture the user's facial expressions and voice tone in real time, and the emotion engine analyzes this data to recognize the user's emotions.

[0271] Sending data

[0272] Once recording begins, the device periodically generates and transmits data packets containing video, audio, text, and emotion data to the server. Each packet is accompanied by metadata such as a timestamp and user ID.

[0273] Data analysis

[0274] The server analyzes the received data packet and performs the following actions:

[0275] 1. Video and audio analysis using generative AI: The server uses generative AI to extract important points from video and audio data, such as clicking on important menu items or operating settings screens.

[0276] 2. Natural language processing: The server analyzes the text data using a natural language processing engine to identify operational procedures and related background information, thereby clarifying the user's intent and purpose.

[0277] 3. Emotion analysis: The server analyzes the emotion data provided by the emotion engine to determine the emotional state of the user at each point of the operation, for example, whether they felt stressed or happy during the operation.

[0278] Context Integration

[0279] Based on the analysis results, the server collects and integrates additional data related to the operation procedure, including other text, image, audio, and emotional data, such as help documents related to the operation and screenshots of the settings screen.

[0280] Generate manual

[0281] The server uses the integrated data to create a manual that explains the operation procedures in an easy-to-understand format. The manual contains the following elements:

[0282] 1. Video Clips: Short video clips are inserted to show user actions.

[0283] 2. Text Instructions: Contains detailed text instructions related to the operating procedure.

[0284] 3. Additional Related Information: Provides additional background information and notes related to the operation.

[0285] 4. Emotion analysis results: Data showing the user's emotional state is inserted as diagrams and text, allowing users to visually understand which parts of the operation procedure the user found difficult or went smoothly.

[0286] Providing manuals

[0287] Finally, the generated manual is sent to the terminal and made accessible to the user. By viewing this manual, the user can check the operation procedures, background information, and their own emotional state while working.

[0288] Specific examples

[0289] For example, if a user records the steps to install new software, the device records the entire installation process as video, audio, text, and emotional data. The server analyzes this data to extract important installation steps and the user's emotional state during the process. The server then integrates additional help information and system requirements related to the installation process and generates a manual that includes these. The manual visually indicates the points where the user felt stressed and the points where the installation went smoothly. Finally, the manual is sent to the user's device, where the user can review the installation steps and refer to the results of their own emotional analysis.

[0290] In this way, by effectively recording, analyzing, and integrating the user's emotional state in addition to their operations and their background information, a detailed and easy-to-understand manual is provided, allowing the user to understand the intentions and concepts behind the operations and work efficiently.

[0291] The processing flow will be explained below.

[0292] Step 1:

[0293] The user starts a specific task on the information processing device and clicks a record button, thereby starting the recording process.

[0294] Step 2:

[0295] The device detects the user's click and starts real-time recording in the following format:

[0296] Video recording: Using screen capture software, the entire user's PC screen is saved as video data.

[0297] Audio recording: Saves the user's voice and environmental sounds as audio data through microphone input.

[0298] Text recording: Using a keylogger or mouse event capture function, keyboard input and mouse operations are saved as text data.

[0299] Emotion Recording: Using a camera and microphone, the system captures the user's facial expressions and voice tone in real time, and the emotion engine analyzes this data to recognize the user's emotions.

[0300] Step 3:

[0301] The terminal divides the recorded data into packets of a certain size and adds metadata such as a timestamp and a user ID to each packet.

[0302] Step 4:

[0303] The terminal sends data packets sequentially to the server, using the SSL / TLS protocol to ensure data security.

[0304] Step 5:

[0305] The server takes the received data packets sequentially and reassembles them into one continuous data set, allowing the entire user interaction to be analyzed as a continuous data stream.

[0306] Step 6:

[0307] The server uses generative AI to analyze the received video and audio data. The AI ​​model does the following:

[0308] It detects and tags important actions from video data, for example identifying the moment a specific button is clicked or a settings menu is opened.

[0309] The voice data is converted into text by a speech recognition engine, and specific instructions or explanations are extracted.

[0310] Step 7:

[0311] The server analyzes the text data with a natural language processing (NLP) engine to identify operational steps and related context, such as when a particular command is entered and what its meaning is.

[0312] Step 8:

[0313] The server analyzes the emotional data provided by the emotion engine and identifies the emotional state the user was in at each point of the operation, for example, whether they were feeling stressed or happy during the operation.

[0314] Step 9:

[0315] Based on the analysis, the server collects additional data related to the operation procedure from internal databases and external APIs, including:

[0316] The URL and content of the website accessed during the user's operation.

[0317] Links and contents of manuals and help documents related to the operation.

[0318] System requirements and configuration information required for operation.

[0319] Step 10:

[0320] The server consolidates the collected data and assembles it into a document structure that provides operational instructions, highlighting background information and important points related to each step.

[0321] Step 11:

[0322] The server applies the generated content to the presentation layer, producing the final manual in multiple formats (PDF, HTML, video files).

[0323] Step 12:

[0324] The server sends the generated manual to the terminal and uploads it to a repository accessible to the user. The user receives a notification and can view the manual on their PC.

[0325] Step 13:

[0326] Users can refer to the generated manual to check the necessary operating procedures, their background information, and their own emotional state, which will deepen their understanding of the operation and improve work efficiency.

[0327] Example 2

[0328] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0329] Conventional information processing devices were able to record user operations and generate manuals based on those operations. However, they did not consider grasping the user's emotional state and incorporating that into the manual. As a result, it was not possible to provide a detailed manual that reflected the stress and difficulties the user experienced during operation. The present invention aims to visualize the difficulties the user experiences by recording and analyzing the user's emotional state in addition to operation data and generating an integrated manual, thereby supporting easier and smoother operation.

[0330] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0331] In this invention, the server includes means for recording in real time operations performed by a user on an information processing device, means for classifying the recorded operation data into video, audio, text, and emotional data, means for analyzing the classified operation data using a generative artificial intelligence to extract important points and the user's emotional state, means for integrating the extracted important points, emotional state, and additional related data to generate a manual, and means for providing the generated manual on the information processing device. This makes it possible to comprehensively grasp the user's operations and the emotions felt at the time, and to provide a more accurate manual.

[0332] The term "user" refers to a person who operates an information processing device.

[0333] "Information processing device" refers to computer equipment or digital devices that users can operate.

[0334] "Real time" refers to processing that is carried out continuously in real time.

[0335] "Recording" refers to saving information such as the user's actions, comments, and facial expressions.

[0336] "Video" refers to data that expresses movement by playing back successive image frames over time.

[0337] "Audio" refers to data that contains recorded human voices and surrounding environmental sounds.

[0338] "Text" refers to data that stores keyboard input and mouse operations as character information.

[0339] "Emotion data" refers to data that indicates the emotional state of a user extracted from facial expressions and vocal tones.

[0340] "Classification" refers to dividing the recorded operation data into video, audio, text, and emotion data.

[0341] "Generative AI" refers to AI technology that analyzes recorded data and recognizes and generates specific information and patterns.

[0342] "Analysis" refers to the detailed examination of recorded data to uncover meanings and patterns.

[0343] "Important points" refer to actions or content that are particularly noteworthy in the operation data.

[0344] "User's emotional state" refers to the type and strength of emotions felt by the user during operation.

[0345] "Synthesis" refers to bringing together extracted key points, emotional states, and related data into a single framework.

[0346] A "manual" refers to a guide document that summarizes user operating procedures, background information, and emotion analysis results.

[0347] "Providing" refers to presenting information such as generated manuals to users in an easy-to-use format.

[0348] This invention relates to a system that records the operations performed by a user on an information processing device in real time, analyzes the recorded operation data, and generates a manual. In particular, it aims to record and analyze the user's emotional state and provide a detailed manual that integrates this data.

[0349] Recording user actions

[0350] First, the user starts a specific task on the information processing device. For example, when the user starts installing new software, the device records the operation in real time in the following format:

[0351] 1. Video recording: The device captures the user's screen and saves it as video data in MP4 format. This video recording allows you to visually confirm the operations that the user actually performed.

[0352] 2. Audio recording: The device will record the user's voice and environmental sounds in WAV format using the built-in microphone or an external microphone, so that what the user says during operation and the surrounding background sounds will be recorded.

[0353] 3. Text logging: The terminal records keyboard and mouse inputs in text format in a log file, which records the commands and text entered by the user.

[0354] 4. Emotion Recording: The device uses a camera and microphone to capture the user's facial expressions and voice tone in real time, and stores them as emotion data through the emotion engine. This emotion data is stored in JSON format.

[0355] Sending data

[0356] Once recording begins, the device will collect data at regular intervals (e.g., every minute) and compile it into a single data packet, which includes a video file, an audio file, a text log, and emotion data.

[0357] The device then sends the generated data packets to the server via an internet connection, using encryption technology such as SSL / TLS.

[0358] Data analysis

[0359] The server receives data packets sent from the device, generates an analysis request for the received data, and sends it to an internal processing engine service. The server then uses a generative artificial intelligence model (e.g., OpenAI's GPT-4) to perform the next analysis.

[0360] Video and audio analysis: Extracts important operation points from video data frame by frame, and picks out instructions and important statements from audio data.

[0361] Natural language processing: Analyzes commands and operation details from text logs and identifies operation procedures based on the context.

[0362] Emotion analysis: Use the output of the emotion engine to identify the emotional state the user was in during each action.

[0363] Context Integration

[0364] Based on the parsed data, the server collects additional relevant information from databases and external resources, such as help documents and FAQ pages for operations.

[0365] The server combines this additional information with the analysis results to create a single integrated dataset, which includes operating instructions, related help documents, sentiment analysis results, etc.

[0366] Generate manual

[0367] The server uses the integrated data to create a manual in a format that is easy for users to understand. This includes the following elements:

[0368] Video Clips: Short video clips are inserted to show user actions.

[0369] Text Instructions: Contains detailed text instructions related to the procedure.

[0370] Additional Related Information: Provides additional background information or notes related to the operation.

[0371] Sentiment analysis results: Data showing the user's emotional state is inserted as diagrams and text, allowing users to visually understand which parts of the operation procedure the user found difficult or went smoothly.

[0372] Providing manuals

[0373] Finally, the generated manual is sent to the terminal and made accessible to the user. By viewing this manual, the user can check the operation procedures, background information, and their own emotional state while working.

[0374] Examples of specific examples and prompts

[0375] For example, if a user records the steps to install new software, they can input the following prompts to the generative AI model:

[0376] "Analyze data recording the steps to install new software and generate a manual that combines operating procedures and emotional states."

[0377] This allows a specific and detailed manual to be automatically generated, supporting the user's operations.

[0378] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0379] Processing Steps

[0380] Step 1:

[0381] A user starts a specific task on an information processing device. The input is a trigger operated by the user to start the task. The output is the start of real-time recording of the user's operations. At this stage, the system starts monitoring the operations.

[0382] Step 2:

[0383] The terminal records operation data in real time. The specific operations are as follows:

[0384] Video recording: The device launches screen capture software and saves the user's screen operations in MP4 format. The input is the user's screen information, and the output is video data.

[0385] Audio recording: The device uses a microphone to record ambient sounds in WAV format. The input is microphone input and the output is audio data.

[0386] Text logging: The terminal records keyboard and mouse activity in a text log. The input is the user's keyboard and mouse actions, and the output is text data.

[0387] Emotion recording: The device uses a camera and microphone to capture the user's facial expressions and voice tone in real time, which are then analyzed by the emotion engine. The input is data from the camera and microphone, and the output is emotion data.

[0388] Step 3:

[0389] The terminal aggregates the recorded data into data packets at regular intervals. The input is video, audio, text, and emotion data collected in real time. The output is the aggregated data packets.

[0390] Step 4:

[0391] The terminal sends the integrated data packet to the server using encryption technology such as SSL / TLS. The input is the data packet, and the output is the data transmission to the server.

[0392] Step 5:

[0393] The server receives data packets: the input is the data packet sent by the terminal, and the output is the acknowledgement.

[0394] Step 6:

[0395] The server analyzes the received data packets to generate requests and sends them to an internal processing engine service. The input is the data packets and the output is the analysis requests.

[0396] Step 7:

[0397] The server analyzes the data using a generative artificial intelligence model, specifically by:

[0398] Video and audio analysis: Extracts important operation points from input data frame by frame and picks out important statements. The output is analyzed operation data.

[0399] Natural language processing: Extracting important commands and operations from text data. The input is text data, and the output is parsed text data.

[0400] Sentiment Analysis: The output of the emotion engine is used to extract the user's emotional state during operation. The input is emotion data, and the output is the analyzed emotion data.

[0401] Step 8:

[0402] The server collects additional relevant information based on the analysis results. The input is the analyzed data, and the output is the additional relevant information.

[0403] Step 9:

[0404] The server integrates the collected additional information and analysis results into a single dataset. The input is the analysis results and additional information, and the output is the integrated dataset.

[0405] Step 10:

[0406] The server uses the integrated dataset to generate a manual, including video clips, text descriptions, additional related information, and sentiment analysis results. The input is the integrated dataset, and the output is the generated manual.

[0407] Step 11:

[0408] The server sends the generated manual to the terminal. The input is the generated manual, and the output is the delivery of the manual to the terminal.

[0409] Step 12:

[0410] The user reads the received manual on the terminal and checks the operation procedures and emotion analysis results. The input is the received manual, and the output is operation support for the user.

[0411] This allows for a comprehensive understanding of the user's operations and the emotions felt at the time, making it possible to provide more accurate manuals.

[0412] (Application example 2)

[0413] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0414] On conventional online shopping sites, users often feel anxious and confused when operating the site for the first time. In particular, new users are often not provided with effective operation guides, which increases the likelihood of operational errors and a lack of understanding of operation procedures. Furthermore, manuals provided without considering the user's emotions make it difficult to alleviate the stress and difficulties experienced during operation. To solve these problems, a system is needed that records and analyzes user operations, and then generates and provides custom operation guides that incorporate the user's emotional state.

[0415] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0416] In this invention, the server includes means for recording in real time the operations performed by a user on an information processing device, means for analyzing the recorded operation data using a generative artificial intelligence to extract important points, means for analyzing the user's emotional state using an emotion engine, means for generating a manual by integrating the extracted important points with additional related data and emotional data, and means for providing the generated manual on the information processing device. This makes it possible to generate a detailed customized manual for the user's operating procedures and provide support that takes into account the user's emotional state.

[0417] A "user" is a subject who operates an information processing device.

[0418] An "information processing device" is a device that processes digital data, such as a computer, smartphone, or tablet.

[0419] "Means for recording in real time" refers to a function or device for instantly recording or recording user operations on the spot.

[0420] "Operation data" is data that records the content of operations performed by a user, and includes video, audio, text, and the like.

[0421] "Generative artificial intelligence" is an AI technology that uses machine learning and deep learning, and is a method of extracting features from data and analyzing them.

[0422] "Important points" are parts of user operations that require special attention, or important events or operations in the system.

[0423] An "emotion engine" is software or a system for analyzing a user's emotional state from their tone of voice and facial expressions.

[0424] "Related data" refers to the operating procedures as well as helpful help documents and additional information related to the operation.

[0425] The "means for generating a manual" refers to a function or software that integrates recorded operation data and related information to create a manual in a format that is easy for the user to understand.

[0426] The "means for providing on an information processing device" refers to a function or method for displaying the generated manual on the information processing device operated by the user, making it viewable.

[0427] This invention is a system that records the operations performed by users on information processing devices in real time, analyzes and integrates the operation data and emotional data, and generates and provides a customized operation manual.

[0428] 1. System Configuration

[0429] Hardware

[0430] Information processing device: A terminal operated by a user, including computers, smartphones, tablets, etc.

[0431] Camera and microphone: Input devices for recording user actions, facial expressions, and voice tones in real time.

[0432] Server: Back-end infrastructure for analyzing operation data and emotion data and generating manuals.

[0433] software

[0434] OpenCV: A library for capturing and processing video and image data.

[0435] Google Cloud Speech-to-Text API: A service for converting voice data into text data.

[0436] TensorFlow: A machine learning library for analyzing emotions from facial expressions and voice.

[0437] Django Framework: A server-side framework for generating and serving operation manuals.

[0438] JavaScript: A front-end technology for displaying manuals on information processing devices.

[0439] 2. System Functions

[0440] Recording user actions

[0441] When a user starts a specific task on the information processing device, the device records the user's actions in real time. The recordings are in the following format:

[0442] Video recording: Use the camera to capture the user's operation screen and save it as video data.

[0443] Audio recording: Records the user's voice and environmental sounds through microphone input and saves them as audio data.

[0444] Emotion recording: Uses a camera and microphone to capture the user's facial expressions and vocal tone, which are then analyzed by the emotion engine.

[0445] Data transmission and analysis

[0446] Once recording begins, the device will periodically generate data packets and send them to the server. The data packets contain video, audio, text, and emotion data. The server will analyze the data and extract key points. The complete analysis process includes the following steps:

[0447] Generative AI video and audio analysis

[0448] Natural Language Processing (NLP)

[0449] Emotion analysis

[0450] Manual generation and provision

[0451] Based on the analysis results, the server integrates additional data related to user operations and generates an intuitive and easy-to-understand operation manual. The generated manual includes the following elements:

[0452] Video Clip: A short video clip showing the procedure

[0453] Text Description: Detailed instructions related to the procedure

[0454] Sentiment analysis results: Data that visually indicates the user's emotional state

[0455] The generated manual is finally sent to the terminal, and the user can refer to this manual as they work.

[0456] 3. Specific Examples

[0457] For example, if a user records the steps of purchasing a product on an online shopping site, the device records the entire purchasing process as video, audio, and emotional data. The server analyzes this data to extract important steps and the user's emotional state during the process. It then integrates additional help information and product reviews related to the purchasing process and generates a custom operation guide that includes them.

[0458] Example prompts for generative AI models

[0459] "On your online store, walk users through the steps of adding an item to their cart and visually demonstrate points of frustration."

[0460] As described above, this system can record and analyze the user's operations and their emotional state in detail, and generate and provide optimal operation guidance.

[0461] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0462] Step 1:

[0463] When a user starts operating an information processing device, the device records the user's operations in real time. At this point, the device's camera and microphone are activated, capturing the user's screen operation as a video and recording the user's voice input. The input data is the user's screen capture video and voice data, which are saved as recorded data.

[0464] Step 2:

[0465] The device uses an emotion engine to analyze the user's facial expressions and voice tone in real time to generate emotion data. Specifically, an emotion recognition model using TensorFlow predicts the user's emotional state (e.g., stress, joy, neutral, etc.) from their facial expressions and voice. This emotion data is stored along with the video and audio data. The input data is the user's facial expressions and voice, and the output data is the analyzed emotional state.

[0466] Step 3:

[0467] After recording for a certain period of time or for each specific operation, the device sends these data packets to the server. The data packets include video, audio, text, and emotion data. The input data is the recorded data packets, and the output data is the data sent to the server. Specifically, the device secures a stable network connection and uploads the data to the server using protocols such as HTTP or WebSocket.

[0468] Step 4:

[0469] The server analyzes the received data packets and begins processing various data. First, it uses generative AI to extract key operation points from the video and audio data. For example, it identifies the icons the user clicked and the text fields they entered. The input data is the video and audio data, and the output data is the analyzed key points. Specifically, it runs a generative AI model (for example, YOLO or DeepSpeech) to detect key operations.

[0470] Step 5:

[0471] The server then analyzes the text data using natural language processing (NLP). This clarifies the intent of the user's input text or command and identifies operational procedures and related background information. The input data is the text data, and the output data is the analyzed operational procedures and background information. Specifically, the server performs text analysis using an NLP engine (e.g., spaCy or BERT).

[0472] Step 6:

[0473] The server analyzes the emotion data provided by the emotion engine and identifies the user's emotional state at each operation point. The input data is emotion data, and the output data is the emotional state corresponding to each operation point. Specifically, the server maps the emotional state at each operation step from the time-series data.

[0474] Step 7:

[0475] Based on the analysis results, the server collects and integrates additional data related to the operation procedures, such as help documents and screenshots related to the operations. The input data is the analyzed operation procedures and emotion data, and the output data is a draft of the integrated operation manual. Specific operations include the process of retrieving related information from databases and external APIs.

[0476] Step 8:

[0477] The server generates a detailed operating manual based on the integrated data. The generated manual includes video clips, text descriptions, additional related information, and sentiment analysis results. The input data is a draft of the integrated operating manual, and the output data is the final generated operating manual. Specifically, the manual document is automatically generated using a template engine.

[0478] Step 9:

[0479] Finally, the server sends the generated operation manual to the terminal, making it accessible to the user. The input data is the finally generated operation manual, and the output data is the data sent to the terminal. Specific operations include providing information to the user through HTTP responses and updating the database.

[0480] As described above, the input data is analyzed and processed at each processing step, and ultimately a detailed custom operation manual is generated and provided to assist the user in their operations.

[0481] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0482] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0483] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0484] [Second embodiment]

[0485] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0486] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0487] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0488] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0489] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0490] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0491] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0492] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0493] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0494] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0495] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0496] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0497] This invention relates to a system that records operations performed by a user on an information processing device in real time, analyzes and integrates the data, and generates a manual. Specific embodiments of this system will be described below.

[0498] Recording user actions

[0499] First, when a user starts a specific task on an information processing device, the device records the user's actions in real time. The recording is done in three forms:

[0500] 1. Video recording: The device captures the user's screen and saves it as video data, allowing you to visually confirm the operations that the user actually performed.

[0501] 2. Audio recording: The device will record the user's voice and environmental sounds through the microphone input, so that what the user says and the background sounds while operating the device will be recorded.

[0502] 3. Text recording: The terminal stores keyboard and mouse inputs as text data, thereby recording the commands and text entered by the user.

[0503] Sending data

[0504] Once recording begins, the device periodically generates and transmits data packets containing video, audio, and text data to the server, each accompanied by metadata such as a timestamp and user ID.

[0505] Data analysis

[0506] The server analyzes the received data packet and performs the following actions:

[0507] 1. Video and audio analysis using generative AI: The server uses generative AI to extract important points from video and audio data, such as clicking on important menu items or operating settings screens.

[0508] 2. Natural language processing: The server analyzes the text data using a natural language processing engine to identify operational procedures and related background information, thereby clarifying the user's intent and purpose.

[0509] Context Integration

[0510] Based on the analysis results, the server collects and integrates additional data related to the operation procedure, including other text, images, and audio, such as help documents related to the operation and screenshots of the settings screen.

[0511] Generate manual

[0512] The server uses the integrated data to create a manual that explains the operation procedures in an easy-to-understand format. The manual contains the following elements:

[0513] 1. Video Clips: Short video clips are inserted to show user actions.

[0514] 2. Text Instructions: Contains detailed text instructions related to the operating procedure.

[0515] 3. Additional Related Information: Provides additional background information and notes related to the operation.

[0516] Providing manuals

[0517] Finally, the generated manual is sent to the terminal and made accessible to the user. By viewing this manual, the user can gain a deeper understanding of the operating procedures and their background, allowing them to work more efficiently.

[0518] Specific examples

[0519] For example, if a user records the process of installing new software, the device will record the entire installation process as video, audio, and text. The server will analyze this data and extract important installation steps. It will then integrate additional help information and system requirements related to the installation process and generate a manual that includes them. Finally, this manual will be sent to the user's device, where the user can follow the installation process while checking the instructions.

[0520] In this way, by effectively recording, analyzing, and integrating user operations and their background information, a detailed and easy-to-understand manual is provided, allowing users to understand the intentions and concepts behind the operations and work efficiently.

[0521] The processing flow will be explained below.

[0522] Step 1:

[0523] The user starts a specific task on the information processing device and clicks the record button, thereby starting the recording process.

[0524] Step 2:

[0525] The device detects the user's clicks and begins recording the user's actions in real time, including by:

[0526] Video recording: Using screen capture software, the entire user's PC screen is saved as video data.

[0527] Audio recording: Environmental sounds and the user's voice are saved as audio data via microphone input.

[0528] Text recording: Using a keylogger or mouse event capture function, keyboard input and mouse operations are saved as text data.

[0529] Step 3:

[0530] The terminal divides the recorded data into packets of a certain size and adds metadata such as a timestamp and a user ID to each packet.

[0531] Step 4:

[0532] The terminal sends data packets sequentially to the server, using the SSL / TLS protocol to ensure data security.

[0533] Step 5:

[0534] The server takes the received data packets sequentially and reassembles them into one continuous data set, allowing the entire user interaction to be analyzed as a continuous data stream.

[0535] Step 6:

[0536] The server uses generative AI to analyze the received video and audio data. The AI ​​model does the following:

[0537] It detects and tags important actions from video data, for example identifying the moment a specific button is clicked or a settings menu is opened.

[0538] The voice data is converted into text by a speech recognition engine, and specific instructions or explanations are extracted.

[0539] Step 7:

[0540] The server analyzes the text data with a natural language processing (NLP) engine to identify operational steps and related context, such as when a particular command is entered and what its meaning is.

[0541] Step 8:

[0542] Based on the analysis, the server collects additional data related to the operation procedure from internal databases and external APIs, including:

[0543] The URL and content of the website accessed during the user's operation.

[0544] Links and contents of manuals and help documents related to the operation.

[0545] System requirements and configuration information required for operation.

[0546] Step 9:

[0547] The server consolidates the collected data and assembles it into a document structure that provides operational instructions, highlighting background information and important points related to each step.

[0548] Step 10:

[0549] The server applies the generated content to the presentation layer, producing the final manual in multiple formats (PDF, HTML, video files).

[0550] Step 11:

[0551] The server sends the generated manual to the terminal and uploads it to a repository accessible to the user. The user receives a notification and can view the manual on their PC.

[0552] Step 12:

[0553] Users can refer to the generated manual to check the necessary operating procedures and related background information, which will deepen their understanding of the operations and improve work efficiency.

[0554] Example 1

[0555] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0556] In the past, recording the operations performed by users on information processing devices and creating manuals to communicate those operating procedures to other users was a time-consuming and labor-intensive process. Furthermore, manual recording and editing is prone to human error and omissions, making it difficult to create efficient procedure manuals. For this reason, there was a demand for a system that could accurately and efficiently record user operations and automatically generate manuals based on that information.

[0557] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0558] In this invention, the server includes means for recording in real time operations performed by a user on an information processing device, means for saving the recorded operation data in video, audio, and text formats, means for analyzing the recorded operation data using a generative AI model and extracting important points, means for generating a manual by integrating the extracted important points with additional related information, and means for providing the generated manual on the information processing device. This makes it possible to automatically and accurately record user operations and quickly generate a manual based on the operation procedures.

[0559] The term "user" refers to a person who performs an operation on an information processing device.

[0560] "Information processing device" refers to a computer or its peripheral devices for inputting, processing, storing, and outputting data.

[0561] "Real-time recording" refers to recording user operations immediately.

[0562] "Operation data" refers to data indicating the history and content of operations performed by a user on an information processing device.

[0563] "Video format" refers to video data captured from the user's screen.

[0564] "Audio format" refers to audio data that records the user's voice or environmental sounds.

[0565] "Text format" refers to character data that records keyboard input and mouse operations.

[0566] A "generative AI model" is a model that uses artificial intelligence to analyze data and extract important information and patterns.

[0567] "Analysis" refers to the act of analyzing recorded data to extract meaning and patterns.

[0568] The "important points" refer to particularly noteworthy or critical parts of the operations performed by the user.

[0569] "Related information" refers to information such as additional text, images, and audio that is related to the user's operating procedure.

[0570] "Integration" refers to the act of bringing together data or information in different formats.

[0571] A "manual" refers to a document or multimedia content that clearly shows and explains user operating procedures.

[0572] "Providing" refers to the act of displaying or saving the generated manual on an information processing device in a form that can be used by the user.

[0573] This invention relates to a system that records operations performed by a user on an information processing device in real time, analyzes and integrates the data, and generates a manual. Specific embodiments of this system will be described below.

[0574] Recording user actions

[0575] First, when a user starts a specific task on an information processing device, the device records the user's actions in real time. The recording is done in three forms:

[0576] 1. Video recording: The device captures the user's screen and saves it as video data using video capture software such as OBS Studio, allowing you to visually confirm the operations that the user actually performed.

[0577] 2. Audio recording: The device records the user's voice and environmental sounds through microphone input. For example, the device uses a built-in microphone or an external microphone to collect audio data, recording what the user says and the background sounds while operating the device.

[0578] 3. Text recording: The terminal stores keyboard and mouse inputs as text data. Logging software is used to record the commands and text entered by the user.

[0579] Sending data

[0580] Once recording begins, the device periodically generates data packets and sends them to the server. The data packets contain video, audio, and text data, and each packet is assigned metadata such as a timestamp and user ID. For example, a packet could be generated every minute and sent to the server via the network.

[0581] Data analysis

[0582] The server analyzes the received data packet and performs the following actions:

[0583] 1. Video and audio analysis using generative AI models: The server uses generative AI models (such as OpenAI's GPT model) to extract important points from video and audio data, such as clicking on important menu items or operating settings screens.

[0584] 2. Natural Language Processing: The server analyzes the text data using a natural language processing engine (e.g., spaCy) to identify operational steps and related background information, thereby clarifying the user's intent and purpose.

[0585] Context Integration

[0586] Based on the analysis results, the server collects and integrates additional data related to the operation procedure, including other text, images, and audio, such as help documents related to the operation and screenshots of the settings screen.

[0587] Generate manual

[0588] The server uses the integrated data to create a manual that explains the operation procedures in an easy-to-understand format. The manual contains the following elements:

[0589] 1. Video Clips: Short video clips are inserted to show user actions.

[0590] 2. Text Instructions: Contains detailed text instructions related to the operating procedure.

[0591] 3. Additional Related Information: Provides additional background information and notes related to the operation.

[0592] Providing manuals

[0593] Finally, the generated manual is sent to the terminal and made accessible to the user. By viewing this manual, the user can gain a deeper understanding of the operating procedures and their background, allowing them to work more efficiently.

[0594] Specific examples

[0595] For example, if a user records the process of installing new software, the device will record the entire installation process as video, audio, and text. The server will analyze this data and extract important installation steps. It will then integrate additional help information and system requirements related to the installation process and generate a manual that includes them. Finally, this manual will be sent to the user's device, allowing the user to easily follow the installation process.

[0596] Prompt Sentence Examples

[0597] "This system records the steps a user takes to install new software, then analyzes and integrates the data to generate a detailed operating manual. Please provide a concrete example of how the generative AI model employed in this process extracts key installation steps and integrates relevant information."

[0598] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0599] Step 1:

[0600] When a user starts a specific task on the information processing device, the device records the user's actions in real time. For example, when starting to install new software, the device launches OBS Studio and starts capturing the screen. It also records the user's voice via microphone and logs keyboard and mouse input data.

[0601] Input: User operation begins

[0602] Output: Video files, audio files, text logs

[0603] Step 2:

[0604] The device packets the collected data at regular intervals and sends it to the server. For example, video, audio, and text data are packetized every minute, and metadata such as timestamps and user IDs are added. A library for packet generation is used to send the data over the network to the server.

[0605] Input: Video files, audio files, text logs

[0606] Output: Data packets

[0607] Step 3:

[0608] The server analyzes the received data packets and extracts key points using generative AI models, such as OpenAI's GPT model, to detect click events from video, commands from audio, and settings from text.

[0609] Input: Data packet

[0610] Output: List of key points

[0611] Step 4:

[0612] The server then collects and integrates additional relevant information based on the extracted key points, using a natural language processing engine (e.g., spaCy) to analyze the text data and collect relevant help documentation and screenshots of configuration screens.

[0613] Input: List of key points

[0614] Output: Consolidated related information

[0615] Step 5:

[0616] The server then uses the integrated data to create easy-to-understand operation procedures in a manual, generating documents including video clips, detailed text explanations, and background information.

[0617] Input: Integrated related information

[0618] Output: Manual document

[0619] Step 6:

[0620] The server sends the generated manual to the terminal, where the user can view the manual and check the operating procedures as they work.

[0621] Input: Manual document

[0622] Output: A user-readable manual

[0623] (Application example 1)

[0624] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0625] Conventional methods for creating manuals for factory robot operation procedures required a great deal of time and effort for manual recording and analysis, making them inefficient. Furthermore, operation procedures were rarely provided in a format that was easy for workers to understand, leading to operational errors and reduced efficiency. Furthermore, it was difficult to check operation procedures on different devices, creating a need for an integrated solution to improve on-site work efficiency.

[0626] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0627] In this invention, the server includes means for recording operations performed by a user on an information processing device in real time, means for analyzing the recorded operation data using artificial intelligence and extracting important points, means for generating a manual by integrating the extracted important points with additional related data, means for providing the generated manual on the information processing device, and means for recording and viewing operation procedures for robot operations in factories via wearable devices such as smart glasses and head-mounted displays. This allows for real-time recording and analysis of operation procedures, enabling workers to instantly check easy-to-understand manuals via wearable devices, thereby enabling efficient and accurate operation.

[0628] A "user" is a person or entity that performs an operation on an information processing device.

[0629] An "information processing device" is a device that inputs, processes, and outputs data, and includes personal computers, tablets, smartphones, etc.

[0630] "Real-time" refers to operations and processing that are recorded and analyzed at the same time as they are performed.

[0631] "Recording" refers to the process of saving user operations as data.

[0632] "Analysis" refers to the process of evaluating recorded data and extracting important information.

[0633] "Generative artificial intelligence" refers to algorithms and software that have the ability to extract meaning and patterns from input data.

[0634] "Important points" refer to parts or steps that require special attention in the user's operations.

[0635] "Extraction" refers to the process of extracting specific information from the overall data.

[0636] "Integration" refers to the process of combining multiple pieces of data or information into one whole.

[0637] A "manual" refers to a document or material that describes operating procedures and guidelines.

[0638] "Providing" refers to the process of displaying the generated manual in a form that is accessible to users.

[0639] "Robot operation in a factory" refers to operating a robot in a factory to perform specific tasks.

[0640] "Smart glasses" refers to a wearable device in the shape of glasses that has the function of displaying information.

[0641] A "head-mounted display" refers to a device worn on the head that displays images and information.

[0642] A "wearable device" refers to an information processing device that can be worn.

[0643] This invention is a system that records and analyzes operations performed by a user on an information processing device in real time to efficiently operate robots in a factory, and provides the results as a manual.

[0644] System Overview

[0645] The system consists of the following elements:

[0646] 1. Information processing devices used by users (PCs, tablets, smartphones, etc.)

[0647] 2. Recording equipment (camera and microphone)

[0648] 3. Generative Artificial Intelligence (AI Model)

[0649] 4. Server (performs data analysis and manual generation)

[0650] 5. Wearable devices (smart glasses, head-mounted displays)

[0651] How it works

[0652] Recording user actions

[0653] When the user starts operating the robot, the camera records the operation video in real time. At the same time, the microphone records the audio, and keyboard input and mouse operations are recorded as text data. This recorded data is periodically sent to the server as data packets.

[0654] Data transmission and analysis

[0655] The server analyzes the received data packets and uses generative artificial intelligence to extract key points from the video and audio data. Specifically, it detects important operation steps, commands, setting changes, etc. At the same time, it also analyzes the text data using a natural language processing engine to clarify operation procedures and intentions.

[0656] Context integration and manual generation

[0657] Based on the analysis results, the server collects and integrates additional related data (help documents, screenshots, configuration details, etc.). Based on the integrated data, an easy-to-understand operation manual is created. The manual includes video clips, text explanations, and additional related information.

[0658] Providing manuals

[0659] The generated manual is sent to the user's information processing device and wearable device, where the user can check the operation procedures in real time via smart glasses or a head-mounted display.

[0660] Hardware and software used

[0661] The system uses the following hardware and software:

[0662] Recording device: Camera (e.g., Logitech C920) and microphone (e.g., Blue Yeti)

[0663] Generative Artificial Intelligence: Azure Cognitive Services Text Analytics API

[0664] Information processing devices: Windows PCs, Macs, tablets, smartphones

[0665] Wearable devices: smart glasses (e.g., Google Glass), head-mounted displays (e.g., Microsoft HoloLens)

[0666] Specific examples

[0667] For example, when a user assembles parts using a robot in a factory, the entire operation is recorded using a camera and microphone. The server analyzes this recorded data, identifies important steps (such as the correct position of some parts or how to tighten screws), and generates a detailed manual. The user can then proceed with the work while viewing the generated manual in real time through smart glasses or a head-mounted display.

[0668] Prompt Sentence Examples

[0669] "Record the steps the robot takes to assemble the parts, extract the important steps, and generate a manual based on them."

[0670] In this way, operating procedures can be recorded and analyzed in real time, and workers can instantly check easy-to-understand manuals via wearable devices, enabling efficient and accurate operations.

[0671] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0672] Step 1:

[0673] User operation initiation and recording

[0674] The user begins operating the factory robot. The device uses a camera and microphone to record video and audio of the operation in real time. At the same time, keyboard and mouse operations are also recorded as text data.

[0675] Input: User operation video, voice, keyboard input, mouse operation

[0676] Data processing: Capture video, record audio, record text input

[0677] Output: Data packets collected in real time

[0678] Specific behavior: Capture video with a camera (e.g. Logitech C920), record audio with a microphone (e.g. Blue Yeti), and save the text to a local file.

[0679] Step 2:

[0680] Creating and sending data packets

[0681] The terminal generates the collected data as data packets at regular intervals and transmits them to the server.

[0682] Input: Data collected in real time (video, audio, text)

[0683] Data processing: Packetizing data (adding timestamps and user IDs)

[0684] Output: Data packet with timestamp and user ID added

[0685] Specific operation: Data packets are generated at regular intervals and sent to the server via the network.

[0686] Step 3:

[0687] Data reception and storage by the server

[0688] The server stores the received data packets, organizing them based on user ID and timestamp.

[0689] Input: Data packets (video, audio, text)

[0690] Data processing: packets are decoded, stored, and organized by metadata

[0691] Output: Organized data (stored in a database or file system)

[0692] Specific operation: Save to a database system (e.g. MySQL), record to the file system.

[0693] Step 4:

[0694] Data analysis

[0695] The server analyzes the stored data using artificial intelligence to extract key points, while voice and text data is analyzed using a natural language processing engine.

[0696] Input: stored video, audio, and text data

[0697] Data calculation: AI model-based video analysis (e.g., object detection), audio analysis (e.g., sound recognition), and text analysis (e.g., key phrase extraction)

[0698] Output: Analysis results (important operation points, key phrases, etc.)

[0699] Specific operation: Uses Azure Cognitive Services' Text Analytics API and object detection models (e.g., YOLO).

[0700] Step 5:

[0701] Context Integration

[0702] Based on the analyzed results, the server collects and integrates additional relevant data (help documentation, screenshots, configuration details, etc.).

[0703] Input: Analysis results (important operation points, key phrases, etc.)

[0704] Data processing: Collecting and integrating additional data (e.g., web crawling, database searching)

[0705] Output: Integrated data set (full operating procedures and related information)

[0706] Specific actions: API calls, database queries.

[0707] Step 6:

[0708] Generate manual

[0709] The server uses the integrated dataset to create easy-to-understand instruction manuals, including video clips, text instructions, and additional relevant information.

[0710] Input: Integrated data set (full operating procedures and related information)

[0711] Data processing: Data formatting and manual generation (using document generation tools)

[0712] Output: Generated manual (PDF, HTML, etc.)

[0713] Specific behavior: Uses a documentation generation tool (e.g., LaTeX, Markdown).

[0714] Step 7:

[0715] Providing manuals

[0716] The server then sends the generated manual to the user's information processing device or wearable device, where it can be displayed. The user can view the operating procedures via smart glasses or a head-mounted display.

[0717] Input: Generated manual (PDF, HTML, etc.)

[0718] Data processing: Sending data (over a network connection)

[0719] Output: Manual displayed on terminal and wearable devices

[0720] Specific operations: Send data to the device, use a display application (e.g., a dedicated viewer).

[0721] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0722] This invention combines a system that records user operations on an information processing device in real time, analyzes and integrates the data, and generates a manual, with an emotion engine. Specific embodiments of this system will be described below.

[0723] Recording user actions

[0724] First, when a user starts a specific task on an information processing device, the device records the user's actions in real time. The recordings are made in the following four forms:

[0725] 1. Video recording: The device captures the user's screen and saves it as video data, allowing you to visually confirm the operations that the user actually performed.

[0726] 2. Audio recording: The device will record the user's voice and environmental sounds through the microphone input, so that what the user says and the background sounds while operating the device will be recorded.

[0727] 3. Text recording: The terminal stores keyboard and mouse inputs as text data, thereby recording the commands and text entered by the user.

[0728] 4. Emotion recording: The device uses a camera and microphone to capture the user's facial expressions and voice tone in real time, and the emotion engine analyzes this data to recognize the user's emotions.

[0729] Sending data

[0730] Once recording begins, the device periodically generates and transmits data packets containing video, audio, text, and emotion data to the server. Each packet is accompanied by metadata such as a timestamp and user ID.

[0731] Data analysis

[0732] The server analyzes the received data packet and performs the following actions:

[0733] 1. Video and audio analysis using generative AI: The server uses generative AI to extract important points from video and audio data, such as clicking on important menu items or operating settings screens.

[0734] 2. Natural language processing: The server analyzes the text data using a natural language processing engine to identify operational procedures and related background information, thereby clarifying the user's intent and purpose.

[0735] 3. Emotion analysis: The server analyzes the emotion data provided by the emotion engine to determine the emotional state of the user at each point of the operation, for example, whether they felt stressed or happy during the operation.

[0736] Context Integration

[0737] Based on the analysis results, the server collects and integrates additional data related to the operation procedure, including other text, image, audio, and emotional data, such as help documents related to the operation and screenshots of the settings screen.

[0738] Generate manual

[0739] The server uses the integrated data to create a manual that explains the operation procedures in an easy-to-understand format. The manual contains the following elements:

[0740] 1. Video Clips: Short video clips are inserted to show user actions.

[0741] 2. Text Instructions: Contains detailed text instructions related to the operating procedure.

[0742] 3. Additional Related Information: Provides additional background information and notes related to the operation.

[0743] 4. Emotion analysis results: Data showing the user's emotional state is inserted as diagrams and text, allowing users to visually understand which parts of the operation procedure the user found difficult or went smoothly.

[0744] Providing manuals

[0745] Finally, the generated manual is sent to the terminal and made accessible to the user. By viewing this manual, the user can check the operation procedures, background information, and their own emotional state while working.

[0746] Specific examples

[0747] For example, if a user records the steps to install new software, the device records the entire installation process as video, audio, text, and emotional data. The server analyzes this data to extract important installation steps and the user's emotional state during the process. The server then integrates additional help information and system requirements related to the installation process and generates a manual that includes these. The manual visually indicates the points where the user felt stressed and the points where the installation went smoothly. Finally, the manual is sent to the user's device, where the user can review the installation steps and refer to the results of their own emotional analysis.

[0748] In this way, by effectively recording, analyzing, and integrating the user's emotional state in addition to their operations and their background information, a detailed and easy-to-understand manual is provided, allowing the user to understand the intentions and concepts behind the operations and work efficiently.

[0749] The processing flow will be explained below.

[0750] Step 1:

[0751] The user starts a specific task on the information processing device and clicks a record button, thereby starting the recording process.

[0752] Step 2:

[0753] The device detects the user's click and starts real-time recording in the following format:

[0754] Video recording: Using screen capture software, the entire user's PC screen is saved as video data.

[0755] Audio recording: Saves the user's voice and environmental sounds as audio data through microphone input.

[0756] Text recording: Using a keylogger or mouse event capture function, keyboard input and mouse operations are saved as text data.

[0757] Emotion Recording: Using a camera and microphone, the system captures the user's facial expressions and voice tone in real time, and the emotion engine analyzes this data to recognize the user's emotions.

[0758] Step 3:

[0759] The terminal divides the recorded data into packets of a certain size and adds metadata such as a timestamp and a user ID to each packet.

[0760] Step 4:

[0761] The terminal sends data packets sequentially to the server, using the SSL / TLS protocol to ensure data security.

[0762] Step 5:

[0763] The server takes the received data packets sequentially and reassembles them into one continuous data set, allowing the entire user interaction to be analyzed as a continuous data stream.

[0764] Step 6:

[0765] The server uses generative AI to analyze the received video and audio data. The AI ​​model does the following:

[0766] It detects and tags important actions from video data, for example identifying the moment a specific button is clicked or a settings menu is opened.

[0767] The voice data is converted into text by a speech recognition engine, and specific instructions or explanations are extracted.

[0768] Step 7:

[0769] The server analyzes the text data with a natural language processing (NLP) engine to identify operational steps and related context, such as when a particular command is entered and what its meaning is.

[0770] Step 8:

[0771] The server analyzes the emotional data provided by the emotion engine and identifies the emotional state the user was in at each point of the operation, for example, whether they were feeling stressed or happy during the operation.

[0772] Step 9:

[0773] Based on the analysis, the server collects additional data related to the operation procedure from internal databases and external APIs, including:

[0774] The URL and content of the website accessed during the user's operation.

[0775] Links and contents of manuals and help documents related to the operation.

[0776] System requirements and configuration information required for operation.

[0777] Step 10:

[0778] The server consolidates the collected data and assembles it into a document structure that provides operational instructions, highlighting background information and important points related to each step.

[0779] Step 11:

[0780] The server applies the generated content to the presentation layer, producing the final manual in multiple formats (PDF, HTML, video files).

[0781] Step 12:

[0782] The server sends the generated manual to the terminal and uploads it to a repository accessible to the user. The user receives a notification and can view the manual on their PC.

[0783] Step 13:

[0784] Users can refer to the generated manual to check the necessary operating procedures, their background information, and their own emotional state, which will deepen their understanding of the operation and improve work efficiency.

[0785] Example 2

[0786] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0787] Conventional information processing devices were able to record user operations and generate manuals based on those operations. However, they did not consider grasping the user's emotional state and incorporating that into the manual. As a result, it was not possible to provide a detailed manual that reflected the stress and difficulties the user experienced during operation. The present invention aims to visualize the difficulties the user experiences by recording and analyzing the user's emotional state in addition to operation data and generating an integrated manual, thereby supporting easier and smoother operation.

[0788] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0789] In this invention, the server includes means for recording in real time operations performed by a user on an information processing device, means for classifying the recorded operation data into video, audio, text, and emotional data, means for analyzing the classified operation data using a generative artificial intelligence to extract important points and the user's emotional state, means for integrating the extracted important points, emotional state, and additional related data to generate a manual, and means for providing the generated manual on the information processing device. This makes it possible to comprehensively grasp the user's operations and the emotions felt at the time, and to provide a more accurate manual.

[0790] The term "user" refers to a person who operates an information processing device.

[0791] "Information processing device" refers to computer equipment or digital devices that users can operate.

[0792] "Real time" refers to processing that is carried out continuously in real time.

[0793] "Recording" refers to saving information such as the user's actions, comments, and facial expressions.

[0794] "Video" refers to data that expresses movement by playing back successive image frames over time.

[0795] "Audio" refers to data that contains recorded human voices and surrounding environmental sounds.

[0796] "Text" refers to data that stores keyboard input and mouse operations as character information.

[0797] "Emotion data" refers to data that indicates the emotional state of a user extracted from facial expressions and vocal tones.

[0798] "Classification" refers to dividing the recorded operation data into video, audio, text, and emotion data.

[0799] "Generative AI" refers to AI technology that analyzes recorded data and recognizes and generates specific information and patterns.

[0800] "Analysis" refers to the detailed examination of recorded data to uncover meanings and patterns.

[0801] "Important points" refer to actions or content that are particularly noteworthy in the operation data.

[0802] "User's emotional state" refers to the type and strength of emotions felt by the user during operation.

[0803] "Synthesis" refers to bringing together extracted key points, emotional states, and related data into a single framework.

[0804] A "manual" refers to a guide document that summarizes user operating procedures, background information, and emotion analysis results.

[0805] "Providing" refers to presenting information such as generated manuals to users in an easy-to-use format.

[0806] This invention relates to a system that records the operations performed by a user on an information processing device in real time, analyzes the recorded operation data, and generates a manual. In particular, it aims to record and analyze the user's emotional state and provide a detailed manual that integrates this data.

[0807] Recording user actions

[0808] First, the user starts a specific task on the information processing device. For example, when the user starts installing new software, the device records the operation in real time in the following format:

[0809] 1. Video recording: The device captures the user's screen and saves it as video data in MP4 format. This video recording allows you to visually confirm the operations that the user actually performed.

[0810] 2. Audio recording: The device will record the user's voice and environmental sounds in WAV format using the built-in microphone or an external microphone, so that what the user says during operation and the surrounding background sounds will be recorded.

[0811] 3. Text logging: The terminal records keyboard and mouse inputs in text format in a log file, which records the commands and text entered by the user.

[0812] 4. Emotion Recording: The device uses a camera and microphone to capture the user's facial expressions and voice tone in real time, and stores them as emotion data through the emotion engine. This emotion data is stored in JSON format.

[0813] Sending data

[0814] Once recording begins, the device will collect data at regular intervals (e.g., every minute) and compile it into a single data packet, which includes a video file, an audio file, a text log, and emotion data.

[0815] The device then sends the generated data packets to the server via an internet connection, using encryption technology such as SSL / TLS.

[0816] Data analysis

[0817] The server receives data packets sent from the device, generates an analysis request for the received data, and sends it to an internal processing engine service. The server then uses a generative artificial intelligence model (e.g., OpenAI's GPT-4) to perform the next analysis.

[0818] Video and audio analysis: Extracts important operation points from video data frame by frame, and picks out instructions and important statements from audio data.

[0819] Natural language processing: Analyzes commands and operation details from text logs and identifies operation procedures based on the context.

[0820] Emotion analysis: Use the output of the emotion engine to identify the emotional state the user was in during each action.

[0821] Context Integration

[0822] Based on the parsed data, the server collects additional relevant information from databases and external resources, such as help documents and FAQ pages for operations.

[0823] The server combines this additional information with the analysis results to create a single integrated dataset, which includes operating instructions, related help documents, sentiment analysis results, etc.

[0824] Generate manual

[0825] The server uses the integrated data to create a manual in a format that is easy for users to understand. This includes the following elements:

[0826] Video Clips: Short video clips are inserted to show user actions.

[0827] Text Instructions: Contains detailed text instructions related to the procedure.

[0828] Additional Related Information: Provides additional background information or notes related to the operation.

[0829] Sentiment analysis results: Data showing the user's emotional state is inserted as diagrams and text, allowing users to visually understand which parts of the operation procedure the user found difficult or went smoothly.

[0830] Providing manuals

[0831] Finally, the generated manual is sent to the terminal and made accessible to the user. By viewing this manual, the user can check the operation procedures, background information, and their own emotional state while working.

[0832] Examples of specific examples and prompts

[0833] For example, if a user records the steps to install new software, they can input the following prompts to the generative AI model:

[0834] "Analyze data recording the steps to install new software and generate a manual that combines operating procedures and emotional states."

[0835] This allows a specific and detailed manual to be automatically generated, supporting the user's operations.

[0836] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0837] Processing Steps

[0838] Step 1:

[0839] A user starts a specific task on an information processing device. The input is a trigger operated by the user to start the task. The output is the start of real-time recording of the user's operations. At this stage, the system starts monitoring the operations.

[0840] Step 2:

[0841] The terminal records operation data in real time. The specific operations are as follows:

[0842] Video recording: The device launches screen capture software and saves the user's screen operations in MP4 format. The input is the user's screen information, and the output is video data.

[0843] Audio recording: The device uses a microphone to record ambient sounds in WAV format. The input is microphone input and the output is audio data.

[0844] Text logging: The terminal records keyboard and mouse activity in a text log. The input is the user's keyboard and mouse actions, and the output is text data.

[0845] Emotion recording: The device uses a camera and microphone to capture the user's facial expressions and voice tone in real time, which are then analyzed by the emotion engine. The input is data from the camera and microphone, and the output is emotion data.

[0846] Step 3:

[0847] The terminal aggregates the recorded data into data packets at regular intervals. The input is video, audio, text, and emotion data collected in real time. The output is the aggregated data packets.

[0848] Step 4:

[0849] The terminal sends the integrated data packet to the server using encryption technology such as SSL / TLS. The input is the data packet, and the output is the data transmission to the server.

[0850] Step 5:

[0851] The server receives data packets: the input is the data packet sent by the terminal, and the output is the acknowledgement.

[0852] Step 6:

[0853] The server analyzes the received data packets to generate requests and sends them to an internal processing engine service. The input is the data packets and the output is the analysis requests.

[0854] Step 7:

[0855] The server analyzes the data using a generative artificial intelligence model, specifically by:

[0856] Video and audio analysis: Extracts important operation points from input data frame by frame and picks out important statements. The output is analyzed operation data.

[0857] Natural language processing: Extracting important commands and operations from text data. The input is text data, and the output is parsed text data.

[0858] Sentiment Analysis: The output of the emotion engine is used to extract the user's emotional state during operation. The input is emotion data, and the output is the analyzed emotion data.

[0859] Step 8:

[0860] The server collects additional relevant information based on the analysis results. The input is the analyzed data, and the output is the additional relevant information.

[0861] Step 9:

[0862] The server integrates the collected additional information and analysis results into a single dataset. The input is the analysis results and additional information, and the output is the integrated dataset.

[0863] Step 10:

[0864] The server uses the integrated dataset to generate a manual, including video clips, text descriptions, additional related information, and sentiment analysis results. The input is the integrated dataset, and the output is the generated manual.

[0865] Step 11:

[0866] The server sends the generated manual to the terminal. The input is the generated manual, and the output is the delivery of the manual to the terminal.

[0867] Step 12:

[0868] The user reads the received manual on the terminal and checks the operation procedures and emotion analysis results. The input is the received manual, and the output is operation support for the user.

[0869] This allows for a comprehensive understanding of the user's operations and the emotions felt at the time, making it possible to provide more accurate manuals.

[0870] (Application example 2)

[0871] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0872] On conventional online shopping sites, users often feel anxious and confused when operating the site for the first time. In particular, new users are often not provided with effective operation guides, which increases the likelihood of operational errors and a lack of understanding of operation procedures. Furthermore, manuals provided without considering the user's emotions make it difficult to alleviate the stress and difficulties experienced during operation. To solve these problems, a system is needed that records and analyzes user operations, and then generates and provides custom operation guides that incorporate the user's emotional state.

[0873] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0874] In this invention, the server includes means for recording in real time the operations performed by a user on an information processing device, means for analyzing the recorded operation data using a generative artificial intelligence to extract important points, means for analyzing the user's emotional state using an emotion engine, means for generating a manual by integrating the extracted important points with additional related data and emotional data, and means for providing the generated manual on the information processing device. This makes it possible to generate a detailed customized manual for the user's operating procedures and provide support that takes into account the user's emotional state.

[0875] A "user" is a subject who operates an information processing device.

[0876] An "information processing device" is a device that processes digital data, such as a computer, smartphone, or tablet.

[0877] "Means for recording in real time" refers to a function or device for instantly recording or recording user operations on the spot.

[0878] "Operation data" is data that records the content of operations performed by a user, and includes video, audio, text, and the like.

[0879] "Generative artificial intelligence" is an AI technology that uses machine learning and deep learning, and is a method of extracting features from data and analyzing them.

[0880] "Important points" are parts of user operations that require special attention, or important events or operations in the system.

[0881] An "emotion engine" is software or a system for analyzing a user's emotional state from their tone of voice and facial expressions.

[0882] "Related data" refers to the operating procedures as well as helpful help documents and additional information related to the operation.

[0883] The "means for generating a manual" refers to a function or software that integrates recorded operation data and related information to create a manual in a format that is easy for the user to understand.

[0884] The "means for providing on an information processing device" refers to a function or method for displaying the generated manual on the information processing device operated by the user, making it viewable.

[0885] This invention is a system that records the operations performed by users on information processing devices in real time, analyzes and integrates the operation data and emotional data, and generates and provides a customized operation manual.

[0886] 1. System Configuration

[0887] Hardware

[0888] Information processing device: A terminal operated by a user, including computers, smartphones, tablets, etc.

[0889] Camera and microphone: Input devices for recording user actions, facial expressions, and voice tones in real time.

[0890] Server: Back-end infrastructure for analyzing operation data and emotion data and generating manuals.

[0891] software

[0892] OpenCV: A library for capturing and processing video and image data.

[0893] Google Cloud Speech-to-Text API: A service for converting voice data into text data.

[0894] TensorFlow: A machine learning library for analyzing emotions from facial expressions and voice.

[0895] Django Framework: A server-side framework for generating and serving operation manuals.

[0896] JavaScript: A front-end technology for displaying manuals on information processing devices.

[0897] 2. System Functions

[0898] Recording user actions

[0899] When a user starts a specific task on the information processing device, the device records the user's actions in real time. The recordings are in the following format:

[0900] Video recording: Use the camera to capture the user's operation screen and save it as video data.

[0901] Audio recording: Records the user's voice and environmental sounds through microphone input and saves them as audio data.

[0902] Emotion recording: Uses a camera and microphone to capture the user's facial expressions and vocal tone, which are then analyzed by the emotion engine.

[0903] Data transmission and analysis

[0904] Once recording begins, the device will periodically generate data packets and send them to the server. The data packets contain video, audio, text, and emotion data. The server will analyze the data and extract key points. The complete analysis process includes the following steps:

[0905] Generative AI video and audio analysis

[0906] Natural Language Processing (NLP)

[0907] Emotion analysis

[0908] Manual generation and provision

[0909] Based on the analysis results, the server integrates additional data related to user operations and generates an intuitive and easy-to-understand operation manual. The generated manual includes the following elements:

[0910] Video Clip: A short video clip showing the procedure

[0911] Text Description: Detailed instructions related to the procedure

[0912] Sentiment analysis results: Data that visually indicates the user's emotional state

[0913] The generated manual is finally sent to the terminal, and the user can refer to this manual as they work.

[0914] 3. Specific Examples

[0915] For example, if a user records the steps of purchasing a product on an online shopping site, the device records the entire purchasing process as video, audio, and emotional data. The server analyzes this data to extract important steps and the user's emotional state during the process. It then integrates additional help information and product reviews related to the purchasing process and generates a custom operation guide that includes them.

[0916] Example prompts for generative AI models

[0917] "On your online store, walk users through the steps of adding an item to their cart and visually demonstrate points of frustration."

[0918] As described above, this system can record and analyze the user's operations and their emotional state in detail, and generate and provide optimal operation guidance.

[0919] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0920] Step 1:

[0921] When a user starts operating an information processing device, the device records the user's operations in real time. At this point, the device's camera and microphone are activated, capturing the user's screen operation as a video and recording the user's voice input. The input data is the user's screen capture video and voice data, which are saved as recorded data.

[0922] Step 2:

[0923] The device uses an emotion engine to analyze the user's facial expressions and voice tone in real time to generate emotion data. Specifically, an emotion recognition model using TensorFlow predicts the user's emotional state (e.g., stress, joy, neutral, etc.) from their facial expressions and voice. This emotion data is stored along with the video and audio data. The input data is the user's facial expressions and voice, and the output data is the analyzed emotional state.

[0924] Step 3:

[0925] After recording for a certain period of time or for each specific operation, the device sends these data packets to the server. The data packets include video, audio, text, and emotion data. The input data is the recorded data packets, and the output data is the data sent to the server. Specifically, the device secures a stable network connection and uploads the data to the server using protocols such as HTTP or WebSocket.

[0926] Step 4:

[0927] The server analyzes the received data packets and begins processing various data. First, it uses generative AI to extract key operation points from the video and audio data. For example, it identifies the icons the user clicked and the text fields they entered. The input data is the video and audio data, and the output data is the analyzed key points. Specifically, it runs a generative AI model (for example, YOLO or DeepSpeech) to detect key operations.

[0928] Step 5:

[0929] The server then analyzes the text data using natural language processing (NLP). This clarifies the intent of the user's input text or command and identifies operational procedures and related background information. The input data is the text data, and the output data is the analyzed operational procedures and background information. Specifically, the server performs text analysis using an NLP engine (e.g., spaCy or BERT).

[0930] Step 6:

[0931] The server analyzes the emotion data provided by the emotion engine and identifies the user's emotional state at each operation point. The input data is emotion data, and the output data is the emotional state corresponding to each operation point. Specifically, the server maps the emotional state at each operation step from the time-series data.

[0932] Step 7:

[0933] Based on the analysis results, the server collects and integrates additional data related to the operation procedures, such as help documents and screenshots related to the operations. The input data is the analyzed operation procedures and emotion data, and the output data is a draft of the integrated operation manual. Specific operations include the process of retrieving related information from databases and external APIs.

[0934] Step 8:

[0935] The server generates a detailed operating manual based on the integrated data. The generated manual includes video clips, text descriptions, additional related information, and sentiment analysis results. The input data is a draft of the integrated operating manual, and the output data is the final generated operating manual. Specifically, the manual document is automatically generated using a template engine.

[0936] Step 9:

[0937] Finally, the server sends the generated operation manual to the terminal, making it accessible to the user. The input data is the finally generated operation manual, and the output data is the data sent to the terminal. Specific operations include providing information to the user through HTTP responses and updating the database.

[0938] As described above, the input data is analyzed and processed at each processing step, and ultimately a detailed custom operation manual is generated and provided to assist the user in their operations.

[0939] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0940] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0941] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0942] [Third embodiment]

[0943] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0944] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0945] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0946] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0947] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0948] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0949] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0950] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0951] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0952] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0953] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0954] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0955] This invention relates to a system that records operations performed by a user on an information processing device in real time, analyzes and integrates the data, and generates a manual. Specific embodiments of this system will be described below.

[0956] Recording user actions

[0957] First, when a user starts a specific task on an information processing device, the device records the user's actions in real time. The recording is done in three forms:

[0958] 1. Video recording: The device captures the user's screen and saves it as video data, allowing you to visually confirm the operations that the user actually performed.

[0959] 2. Audio recording: The device will record the user's voice and environmental sounds through the microphone input, so that what the user says and the background sounds while operating the device will be recorded.

[0960] 3. Text recording: The terminal stores keyboard and mouse inputs as text data, thereby recording the commands and text entered by the user.

[0961] Sending data

[0962] Once recording begins, the device periodically generates and transmits data packets containing video, audio, and text data to the server, each accompanied by metadata such as a timestamp and user ID.

[0963] Data analysis

[0964] The server analyzes the received data packet and performs the following actions:

[0965] 1. Video and audio analysis using generative AI: The server uses generative AI to extract important points from video and audio data, such as clicking on important menu items or operating settings screens.

[0966] 2. Natural language processing: The server analyzes the text data using a natural language processing engine to identify operational procedures and related background information, thereby clarifying the user's intent and purpose.

[0967] Context Integration

[0968] Based on the analysis results, the server collects and integrates additional data related to the operation procedure, including other text, images, and audio, such as help documents related to the operation and screenshots of the settings screen.

[0969] Generate manual

[0970] The server uses the integrated data to create a manual that explains the operation procedures in an easy-to-understand format. The manual contains the following elements:

[0971] 1. Video Clips: Short video clips are inserted to show user actions.

[0972] 2. Text Instructions: Contains detailed text instructions related to the operating procedure.

[0973] 3. Additional Related Information: Provides additional background information and notes related to the operation.

[0974] Providing manuals

[0975] Finally, the generated manual is sent to the terminal and made accessible to the user. By viewing this manual, the user can gain a deeper understanding of the operating procedures and their background, allowing them to work more efficiently.

[0976] Specific examples

[0977] For example, if a user records the process of installing new software, the device will record the entire installation process as video, audio, and text. The server will analyze this data and extract important installation steps. It will then integrate additional help information and system requirements related to the installation process and generate a manual that includes them. Finally, this manual will be sent to the user's device, where the user can follow the installation process while checking the instructions.

[0978] In this way, by effectively recording, analyzing, and integrating user operations and their background information, a detailed and easy-to-understand manual is provided, allowing users to understand the intentions and concepts behind the operations and work efficiently.

[0979] The processing flow will be explained below.

[0980] Step 1:

[0981] The user starts a specific task on the information processing device and clicks the record button, thereby starting the recording process.

[0982] Step 2:

[0983] The device detects the user's clicks and begins recording the user's actions in real time, including by:

[0984] Video recording: Using screen capture software, the entire user's PC screen is saved as video data.

[0985] Audio recording: Environmental sounds and the user's voice are saved as audio data via microphone input.

[0986] Text recording: Using a keylogger or mouse event capture function, keyboard input and mouse operations are saved as text data.

[0987] Step 3:

[0988] The terminal divides the recorded data into packets of a certain size and adds metadata such as a timestamp and a user ID to each packet.

[0989] Step 4:

[0990] The terminal sends data packets sequentially to the server, using the SSL / TLS protocol to ensure data security.

[0991] Step 5:

[0992] The server takes the received data packets sequentially and reassembles them into one continuous data set, allowing the entire user interaction to be analyzed as a continuous data stream.

[0993] Step 6:

[0994] The server uses generative AI to analyze the received video and audio data. The AI ​​model does the following:

[0995] It detects and tags important actions from video data, for example identifying the moment a specific button is clicked or a settings menu is opened.

[0996] The voice data is converted into text by a speech recognition engine, and specific instructions or explanations are extracted.

[0997] Step 7:

[0998] The server analyzes the text data with a natural language processing (NLP) engine to identify operational steps and related context, such as when a particular command is entered and what its meaning is.

[0999] Step 8:

[1000] Based on the analysis, the server collects additional data related to the operation procedure from internal databases and external APIs, including:

[1001] The URL and content of the website accessed during the user's operation.

[1002] Links and contents of manuals and help documents related to the operation.

[1003] System requirements and configuration information required for operation.

[1004] Step 9:

[1005] The server consolidates the collected data and assembles it into a document structure that provides operational instructions, highlighting background information and important points related to each step.

[1006] Step 10:

[1007] The server applies the generated content to the presentation layer, producing the final manual in multiple formats (PDF, HTML, video files).

[1008] Step 11:

[1009] The server sends the generated manual to the terminal and uploads it to a repository accessible to the user. The user receives a notification and can view the manual on their PC.

[1010] Step 12:

[1011] Users can refer to the generated manual to check the necessary operating procedures and related background information, which will deepen their understanding of the operations and improve work efficiency.

[1012] Example 1

[1013] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1014] In the past, recording the operations performed by users on information processing devices and creating manuals to communicate those operating procedures to other users was a time-consuming and labor-intensive process. Furthermore, manual recording and editing is prone to human error and omissions, making it difficult to create efficient procedure manuals. For this reason, there was a demand for a system that could accurately and efficiently record user operations and automatically generate manuals based on that information.

[1015] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1016] In this invention, the server includes means for recording in real time operations performed by a user on an information processing device, means for saving the recorded operation data in video, audio, and text formats, means for analyzing the recorded operation data using a generative AI model and extracting important points, means for generating a manual by integrating the extracted important points with additional related information, and means for providing the generated manual on the information processing device. This makes it possible to automatically and accurately record user operations and quickly generate a manual based on the operation procedures.

[1017] The term "user" refers to a person who performs an operation on an information processing device.

[1018] "Information processing device" refers to a computer or its peripheral devices for inputting, processing, storing, and outputting data.

[1019] "Real-time recording" refers to recording user operations immediately.

[1020] "Operation data" refers to data indicating the history and content of operations performed by a user on an information processing device.

[1021] "Video format" refers to video data captured from the user's screen.

[1022] "Audio format" refers to audio data that records the user's voice or environmental sounds.

[1023] "Text format" refers to character data that records keyboard input and mouse operations.

[1024] A "generative AI model" is a model that uses artificial intelligence to analyze data and extract important information and patterns.

[1025] "Analysis" refers to the act of analyzing recorded data to extract meaning and patterns.

[1026] The "important points" refer to particularly noteworthy or critical parts of the operations performed by the user.

[1027] "Related information" refers to information such as additional text, images, and audio that is related to the user's operating procedure.

[1028] "Integration" refers to the act of bringing together data or information in different formats.

[1029] A "manual" refers to a document or multimedia content that clearly shows and explains user operating procedures.

[1030] "Providing" refers to the act of displaying or saving the generated manual on an information processing device in a form that can be used by the user.

[1031] This invention relates to a system that records operations performed by a user on an information processing device in real time, analyzes and integrates the data, and generates a manual. Specific embodiments of this system will be described below.

[1032] Recording user actions

[1033] First, when a user starts a specific task on an information processing device, the device records the user's actions in real time. The recording is done in three forms:

[1034] 1. Video recording: The device captures the user's screen and saves it as video data using video capture software such as OBS Studio, allowing you to visually confirm the operations that the user actually performed.

[1035] 2. Audio recording: The device records the user's voice and environmental sounds through microphone input. For example, the device uses a built-in microphone or an external microphone to collect audio data, recording what the user says and the background sounds while operating the device.

[1036] 3. Text recording: The terminal stores keyboard and mouse inputs as text data. Logging software is used to record the commands and text entered by the user.

[1037] Sending data

[1038] Once recording begins, the device periodically generates data packets and sends them to the server. The data packets contain video, audio, and text data, and each packet is assigned metadata such as a timestamp and user ID. For example, a packet could be generated every minute and sent to the server via the network.

[1039] Data analysis

[1040] The server analyzes the received data packet and performs the following actions:

[1041] 1. Video and audio analysis using generative AI models: The server uses generative AI models (such as OpenAI's GPT model) to extract important points from video and audio data, such as clicking on important menu items or operating settings screens.

[1042] 2. Natural Language Processing: The server analyzes the text data using a natural language processing engine (e.g., spaCy) to identify operational steps and related background information, thereby clarifying the user's intent and purpose.

[1043] Context Integration

[1044] Based on the analysis results, the server collects and integrates additional data related to the operation procedure, including other text, images, and audio, such as help documents related to the operation and screenshots of the settings screen.

[1045] Generate manual

[1046] The server uses the integrated data to create a manual that explains the operation procedures in an easy-to-understand format. The manual contains the following elements:

[1047] 1. Video Clips: Short video clips are inserted to show user actions.

[1048] 2. Text Instructions: Contains detailed text instructions related to the operating procedure.

[1049] 3. Additional Related Information: Provides additional background information and notes related to the operation.

[1050] Providing manuals

[1051] Finally, the generated manual is sent to the terminal and made accessible to the user. By viewing this manual, the user can gain a deeper understanding of the operating procedures and their background, allowing them to work more efficiently.

[1052] Specific examples

[1053] For example, if a user records the process of installing new software, the device will record the entire installation process as video, audio, and text. The server will analyze this data and extract important installation steps. It will then integrate additional help information and system requirements related to the installation process and generate a manual that includes them. Finally, this manual will be sent to the user's device, allowing the user to easily follow the installation process.

[1054] Prompt Sentence Examples

[1055] "This system records the steps a user takes to install new software, then analyzes and integrates the data to generate a detailed operating manual. Please provide a concrete example of how the generative AI model employed in this process extracts key installation steps and integrates relevant information."

[1056] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1057] Step 1:

[1058] When a user starts a specific task on the information processing device, the device records the user's actions in real time. For example, when starting to install new software, the device launches OBS Studio and starts capturing the screen. It also records the user's voice via microphone and logs keyboard and mouse input data.

[1059] Input: User operation begins

[1060] Output: Video files, audio files, text logs

[1061] Step 2:

[1062] The device packets the collected data at regular intervals and sends it to the server. For example, video, audio, and text data are packetized every minute, and metadata such as timestamps and user IDs are added. A library for packet generation is used to send the data over the network to the server.

[1063] Input: Video files, audio files, text logs

[1064] Output: Data packets

[1065] Step 3:

[1066] The server analyzes the received data packets and extracts key points using generative AI models, such as OpenAI's GPT model, to detect click events from video, commands from audio, and settings from text.

[1067] Input: Data packet

[1068] Output: List of key points

[1069] Step 4:

[1070] The server then collects and integrates additional relevant information based on the extracted key points, using a natural language processing engine (e.g., spaCy) to analyze the text data and collect relevant help documentation and screenshots of configuration screens.

[1071] Input: List of key points

[1072] Output: Consolidated related information

[1073] Step 5:

[1074] The server then uses the integrated data to create easy-to-understand operation procedures in a manual, generating documents including video clips, detailed text explanations, and background information.

[1075] Input: Integrated related information

[1076] Output: Manual document

[1077] Step 6:

[1078] The server sends the generated manual to the terminal, where the user can view the manual and check the operating procedures as they work.

[1079] Input: Manual document

[1080] Output: A user-readable manual

[1081] (Application example 1)

[1082] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1083] Conventional methods for creating manuals for factory robot operation procedures required a great deal of time and effort for manual recording and analysis, making them inefficient. Furthermore, operation procedures were rarely provided in a format that was easy for workers to understand, leading to operational errors and reduced efficiency. Furthermore, it was difficult to check operation procedures on different devices, creating a need for an integrated solution to improve on-site work efficiency.

[1084] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1085] In this invention, the server includes means for recording operations performed by a user on an information processing device in real time, means for analyzing the recorded operation data using artificial intelligence and extracting important points, means for generating a manual by integrating the extracted important points with additional related data, means for providing the generated manual on the information processing device, and means for recording and viewing operation procedures for robot operations in factories via wearable devices such as smart glasses and head-mounted displays. This allows for real-time recording and analysis of operation procedures, enabling workers to instantly check easy-to-understand manuals via wearable devices, thereby enabling efficient and accurate operation.

[1086] A "user" is a person or entity that performs an operation on an information processing device.

[1087] An "information processing device" is a device that inputs, processes, and outputs data, and includes personal computers, tablets, smartphones, etc.

[1088] "Real-time" refers to operations and processing that are recorded and analyzed at the same time as they are performed.

[1089] "Recording" refers to the process of saving user operations as data.

[1090] "Analysis" refers to the process of evaluating recorded data and extracting important information.

[1091] "Generative artificial intelligence" refers to algorithms and software that have the ability to extract meaning and patterns from input data.

[1092] "Important points" refer to parts or steps that require special attention in the user's operations.

[1093] "Extraction" refers to the process of extracting specific information from the overall data.

[1094] "Integration" refers to the process of combining multiple pieces of data or information into one whole.

[1095] A "manual" refers to a document or material that describes operating procedures and guidelines.

[1096] "Providing" refers to the process of displaying the generated manual in a form that is accessible to users.

[1097] "Robot operation in a factory" refers to operating a robot in a factory to perform specific tasks.

[1098] "Smart glasses" refers to a wearable device in the shape of glasses that has the function of displaying information.

[1099] A "head-mounted display" refers to a device worn on the head that displays images and information.

[1100] A "wearable device" refers to an information processing device that can be worn.

[1101] This invention is a system that records and analyzes operations performed by a user on an information processing device in real time to efficiently operate robots in a factory, and provides the results as a manual.

[1102] System Overview

[1103] The system consists of the following elements:

[1104] 1. Information processing devices used by users (PCs, tablets, smartphones, etc.)

[1105] 2. Recording equipment (camera and microphone)

[1106] 3. Generative Artificial Intelligence (AI Model)

[1107] 4. Server (performs data analysis and manual generation)

[1108] 5. Wearable devices (smart glasses, head-mounted displays)

[1109] How it works

[1110] Recording user actions

[1111] When the user starts operating the robot, the camera records the operation video in real time. At the same time, the microphone records the audio, and keyboard input and mouse operations are recorded as text data. This recorded data is periodically sent to the server as data packets.

[1112] Data transmission and analysis

[1113] The server analyzes the received data packets and uses generative artificial intelligence to extract key points from the video and audio data. Specifically, it detects important operation steps, commands, setting changes, etc. At the same time, it also analyzes the text data using a natural language processing engine to clarify operation procedures and intentions.

[1114] Context integration and manual generation

[1115] Based on the analysis results, the server collects and integrates additional related data (help documents, screenshots, configuration details, etc.). Based on the integrated data, an easy-to-understand operation manual is created. The manual includes video clips, text explanations, and additional related information.

[1116] Providing manuals

[1117] The generated manual is sent to the user's information processing device and wearable device, where the user can check the operation procedures in real time via smart glasses or a head-mounted display.

[1118] Hardware and software used

[1119] The system uses the following hardware and software:

[1120] Recording device: Camera (e.g., Logitech C920) and microphone (e.g., Blue Yeti)

[1121] Generative Artificial Intelligence: Azure Cognitive Services Text Analytics API

[1122] Information processing devices: Windows PCs, Macs, tablets, smartphones

[1123] Wearable devices: smart glasses (e.g., Google Glass), head-mounted displays (e.g., Microsoft HoloLens)

[1124] Specific examples

[1125] For example, when a user assembles parts using a robot in a factory, the entire operation is recorded using a camera and microphone. The server analyzes this recorded data, identifies important steps (such as the correct position of some parts or how to tighten screws), and generates a detailed manual. The user can then proceed with the work while viewing the generated manual in real time through smart glasses or a head-mounted display.

[1126] Prompt Sentence Examples

[1127] "Record the steps the robot takes to assemble the parts, extract the important steps, and generate a manual based on them."

[1128] In this way, operating procedures can be recorded and analyzed in real time, and workers can instantly check easy-to-understand manuals via wearable devices, enabling efficient and accurate operations.

[1129] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1130] Step 1:

[1131] User operation initiation and recording

[1132] The user begins operating the factory robot. The device uses a camera and microphone to record video and audio of the operation in real time. At the same time, keyboard and mouse operations are also recorded as text data.

[1133] Input: User operation video, voice, keyboard input, mouse operation

[1134] Data processing: Capture video, record audio, record text input

[1135] Output: Data packets collected in real time

[1136] Specific behavior: Capture video with a camera (e.g. Logitech C920), record audio with a microphone (e.g. Blue Yeti), and save the text to a local file.

[1137] Step 2:

[1138] Creating and sending data packets

[1139] The terminal generates the collected data as data packets at regular intervals and transmits them to the server.

[1140] Input: Data collected in real time (video, audio, text)

[1141] Data processing: Packetizing data (adding timestamps and user IDs)

[1142] Output: Data packet with timestamp and user ID added

[1143] Specific operation: Data packets are generated at regular intervals and sent to the server via the network.

[1144] Step 3:

[1145] Data reception and storage by the server

[1146] The server stores the received data packets, organizing them based on user ID and timestamp.

[1147] Input: Data packets (video, audio, text)

[1148] Data processing: packets are decoded, stored, and organized by metadata

[1149] Output: Organized data (stored in a database or file system)

[1150] Specific operation: Save to a database system (e.g. MySQL), record to the file system.

[1151] Step 4:

[1152] Data analysis

[1153] The server analyzes the stored data using artificial intelligence to extract key points, while voice and text data is analyzed using a natural language processing engine.

[1154] Input: stored video, audio, and text data

[1155] Data calculation: AI model-based video analysis (e.g., object detection), audio analysis (e.g., sound recognition), and text analysis (e.g., key phrase extraction)

[1156] Output: Analysis results (important operation points, key phrases, etc.)

[1157] Specific operation: Uses Azure Cognitive Services' Text Analytics API and object detection models (e.g., YOLO).

[1158] Step 5:

[1159] Context Integration

[1160] Based on the analyzed results, the server collects and integrates additional relevant data (help documentation, screenshots, configuration details, etc.).

[1161] Input: Analysis results (important operation points, key phrases, etc.)

[1162] Data processing: Collecting and integrating additional data (e.g., web crawling, database searching)

[1163] Output: Integrated data set (full operating procedures and related information)

[1164] Specific actions: API calls, database queries.

[1165] Step 6:

[1166] Generate manual

[1167] The server uses the integrated dataset to create easy-to-understand instruction manuals, including video clips, text instructions, and additional relevant information.

[1168] Input: Integrated data set (full operating procedures and related information)

[1169] Data processing: Data formatting and manual generation (using document generation tools)

[1170] Output: Generated manual (PDF, HTML, etc.)

[1171] Specific behavior: Uses a documentation generation tool (e.g., LaTeX, Markdown).

[1172] Step 7:

[1173] Providing manuals

[1174] The server then sends the generated manual to the user's information processing device or wearable device, where it can be displayed. The user can view the operating procedures via smart glasses or a head-mounted display.

[1175] Input: Generated manual (PDF, HTML, etc.)

[1176] Data processing: Sending data (over a network connection)

[1177] Output: Manual displayed on terminal and wearable devices

[1178] Specific operations: Send data to the device, use a display application (e.g., a dedicated viewer).

[1179] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1180] This invention combines a system that records user operations on an information processing device in real time, analyzes and integrates the data, and generates a manual, with an emotion engine. Specific embodiments of this system will be described below.

[1181] Recording user actions

[1182] First, when a user starts a specific task on an information processing device, the device records the user's actions in real time. The recordings are made in the following four forms:

[1183] 1. Video recording: The device captures the user's screen and saves it as video data, allowing you to visually confirm the operations that the user actually performed.

[1184] 2. Audio recording: The device will record the user's voice and environmental sounds through the microphone input, so that what the user says and the background sounds while operating the device will be recorded.

[1185] 3. Text recording: The terminal stores keyboard and mouse inputs as text data, thereby recording the commands and text entered by the user.

[1186] 4. Emotion recording: The device uses a camera and microphone to capture the user's facial expressions and voice tone in real time, and the emotion engine analyzes this data to recognize the user's emotions.

[1187] Sending data

[1188] Once recording begins, the device periodically generates and transmits data packets containing video, audio, text, and emotion data to the server. Each packet is accompanied by metadata such as a timestamp and user ID.

[1189] Data analysis

[1190] The server analyzes the received data packet and performs the following actions:

[1191] 1. Video and audio analysis using generative AI: The server uses generative AI to extract important points from video and audio data, such as clicking on important menu items or operating settings screens.

[1192] 2. Natural language processing: The server analyzes the text data using a natural language processing engine to identify operational procedures and related background information, thereby clarifying the user's intent and purpose.

[1193] 3. Emotion analysis: The server analyzes the emotion data provided by the emotion engine to determine the emotional state of the user at each point of the operation, for example, whether they felt stressed or happy during the operation.

[1194] Context Integration

[1195] Based on the analysis results, the server collects and integrates additional data related to the operation procedure, including other text, image, audio, and emotional data, such as help documents related to the operation and screenshots of the settings screen.

[1196] Generate manual

[1197] The server uses the integrated data to create a manual that explains the operation procedures in an easy-to-understand format. The manual contains the following elements:

[1198] 1. Video Clips: Short video clips are inserted to show user actions.

[1199] 2. Text Instructions: Contains detailed text instructions related to the operating procedure.

[1200] 3. Additional Related Information: Provides additional background information and notes related to the operation.

[1201] 4. Emotion analysis results: Data showing the user's emotional state is inserted as diagrams and text, allowing users to visually understand which parts of the operation procedure the user found difficult or went smoothly.

[1202] Providing manuals

[1203] Finally, the generated manual is sent to the terminal and made accessible to the user. By viewing this manual, the user can check the operation procedures, background information, and their own emotional state while working.

[1204] Specific examples

[1205] For example, if a user records the steps to install new software, the device records the entire installation process as video, audio, text, and emotional data. The server analyzes this data to extract important installation steps and the user's emotional state during the process. The server then integrates additional help information and system requirements related to the installation process and generates a manual that includes these. The manual visually indicates the points where the user felt stressed and the points where the installation went smoothly. Finally, the manual is sent to the user's device, where the user can review the installation steps and refer to the results of their own emotional analysis.

[1206] In this way, by effectively recording, analyzing, and integrating the user's emotional state in addition to their operations and their background information, a detailed and easy-to-understand manual is provided, allowing the user to understand the intentions and concepts behind the operations and work efficiently.

[1207] The processing flow will be explained below.

[1208] Step 1:

[1209] The user starts a specific task on the information processing device and clicks a record button, thereby starting the recording process.

[1210] Step 2:

[1211] The device detects the user's click and starts real-time recording in the following format:

[1212] Video recording: Using screen capture software, the entire user's PC screen is saved as video data.

[1213] Audio recording: Saves the user's voice and environmental sounds as audio data through microphone input.

[1214] Text recording: Using a keylogger or mouse event capture function, keyboard input and mouse operations are saved as text data.

[1215] Emotion Recording: Using a camera and microphone, the system captures the user's facial expressions and voice tone in real time, and the emotion engine analyzes this data to recognize the user's emotions.

[1216] Step 3:

[1217] The terminal divides the recorded data into packets of a certain size and adds metadata such as a timestamp and a user ID to each packet.

[1218] Step 4:

[1219] The terminal sends data packets sequentially to the server, using the SSL / TLS protocol to ensure data security.

[1220] Step 5:

[1221] The server takes the received data packets sequentially and reassembles them into one continuous data set, allowing the entire user interaction to be analyzed as a continuous data stream.

[1222] Step 6:

[1223] The server uses generative AI to analyze the received video and audio data. The AI ​​model does the following:

[1224] It detects and tags important actions from video data, for example identifying the moment a specific button is clicked or a settings menu is opened.

[1225] The voice data is converted into text by a speech recognition engine, and specific instructions or explanations are extracted.

[1226] Step 7:

[1227] The server analyzes the text data with a natural language processing (NLP) engine to identify operational steps and related context, such as when a particular command is entered and what its meaning is.

[1228] Step 8:

[1229] The server analyzes the emotional data provided by the emotion engine and identifies the emotional state the user was in at each point of the operation, for example, whether they were feeling stressed or happy during the operation.

[1230] Step 9:

[1231] Based on the analysis, the server collects additional data related to the operation procedure from internal databases and external APIs, including:

[1232] The URL and content of the website accessed during the user's operation.

[1233] Links and contents of manuals and help documents related to the operation.

[1234] System requirements and configuration information required for operation.

[1235] Step 10:

[1236] The server consolidates the collected data and assembles it into a document structure that provides operational instructions, highlighting background information and important points related to each step.

[1237] Step 11:

[1238] The server applies the generated content to the presentation layer, producing the final manual in multiple formats (PDF, HTML, video files).

[1239] Step 12:

[1240] The server sends the generated manual to the terminal and uploads it to a repository accessible to the user. The user receives a notification and can view the manual on their PC.

[1241] Step 13:

[1242] Users can refer to the generated manual to check the necessary operating procedures, their background information, and their own emotional state, which will deepen their understanding of the operation and improve work efficiency.

[1243] Example 2

[1244] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1245] Conventional information processing devices were able to record user operations and generate manuals based on those operations. However, they did not consider grasping the user's emotional state and incorporating that into the manual. As a result, it was not possible to provide a detailed manual that reflected the stress and difficulties the user experienced during operation. The present invention aims to visualize the difficulties the user experiences by recording and analyzing the user's emotional state in addition to operation data and generating an integrated manual, thereby supporting easier and smoother operation.

[1246] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1247] In this invention, the server includes means for recording in real time operations performed by a user on an information processing device, means for classifying the recorded operation data into video, audio, text, and emotional data, means for analyzing the classified operation data using a generative artificial intelligence to extract important points and the user's emotional state, means for integrating the extracted important points, emotional state, and additional related data to generate a manual, and means for providing the generated manual on the information processing device. This makes it possible to comprehensively grasp the user's operations and the emotions felt at the time, and to provide a more accurate manual.

[1248] The term "user" refers to a person who operates an information processing device.

[1249] "Information processing device" refers to computer equipment or digital devices that users can operate.

[1250] "Real time" refers to processing that is carried out continuously in real time.

[1251] "Recording" refers to saving information such as the user's actions, comments, and facial expressions.

[1252] "Video" refers to data that expresses movement by playing back successive image frames over time.

[1253] "Audio" refers to data that contains recorded human voices and surrounding environmental sounds.

[1254] "Text" refers to data that stores keyboard input and mouse operations as character information.

[1255] "Emotion data" refers to data that indicates the emotional state of a user extracted from facial expressions and vocal tones.

[1256] "Classification" refers to dividing the recorded operation data into video, audio, text, and emotion data.

[1257] "Generative AI" refers to AI technology that analyzes recorded data and recognizes and generates specific information and patterns.

[1258] "Analysis" refers to the detailed examination of recorded data to uncover meanings and patterns.

[1259] "Important points" refer to actions or content that are particularly noteworthy in the operation data.

[1260] "User's emotional state" refers to the type and strength of emotions felt by the user during operation.

[1261] "Synthesis" refers to bringing together extracted key points, emotional states, and related data into a single framework.

[1262] A "manual" refers to a guide document that summarizes user operating procedures, background information, and emotion analysis results.

[1263] "Providing" refers to presenting information such as generated manuals to users in an easy-to-use format.

[1264] This invention relates to a system that records the operations performed by a user on an information processing device in real time, analyzes the recorded operation data, and generates a manual. In particular, it aims to record and analyze the user's emotional state and provide a detailed manual that integrates this data.

[1265] Recording user actions

[1266] First, the user starts a specific task on the information processing device. For example, when the user starts installing new software, the device records the operation in real time in the following format:

[1267] 1. Video recording: The device captures the user's screen and saves it as video data in MP4 format. This video recording allows you to visually confirm the operations that the user actually performed.

[1268] 2. Audio recording: The device will record the user's voice and environmental sounds in WAV format using the built-in microphone or an external microphone, so that what the user says during operation and the surrounding background sounds will be recorded.

[1269] 3. Text logging: The terminal records keyboard and mouse inputs in text format in a log file, which records the commands and text entered by the user.

[1270] 4. Emotion Recording: The device uses a camera and microphone to capture the user's facial expressions and voice tone in real time, and stores them as emotion data through the emotion engine. This emotion data is stored in JSON format.

[1271] Sending data

[1272] Once recording begins, the device will collect data at regular intervals (e.g., every minute) and compile it into a single data packet, which includes a video file, an audio file, a text log, and emotion data.

[1273] The device then sends the generated data packets to the server via an internet connection, using encryption technology such as SSL / TLS.

[1274] Data analysis

[1275] The server receives data packets sent from the device, generates an analysis request for the received data, and sends it to an internal processing engine service. The server then uses a generative artificial intelligence model (e.g., OpenAI's GPT-4) to perform the next analysis.

[1276] Video and audio analysis: Extracts important operation points from video data frame by frame, and picks out instructions and important statements from audio data.

[1277] Natural language processing: Analyzes commands and operation details from text logs and identifies operation procedures based on the context.

[1278] Emotion analysis: Use the output of the emotion engine to identify the emotional state the user was in during each action.

[1279] Context Integration

[1280] Based on the parsed data, the server collects additional relevant information from databases and external resources, such as help documents and FAQ pages for operations.

[1281] The server combines this additional information with the analysis results to create a single integrated dataset, which includes operating instructions, related help documents, sentiment analysis results, etc.

[1282] Generate manual

[1283] The server uses the integrated data to create a manual in a format that is easy for users to understand. This includes the following elements:

[1284] Video Clips: Short video clips are inserted to show user actions.

[1285] Text Instructions: Contains detailed text instructions related to the procedure.

[1286] Additional Related Information: Provides additional background information or notes related to the operation.

[1287] Sentiment analysis results: Data showing the user's emotional state is inserted as diagrams and text, allowing users to visually understand which parts of the operation procedure the user found difficult or went smoothly.

[1288] Providing manuals

[1289] Finally, the generated manual is sent to the terminal and made accessible to the user. By viewing this manual, the user can check the operation procedures, background information, and their own emotional state while working.

[1290] Examples of specific examples and prompts

[1291] For example, if a user records the steps to install new software, they can input the following prompts to the generative AI model:

[1292] "Analyze data recording the steps to install new software and generate a manual that combines operating procedures and emotional states."

[1293] This allows a specific and detailed manual to be automatically generated, supporting the user's operations.

[1294] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1295] Processing Steps

[1296] Step 1:

[1297] A user starts a specific task on an information processing device. The input is a trigger operated by the user to start the task. The output is the start of real-time recording of the user's operations. At this stage, the system starts monitoring the operations.

[1298] Step 2:

[1299] The terminal records operation data in real time. The specific operations are as follows:

[1300] Video recording: The device launches screen capture software and saves the user's screen operations in MP4 format. The input is the user's screen information, and the output is video data.

[1301] Audio recording: The device uses a microphone to record ambient sounds in WAV format. The input is microphone input and the output is audio data.

[1302] Text logging: The terminal records keyboard and mouse activity in a text log. The input is the user's keyboard and mouse actions, and the output is text data.

[1303] Emotion recording: The device uses a camera and microphone to capture the user's facial expressions and voice tone in real time, which are then analyzed by the emotion engine. The input is data from the camera and microphone, and the output is emotion data.

[1304] Step 3:

[1305] The terminal aggregates the recorded data into data packets at regular intervals. The input is video, audio, text, and emotion data collected in real time. The output is the aggregated data packets.

[1306] Step 4:

[1307] The terminal sends the integrated data packet to the server using encryption technology such as SSL / TLS. The input is the data packet, and the output is the data transmission to the server.

[1308] Step 5:

[1309] The server receives data packets: the input is the data packet sent by the terminal, and the output is the acknowledgement.

[1310] Step 6:

[1311] The server analyzes the received data packets to generate requests and sends them to an internal processing engine service. The input is the data packets and the output is the analysis requests.

[1312] Step 7:

[1313] The server analyzes the data using a generative artificial intelligence model, specifically by:

[1314] Video and audio analysis: Extracts important operation points from input data frame by frame and picks out important statements. The output is analyzed operation data.

[1315] Natural language processing: Extracting important commands and operations from text data. The input is text data, and the output is parsed text data.

[1316] Sentiment Analysis: The output of the emotion engine is used to extract the user's emotional state during operation. The input is emotion data, and the output is the analyzed emotion data.

[1317] Step 8:

[1318] The server collects additional relevant information based on the analysis results. The input is the analyzed data, and the output is the additional relevant information.

[1319] Step 9:

[1320] The server integrates the collected additional information and analysis results into a single dataset. The input is the analysis results and additional information, and the output is the integrated dataset.

[1321] Step 10:

[1322] The server uses the integrated dataset to generate a manual, including video clips, text descriptions, additional related information, and sentiment analysis results. The input is the integrated dataset, and the output is the generated manual.

[1323] Step 11:

[1324] The server sends the generated manual to the terminal. The input is the generated manual, and the output is the delivery of the manual to the terminal.

[1325] Step 12:

[1326] The user reads the received manual on the terminal and checks the operation procedures and emotion analysis results. The input is the received manual, and the output is operation support for the user.

[1327] This allows for a comprehensive understanding of the user's operations and the emotions felt at the time, making it possible to provide more accurate manuals.

[1328] (Application example 2)

[1329] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1330] On conventional online shopping sites, users often feel anxious and confused when operating the site for the first time. In particular, new users are often not provided with effective operation guides, which increases the likelihood of operational errors and a lack of understanding of operation procedures. Furthermore, manuals provided without considering the user's emotions make it difficult to alleviate the stress and difficulties experienced during operation. To solve these problems, a system is needed that records and analyzes user operations, and then generates and provides custom operation guides that incorporate the user's emotional state.

[1331] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1332] In this invention, the server includes means for recording in real time the operations performed by a user on an information processing device, means for analyzing the recorded operation data using a generative artificial intelligence to extract important points, means for analyzing the user's emotional state using an emotion engine, means for generating a manual by integrating the extracted important points with additional related data and emotional data, and means for providing the generated manual on the information processing device. This makes it possible to generate a detailed customized manual for the user's operating procedures and provide support that takes into account the user's emotional state.

[1333] A "user" is a subject who operates an information processing device.

[1334] An "information processing device" is a device that processes digital data, such as a computer, smartphone, or tablet.

[1335] "Means for recording in real time" refers to a function or device for instantly recording or recording user operations on the spot.

[1336] "Operation data" is data that records the content of operations performed by a user, and includes video, audio, text, and the like.

[1337] "Generative artificial intelligence" is an AI technology that uses machine learning and deep learning, and is a method of extracting features from data and analyzing them.

[1338] "Important points" are parts of user operations that require special attention, or important events or operations in the system.

[1339] An "emotion engine" is software or a system for analyzing a user's emotional state from their tone of voice and facial expressions.

[1340] "Related data" refers to the operating procedures as well as helpful help documents and additional information related to the operation.

[1341] The "means for generating a manual" refers to a function or software that integrates recorded operation data and related information to create a manual in a format that is easy for the user to understand.

[1342] The "means for providing on an information processing device" refers to a function or method for displaying the generated manual on the information processing device operated by the user, making it viewable.

[1343] This invention is a system that records the operations performed by users on information processing devices in real time, analyzes and integrates the operation data and emotional data, and generates and provides a customized operation manual.

[1344] 1. System Configuration

[1345] Hardware

[1346] Information processing device: A terminal operated by a user, including computers, smartphones, tablets, etc.

[1347] Camera and microphone: Input devices for recording user actions, facial expressions, and voice tones in real time.

[1348] Server: Back-end infrastructure for analyzing operation data and emotion data and generating manuals.

[1349] software

[1350] OpenCV: A library for capturing and processing video and image data.

[1351] Google Cloud Speech-to-Text API: A service for converting voice data into text data.

[1352] TensorFlow: A machine learning library for analyzing emotions from facial expressions and voice.

[1353] Django Framework: A server-side framework for generating and serving operation manuals.

[1354] JavaScript: A front-end technology for displaying manuals on information processing devices.

[1355] 2. System Functions

[1356] Recording user actions

[1357] When a user starts a specific task on the information processing device, the device records the user's actions in real time. The recordings are in the following format:

[1358] Video recording: Use the camera to capture the user's operation screen and save it as video data.

[1359] Audio recording: Records the user's voice and environmental sounds through microphone input and saves them as audio data.

[1360] Emotion recording: Uses a camera and microphone to capture the user's facial expressions and vocal tone, which are then analyzed by the emotion engine.

[1361] Data transmission and analysis

[1362] Once recording begins, the device will periodically generate data packets and send them to the server. The data packets contain video, audio, text, and emotion data. The server will analyze the data and extract key points. The complete analysis process includes the following steps:

[1363] Generative AI video and audio analysis

[1364] Natural Language Processing (NLP)

[1365] Emotion analysis

[1366] Manual generation and provision

[1367] Based on the analysis results, the server integrates additional data related to user operations and generates an intuitive and easy-to-understand operation manual. The generated manual includes the following elements:

[1368] Video Clip: A short video clip showing the procedure

[1369] Text Description: Detailed instructions related to the procedure

[1370] Sentiment analysis results: Data that visually indicates the user's emotional state

[1371] The generated manual is finally sent to the terminal, and the user can refer to this manual as they work.

[1372] 3. Specific Examples

[1373] For example, if a user records the steps of purchasing a product on an online shopping site, the device records the entire purchasing process as video, audio, and emotional data. The server analyzes this data to extract important steps and the user's emotional state during the process. It then integrates additional help information and product reviews related to the purchasing process and generates a custom operation guide that includes them.

[1374] Example prompts for generative AI models

[1375] "On your online store, walk users through the steps of adding an item to their cart and visually demonstrate points of frustration."

[1376] As described above, this system can record and analyze the user's operations and their emotional state in detail, and generate and provide optimal operation guidance.

[1377] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1378] Step 1:

[1379] When a user starts operating an information processing device, the device records the user's operations in real time. At this point, the device's camera and microphone are activated, capturing the user's screen operation as a video and recording the user's voice input. The input data is the user's screen capture video and voice data, which are saved as recorded data.

[1380] Step 2:

[1381] The device uses an emotion engine to analyze the user's facial expressions and voice tone in real time to generate emotion data. Specifically, an emotion recognition model using TensorFlow predicts the user's emotional state (e.g., stress, joy, neutral, etc.) from their facial expressions and voice. This emotion data is stored along with the video and audio data. The input data is the user's facial expressions and voice, and the output data is the analyzed emotional state.

[1382] Step 3:

[1383] After recording for a certain period of time or for each specific operation, the device sends these data packets to the server. The data packets include video, audio, text, and emotion data. The input data is the recorded data packets, and the output data is the data sent to the server. Specifically, the device secures a stable network connection and uploads the data to the server using protocols such as HTTP or WebSocket.

[1384] Step 4:

[1385] The server analyzes the received data packets and begins processing various data. First, it uses generative AI to extract key operation points from the video and audio data. For example, it identifies the icons the user clicked and the text fields they entered. The input data is the video and audio data, and the output data is the analyzed key points. Specifically, it runs a generative AI model (for example, YOLO or DeepSpeech) to detect key operations.

[1386] Step 5:

[1387] The server then analyzes the text data using natural language processing (NLP). This clarifies the intent of the user's input text or command and identifies operational procedures and related background information. The input data is the text data, and the output data is the analyzed operational procedures and background information. Specifically, the server performs text analysis using an NLP engine (e.g., spaCy or BERT).

[1388] Step 6:

[1389] The server analyzes the emotion data provided by the emotion engine and identifies the user's emotional state at each operation point. The input data is emotion data, and the output data is the emotional state corresponding to each operation point. Specifically, the server maps the emotional state at each operation step from the time-series data.

[1390] Step 7:

[1391] Based on the analysis results, the server collects and integrates additional data related to the operation procedures, such as help documents and screenshots related to the operations. The input data is the analyzed operation procedures and emotion data, and the output data is a draft of the integrated operation manual. Specific operations include the process of retrieving related information from databases and external APIs.

[1392] Step 8:

[1393] The server generates a detailed operating manual based on the integrated data. The generated manual includes video clips, text descriptions, additional related information, and sentiment analysis results. The input data is a draft of the integrated operating manual, and the output data is the final generated operating manual. Specifically, the manual document is automatically generated using a template engine.

[1394] Step 9:

[1395] Finally, the server sends the generated operation manual to the terminal, making it accessible to the user. The input data is the finally generated operation manual, and the output data is the data sent to the terminal. Specific operations include providing information to the user through HTTP responses and updating the database.

[1396] As described above, the input data is analyzed and processed at each processing step, and ultimately a detailed custom operation manual is generated and provided to assist the user in their operations.

[1397] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1398] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1399] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1400] [Fourth embodiment]

[1401] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1402] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1403] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1404] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1405] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1406] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1407] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1408] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1409] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1410] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1411] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1412] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1413] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1414] This invention relates to a system that records operations performed by a user on an information processing device in real time, analyzes and integrates the data, and generates a manual. Specific embodiments of this system will be described below.

[1415] Recording user actions

[1416] First, when a user starts a specific task on an information processing device, the device records the user's actions in real time. The recording is done in three forms:

[1417] 1. Video recording: The device captures the user's screen and saves it as video data, allowing you to visually confirm the operations that the user actually performed.

[1418] 2. Audio recording: The device will record the user's voice and environmental sounds through the microphone input, so that what the user says and the background sounds while operating the device will be recorded.

[1419] 3. Text recording: The terminal stores keyboard and mouse inputs as text data, thereby recording the commands and text entered by the user.

[1420] Sending data

[1421] Once recording begins, the device periodically generates and transmits data packets containing video, audio, and text data to the server, each accompanied by metadata such as a timestamp and user ID.

[1422] Data analysis

[1423] The server analyzes the received data packet and performs the following actions:

[1424] 1. Video and audio analysis using generative AI: The server uses generative AI to extract important points from video and audio data, such as clicking on important menu items or operating settings screens.

[1425] 2. Natural language processing: The server analyzes the text data using a natural language processing engine to identify operational procedures and related background information, thereby clarifying the user's intent and purpose.

[1426] Context Integration

[1427] Based on the analysis results, the server collects and integrates additional data related to the operation procedure, including other text, images, and audio, such as help documents related to the operation and screenshots of the settings screen.

[1428] Generate manual

[1429] The server uses the integrated data to create a manual that explains the operation procedures in an easy-to-understand format. The manual contains the following elements:

[1430] 1. Video Clips: Short video clips are inserted to show user actions.

[1431] 2. Text Instructions: Contains detailed text instructions related to the operating procedure.

[1432] 3. Additional Related Information: Provides additional background information and notes related to the operation.

[1433] Providing manuals

[1434] Finally, the generated manual is sent to the terminal and made accessible to the user. By viewing this manual, the user can gain a deeper understanding of the operating procedures and their background, allowing them to work more efficiently.

[1435] Specific examples

[1436] For example, if a user records the process of installing new software, the device will record the entire installation process as video, audio, and text. The server will analyze this data and extract important installation steps. It will then integrate additional help information and system requirements related to the installation process and generate a manual that includes them. Finally, this manual will be sent to the user's device, where the user can follow the installation process while checking the instructions.

[1437] In this way, by effectively recording, analyzing, and integrating user operations and their background information, a detailed and easy-to-understand manual is provided, allowing users to understand the intentions and concepts behind the operations and work efficiently.

[1438] The processing flow will be explained below.

[1439] Step 1:

[1440] The user starts a specific task on the information processing device and clicks the record button, thereby starting the recording process.

[1441] Step 2:

[1442] The device detects the user's clicks and begins recording the user's actions in real time, including by:

[1443] Video recording: Using screen capture software, the entire user's PC screen is saved as video data.

[1444] Audio recording: Environmental sounds and the user's voice are saved as audio data via microphone input.

[1445] Text recording: Using a keylogger or mouse event capture function, keyboard input and mouse operations are saved as text data.

[1446] Step 3:

[1447] The terminal divides the recorded data into packets of a certain size and adds metadata such as a timestamp and a user ID to each packet.

[1448] Step 4:

[1449] The terminal sends data packets sequentially to the server, using the SSL / TLS protocol to ensure data security.

[1450] Step 5:

[1451] The server takes the received data packets sequentially and reassembles them into one continuous data set, allowing the entire user interaction to be analyzed as a continuous data stream.

[1452] Step 6:

[1453] The server uses generative AI to analyze the received video and audio data. The AI ​​model does the following:

[1454] It detects and tags important actions from video data, for example identifying the moment a specific button is clicked or a settings menu is opened.

[1455] The voice data is converted into text by a speech recognition engine, and specific instructions or explanations are extracted.

[1456] Step 7:

[1457] The server analyzes the text data with a natural language processing (NLP) engine to identify operational steps and related context, such as when a particular command is entered and what its meaning is.

[1458] Step 8:

[1459] Based on the analysis, the server collects additional data related to the operation procedure from internal databases and external APIs, including:

[1460] The URL and content of the website accessed during the user's operation.

[1461] Links and contents of manuals and help documents related to the operation.

[1462] System requirements and configuration information required for operation.

[1463] Step 9:

[1464] The server consolidates the collected data and assembles it into a document structure that provides operational instructions, highlighting background information and important points related to each step.

[1465] Step 10:

[1466] The server applies the generated content to the presentation layer, producing the final manual in multiple formats (PDF, HTML, video files).

[1467] Step 11:

[1468] The server sends the generated manual to the terminal and uploads it to a repository accessible to the user. The user receives a notification and can view the manual on their PC.

[1469] Step 12:

[1470] Users can refer to the generated manual to check the necessary operating procedures and related background information, which will deepen their understanding of the operations and improve work efficiency.

[1471] Example 1

[1472] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1473] In the past, recording the operations performed by users on information processing devices and creating manuals to communicate those operating procedures to other users was a time-consuming and labor-intensive process. Furthermore, manual recording and editing is prone to human error and omissions, making it difficult to create efficient procedure manuals. For this reason, there was a demand for a system that could accurately and efficiently record user operations and automatically generate manuals based on that information.

[1474] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1475] In this invention, the server includes means for recording in real time operations performed by a user on an information processing device, means for saving the recorded operation data in video, audio, and text formats, means for analyzing the recorded operation data using a generative AI model and extracting important points, means for generating a manual by integrating the extracted important points with additional related information, and means for providing the generated manual on the information processing device. This makes it possible to automatically and accurately record user operations and quickly generate a manual based on the operation procedures.

[1476] The term "user" refers to a person who performs an operation on an information processing device.

[1477] "Information processing device" refers to a computer or its peripheral devices for inputting, processing, storing, and outputting data.

[1478] "Real-time recording" refers to recording user operations immediately.

[1479] "Operation data" refers to data indicating the history and content of operations performed by a user on an information processing device.

[1480] "Video format" refers to video data captured from the user's screen.

[1481] "Audio format" refers to audio data that records the user's voice or environmental sounds.

[1482] "Text format" refers to character data that records keyboard input and mouse operations.

[1483] A "generative AI model" is a model that uses artificial intelligence to analyze data and extract important information and patterns.

[1484] "Analysis" refers to the act of analyzing recorded data to extract meaning and patterns.

[1485] The "important points" refer to particularly noteworthy or critical parts of the operations performed by the user.

[1486] "Related information" refers to information such as additional text, images, and audio that is related to the user's operating procedure.

[1487] "Integration" refers to the act of bringing together data or information in different formats.

[1488] A "manual" refers to a document or multimedia content that clearly shows and explains user operating procedures.

[1489] "Providing" refers to the act of displaying or saving the generated manual on an information processing device in a form that can be used by the user.

[1490] This invention relates to a system that records operations performed by a user on an information processing device in real time, analyzes and integrates the data, and generates a manual. Specific embodiments of this system will be described below.

[1491] Recording user actions

[1492] First, when a user starts a specific task on an information processing device, the device records the user's actions in real time. The recording is done in three forms:

[1493] 1. Video recording: The device captures the user's screen and saves it as video data using video capture software such as OBS Studio, allowing you to visually confirm the operations that the user actually performed.

[1494] 2. Audio recording: The device records the user's voice and environmental sounds through microphone input. For example, the device uses a built-in microphone or an external microphone to collect audio data, recording what the user says and the background sounds while operating the device.

[1495] 3. Text recording: The terminal stores keyboard and mouse inputs as text data. Logging software is used to record the commands and text entered by the user.

[1496] Sending data

[1497] Once recording begins, the device periodically generates data packets and sends them to the server. The data packets contain video, audio, and text data, and each packet is assigned metadata such as a timestamp and user ID. For example, a packet could be generated every minute and sent to the server via the network.

[1498] Data analysis

[1499] The server analyzes the received data packet and performs the following actions:

[1500] 1. Video and audio analysis using generative AI models: The server uses generative AI models (such as OpenAI's GPT model) to extract important points from video and audio data, such as clicking on important menu items or operating settings screens.

[1501] 2. Natural Language Processing: The server analyzes the text data using a natural language processing engine (e.g., spaCy) to identify operational steps and related background information, thereby clarifying the user's intent and purpose.

[1502] Context Integration

[1503] Based on the analysis results, the server collects and integrates additional data related to the operation procedure, including other text, images, and audio, such as help documents related to the operation and screenshots of the settings screen.

[1504] Generate manual

[1505] The server uses the integrated data to create a manual that explains the operation procedures in an easy-to-understand format. The manual contains the following elements:

[1506] 1. Video Clips: Short video clips are inserted to show user actions.

[1507] 2. Text Instructions: Contains detailed text instructions related to the operating procedure.

[1508] 3. Additional Related Information: Provides additional background information and notes related to the operation.

[1509] Providing manuals

[1510] Finally, the generated manual is sent to the terminal and made accessible to the user. By viewing this manual, the user can gain a deeper understanding of the operating procedures and their background, allowing them to work more efficiently.

[1511] Specific examples

[1512] For example, if a user records the process of installing new software, the device will record the entire installation process as video, audio, and text. The server will analyze this data and extract important installation steps. It will then integrate additional help information and system requirements related to the installation process and generate a manual that includes them. Finally, this manual will be sent to the user's device, allowing the user to easily follow the installation process.

[1513] Prompt Sentence Examples

[1514] "This system records the steps a user takes to install new software, then analyzes and integrates the data to generate a detailed operating manual. Please provide a concrete example of how the generative AI model employed in this process extracts key installation steps and integrates relevant information."

[1515] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1516] Step 1:

[1517] When a user starts a specific task on the information processing device, the device records the user's actions in real time. For example, when starting to install new software, the device launches OBS Studio and starts capturing the screen. It also records the user's voice via microphone and logs keyboard and mouse input data.

[1518] Input: User operation begins

[1519] Output: Video files, audio files, text logs

[1520] Step 2:

[1521] The device packets the collected data at regular intervals and sends it to the server. For example, video, audio, and text data are packetized every minute, and metadata such as timestamps and user IDs are added. A library for packet generation is used to send the data over the network to the server.

[1522] Input: Video files, audio files, text logs

[1523] Output: Data packets

[1524] Step 3:

[1525] The server analyzes the received data packets and extracts key points using generative AI models, such as OpenAI's GPT model, to detect click events from video, commands from audio, and settings from text.

[1526] Input: Data packet

[1527] Output: List of key points

[1528] Step 4:

[1529] The server then collects and integrates additional relevant information based on the extracted key points, using a natural language processing engine (e.g., spaCy) to analyze the text data and collect relevant help documentation and screenshots of configuration screens.

[1530] Input: List of key points

[1531] Output: Consolidated related information

[1532] Step 5:

[1533] The server then uses the integrated data to create easy-to-understand operation procedures in a manual, generating documents including video clips, detailed text explanations, and background information.

[1534] Input: Integrated related information

[1535] Output: Manual document

[1536] Step 6:

[1537] The server sends the generated manual to the terminal, where the user can view the manual and check the operating procedures as they work.

[1538] Input: Manual document

[1539] Output: A user-readable manual

[1540] (Application example 1)

[1541] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1542] Conventional methods for creating manuals for factory robot operation procedures required a great deal of time and effort for manual recording and analysis, making them inefficient. Furthermore, operation procedures were rarely provided in a format that was easy for workers to understand, leading to operational errors and reduced efficiency. Furthermore, it was difficult to check operation procedures on different devices, creating a need for an integrated solution to improve on-site work efficiency.

[1543] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1544] In this invention, the server includes means for recording operations performed by a user on an information processing device in real time, means for analyzing the recorded operation data using artificial intelligence and extracting important points, means for generating a manual by integrating the extracted important points with additional related data, means for providing the generated manual on the information processing device, and means for recording and viewing operation procedures for robot operations in factories via wearable devices such as smart glasses and head-mounted displays. This allows for real-time recording and analysis of operation procedures, enabling workers to instantly check easy-to-understand manuals via wearable devices, thereby enabling efficient and accurate operation.

[1545] A "user" is a person or entity that performs an operation on an information processing device.

[1546] An "information processing device" is a device that inputs, processes, and outputs data, and includes personal computers, tablets, smartphones, etc.

[1547] "Real-time" refers to operations and processing that are recorded and analyzed at the same time as they are performed.

[1548] "Recording" refers to the process of saving user operations as data.

[1549] "Analysis" refers to the process of evaluating recorded data and extracting important information.

[1550] "Generative artificial intelligence" refers to algorithms and software that have the ability to extract meaning and patterns from input data.

[1551] "Important points" refer to parts or steps that require special attention in the user's operations.

[1552] "Extraction" refers to the process of extracting specific information from the overall data.

[1553] "Integration" refers to the process of combining multiple pieces of data or information into one whole.

[1554] A "manual" refers to a document or material that describes operating procedures and guidelines.

[1555] "Providing" refers to the process of displaying the generated manual in a form that is accessible to users.

[1556] "Robot operation in a factory" refers to operating a robot in a factory to perform specific tasks.

[1557] "Smart glasses" refers to a wearable device in the shape of glasses that has the function of displaying information.

[1558] A "head-mounted display" refers to a device worn on the head that displays images and information.

[1559] A "wearable device" refers to an information processing device that can be worn.

[1560] This invention is a system that records and analyzes operations performed by a user on an information processing device in real time to efficiently operate robots in a factory, and provides the results as a manual.

[1561] System Overview

[1562] The system consists of the following elements:

[1563] 1. Information processing devices used by users (PCs, tablets, smartphones, etc.)

[1564] 2. Recording equipment (camera and microphone)

[1565] 3. Generative Artificial Intelligence (AI Model)

[1566] 4. Server (performs data analysis and manual generation)

[1567] 5. Wearable devices (smart glasses, head-mounted displays)

[1568] How it works

[1569] Recording user actions

[1570] When the user starts operating the robot, the camera records the operation video in real time. At the same time, the microphone records the audio, and keyboard input and mouse operations are recorded as text data. This recorded data is periodically sent to the server as data packets.

[1571] Data transmission and analysis

[1572] The server analyzes the received data packets and uses generative artificial intelligence to extract key points from the video and audio data. Specifically, it detects important operation steps, commands, setting changes, etc. At the same time, it also analyzes the text data using a natural language processing engine to clarify operation procedures and intentions.

[1573] Context integration and manual generation

[1574] Based on the analysis results, the server collects and integrates additional related data (help documents, screenshots, configuration details, etc.). Based on the integrated data, an easy-to-understand operation manual is created. The manual includes video clips, text explanations, and additional related information.

[1575] Providing manuals

[1576] The generated manual is sent to the user's information processing device and wearable device, where the user can check the operation procedures in real time via smart glasses or a head-mounted display.

[1577] Hardware and software used

[1578] The system uses the following hardware and software:

[1579] Recording device: Camera (e.g., Logitech C920) and microphone (e.g., Blue Yeti)

[1580] Generative Artificial Intelligence: Azure Cognitive Services Text Analytics API

[1581] Information processing devices: Windows PCs, Macs, tablets, smartphones

[1582] Wearable devices: smart glasses (e.g., Google Glass), head-mounted displays (e.g., Microsoft HoloLens)

[1583] Specific examples

[1584] For example, when a user assembles parts using a robot in a factory, the entire operation is recorded using a camera and microphone. The server analyzes this recorded data, identifies important steps (such as the correct position of some parts or how to tighten screws), and generates a detailed manual. The user can then proceed with the work while viewing the generated manual in real time through smart glasses or a head-mounted display.

[1585] Prompt Sentence Examples

[1586] "Record the steps the robot takes to assemble the parts, extract the important steps, and generate a manual based on them."

[1587] In this way, operating procedures can be recorded and analyzed in real time, and workers can instantly check easy-to-understand manuals via wearable devices, enabling efficient and accurate operations.

[1588] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1589] Step 1:

[1590] User operation initiation and recording

[1591] The user begins operating the factory robot. The device uses a camera and microphone to record video and audio of the operation in real time. At the same time, keyboard and mouse operations are also recorded as text data.

[1592] Input: User operation video, voice, keyboard input, mouse operation

[1593] Data processing: Capture video, record audio, record text input

[1594] Output: Data packets collected in real time

[1595] Specific behavior: Capture video with a camera (e.g. Logitech C920), record audio with a microphone (e.g. Blue Yeti), and save the text to a local file.

[1596] Step 2:

[1597] Creating and sending data packets

[1598] The terminal generates the collected data as data packets at regular intervals and transmits them to the server.

[1599] Input: Data collected in real time (video, audio, text)

[1600] Data processing: Packetizing data (adding timestamps and user IDs)

[1601] Output: Data packet with timestamp and user ID added

[1602] Specific operation: Data packets are generated at regular intervals and sent to the server via the network.

[1603] Step 3:

[1604] Data reception and storage by the server

[1605] The server stores the received data packets, organizing them based on user ID and timestamp.

[1606] Input: Data packets (video, audio, text)

[1607] Data processing: packets are decoded, stored, and organized by metadata

[1608] Output: Organized data (stored in a database or file system)

[1609] Specific operation: Save to a database system (e.g. MySQL), record to the file system.

[1610] Step 4:

[1611] Data analysis

[1612] The server analyzes the stored data using artificial intelligence to extract key points, while voice and text data is analyzed using a natural language processing engine.

[1613] Input: stored video, audio, and text data

[1614] Data calculation: AI model-based video analysis (e.g., object detection), audio analysis (e.g., sound recognition), and text analysis (e.g., key phrase extraction)

[1615] Output: Analysis results (important operation points, key phrases, etc.)

[1616] Specific operation: Uses Azure Cognitive Services' Text Analytics API and object detection models (e.g., YOLO).

[1617] Step 5:

[1618] Context Integration

[1619] Based on the analyzed results, the server collects and integrates additional relevant data (help documentation, screenshots, configuration details, etc.).

[1620] Input: Analysis results (important operation points, key phrases, etc.)

[1621] Data processing: Collecting and integrating additional data (e.g., web crawling, database searching)

[1622] Output: Integrated data set (full operating procedures and related information)

[1623] Specific actions: API calls, database queries.

[1624] Step 6:

[1625] Generate manual

[1626] The server uses the integrated dataset to create easy-to-understand instruction manuals, including video clips, text instructions, and additional relevant information.

[1627] Input: Integrated data set (full operating procedures and related information)

[1628] Data processing: Data formatting and manual generation (using document generation tools)

[1629] Output: Generated manual (PDF, HTML, etc.)

[1630] Specific behavior: Uses a documentation generation tool (e.g., LaTeX, Markdown).

[1631] Step 7:

[1632] Providing manuals

[1633] The server then sends the generated manual to the user's information processing device or wearable device, where it can be displayed. The user can view the operating procedures via smart glasses or a head-mounted display.

[1634] Input: Generated manual (PDF, HTML, etc.)

[1635] Data processing: Sending data (over a network connection)

[1636] Output: Manual displayed on terminal and wearable devices

[1637] Specific operations: Send data to the device, use a display application (e.g., a dedicated viewer).

[1638] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1639] This invention combines a system that records user operations on an information processing device in real time, analyzes and integrates the data, and generates a manual, with an emotion engine. Specific embodiments of this system will be described below.

[1640] Recording user actions

[1641] First, when a user starts a specific task on an information processing device, the device records the user's actions in real time. The recordings are made in the following four forms:

[1642] 1. Video recording: The device captures the user's screen and saves it as video data, allowing you to visually confirm the operations that the user actually performed.

[1643] 2. Audio recording: The device will record the user's voice and environmental sounds through the microphone input, so that what the user says and the background sounds while operating the device will be recorded.

[1644] 3. Text recording: The terminal stores keyboard and mouse inputs as text data, thereby recording the commands and text entered by the user.

[1645] 4. Emotion recording: The device uses a camera and microphone to capture the user's facial expressions and voice tone in real time, and the emotion engine analyzes this data to recognize the user's emotions.

[1646] Sending data

[1647] Once recording begins, the device periodically generates and transmits data packets containing video, audio, text, and emotion data to the server. Each packet is accompanied by metadata such as a timestamp and user ID.

[1648] Data analysis

[1649] The server analyzes the received data packet and performs the following actions:

[1650] 1. Video and audio analysis using generative AI: The server uses generative AI to extract important points from video and audio data, such as clicking on important menu items or operating settings screens.

[1651] 2. Natural language processing: The server analyzes the text data using a natural language processing engine to identify operational procedures and related background information, thereby clarifying the user's intent and purpose.

[1652] 3. Emotion analysis: The server analyzes the emotion data provided by the emotion engine to determine the emotional state of the user at each point of the operation, for example, whether they felt stressed or happy during the operation.

[1653] Context Integration

[1654] Based on the analysis results, the server collects and integrates additional data related to the operation procedure, including other text, image, audio, and emotional data, such as help documents related to the operation and screenshots of the settings screen.

[1655] Generate manual

[1656] The server uses the integrated data to create a manual that explains the operation procedures in an easy-to-understand format. The manual contains the following elements:

[1657] 1. Video Clips: Short video clips are inserted to show user actions.

[1658] 2. Text Instructions: Contains detailed text instructions related to the operating procedure.

[1659] 3. Additional Related Information: Provides additional background information and notes related to the operation.

[1660] 4. Emotion analysis results: Data showing the user's emotional state is inserted as diagrams and text, allowing users to visually understand which parts of the operation procedure the user found difficult or went smoothly.

[1661] Providing manuals

[1662] Finally, the generated manual is sent to the terminal and made accessible to the user. By viewing this manual, the user can check the operation procedures, background information, and their own emotional state while working.

[1663] Specific examples

[1664] For example, if a user records the steps to install new software, the device records the entire installation process as video, audio, text, and emotional data. The server analyzes this data to extract important installation steps and the user's emotional state during the process. The server then integrates additional help information and system requirements related to the installation process and generates a manual that includes these. The manual visually indicates the points where the user felt stressed and the points where the installation went smoothly. Finally, the manual is sent to the user's device, where the user can review the installation steps and refer to the results of their own emotional analysis.

[1665] In this way, by effectively recording, analyzing, and integrating the user's emotional state in addition to their operations and their background information, a detailed and easy-to-understand manual is provided, allowing the user to understand the intentions and concepts behind the operations and work efficiently.

[1666] The processing flow will be explained below.

[1667] Step 1:

[1668] The user starts a specific task on the information processing device and clicks a record button, thereby starting the recording process.

[1669] Step 2:

[1670] The device detects the user's click and starts real-time recording in the following format:

[1671] Video recording: Using screen capture software, the entire user's PC screen is saved as video data.

[1672] Audio recording: Saves the user's voice and environmental sounds as audio data through microphone input.

[1673] Text recording: Using a keylogger or mouse event capture function, keyboard input and mouse operations are saved as text data.

[1674] Emotion Recording: Using a camera and microphone, the system captures the user's facial expressions and voice tone in real time, and the emotion engine analyzes this data to recognize the user's emotions.

[1675] Step 3:

[1676] The terminal divides the recorded data into packets of a certain size and adds metadata such as a timestamp and a user ID to each packet.

[1677] Step 4:

[1678] The terminal sends data packets sequentially to the server, using the SSL / TLS protocol to ensure data security.

[1679] Step 5:

[1680] The server takes the received data packets sequentially and reassembles them into one continuous data set, allowing the entire user interaction to be analyzed as a continuous data stream.

[1681] Step 6:

[1682] The server uses generative AI to analyze the received video and audio data. The AI ​​model does the following:

[1683] It detects and tags important actions from video data, for example identifying the moment a specific button is clicked or a settings menu is opened.

[1684] The voice data is converted into text by a speech recognition engine, and specific instructions or explanations are extracted.

[1685] Step 7:

[1686] The server analyzes the text data with a natural language processing (NLP) engine to identify operational steps and related context, such as when a particular command is entered and what its meaning is.

[1687] Step 8:

[1688] The server analyzes the emotional data provided by the emotion engine and identifies the emotional state the user was in at each point of the operation, for example, whether they were feeling stressed or happy during the operation.

[1689] Step 9:

[1690] Based on the analysis, the server collects additional data related to the operation procedure from internal databases and external APIs, including:

[1691] The URL and content of the website accessed during the user's operation.

[1692] Links and contents of manuals and help documents related to the operation.

[1693] System requirements and configuration information required for operation.

[1694] Step 10:

[1695] The server consolidates the collected data and assembles it into a document structure that provides operational instructions, highlighting background information and important points related to each step.

[1696] Step 11:

[1697] The server applies the generated content to the presentation layer, producing the final manual in multiple formats (PDF, HTML, video files).

[1698] Step 12:

[1699] The server sends the generated manual to the terminal and uploads it to a repository accessible to the user. The user receives a notification and can view the manual on their PC.

[1700] Step 13:

[1701] Users can refer to the generated manual to check the necessary operating procedures, their background information, and their own emotional state, which will deepen their understanding of the operation and improve work efficiency.

[1702] Example 2

[1703] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1704] Conventional information processing devices were able to record user operations and generate manuals based on those operations. However, they did not consider grasping the user's emotional state and incorporating that into the manual. As a result, it was not possible to provide a detailed manual that reflected the stress and difficulties the user experienced during operation. The present invention aims to visualize the difficulties the user experiences by recording and analyzing the user's emotional state in addition to operation data and generating an integrated manual, thereby supporting easier and smoother operation.

[1705] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1706] In this invention, the server includes means for recording in real time operations performed by a user on an information processing device, means for classifying the recorded operation data into video, audio, text, and emotional data, means for analyzing the classified operation data using a generative artificial intelligence to extract important points and the user's emotional state, means for integrating the extracted important points, emotional state, and additional related data to generate a manual, and means for providing the generated manual on the information processing device. This makes it possible to comprehensively grasp the user's operations and the emotions felt at the time, and to provide a more accurate manual.

[1707] The term "user" refers to a person who operates an information processing device.

[1708] "Information processing device" refers to computer equipment or digital devices that users can operate.

[1709] "Real time" refers to processing that is carried out continuously in real time.

[1710] "Recording" refers to saving information such as the user's actions, comments, and facial expressions.

[1711] "Video" refers to data that expresses movement by playing back successive image frames over time.

[1712] "Audio" refers to data that contains recorded human voices and surrounding environmental sounds.

[1713] "Text" refers to data that stores keyboard input and mouse operations as character information.

[1714] "Emotion data" refers to data that indicates the emotional state of a user extracted from facial expressions and vocal tones.

[1715] "Classification" refers to dividing the recorded operation data into video, audio, text, and emotion data.

[1716] "Generative AI" refers to AI technology that analyzes recorded data and recognizes and generates specific information and patterns.

[1717] "Analysis" refers to the detailed examination of recorded data to uncover meanings and patterns.

[1718] "Important points" refer to actions or content that are particularly noteworthy in the operation data.

[1719] "User's emotional state" refers to the type and strength of emotions felt by the user during operation.

[1720] "Synthesis" refers to bringing together extracted key points, emotional states, and related data into a single framework.

[1721] A "manual" refers to a guide document that summarizes user operating procedures, background information, and emotion analysis results.

[1722] "Providing" refers to presenting information such as generated manuals to users in an easy-to-use format.

[1723] This invention relates to a system that records the operations performed by a user on an information processing device in real time, analyzes the recorded operation data, and generates a manual. In particular, it aims to record and analyze the user's emotional state and provide a detailed manual that integrates this data.

[1724] Recording user actions

[1725] First, the user starts a specific task on the information processing device. For example, when the user starts installing new software, the device records the operation in real time in the following format:

[1726] 1. Video recording: The device captures the user's screen and saves it as video data in MP4 format. This video recording allows you to visually confirm the operations that the user actually performed.

[1727] 2. Audio recording: The device will record the user's voice and environmental sounds in WAV format using the built-in microphone or an external microphone, so that what the user says during operation and the surrounding background sounds will be recorded.

[1728] 3. Text logging: The terminal records keyboard and mouse inputs in text format in a log file, which records the commands and text entered by the user.

[1729] 4. Emotion Recording: The device uses a camera and microphone to capture the user's facial expressions and voice tone in real time, and stores them as emotion data through the emotion engine. This emotion data is stored in JSON format.

[1730] Sending data

[1731] Once recording begins, the device will collect data at regular intervals (e.g., every minute) and compile it into a single data packet, which includes a video file, an audio file, a text log, and emotion data.

[1732] The device then sends the generated data packets to the server via an internet connection, using encryption technology such as SSL / TLS.

[1733] Data analysis

[1734] The server receives data packets sent from the device, generates an analysis request for the received data, and sends it to an internal processing engine service. The server then uses a generative artificial intelligence model (e.g., OpenAI's GPT-4) to perform the next analysis.

[1735] Video and audio analysis: Extracts important operation points from video data frame by frame, and picks out instructions and important statements from audio data.

[1736] Natural language processing: Analyzes commands and operation details from text logs and identifies operation procedures based on the context.

[1737] Emotion analysis: Use the output of the emotion engine to identify the emotional state the user was in during each action.

[1738] Context Integration

[1739] Based on the parsed data, the server collects additional relevant information from databases and external resources, such as help documents and FAQ pages for operations.

[1740] The server combines this additional information with the analysis results to create a single integrated dataset, which includes operating instructions, related help documents, sentiment analysis results, etc.

[1741] Generate manual

[1742] The server uses the integrated data to create a manual in a format that is easy for users to understand. This includes the following elements:

[1743] Video Clips: Short video clips are inserted to show user actions.

[1744] Text Instructions: Contains detailed text instructions related to the procedure.

[1745] Additional Related Information: Provides additional background information or notes related to the operation.

[1746] Sentiment analysis results: Data showing the user's emotional state is inserted as diagrams and text, allowing users to visually understand which parts of the operation procedure the user found difficult or went smoothly.

[1747] Providing manuals

[1748] Finally, the generated manual is sent to the terminal and made accessible to the user. By viewing this manual, the user can check the operation procedures, background information, and their own emotional state while working.

[1749] Examples of specific examples and prompts

[1750] For example, if a user records the steps to install new software, they can input the following prompts to the generative AI model:

[1751] "Analyze data recording the steps to install new software and generate a manual that combines operating procedures and emotional states."

[1752] This allows a specific and detailed manual to be automatically generated, supporting the user's operations.

[1753] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1754] Processing Steps

[1755] Step 1:

[1756] A user starts a specific task on an information processing device. The input is a trigger operated by the user to start the task. The output is the start of real-time recording of the user's operations. At this stage, the system starts monitoring the operations.

[1757] Step 2:

[1758] The terminal records operation data in real time. The specific operations are as follows:

[1759] Video recording: The device launches screen capture software and saves the user's screen operations in MP4 format. The input is the user's screen information, and the output is video data.

[1760] Audio recording: The device uses a microphone to record ambient sounds in WAV format. The input is microphone input and the output is audio data.

[1761] Text logging: The terminal records keyboard and mouse activity in a text log. The input is the user's keyboard and mouse actions, and the output is text data.

[1762] Emotion recording: The device uses a camera and microphone to capture the user's facial expressions and voice tone in real time, which are then analyzed by the emotion engine. The input is data from the camera and microphone, and the output is emotion data.

[1763] Step 3:

[1764] The terminal aggregates the recorded data into data packets at regular intervals. The input is video, audio, text, and emotion data collected in real time. The output is the aggregated data packets.

[1765] Step 4:

[1766] The terminal sends the integrated data packet to the server using encryption technology such as SSL / TLS. The input is the data packet, and the output is the data transmission to the server.

[1767] Step 5:

[1768] The server receives data packets: the input is the data packet sent by the terminal, and the output is the acknowledgement.

[1769] Step 6:

[1770] The server analyzes the received data packets to generate requests and sends them to an internal processing engine service. The input is the data packets and the output is the analysis requests.

[1771] Step 7:

[1772] The server analyzes the data using a generative artificial intelligence model, specifically by:

[1773] Video and audio analysis: Extracts important operation points from input data frame by frame and picks out important statements. The output is analyzed operation data.

[1774] Natural language processing: Extracting important commands and operations from text data. The input is text data, and the output is parsed text data.

[1775] Sentiment Analysis: The output of the emotion engine is used to extract the user's emotional state during operation. The input is emotion data, and the output is the analyzed emotion data.

[1776] Step 8:

[1777] The server collects additional relevant information based on the analysis results. The input is the analyzed data, and the output is the additional relevant information.

[1778] Step 9:

[1779] The server integrates the collected additional information and analysis results into a single dataset. The input is the analysis results and additional information, and the output is the integrated dataset.

[1780] Step 10:

[1781] The server uses the integrated dataset to generate a manual, including video clips, text descriptions, additional related information, and sentiment analysis results. The input is the integrated dataset, and the output is the generated manual.

[1782] Step 11:

[1783] The server sends the generated manual to the terminal. The input is the generated manual, and the output is the delivery of the manual to the terminal.

[1784] Step 12:

[1785] The user reads the received manual on the terminal and checks the operation procedures and emotion analysis results. The input is the received manual, and the output is operation support for the user.

[1786] This allows for a comprehensive understanding of the user's operations and the emotions felt at the time, making it possible to provide more accurate manuals.

[1787] (Application example 2)

[1788] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1789] On conventional online shopping sites, users often feel anxious and confused when operating the site for the first time. In particular, new users are often not provided with effective operation guides, which increases the likelihood of operational errors and a lack of understanding of operation procedures. Furthermore, manuals provided without considering the user's emotions make it difficult to alleviate the stress and difficulties experienced during operation. To solve these problems, a system is needed that records and analyzes user operations, and then generates and provides custom operation guides that incorporate the user's emotional state.

[1790] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1791] In this invention, the server includes means for recording in real time the operations performed by a user on an information processing device, means for analyzing the recorded operation data using a generative artificial intelligence to extract important points, means for analyzing the user's emotional state using an emotion engine, means for generating a manual by integrating the extracted important points with additional related data and emotional data, and means for providing the generated manual on the information processing device. This makes it possible to generate a detailed customized manual for the user's operating procedures and provide support that takes into account the user's emotional state.

[1792] A "user" is a subject who operates an information processing device.

[1793] An "information processing device" is a device that processes digital data, such as a computer, smartphone, or tablet.

[1794] "Means for recording in real time" refers to a function or device for instantly recording or recording user operations on the spot.

[1795] "Operation data" is data that records the content of operations performed by a user, and includes video, audio, text, and the like.

[1796] "Generative artificial intelligence" is an AI technology that uses machine learning and deep learning, and is a method of extracting features from data and analyzing them.

[1797] "Important points" are parts of user operations that require special attention, or important events or operations in the system.

[1798] An "emotion engine" is software or a system for analyzing a user's emotional state from their tone of voice and facial expressions.

[1799] "Related data" refers to the operating procedures as well as helpful help documents and additional information related to the operation.

[1800] The "means for generating a manual" refers to a function or software that integrates recorded operation data and related information to create a manual in a format that is easy for the user to understand.

[1801] The "means for providing on an information processing device" refers to a function or method for displaying the generated manual on the information processing device operated by the user, making it viewable.

[1802] This invention is a system that records the operations performed by users on information processing devices in real time, analyzes and integrates the operation data and emotional data, and generates and provides a customized operation manual.

[1803] 1. System Configuration

[1804] Hardware

[1805] Information processing device: A terminal operated by a user, including computers, smartphones, tablets, etc.

[1806] Camera and microphone: Input devices for recording user actions, facial expressions, and voice tones in real time.

[1807] Server: Back-end infrastructure for analyzing operation data and emotion data and generating manuals.

[1808] software

[1809] OpenCV: A library for capturing and processing video and image data.

[1810] Google Cloud Speech-to-Text API: A service for converting voice data into text data.

[1811] TensorFlow: A machine learning library for analyzing emotions from facial expressions and voice.

[1812] Django Framework: A server-side framework for generating and serving operation manuals.

[1813] JavaScript: A front-end technology for displaying manuals on information processing devices.

[1814] 2. System Functions

[1815] Recording user actions

[1816] When a user starts a specific task on the information processing device, the device records the user's actions in real time. The recordings are in the following format:

[1817] Video recording: Use the camera to capture the user's operation screen and save it as video data.

[1818] Audio recording: Records the user's voice and environmental sounds through microphone input and saves them as audio data.

[1819] Emotion recording: Uses a camera and microphone to capture the user's facial expressions and vocal tone, which are then analyzed by the emotion engine.

[1820] Data transmission and analysis

[1821] Once recording begins, the device will periodically generate data packets and send them to the server. The data packets contain video, audio, text, and emotion data. The server will analyze the data and extract key points. The complete analysis process includes the following steps:

[1822] Generative AI video and audio analysis

[1823] Natural Language Processing (NLP)

[1824] Emotion analysis

[1825] Manual generation and provision

[1826] Based on the analysis results, the server integrates additional data related to user operations and generates an intuitive and easy-to-understand operation manual. The generated manual includes the following elements:

[1827] Video Clip: A short video clip showing the procedure

[1828] Text Description: Detailed instructions related to the procedure

[1829] Sentiment analysis results: Data that visually indicates the user's emotional state

[1830] The generated manual is finally sent to the terminal, and the user can refer to this manual as they work.

[1831] 3. Specific Examples

[1832] For example, if a user records the steps of purchasing a product on an online shopping site, the device records the entire purchasing process as video, audio, and emotional data. The server analyzes this data to extract important steps and the user's emotional state during the process. It then integrates additional help information and product reviews related to the purchasing process and generates a custom operation guide that includes them.

[1833] Example prompts for generative AI models

[1834] "On your online store, walk users through the steps of adding an item to their cart and visually demonstrate points of frustration."

[1835] As described above, this system can record and analyze the user's operations and their emotional state in detail, and generate and provide optimal operation guidance.

[1836] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1837] Step 1:

[1838] When a user starts operating an information processing device, the device records the user's operations in real time. At this point, the device's camera and microphone are activated, capturing the user's screen operation as a video and recording the user's voice input. The input data is the user's screen capture video and voice data, which are saved as recorded data.

[1839] Step 2:

[1840] The device uses an emotion engine to analyze the user's facial expressions and voice tone in real time to generate emotion data. Specifically, an emotion recognition model using TensorFlow predicts the user's emotional state (e.g., stress, joy, neutral, etc.) from their facial expressions and voice. This emotion data is stored along with the video and audio data. The input data is the user's facial expressions and voice, and the output data is the analyzed emotional state.

[1841] Step 3:

[1842] After recording for a certain period of time or for each specific operation, the device sends these data packets to the server. The data packets include video, audio, text, and emotion data. The input data is the recorded data packets, and the output data is the data sent to the server. Specifically, the device secures a stable network connection and uploads the data to the server using protocols such as HTTP or WebSocket.

[1843] Step 4:

[1844] The server analyzes the received data packets and begins processing various data. First, it uses generative AI to extract key operation points from the video and audio data. For example, it identifies the icons the user clicked and the text fields they entered. The input data is the video and audio data, and the output data is the analyzed key points. Specifically, it runs a generative AI model (for example, YOLO or DeepSpeech) to detect key operations.

[1845] Step 5:

[1846] The server then analyzes the text data using natural language processing (NLP). This clarifies the intent of the user's input text or command and identifies operational procedures and related background information. The input data is the text data, and the output data is the analyzed operational procedures and background information. Specifically, the server performs text analysis using an NLP engine (e.g., spaCy or BERT).

[1847] Step 6:

[1848] The server analyzes the emotion data provided by the emotion engine and identifies the user's emotional state at each operation point. The input data is emotion data, and the output data is the emotional state corresponding to each operation point. Specifically, the server maps the emotional state at each operation step from the time-series data.

[1849] Step 7:

[1850] Based on the analysis results, the server collects and integrates additional data related to the operation procedures, such as help documents and screenshots related to the operations. The input data is the analyzed operation procedures and emotion data, and the output data is a draft of the integrated operation manual. Specific operations include the process of retrieving related information from databases and external APIs.

[1851] Step 8:

[1852] The server generates a detailed operating manual based on the integrated data. The generated manual includes video clips, text descriptions, additional related information, and sentiment analysis results. The input data is a draft of the integrated operating manual, and the output data is the final generated operating manual. Specifically, the manual document is automatically generated using a template engine.

[1853] Step 9:

[1854] Finally, the server sends the generated operation manual to the terminal, making it accessible to the user. The input data is the finally generated operation manual, and the output data is the data sent to the terminal. Specific operations include providing information to the user through HTTP responses and updating the database.

[1855] As described above, the input data is analyzed and processed at each processing step, and ultimately a detailed custom operation manual is generated and provided to assist the user in their operations.

[1856] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1857] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1858] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1859] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1860] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1861] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1862] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1863] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, motorcycles, and other devices, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1864] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1865] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1866] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1867] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1868] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1869] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1870] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1871] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1872] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1873] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1874] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1875] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1876] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1877] The following is further disclosed regarding the above embodiment.

[1878] (Claim 1)

[1879] means for recording in real time operations performed by a user on an information processing device;

[1880] A means for analyzing the recorded operation data using artificial intelligence to extract important points;

[1881] A means of integrating the extracted key points with additional relevant data to generate a manual;

[1882] means for providing the generated manual on an information processing device;

[1883] A system including:

[1884] (Claim 2)

[1885] The system according to claim 1, which records user operations as video.

[1886] (Claim 3)

[1887] 10. The system of claim 1, which uses generative artificial intelligence to extract key points from operational data.

[1888] "Example 1"

[1889] (Claim 1)

[1890] means for recording in real time operations performed by a user on an information processing device;

[1891] A means for saving the recorded operation data in video, audio, and text formats;

[1892] A means of analyzing the recorded operation data using a generative AI model and extracting important points;

[1893] A means of integrating the extracted key points with additional relevant information to generate a manual;

[1894] means for providing the generated manual on an information processing device;

[1895] A system including:

[1896] (Claim 2)

[1897] 2. The system according to claim 1, wherein the user's operations are recorded as video, audio, or text.

[1898] (Claim 3)

[1899] 10. The system of claim 1, wherein a generative AI model is used to extract key points from operational data.

[1900] "Application Example 1"

[1901] (Claim 1)

[1902] means for recording in real time operations performed by a user on an information processing device;

[1903] A means for analyzing the recorded operation data using artificial intelligence to extract important points;

[1904] A means of integrating the extracted key points with additional relevant data to generate a manual;

[1905] means for providing the generated manual on an information processing device;

[1906] It is aimed at robot operations in factories, and provides a means to record and view the operating procedures via wearable devices such as smart glasses and head-mounted displays.

[1907] A system including:

[1908] (Claim 2)

[1909] 2. The system according to claim 1, wherein the system records user operations as video.

[1910] (Claim 3)

[1911] 10. The system of claim 1, which uses generative artificial intelligence to extract key points from operational data.

[1912] "Example 2: Combining Emotion Engines"

[1913] (Claim 1)

[1914] means for recording in real time operations performed by a user on an information processing device;

[1915] means for classifying the recorded operation data as video, audio, text, and emotion data;

[1916] A means for analyzing the classified operation data using a generating artificial intelligence to extract important points and the user's emotional state;

[1917] A means of integrating the extracted key points, emotional state, and additional relevant data to generate a manual; and

[1918] means for providing the generated manual on an information processing device;

[1919] A system including:

[1920] (Claim 2)

[1921] 2. The system according to claim 1, wherein the user's operations are recorded as video, and the user's emotional state is also recorded.

[1922] (Claim 3)

[1923] 10. The system of claim 1, which uses generative artificial intelligence to extract key points and the user's emotional state from the operation data and emotion data.

[1924] "Application example 2 when combining emotion engines"

[1925] (Claim 1)

[1926] means for recording in real time operations performed by a user on an information processing device;

[1927] A means for analyzing the recorded operation data using artificial intelligence to extract important points;

[1928] means for analyzing a user's emotional state using an emotion engine;

[1929] a means for integrating the extracted key points with additional relevant data and sentiment data to generate a manual;

[1930] means for providing the generated manual on an information processing device;

[1931] A system including:

[1932] (Claim 2)

[1933] 2. The system according to claim 1, wherein the system records user operations as video.

[1934] (Claim 3)

[1935] 10. The system of claim 1, wherein generative artificial intelligence is used to extract key points from the operational data and emotion data. [Explanation of symbols]

[1936] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for recording in real time operations performed by a user on an information processing device; A means for analyzing the recorded operation data using artificial intelligence to extract important points; A means of integrating the extracted key points with additional relevant data to generate a manual; means for providing the generated manual on an information processing device; A system including:

2. 2. The system according to claim 1, wherein the user's operations are recorded as video.

3. 10. The system of claim 1, wherein generative artificial intelligence is used to extract key points from operational data.

Images