system

Abstract

We provide the system. [Solution] A means connected to a business processing device for collecting user operation record data and screen interface update history, A means for analyzing the collected record data and update history and extracting the operating procedure, A means for generating documents based on extracted operating procedures by applying natural language generation technology, Means for acquiring image data related to the said document and integrating it into the said document, A means of receiving user feedback and revising the aforementioned document, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, the method including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In recent years, improvements and updates to business processing systems have been progressing rapidly, and manuals documenting their operation procedures and function descriptions also need to be frequently updated. However, conventional manual creation requires a great deal of time and human resources, and it is not easy to accurately grasp the changed operation procedures. As a result, there are problems such as a decline in the quality of the manuals and an increase in support requests due to users' mistakes in operation methods. The object of the present invention is to solve such problems and provide a method for efficiently providing high-quality support documents.

Means for Solving the Problems

[0005] This invention provides a system that automatically collects user operation record data and screen interface update history from a business processing device, analyzes this data, and extracts new operating procedures. Furthermore, this system documents the extracted procedures using natural language generation technology and integrates image captures to create a visually easy-to-understand manual. Because it has the ability to incorporate user feedback, the document content can be continuously improved, ensuring that the latest operating information is always provided.

[0006] A "business processing system" is a computer system used by companies and individuals to carry out their work, and is a device equipped with functions to streamline the progress of those tasks.

[0007] "User operation record data" refers to data that includes information such as the specific operations performed by a user when operating a business processing device, the date and time, and the function being operated.

[0008] "Screen interface update history" refers to a record of changes to the user interface of a business processing device, and is data that includes information on changed elements and update history.

[0009] "Natural language generation technology" is a technology for automatically generating natural-sounding text that humans can read, giving computers the ability to construct meaningful sentences.

[0010] "Image data" refers to a collection of digital information used for display as visual information, and is typically recorded as diagrams or photographs.

[0011] "Feedback" refers to information provided by users regarding their opinions and suggestions for improvement, including their evaluation of a product or service. [Brief explanation of the drawing]

[0012] [Figure 1]This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2, which combines an emotion engine. [Modes for carrying out the invention]

[0013] Hereinafter, an example of an embodiment of the system relating to the technology of this disclosure will be described with reference to the attached drawings.

[0014] First, the terms used in the following description will be explained.

[0015] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0016] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0017] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0018] In the following embodiments, the numbered communication I / F (Interface) is an interface including a communication processor and an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark), etc.

[0019] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0020] [First Embodiment]

[0021] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0022] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0023] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0024] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0025] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0026] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0027] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0028] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0029] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0030] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0031] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0032] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0033] The system of the present invention is centered around a server connected to a business processing device, and automatically collects user operation logs and interface update history, and generates and updates operation manuals based on them. This system can quickly reflect changes in procedures that occur when companies or individuals perform their work, and ensures that users always have access to the latest operating procedures.

[0034] Specifically, the server periodically retrieves operation logs from the business processing unit. This includes detailed information such as which user performed which operation and which screens were displayed. Furthermore, by obtaining the update history of the screen interface, it is possible to understand which functions have been changed.

[0035] The server then analyzes the collected data to extract typical operating procedures and new operating methods. Using natural language generation technology, these procedures are documented in an easy-to-understand format to create a manual. This manual also integrates relevant screen captures, making it easier for users to visually understand the procedures.

[0036] Users can view this manual through their devices and provide feedback, which will help continuously improve the manual's content. This feedback is analyzed by the server, and the manual is revised as needed to provide better operational support.

[0037] As a concrete example, consider the case where an audit function is added. The server automatically extracts the operational steps for this new function and updates the manual in the format of "Step 1: Open the settings menu" and "Step 2: Select the audit settings." This allows users to quickly adapt when new functions are introduced, supporting smooth business operations.

[0038] The following describes the processing flow.

[0039] Step 1:

[0040] The server automatically collects user operation logs and screen interface update history for the business processing unit at regular intervals. This includes the date and time of the operation, the user ID, and details of the operation performed. The collected data is stored in a database on the server.

[0041] Step 2:

[0042] The server analyzes the collected operation logs and update history. Here, it extracts frequently used operation patterns and procedures related to newly added or modified features. This analysis allows for understanding trends in how each feature is being used.

[0043] Step 3:

[0044] The server uses natural language generation technology based on the analysis results to create a manual document that includes new operating procedures. This document will describe the operating procedures in easy-to-understand language and will include relevant screen captures and videos.

[0045] Step 4:

[0046] The terminal displays the generated manual documents in a format accessible to the user. Users can access these manuals at any time through the terminal and check the necessary operating procedures for their work.

[0047] Step 5:

[0048] Users can provide feedback on any unclear points or areas for improvement they find while using the manual. This feedback is submitted from their device via a simple form.

[0049] Step 6:

[0050] The server receives and analyzes user feedback to identify areas that need improvement. If necessary, it revises the manual documents to improve their accuracy. The revised content is then provided to the user again via the terminal.

[0051] (Example 1)

[0052] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0053] Traditional business systems often suffered from delays in updating manuals in response to changes in operating procedures or the addition of new features, making it difficult for users to access the latest procedures. Furthermore, the lack of mechanisms to appropriately incorporate user feedback and consistently provide the most up-to-date and optimal operating procedures sometimes led to decreased operational efficiency.

[0054] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0055] In this invention, the server is connected to a business processing device and includes means for collecting user operation logs and display interface update history, means for analyzing the collected operation logs and update history to extract typical operation procedures, and means for applying a generation AI model to create documents based on the extracted operation procedures. This enables quick response to changes in operation procedures and the addition of new functions, and allows users to always access the latest and most visually easy-to-understand operation manual.

[0056] A "business processing device" is an electronic device used to perform operations necessary for carrying out business, and it is a device that generates user operation logs and screen interface update history.

[0057] "User" refers to an individual or organization that uses the business processing device to perform business operations, and is the entity that directly utilizes the information provided by the system.

[0058] An "operation log" is an electronic record of a series of operations performed by a user on a business processing device, and includes data such as the date and time and the details of the operations performed.

[0059] "Display interface" refers to the overall structure and elements on a screen that a business processing device uses to provide information to a user, enabling interaction with the user.

[0060] "Update history" refers to data that shows the history of changes and revisions to the display interface and the functions of the business processing device, and it holds information about past changes to the system.

[0061] A "generative AI model" is a mathematical model that uses artificial intelligence to generate natural language and data, and has the function of creating a desired output for a specific input.

[0062] A "typical operating procedure" refers to a series of frequently performed steps extracted from user operation logs, representing an optimized method of operation for smooth business execution.

[0063] A "document" refers to a collection of information generated based on extracted operating procedures, and is provided in a format that users can understand, such as an instruction manual or guide.

[0064] "Visual data" refers to digital information such as images and diagrams that users can visually refer to, and it plays a role in complementing operating procedures when integrated into documents.

[0065] The system of the present invention is centered around a server connected to a business processing device, and automatically collects user operation logs and display interface update history, and generates and updates operation manuals based on them.

[0066] The server periodically retrieves operation logs from the business processing unit. These operation logs contain detailed information such as which user performed which operation and which screen was displayed. The collected operation logs and update history are stored in a database on the server. The server then uses machine learning algorithms to analyze the collected data and extract typical operation procedures. This analysis utilizes Python's data analysis library Pandas and pattern recognition techniques.

[0067] Based on the extracted operating procedures, the server uses a generative AI model to create an operation manual. The generative AI model employed is an artificial intelligence model widely used to realize natural language generation technology. In this process, the operating procedures are documented in a user-friendly format, and related video data is also integrated.

[0068] Users can view operation manuals generated through their terminals. Furthermore, users can provide feedback on the manuals, which are then analyzed by the server, and the manuals are revised as needed. This process ensures users always have access to the latest information, improving work efficiency.

[0069] As a concrete example, consider the case where a new auditing function is added. The server automatically extracts the operational steps related to this new function and updates the manual with steps such as "Open the settings menu" and "Select audit settings." Users can then use this manual to smoothly operate the new function.

[0070] An example of a prompt message is as follows: "Please provide instructions for an AI program that automatically updates the manual so that users can easily understand how to use the system when new features are added to an enterprise system. Specifically, please provide a clear explanation, including the steps to take when a new auditing function is added." Based on this prompt, the AI ​​model generates an appropriate operation manual.

[0071] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0072] Step 1:

[0073] The server connects to the business processing unit and retrieves user operation logs and display interface update history. Inputs include real-time operation data and update information sent from the business processing unit. This information is sent to the server via a communication protocol and stored in a database. Specifically, it uses a particular API to retrieve data and organize log information in storage.

[0074] Step 2:

[0075] The server analyzes the collected operation logs and update history. The input for this step is the operation logs and update history stored in the database. Based on this data, the server applies machine learning algorithms to clean the data and extract frequently occurring operation patterns. As output, the server generates a list of the extracted typical operation procedures. Specifically, it constructs a dataframe using Python's Pandas and analyzes the frequency of operations.

[0076] Step 3:

[0077] The server utilizes a generative AI model to generate an operation manual based on the analysis results. The input is a list of typical operating procedures. The server uses natural language generation technology to convert the procedures into an easy-to-understand document format. The output is an operation manual in a user-friendly format. Specifically, prompts are input to the generative AI model, and the generated text is saved in HTML or PDF format.

[0078] Step 4:

[0079] Users view the operation manual generated through the terminal. The terminal downloads the latest manual from the server and displays it on the user interface. Specifically, users can access the manual using a web browser and check the procedures.

[0080] Step 5:

[0081] Users provide feedback on the operation manual. Input includes user opinions and suggestions for improvement. The terminal sends this information to the server via a dedicated feedback form. The server analyzes the received feedback and generates new improvement suggestions. The output is a revised operation manual based on the feedback results. The server analyzes this information and incorporates it into the next manual generation.

[0082] (Application Example 1)

[0083] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0084] In business operations, frequent updates to operating procedures and systems can make it difficult for employees to keep up with the latest procedures, leading to decreased work efficiency. Furthermore, existing manuals are static, making it difficult to quickly adapt to screen changes or new features, resulting in longer learning and adaptation times for users.

[0085] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0086] In this invention, the server is connected to a business processing terminal and includes means for collecting user operation record data and screen interface update information, means for analyzing the collected record data and update information and extracting usage procedures, and means for displaying the generated document on an augmented reality display device and providing it to the user. As a result, users can always perform their tasks based on the latest information, enabling improved work efficiency and enhanced adaptability.

[0087] A "business processing terminal" is a computer device used by users when performing business tasks, and is capable of recording operation logs and interface update information.

[0088] "User" refers to an individual or organization that operates a business processing terminal and performs business through the system.

[0089] "Recorded data" refers to data that shows detailed information about the operations performed by the user on the business processing terminal.

[0090] "Screen interface update information" refers to data about the history of changes to the screen configuration and functions within the system.

[0091] "Analysis" is the process of analyzing collected data and deriving meaning and patterns from it.

[0092] "Usage instructions" refers to a procedure that includes step-by-step operating instructions for performing a task.

[0093] "Natural language generation technology" is a technology that converts machine-generated data into a form of natural language that is easy for the reader to understand.

[0094] "Image information" refers to visual data related to a document, used to visually illustrate the usage procedure.

[0095] "Evaluation" refers to feedback provided by users, which is information used to improve the system.

[0096] An "augmented reality display device" is a device that overlays digital information onto physical space and is used to provide visual support for work.

[0097] This invention is a system that utilizes a business processing terminal and a server to collect and analyze user operation data and screen interface update information, and to display the latest operating procedures. The hardware used includes a business processing terminal, a server, and an augmented reality display device (e.g., smart glasses). The software includes a data analysis platform and a natural language generation tool (e.g., OpenAI® GPT-3®).

[0098] The server instantly acquires operation log data and screen interface update information from business processing terminals via the network. This data is processed through the server's analysis platform to extract frequently used operation patterns and procedures for using newly added functions. Using natural language generation technology, these procedures are converted into user-friendly document formats, and relevant image information is added.

[0099] The generated documents are sent to an augmented reality display device, allowing users to perform their tasks while visually confirming the latest operating procedures in real time. For example, when new equipment is introduced, the usage procedures are automatically updated, and employees can understand "what to do next" in real time through smart glasses.

[0100] Using a generative AI model, natural language generation is performed with prompts like the following:

[0101] "We need new manual content. Please generate an easy-to-understand operation guide for manufacturing line workers based on the most efficient procedures collected from operation logs. Please include specific steps and relevant screen captures."

[0102] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0103] Step 1:

[0104] The server instantly acquires operation log data and screen interface update information from business processing terminals via the network. Inputs are operation logs and update history from the terminals, and output is a dataset stored on the server. Specifically, monitoring software on the terminal captures logs and sends them to the server.

[0105] Step 2:

[0106] The system analyzes the dataset collected by the server to extract frequently used operation patterns and modified functions. The input is the accumulated operation logs, and the output is a list of analyzed usage procedures. As a data processing step, data mining techniques are used to statistically analyze the frequency of operations and extract patterns.

[0107] Step 3:

[0108] The server uses a generative AI model to convert the analyzed usage instructions into a natural language-based document format. The input is a list of operating procedures, and the output is an operation manual in text format. Specifically, the process involves providing prompts to the generative AI model, which then outputs an easily understandable document based on those prompts.

[0109] Step 4:

[0110] The server integrates image information related to the document, visually enhancing it. The input is the generated operation manual and associated image data, and the output is a comprehensive operation guide incorporating the images. Specifically, the system searches for relevant images from a database and automatically places them within the document.

[0111] Step 5:

[0112] The server sends a generated comprehensive operation guide to an augmented reality display device, allowing the user to view the latest instructions in real time. The input is an image-integrated operation guide, and the output is a guide displayed in the user's field of view. Specifically, wireless communication technology is used to transmit data to the display device, enabling the user to deepen their visual understanding.

[0113] Step 6:

[0114] The user performs tasks based on the operation guide and then sends their evaluation of the guide as feedback to the server. The input is the user's evaluation information, and the output is stored on the server as feedback data. Specifically, the user evaluates the quality of the displayed guide, and the evaluation system analyzes this information.

[0115] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0116] This invention combines a system that uses a server connected to a business processing device to collect user operation records and screen interface update information, and generates operation manuals based on this information, with an emotion engine. This system can recognize the user's emotional state and adaptively adjust the content of the generated documents accordingly.

[0117] Specifically, the server automatically collects user operation logs and screen interface update history from the business processing unit. This data is analyzed to understand how users are using the system and what changes have been made.

[0118] Furthermore, this invention incorporates an emotion engine that analyzes the user's facial expression data and input information to recognize their emotional state. Based on this, the server creates an operation manual in a format that is easy for the user to understand. For example, if the system detects that the user is confused, it can adjust the manual to provide more detailed explanations or additional guidance.

[0119] The terminal displays the generated manual in a user-friendly format. Users can check the operating procedures through this manual and also contribute to system improvements through feedback.

[0120] To give a specific example, if a user expresses dissatisfaction while using a certain function, the server detects this emotion and generates a manual that clearly restructures the procedure for that function. In this process, the server receives negative feedback from the user and suggests appropriate improvements, thereby enhancing the user experience. As a result, users can always use the system comfortably, and it also contributes to reducing support requests.

[0121] The following describes the processing flow.

[0122] Step 1:

[0123] The server automatically collects user operation logs and screen interface update history from the business processing unit. This data will include information about all operations performed by the user and any changes to the interface.

[0124] Step 2:

[0125] The server analyzes the collected data to extract frequently used user patterns and operating procedures related to newly introduced features. This analysis identifies characteristics such as the frequency of frequently occurring operations and the usage frequency of new features.

[0126] Step 3:

[0127] The server uses an emotion engine to analyze the user's emotions during operation. This analysis is based on facial expression data captured by the camera, as well as operation data such as keystrokes and mouse movements, and evaluates the user's stress level and satisfaction level.

[0128] Step 4:

[0129] The server adjusts the content of the generated operation manual based on the recognized emotional state. For example, if a negative emotion is detected, the procedure is made simpler and additional explanations are added to help the user understand it better.

[0130] Step 5:

[0131] The terminal displays the generated operation manual on the user's screen. The user can review the operating procedures through this manual and revisit them if necessary.

[0132] Step 6:

[0133] Users provide feedback on the manual's content. This feedback includes their understanding of the operation and their opinions on the manual, and is sent from the terminal to the server.

[0134] Step 7:

[0135] The server analyzes user feedback and revises the manual as needed. By reviewing and updating the manual content to address the issues indicated by the feedback, a better user experience is provided.

[0136] (Example 2)

[0137] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0138] Conventional user manual generation systems lack information that takes into account the user's emotional state, sometimes leading to confusion and frustration when using the system. This results in a degraded user experience and an increase in support requests. These problems need to be solved by providing customized support that takes into account the user's emotional state.

[0139] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0140] In this invention, the server is connected to a business processing device and includes means for collecting user operation record data and screen interface update history, means for analyzing user facial expression data and input information to recognize emotional states, and means for generating documents based on extracted operation procedures and emotional states by applying natural language generation technology. This makes it possible to quickly and appropriately resolve operational problems faced by the user and improve the user experience.

[0141] A "business processing system" is a computer system used to efficiently process and manage the business operations of a company or organization.

[0142] "User operation log data" refers to information that stores a log of a series of operations performed by a user on the system.

[0143] "Screen interface update history" refers to data that records the history of changes to the screen displayed to the user.

[0144] "Analysis" is the process of thoroughly examining data and information to clarify its structure and relationships.

[0145] An "operating procedure" is a set of steps or actions performed to complete a specific task.

[0146] "Emotional state" refers to information that indicates the user's psychological and emotional state.

[0147] "Facial expression data" refers to information captured from a user's facial expressions and is used to analyze their emotions.

[0148] "Natural language generation technology" is a technology that uses computers to automatically generate meaningful sentences in human language.

[0149] "Feedback" refers to user evaluations and opinions regarding a system or service.

[0150] This invention is a system that uses a server connected to a business processing device to collect user operation record data and screen interface update history, and generates an operation manual based on this data. The system incorporates an emotion engine that recognizes the user's emotional state and can adaptively adjust the content of the generated manual.

[0151] The server first automatically collects user operation record data and screen interface update history from the business processing unit. During this process, data is obtained in real time via an API through the client application. The collected data is stored in a database management system (e.g., MySQL® or PostgreSQL) and then analyzed using Python's pandas library, among others.

[0152] The server is equipped with an emotion engine that recognizes the user's emotional state by analyzing facial expression data and input information. This process utilizes video data acquired from a camera and performs analysis using facial recognition technology (e.g., OpenCV or Amazon Rekognition). The recognized emotional state is then considered during the document generation process.

[0153] Next, the server utilizes a generative AI model (e.g., GPT-3 or BERT) to generate an operation manual using natural language generation technology based on the analysis results. This operation manual is composed of appropriate content and tone according to the user's emotional state and the functions being used. For example, by inputting a prompt sentence such as "The user is confused about using the new function, please provide an easy-to-understand guide" into the generative AI model, a specific and user-friendly manual is generated.

[0154] The generated manual is displayed on the terminal and made available to users in an easily accessible format. The terminal presents the manual in PDF or interactive HTML format using a web browser or dedicated application.

[0155] Users can use this manual to check operating procedures and contribute to system improvements through feedback. This feedback information is sent to the server and will be applied to future manual generation processes, thereby improving the overall user experience of the system.

[0156] In this way, the system aims to effectively resolve operational problems faced by users and significantly improve the user experience.

[0157] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0158] Step 1:

[0159] The server collects user operation record data and screen interface update history from the business processing unit. It uses an API via the client application as input to retrieve the record data and update history in real time. This data is stored in a database management system for subsequent analysis. Specifically, the data is stored in MySQL or PostgreSQL.

[0160] Step 2:

[0161] The server analyzes the collected record data and update history. The input here is the record data and update history obtained in step 1. The data is processed using an analysis tool (e.g., the pandas library in Python) to extract frequently performed operations and user usage patterns. The output is the operation patterns for use in the next process as analysis results. Specific actions include data cleaning, filtering, and statistical analysis.

[0162] Step 3:

[0163] The server uses an emotion engine to recognize the user's emotional state. User facial expression data and input information are used as input. This data is acquired from a camera or input device and analyzed by facial recognition software (e.g., OpenCV or Amazon Rekognition). The output is information indicating the user's current emotional state, which is considered when generating manuals. Specific operations include facial expression recognition and emotion categorization.

[0164] Step 4:

[0165] The server uses a generative AI model to generate an operation manual based on the analysis results and emotional state. The inputs are the operation patterns extracted in step 2 and the emotional state obtained in step 3. Using natural language generation technology, the server creates a manual tailored to the user based on this information. The output is an operation manual that is adapted to the user's situation. Specifically, the output is text generated using GPT-3 or BERT.

[0166] Step 5:

[0167] The terminal displays the user the operation manual received from the server. The input is the operation manual generated in step 4. The output is a document formatted for user viewing and can be displayed in a web browser or dedicated application. Specific operations include displaying the manual in PDF or HTML format.

[0168] Step 6:

[0169] Users refer to the operation manual and provide feedback on the system. Input consists of user evaluations of the operations performed and their results. Output is user feedback information sent to the server. Specific actions include submitting feedback forms and completing questionnaires. This information is used to improve the system.

[0170] (Application Example 2)

[0171] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0172] Providing appropriate guidance and support to customers who are confused or lost in physical stores can be challenging. While store staff need to directly assist in such situations, staff shortages and a lack of specialized knowledge can prevent them from providing adequate support. Furthermore, there is a need to provide personalized support tailored to each customer's feelings and level of understanding. Therefore, improving customer satisfaction and streamlining store operations are key challenges.

[0173] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0174] In this invention, the server is connected to a business processing device and includes means for collecting user operation record information and screen interface update information, means for analyzing the collected record information and update information and extracting operation procedures, and means for applying natural language generation technology to generate documents based on the extracted operation procedures. This makes it possible to analyze the customer's emotional state in real time and automatically provide appropriate guidance and product suggestions according to that emotion.

[0175] A "business processing device" is a device used to collect user operation records and screen interface update information.

[0176] "Recorded information" refers to data obtained when a user performs an operation, and it shows the history of that operation.

[0177] "Update information" refers to the history of changes to the screen interface, tracking how users interacted with the interface.

[0178] "Analysis means" refers to techniques for analyzing recorded information and updated information to extract meaningful patterns and operating procedures.

[0179] "Natural language generation technology" is a technology that automatically creates documents that are easy for humans to understand based on extracted operating procedures.

[0180] "Means for recognizing emotional states" refers to technologies that analyze a user's facial expression data and input information to identify their emotions.

[0181] "Means of receiving feedback" refers to the process of receiving opinions and evaluations from users and improving the generated documents.

[0182] "Adaptive adjustment mechanisms" refer to functions that modify generated documents and guidance content based on recognized user emotions to provide more effective support.

[0183] A system implementing this invention can be realized with the following main components: The server is connected to the business processing device and has the function of collecting user operation record information and screen interface update information. This information is used to track the user's operation history and analyze what operations were performed.

[0184] Next, the server analyzes the collected record and update information to extract operation procedures. Advanced data analysis techniques are used for the analysis to identify frequently occurring operation patterns. Furthermore, emotion recognition technology is incorporated to analyze facial expression data and input information to recognize the user's emotional state. This utilizes software known as emotion analysis libraries (e.g., Microsoft® Azure® Face API, Google® Cloud Vision API, etc.).

[0185] The documents displayed by the device are created using natural language generation technology based on extracted operating procedures. The natural language generation model translates the operating procedures into clear and easily understandable language for humans. Furthermore, the device receives user feedback and revises the generated documents to improve the system. By adaptively adjusting the content of the documents according to the user's emotional state, it is possible to provide more effective guidance.

[0186] As a concrete example, in a physical store, if a customer shows a confused expression, this system recognizes the customer's emotions in real time and provides a detailed product description based on the customer's emotional state. This can improve customer satisfaction and reduce the burden on store staff.

[0187] An example of a prompt message is: "Send to the generating AI model: 'Adjust the guidance appropriately to match the product category selected by the customer, and display promotions for related products when the customer's facial expression is smiling.'"

[0188] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0189] Step 1:

[0190] The server connects to the business processing unit and collects user operation records and screen interface update information in real time. It receives operation log data and update history from the business processing unit as input and stores this data in a database. This makes it possible to systematically record what operations were performed.

[0191] Step 2:

[0192] The server analyzes the collected record information and update information. It takes operation logs and update history stored in the database as input and extracts operation patterns using advanced data analysis algorithms. As output, it identifies frequently performed operation procedures and abnormal operation patterns and formats them into a report format. This allows for an understanding of operation trends and problems.

[0193] Step 3:

[0194] The server analyzes facial expression data acquired from the device's camera and user input information to recognize the user's emotional state. It receives real-time facial image and text input as input and generates emotional data using an emotion analysis library (e.g., Microsoft Azure Face API). As output, it identifies the user's emotional state (e.g., confused, satisfied) and outputs it as a digital signal. This allows the server to determine the next action based on the user's emotions.

[0195] Step 4:

[0196] The server uses natural language generation technology to create documents based on analyzed operating procedures and sentiment data. It receives identified operating procedures and the user's emotional state as input, and utilizes a generative AI model to generate customized guidance documents. The output is easy-to-understand documents and explanations, which are then sent to the user's terminal. This allows users to receive support tailored to their specific situation.

[0197] Step 5:

[0198] The terminal displays documents sent from the server to the user and also receives feedback. It receives the displayed document and user feedback as input, and sends the collected feedback to the server as output. This feedback information can then be used to further improve the document content.

[0199] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0200] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0201] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0202] [Second Embodiment]

[0203] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0204] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0205] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0206] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0207] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0208] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0209] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0210] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0211] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0212] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0213] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0214] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0215] The system of the present invention is centered around a server connected to a business processing device, and automatically collects user operation logs and interface update history, and generates and updates operation manuals based on them. This system can quickly reflect changes in procedures that occur when companies or individuals perform their work, and ensures that users always have access to the latest operating procedures.

[0216] Specifically, the server periodically retrieves operation logs from the business processing unit. This includes detailed information such as which user performed which operation and which screens were displayed. Furthermore, by obtaining the update history of the screen interface, it is possible to understand which functions have been changed.

[0217] The server then analyzes the collected data to extract typical operating procedures and new operating methods. Using natural language generation technology, these procedures are documented in an easy-to-understand format to create a manual. This manual also integrates relevant screen captures, making it easier for users to visually understand the procedures.

[0218] Users can view this manual through their devices and provide feedback, which will help continuously improve the manual's content. This feedback is analyzed by the server, and the manual is revised as needed to provide better operational support.

[0219] As a concrete example, consider the case where an audit function is added. The server automatically extracts the operational steps for this new function and updates the manual in the format of "Step 1: Open the settings menu" and "Step 2: Select the audit settings." This allows users to quickly adapt when new functions are introduced, supporting smooth business operations.

[0220] The following describes the processing flow.

[0221] Step 1:

[0222] The server automatically collects user operation logs and screen interface update history for the business processing unit at regular intervals. This includes the date and time of the operation, the user ID, and details of the operation performed. The collected data is stored in a database on the server.

[0223] Step 2:

[0224] The server analyzes the collected operation logs and update history. Here, it extracts frequently used operation patterns and procedures related to newly added or modified features. This analysis allows for understanding trends in how each feature is being used.

[0225] Step 3:

[0226] The server uses natural language generation technology based on the analysis results to create a manual document that includes new operating procedures. This document will describe the operating procedures in easy-to-understand language and will include relevant screen captures and videos.

[0227] Step 4:

[0228] The terminal displays the generated manual documents in a format accessible to the user. Users can access these manuals at any time through the terminal and check the necessary operating procedures for their work.

[0229] Step 5:

[0230] Users can provide feedback on any unclear points or areas for improvement they find while using the manual. This feedback is submitted from their device via a simple form.

[0231] Step 6:

[0232] The server receives and analyzes user feedback to identify areas that need improvement. If necessary, it revises the manual documents to improve their accuracy. The revised content is then provided to the user again via the terminal.

[0233] (Example 1)

[0234] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0235] Traditional business systems often suffered from delays in updating manuals in response to changes in operating procedures or the addition of new features, making it difficult for users to access the latest procedures. Furthermore, the lack of mechanisms to appropriately incorporate user feedback and consistently provide the most up-to-date and optimal operating procedures sometimes led to decreased operational efficiency.

[0236] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0237] In this invention, the server is connected to a business processing device and includes means for collecting user operation logs and display interface update history, means for analyzing the collected operation logs and update history to extract typical operation procedures, and means for applying a generation AI model to create documents based on the extracted operation procedures. This enables quick response to changes in operation procedures and the addition of new functions, and allows users to always access the latest and most visually easy-to-understand operation manual.

[0238] A "business processing device" is an electronic device used to perform operations necessary for carrying out business tasks, and it is a device that generates user operation logs and screen interface update history.

[0239] "User" refers to an individual or organization that uses the business processing device to perform business operations, and is the entity that directly utilizes the information provided by the system.

[0240] An "operation log" is an electronic record of a series of operations performed by a user on a business processing device, and includes data such as the date and time and the details of the operations performed.

[0241] "Display interface" refers to the overall structure and elements on a screen that a business processing device uses to provide information to a user, enabling interaction with the user.

[0242] "Update history" refers to data that shows the history of changes and revisions to the display interface and the functions of the business processing device, and it holds information about past changes to the system.

[0243] A "generative AI model" is a mathematical model that uses artificial intelligence to generate natural language and data, and has the function of creating a desired output for a specific input.

[0244] A "typical operating procedure" refers to a series of frequently performed steps extracted from user operation logs, representing an optimized method of operation for smooth business operations.

[0245] A "document" refers to a collection of information generated based on extracted operating procedures, and is provided in a format that users can understand, such as an instruction manual or guide.

[0246] "Visual data" refers to digital information such as images and diagrams that users can visually refer to, and it plays a role in complementing operating procedures when integrated into documents.

[0247] The system of the present invention is centered around a server connected to a business processing device, and automatically collects user operation logs and display interface update history, and generates and updates operation manuals based on them.

[0248] The server periodically retrieves operation logs from the business processing unit. These operation logs contain detailed information such as which user performed which operation and which screen was displayed. The collected operation logs and update history are stored in a database on the server. The server then uses machine learning algorithms to analyze the collected data and extract typical operation procedures. This analysis utilizes Python's data analysis library Pandas and pattern recognition techniques.

[0249] Based on the extracted operating procedures, the server uses a generative AI model to create an operation manual. The generative AI model employed is an artificial intelligence model widely used to realize natural language generation technology. In this process, the operating procedures are documented in a user-friendly format, and related video data is also integrated.

[0250] Users can view operation manuals generated through their terminals. Furthermore, users can provide feedback on the manuals, which are then analyzed by the server, and the manuals are revised as needed. This process ensures users always have access to the latest information, improving work efficiency.

[0251] As a concrete example, consider the case where a new auditing function is added. The server automatically extracts the operational steps related to this new function and updates the manual with steps such as "Open the settings menu" and "Select audit settings." Users can then use this manual to smoothly operate the new function.

[0252] An example of a prompt message is as follows: "Please provide instructions for an AI program that automatically updates the manual so that users can easily understand how to use the system when new features are added to an enterprise system. Specifically, please provide a clear explanation, including the steps to take when a new auditing function is added." Based on this prompt, the AI ​​model generates an appropriate operation manual.

[0253] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0254] Step 1:

[0255] The server connects to the business processing unit and retrieves user operation logs and display interface update history. Inputs include real-time operation data and update information sent from the business processing unit. This information is sent to the server via a communication protocol and stored in a database. Specifically, it uses a particular API to retrieve data and organize log information in storage.

[0256] Step 2:

[0257] The server analyzes the collected operation logs and update history. The input for this step is the operation logs and update history stored in the database. Based on this data, the server applies machine learning algorithms to clean the data and extract frequently occurring operation patterns. As output, the server generates a list of the extracted typical operation procedures. Specifically, it constructs a dataframe using Python's Pandas and analyzes the frequency of operations.

[0258] Step 3:

[0259] The server utilizes a generative AI model to generate an operation manual based on the analysis results. The input is a list of typical operating procedures. The server uses natural language generation technology to convert the procedures into an easy-to-understand document format. The output is an operation manual in a user-friendly format. Specifically, prompts are input to the generative AI model, and the generated text is saved in HTML or PDF format.

[0260] Step 4:

[0261] Users view the operation manual generated through the terminal. The terminal downloads the latest manual from the server and displays it on the user interface. Specifically, users can access the manual using a web browser and check the procedures.

[0262] Step 5:

[0263] Users provide feedback on the operation manual. Input includes user opinions and suggestions for improvement. The terminal sends this information to the server via a dedicated feedback form. The server analyzes the received feedback and generates new improvement suggestions. The output is a revised operation manual based on the feedback results. The server analyzes this information and incorporates it into the next manual generation.

[0264] (Application Example 1)

[0265] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0266] In business operations, frequent updates to operating procedures and systems can make it difficult for employees to keep up with the latest procedures, leading to decreased work efficiency. Furthermore, existing manuals are static, making it difficult to quickly adapt to screen changes or new features, resulting in longer learning and adaptation times for users.

[0267] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0268] In this invention, the server is connected to a business processing terminal and includes means for collecting user operation record data and screen interface update information, means for analyzing the collected record data and update information and extracting usage procedures, and means for displaying the generated document on an augmented reality display device and providing it to the user. As a result, users can always perform their tasks based on the latest information, enabling improved work efficiency and enhanced adaptability.

[0269] A "business processing terminal" is a computer device used by users when performing business tasks, and is capable of recording operation logs and interface update information.

[0270] "User" refers to an individual or organization that operates a business processing terminal and performs business through the system.

[0271] "Recorded data" refers to data that shows detailed information about the operations performed by the user on the business processing terminal.

[0272] "Screen interface update information" refers to data about the history of changes to the screen configuration and functions within the system.

[0273] "Analysis" is the process of analyzing collected data and deriving meaning and patterns from it.

[0274] "Usage instructions" refers to a procedure that includes step-by-step operating instructions for performing a task.

[0275] "Natural language generation technology" is a technology that converts machine-generated data into a form of natural language that is easy for the reader to understand.

[0276] "Image information" refers to visual data related to a document, used to visually illustrate the usage procedure.

[0277] "Evaluation" refers to feedback provided by users, which is information used to improve the system.

[0278] An "augmented reality display device" is a device that overlays digital information onto physical space and is used to provide visual support for work.

[0279] This invention is a system that utilizes a business processing terminal and a server to collect and analyze user operation data and screen interface update information, and to display the latest operating procedures. The hardware used includes a business processing terminal, a server, and an augmented reality display device (e.g., smart glasses). The software includes a data analysis platform and a natural language generation tool (e.g., OpenAI GPT-3).

[0280] The server immediately obtains the operation record data and the update information of the screen interface from the business processing terminal via the network. This data is processed through the server's analysis platform to extract frequently used operation patterns and the usage procedures of newly added functions. Using natural language generation technology, these procedures are converted into a document format that is easy for users to understand, and relevant image information is added.

[0281] The generated document is sent to the augmented reality display device, and the user can perform the business while visually checking the latest operation procedures displayed on the spot. For example, when new equipment is introduced, the usage procedure is automatically updated, and employees can understand "what to do next" in real time through smart glasses.

[0282] Using the generated AI model, natural language generation is performed with the following prompt text:

[0283] "New manual content is needed. Based on the most efficient procedures collected from the operation logs, please generate an easy-to-understand operation guide for manufacturing line workers. Include specific procedures and related screen captures."

[0284] The flow of the specific process in Application Example 1 will be described using FIG. 12.

[0285] Step 1:

[0286] The server immediately obtains the operation record data and the update information of the screen interface from the business processing terminal via the network. The input is the operation log and update history from the terminal, and the output is a data set accumulated on the server. As a specific operation, there is a process where the monitoring software on the terminal captures the log and sends it to the server.

[0287] Step 2:

[0288] The system analyzes the dataset collected by the server to extract frequently used operation patterns and modified functions. The input is the accumulated operation logs, and the output is a list of analyzed usage procedures. As a data processing step, data mining techniques are used to statistically analyze the frequency of operations and extract patterns.

[0289] Step 3:

[0290] The server uses a generative AI model to convert the analyzed usage instructions into a natural language-based document format. The input is a list of operating procedures, and the output is an operation manual in text format. Specifically, the process involves providing prompts to the generative AI model, which then outputs an easily understandable document based on those prompts.

[0291] Step 4:

[0292] The server integrates image information related to the document, visually enhancing it. The input is the generated operation manual and associated image data, and the output is a comprehensive operation guide incorporating the images. Specifically, the system searches for relevant images from a database and automatically places them within the document.

[0293] Step 5:

[0294] The server sends a generated comprehensive operation guide to an augmented reality display device, allowing the user to view the latest instructions in real time. The input is an image-integrated operation guide, and the output is a guide displayed in the user's field of view. Specifically, wireless communication technology is used to transmit data to the display device, enabling the user to deepen their visual understanding.

[0295] Step 6:

[0296] The user performs tasks based on the operation guide and then sends their evaluation of the guide as feedback to the server. The input is the user's evaluation information, and the output is stored on the server as feedback data. Specifically, the user evaluates the quality of the displayed guide, and the evaluation system analyzes this information.

[0297] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0298] This invention combines a system that uses a server connected to a business processing device to collect user operation records and screen interface update information, and generates operation manuals based on this information, with an emotion engine. This system can recognize the user's emotional state and adaptively adjust the content of the generated documents accordingly.

[0299] Specifically, the server automatically collects user operation logs and screen interface update history from the business processing unit. This data is analyzed to understand how users are using the system and what changes have been made.

[0300] Furthermore, this invention incorporates an emotion engine that analyzes the user's facial expression data and input information to recognize their emotional state. Based on this, the server creates an operation manual in a format that is easy for the user to understand. For example, if the system detects that the user is confused, it can adjust the manual to provide more detailed explanations or additional guidance.

[0301] The terminal displays the generated manual in a user-friendly format. Users can check the operating procedures through this manual and also contribute to system improvements through feedback.

[0302] Specifically, when a user shows dissatisfaction during the use of a certain function, the server detects this emotion and generates a manual that reorganizes the procedures related to that function in an easy-to-understand manner. In this process, by receiving the user's negative feedback and presenting appropriate improvement suggestions, the user experience is enhanced. As a result, the user can always use the system comfortably, which also contributes to reducing support requests.

[0303] The following describes the processing flow.

[0304] Step 1:

[0305] The server automatically collects the user's operation logs and the update history of the screen interface from the business processing device. This data will include all the operations performed by the user and information regarding changes to the interface.

[0306] Step 2:

[0307] The server analyzes the collected data and extracts the operation patterns frequently used by the user and the operation procedures related to newly introduced functions. In this analysis, features such as frequently occurring operations and the usage frequency of new functions are identified.

[0308] Step 3:

[0309] The server uses an emotion engine to analyze the emotions of the user during operation. This analysis is based on operation data such as facial expression data captured by a camera, keystrokes, and mouse movements, and evaluates the user's stress level and satisfaction.

[0310] Step 4:

[0311] The server adjusts the content of the generated operation manual based on the recognized emotional state. For example, when a negative emotion is detected, the procedures are made more concise and additional explanations are added to enable the user to deepen their understanding.

[0312] Step 5:

[0313] The terminal displays the generated operation manual on the user's screen. The user can review the operating procedures through this manual and revisit them if necessary.

[0314] Step 6:

[0315] Users provide feedback on the manual's content. This feedback includes their understanding of the operation and their opinions on the manual, and is sent from the terminal to the server.

[0316] Step 7:

[0317] The server analyzes user feedback and revises the manual as needed. By reviewing and updating the manual content to address the issues indicated by the feedback, a better user experience is provided.

[0318] (Example 2)

[0319] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0320] Conventional user manual generation systems lack information that takes into account the user's emotional state, sometimes leading to confusion and frustration when using the system. This results in a degraded user experience and an increase in support requests. These problems need to be solved by providing customized support that takes into account the user's emotional state.

[0321] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0322] In this invention, the server is connected to a business processing device and includes means for collecting user operation record data and screen interface update history, means for analyzing user facial expression data and input information to recognize emotional states, and means for generating documents based on extracted operation procedures and emotional states by applying natural language generation technology. This makes it possible to quickly and appropriately resolve operational problems faced by the user and improve the user experience.

[0323] A "business processing system" is a computer system used to efficiently process and manage the business operations of a company or organization.

[0324] "User operation log data" refers to information that stores a log of a series of operations performed by a user on the system.

[0325] "Screen interface update history" refers to data that records the history of changes to the screen displayed to the user.

[0326] "Analysis" is the process of thoroughly examining data and information to clarify its structure and relationships.

[0327] An "operating procedure" is a set of steps or actions performed to complete a specific task.

[0328] "Emotional state" refers to information that indicates the user's psychological and emotional state.

[0329] "Facial expression data" refers to information captured from a user's facial expressions and is used to analyze their emotions.

[0330] "Natural language generation technology" is a technology that uses computers to automatically generate meaningful sentences in human language.

[0331] "Feedback" refers to user evaluations and opinions regarding a system or service.

[0332] This invention is a system that uses a server connected to a business processing device to collect user operation record data and screen interface update history, and generates an operation manual based on this data. The system incorporates an emotion engine that recognizes the user's emotional state and can adaptively adjust the content of the generated manual.

[0333] The server first automatically collects user operation record data and screen interface update history from the business processing unit. During this process, data is obtained in real time via an API through the client application. The collected data is stored in a database management system (e.g., MySQL or PostgreSQL) and then analyzed using Python's pandas library, among others.

[0334] The server is equipped with an emotion engine that recognizes the user's emotional state by analyzing facial expression data and input information. This process utilizes video data acquired from a camera and performs analysis using facial recognition technology (e.g., OpenCV or Amazon Rekognition). The recognized emotional state is then considered during the document generation process.

[0335] Next, the server utilizes a generative AI model (e.g., GPT-3 or BERT) to generate an operation manual using natural language generation technology based on the analysis results. This operation manual is composed of appropriate content and tone according to the user's emotional state and the functions being used. For example, by inputting a prompt sentence such as "The user is confused about using the new function, please provide an easy-to-understand guide" into the generative AI model, a specific and user-friendly manual is generated.

[0336] The generated manual is displayed on the terminal and made available to users in an easily accessible format. The terminal presents the manual in PDF or interactive HTML format using a web browser or dedicated application.

[0337] Users can use this manual to check operating procedures and contribute to system improvements through feedback. This feedback information is sent to the server and will be applied to future manual generation processes, thereby improving the overall user experience of the system.

[0338] In this way, the system aims to effectively resolve operational problems faced by users and significantly improve the user experience.

[0339] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0340] Step 1:

[0341] The server collects user operation record data and screen interface update history from the business processing unit. It uses an API via the client application as input to retrieve the record data and update history in real time. This data is stored in a database management system for subsequent analysis. Specifically, the data is stored in MySQL or PostgreSQL.

[0342] Step 2:

[0343] The server analyzes the collected record data and update history. The input here is the record data and update history obtained in step 1. The data is processed using an analysis tool (e.g., the pandas library in Python) to extract frequently performed operations and user usage patterns. The output is the operation patterns for use in the next process as analysis results. Specific actions include data cleaning, filtering, and statistical analysis.

[0344] Step 3:

[0345] The server uses an emotion engine to recognize the user's emotional state. User facial expression data and input information are used as input. This data is acquired from a camera or input device and analyzed by facial recognition software (e.g., OpenCV or Amazon Rekognition). The output is information indicating the user's current emotional state, which is considered when generating manuals. Specific operations include facial expression recognition and emotion categorization.

[0346] Step 4:

[0347] The server uses a generative AI model to generate an operation manual based on the analysis results and emotional state. The inputs are the operation patterns extracted in step 2 and the emotional state obtained in step 3. Using natural language generation technology, the server creates a manual tailored to the user based on this information. The output is an operation manual that is adapted to the user's situation. Specifically, the output is text generated using GPT-3 or BERT.

[0348] Step 5:

[0349] The terminal displays the user the operation manual received from the server. The input is the operation manual generated in step 4. The output is a document formatted for user viewing and can be displayed in a web browser or dedicated application. Specific operations include displaying the manual in PDF or HTML format.

[0350] Step 6:

[0351] Users refer to the operation manual and provide feedback on the system. Input consists of user evaluations of the operations performed and their results. Output is user feedback information sent to the server. Specific actions include submitting feedback forms and completing questionnaires. This information is used to improve the system.

[0352] (Application Example 2)

[0353] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0354] Providing appropriate guidance and support to customers who are confused or lost in physical stores can be challenging. While store staff need to directly assist in such situations, staff shortages and a lack of specialized knowledge can prevent them from providing adequate support. Furthermore, there is a need to provide personalized support tailored to each customer's feelings and level of understanding. Therefore, improving customer satisfaction and streamlining store operations are key challenges.

[0355] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0356] In this invention, the server is connected to a business processing device and includes means for collecting user operation record information and screen interface update information, means for analyzing the collected record information and update information and extracting operation procedures, and means for applying natural language generation technology to generate documents based on the extracted operation procedures. This makes it possible to analyze the customer's emotional state in real time and automatically provide appropriate guidance and product suggestions according to that emotion.

[0357] A "business processing device" is a device used to collect user operation records and screen interface update information.

[0358] "Recorded information" refers to data obtained when a user performs an operation, and it shows the history of that operation.

[0359] "Update information" refers to the history of changes to the screen interface, tracking how users interacted with the interface.

[0360] "Analysis means" refers to techniques for analyzing recorded information and updated information to extract meaningful patterns and operating procedures.

[0361] "Natural language generation technology" is a technology that automatically creates documents that are easy for humans to understand based on extracted operating procedures.

[0362] "Means for recognizing emotional states" refers to technologies that analyze a user's facial expression data and input information to identify their emotions.

[0363] "Means of receiving feedback" refers to the process of receiving opinions and evaluations from users and improving the generated documents.

[0364] "Adaptive adjustment mechanisms" refer to functions that modify generated documents and guidance content based on recognized user emotions to provide more effective support.

[0365] A system implementing this invention can be realized with the following main components: The server is connected to the business processing device and has the function of collecting user operation record information and screen interface update information. This information is used to track the user's operation history and analyze what operations were performed.

[0366] Next, the server analyzes the collected record and update information to extract operation procedures. Advanced data analysis techniques are used for the analysis to identify frequently occurring operation patterns. Furthermore, emotion recognition technology is incorporated to analyze facial expression data and input information to recognize the user's emotional state. This utilizes software known as emotion analysis libraries (e.g., Microsoft Azure Face API, Google Cloud Vision API, etc.).

[0367] The documents displayed by the device are created using natural language generation technology based on extracted operating procedures. The natural language generation model translates the operating procedures into clear and easily understandable language for humans. Furthermore, the device receives user feedback and revises the generated documents to improve the system. By adaptively adjusting the content of the documents according to the user's emotional state, it is possible to provide more effective guidance.

[0368] As a concrete example, in a physical store, if a customer shows a confused expression, this system recognizes the customer's emotions in real time and provides a detailed product description based on the customer's emotional state. This can improve customer satisfaction and reduce the burden on store staff.

[0369] An example of a prompt message is: "Send to the generating AI model: 'Adjust the guidance appropriately to match the product category selected by the customer, and display promotions for related products when the customer's facial expression is smiling.'"

[0370] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0371] Step 1:

[0372] The server connects to the business processing unit and collects user operation records and screen interface update information in real time. It receives operation log data and update history from the business processing unit as input and stores this data in a database. This makes it possible to systematically record what operations were performed.

[0373] Step 2:

[0374] The server analyzes the collected record information and update information. It takes operation logs and update history stored in the database as input and extracts operation patterns using advanced data analysis algorithms. As output, it identifies frequently performed operation procedures and abnormal operation patterns and formats them into a report format. This allows for an understanding of operation trends and problems.

[0375] Step 3:

[0376] The server analyzes facial expression data acquired from the device's camera and user input information to recognize the user's emotional state. It receives real-time facial image and text input as input and generates emotional data using an emotion analysis library (e.g., Microsoft Azure Face API). As output, it identifies the user's emotional state (e.g., confused, satisfied) and outputs it as a digital signal. This allows the server to determine the next action based on the user's emotions.

[0377] Step 4:

[0378] The server uses natural language generation technology to create documents based on analyzed operating procedures and sentiment data. It receives identified operating procedures and the user's emotional state as input, and utilizes a generative AI model to generate customized guidance documents. The output is easy-to-understand documents and explanations, which are then sent to the user's terminal. This allows users to receive support tailored to their specific situation.

[0379] Step 5:

[0380] The terminal displays documents sent from the server to the user and also receives feedback. It receives the displayed document and user feedback as input, and sends the collected feedback to the server as output. This feedback information can then be used to further improve the document content.

[0381] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0382] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0383] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0384] [Third Embodiment]

[0385] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0386] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0387] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0388] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0389] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0390] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0391] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0392] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0393] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0394] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0395] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0396] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0397] The system of the present invention is centered around a server connected to a business processing device, and automatically collects user operation logs and interface update history, and generates and updates operation manuals based on them. This system can quickly reflect changes in procedures that occur when companies or individuals perform their work, and ensures that users always have access to the latest operating procedures.

[0398] Specifically, the server periodically retrieves operation logs from the business processing unit. This includes detailed information such as which user performed which operation and which screens were displayed. Furthermore, by obtaining the update history of the screen interface, it is possible to understand which functions have been changed.

[0399] The server then analyzes the collected data to extract typical operating procedures and new operating methods. Using natural language generation technology, these procedures are documented in an easy-to-understand format to create a manual. This manual also integrates relevant screen captures, making it easier for users to visually understand the procedures.

[0400] Users can view this manual through their devices and provide feedback, which will help continuously improve the manual's content. This feedback is analyzed by the server, and the manual is revised as needed to provide better operational support.

[0401] As a concrete example, consider the case where an audit function is added. The server automatically extracts the operational steps for this new function and updates the manual in the format of "Step 1: Open the settings menu" and "Step 2: Select the audit settings." This allows users to quickly adapt when new functions are introduced, supporting smooth business operations.

[0402] The following describes the processing flow.

[0403] Step 1:

[0404] The server automatically collects user operation logs and screen interface update history for the business processing unit at regular intervals. This includes the date and time of the operation, the user ID, and details of the operation performed. The collected data is stored in a database on the server.

[0405] Step 2:

[0406] The server analyzes the collected operation logs and update history. Here, it extracts frequently used operation patterns and procedures related to newly added or modified features. This analysis allows for understanding trends in how each feature is being used.

[0407] Step 3:

[0408] The server uses natural language generation technology based on the analysis results to create a manual document that includes new operating procedures. This document will describe the operating procedures in easy-to-understand language and will include relevant screen captures and videos.

[0409] Step 4:

[0410] The terminal displays the generated manual documents in a format accessible to the user. Users can access these manuals at any time through the terminal and check the necessary operating procedures for their work.

[0411] Step 5:

[0412] Users can provide feedback on any unclear points or areas for improvement they find while using the manual. This feedback is submitted from their device via a simple form.

[0413] Step 6:

[0414] The server receives and analyzes user feedback to identify areas that need improvement. If necessary, it revises the manual documents to improve their accuracy. The revised content is then provided to the user again via the terminal.

[0415] (Example 1)

[0416] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0417] Traditional business systems often suffered from delays in updating manuals in response to changes in operating procedures or the addition of new features, making it difficult for users to access the latest procedures. Furthermore, the lack of mechanisms to appropriately incorporate user feedback and consistently provide the most up-to-date and optimal operating procedures sometimes led to decreased operational efficiency.

[0418] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0419] In this invention, the server is connected to a business processing device and includes means for collecting user operation logs and display interface update history, means for analyzing the collected operation logs and update history to extract typical operation procedures, and means for applying a generation AI model to create documents based on the extracted operation procedures. This enables quick response to changes in operation procedures and the addition of new functions, and allows users to always access the latest and most visually easy-to-understand operation manual.

[0420] A "business processing device" is an electronic device used to perform operations necessary for carrying out business tasks, and it is a device that generates user operation logs and screen interface update history.

[0421] "User" refers to an individual or organization that uses the business processing device to perform business operations, and is the entity that directly utilizes the information provided by the system.

[0422] An "operation log" is an electronic record of a series of operations performed by a user on a business processing device, and includes data such as the date and time and the details of the operations performed.

[0423] "Display interface" refers to the overall structure and elements on a screen that a business processing device uses to provide information to a user, enabling interaction with the user.

[0424] "Update history" refers to data that shows the history of changes and revisions to the display interface and the functions of the business processing device, and it holds information about past changes to the system.

[0425] A "generative AI model" is a mathematical model that uses artificial intelligence to generate natural language and data, and has the function of creating a desired output for a specific input.

[0426] A "typical operating procedure" refers to a series of frequently performed steps extracted from user operation logs, representing an optimized method of operation for smooth business operations.

[0427] A "document" refers to a collection of information generated based on extracted operating procedures, and is provided in a format that users can understand, such as an instruction manual or guide.

[0428] "Visual data" refers to digital information such as images and diagrams that users can visually refer to, and it plays a role in complementing operating procedures when integrated into documents.

[0429] The system of the present invention is centered around a server connected to a business processing device, and automatically collects user operation logs and display interface update history, and generates and updates operation manuals based on them.

[0430] The server periodically retrieves operation logs from the business processing unit. These operation logs contain detailed information such as which user performed which operation and which screen was displayed. The collected operation logs and update history are stored in a database on the server. The server then uses machine learning algorithms to analyze the collected data and extract typical operation procedures. This analysis utilizes Python's data analysis library Pandas and pattern recognition techniques.

[0431] Based on the extracted operating procedures, the server uses a generative AI model to create an operation manual. The generative AI model employed is an artificial intelligence model widely used to realize natural language generation technology. In this process, the operating procedures are documented in a user-friendly format, and related video data is also integrated.

[0432] Users can view operation manuals generated through their terminals. Furthermore, users can provide feedback on the manuals, which are then analyzed by the server, and the manuals are revised as needed. This process ensures users always have access to the latest information, improving work efficiency.

[0433] As a concrete example, consider the case where a new auditing function is added. The server automatically extracts the operational steps related to this new function and updates the manual with steps such as "Open the settings menu" and "Select audit settings." Users can then use this manual to smoothly operate the new function.

[0434] An example of a prompt message is as follows: "Please provide instructions for an AI program that automatically updates the manual so that users can easily understand how to use the system when new features are added to an enterprise system. Specifically, please provide a clear explanation, including the steps to take when a new auditing function is added." Based on this prompt, the AI ​​model generates an appropriate operation manual.

[0435] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0436] Step 1:

[0437] The server connects to the business processing unit and retrieves user operation logs and display interface update history. Inputs include real-time operation data and update information sent from the business processing unit. This information is sent to the server via a communication protocol and stored in a database. Specifically, it uses a particular API to retrieve data and organize log information in storage.

[0438] Step 2:

[0439] The server analyzes the collected operation logs and update history. The input for this step is the operation logs and update history stored in the database. Based on this data, the server applies machine learning algorithms to clean the data and extract frequently occurring operation patterns. As output, the server generates a list of the extracted typical operation procedures. Specifically, it constructs a dataframe using Python's Pandas and analyzes the frequency of operations.

[0440] Step 3:

[0441] The server utilizes a generative AI model to generate an operation manual based on the analysis results. The input is a list of typical operating procedures. The server uses natural language generation technology to convert the procedures into an easy-to-understand document format. The output is an operation manual in a user-friendly format. Specifically, prompts are input to the generative AI model, and the generated text is saved in HTML or PDF format.

[0442] Step 4:

[0443] Users view the operation manual generated through the terminal. The terminal downloads the latest manual from the server and displays it on the user interface. Specifically, users can access the manual using a web browser and check the procedures.

[0444] Step 5:

[0445] Users provide feedback on the operation manual. Input includes user opinions and suggestions for improvement. The terminal sends this information to the server via a dedicated feedback form. The server analyzes the received feedback and generates new improvement suggestions. The output is a revised operation manual based on the feedback results. The server analyzes this information and incorporates it into the next manual generation.

[0446] (Application Example 1)

[0447] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0448] In business operations, frequent updates to operating procedures and systems can make it difficult for employees to keep up with the latest procedures, leading to decreased work efficiency. Furthermore, existing manuals are static, making it difficult to quickly adapt to screen changes or new features, resulting in longer learning and adaptation times for users.

[0449] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0450] In this invention, the server is connected to a business processing terminal and includes means for collecting user operation record data and screen interface update information, means for analyzing the collected record data and update information and extracting usage procedures, and means for displaying the generated document on an augmented reality display device and providing it to the user. As a result, users can always perform their tasks based on the latest information, enabling improved work efficiency and enhanced adaptability.

[0451] A "business processing terminal" is a computer device used by users when performing business tasks, and is capable of recording operation logs and interface update information.

[0452] "User" refers to an individual or organization that operates a business processing terminal and performs business through the system.

[0453] "Recorded data" refers to data that shows detailed information about the operations performed by the user on the business processing terminal.

[0454] "Screen interface update information" refers to data about the history of changes to the screen configuration and functions within the system.

[0455] "Analysis" is the process of analyzing collected data and deriving meaning and patterns from it.

[0456] "Usage instructions" refers to a procedure that includes step-by-step operating instructions for performing a task.

[0457] "Natural language generation technology" is a technology that converts machine-generated data into a form of natural language that is easy for the reader to understand.

[0458] "Image information" refers to visual data related to a document, used to visually illustrate the usage procedure.

[0459] "Evaluation" refers to feedback provided by users, which is information used to improve the system.

[0460] An "augmented reality display device" is a device that overlays digital information onto physical space and is used to provide visual support for work.

[0461] This invention is a system that utilizes a business processing terminal and a server to collect and analyze user operation data and screen interface update information, and to display the latest operating procedures. The hardware used includes a business processing terminal, a server, and an augmented reality display device (e.g., smart glasses). The software includes a data analysis platform and a natural language generation tool (e.g., OpenAI GPT-3).

[0462] The server instantly acquires operation log data and screen interface update information from business processing terminals via the network. This data is processed through the server's analysis platform to extract frequently used operation patterns and procedures for using newly added functions. Using natural language generation technology, these procedures are converted into user-friendly document formats, and relevant image information is added.

[0463] The generated documents are sent to an augmented reality display device, allowing users to perform their tasks while visually confirming the latest operating procedures in real time. For example, when new equipment is introduced, the usage procedures are automatically updated, and employees can understand "what to do next" in real time through smart glasses.

[0464] Using a generative AI model, natural language generation is performed with prompts like the following:

[0465] "We need new manual content. Please generate an easy-to-understand operation guide for manufacturing line workers, based on the most efficient procedures collected from operation logs. Please include specific steps and relevant screen captures."

[0466] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0467] Step 1:

[0468] The server instantly acquires operation log data and screen interface update information from business processing terminals via the network. Inputs are operation logs and update history from the terminals, and output is a dataset stored on the server. Specifically, monitoring software on the terminal captures logs and sends them to the server.

[0469] Step 2:

[0470] The system analyzes the dataset collected by the server to extract frequently used operation patterns and modified functions. The input is the accumulated operation logs, and the output is a list of analyzed usage procedures. As a data processing step, data mining techniques are used to statistically analyze the frequency of operations and extract patterns.

[0471] Step 3:

[0472] The server uses a generative AI model to convert the analyzed usage instructions into a natural language-based document format. The input is a list of operating procedures, and the output is an operation manual in text format. Specifically, the process involves providing prompts to the generative AI model, which then outputs an easily understandable document based on those prompts.

[0473] Step 4:

[0474] The server integrates image information related to the document, visually enhancing it. The input is the generated operation manual and associated image data, and the output is a comprehensive operation guide incorporating the images. Specifically, the system searches for relevant images from a database and automatically places them within the document.

[0475] Step 5:

[0476] The server sends a generated comprehensive operation guide to an augmented reality display device, allowing the user to view the latest instructions in real time. The input is an image-integrated operation guide, and the output is a guide displayed in the user's field of view. Specifically, wireless communication technology is used to transmit data to the display device, enabling the user to deepen their visual understanding.

[0477] Step 6:

[0478] The user performs tasks based on the operation guide and then sends their evaluation of the guide as feedback to the server. The input is the user's evaluation information, and the output is stored on the server as feedback data. Specifically, the user evaluates the quality of the displayed guide, and the evaluation system analyzes this information.

[0479] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0480] This invention combines a system that uses a server connected to a business processing device to collect user operation records and screen interface update information, and generates operation manuals based on this information, with an emotion engine. This system can recognize the user's emotional state and adaptively adjust the content of the generated documents accordingly.

[0481] Specifically, the server automatically collects user operation logs and screen interface update history from the business processing unit. This data is analyzed to understand how users are using the system and what changes have been made.

[0482] Furthermore, this invention incorporates an emotion engine that analyzes the user's facial expression data and input information to recognize their emotional state. Based on this, the server creates an operation manual in a format that is easy for the user to understand. For example, if the system detects that the user is confused, it can adjust the manual to provide more detailed explanations or additional guidance.

[0483] The terminal displays the generated manual in a user-friendly format. Users can check the operating procedures through this manual and also contribute to system improvements through feedback.

[0484] To give a specific example, if a user expresses dissatisfaction while using a certain function, the server detects this emotion and generates a manual that clearly restructures the procedure for that function. In this process, the server receives negative feedback from the user and suggests appropriate improvements, thereby enhancing the user experience. As a result, users can always use the system comfortably, and it also contributes to reducing support requests.

[0485] The following describes the processing flow.

[0486] Step 1:

[0487] The server automatically collects user operation logs and screen interface update history from the business processing unit. This data will include information about all operations performed by the user and any changes to the interface.

[0488] Step 2:

[0489] The server analyzes the collected data to extract frequently used user patterns and operating procedures related to newly introduced features. This analysis identifies characteristics such as the frequency of frequently occurring operations and the usage frequency of new features.

[0490] Step 3:

[0491] The server uses an emotion engine to analyze the user's emotions during operation. This analysis is based on facial expression data captured by the camera, as well as operation data such as keystrokes and mouse movements, and evaluates the user's stress level and satisfaction level.

[0492] Step 4:

[0493] The server adjusts the content of the generated operation manual based on the recognized emotional state. For example, if a negative emotion is detected, the procedure is made simpler and additional explanations are added to help the user understand it better.

[0494] Step 5:

[0495] The terminal displays the generated operation manual on the user's screen. The user can review the operating procedures through this manual and revisit them if necessary.

[0496] Step 6:

[0497] Users provide feedback on the manual's content. This feedback includes their understanding of the operation and their opinions on the manual, and is sent from the terminal to the server.

[0498] Step 7:

[0499] The server analyzes user feedback and revises the manual as needed. By reviewing and updating the manual content to address the issues indicated by the feedback, a better user experience is provided.

[0500] (Example 2)

[0501] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0502] Conventional user manual generation systems lack information that takes into account the user's emotional state, sometimes leading to confusion and frustration when using the system. This results in a degraded user experience and an increase in support requests. These problems need to be solved by providing customized support that takes into account the user's emotional state.

[0503] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0504] In this invention, the server is connected to a business processing device and includes means for collecting user operation record data and screen interface update history, means for analyzing user facial expression data and input information to recognize emotional states, and means for generating documents based on extracted operation procedures and emotional states by applying natural language generation technology. This makes it possible to quickly and appropriately resolve operational problems faced by the user and improve the user experience.

[0505] A "business processing system" is a computer system used to efficiently process and manage the business operations of a company or organization.

[0506] "User operation log data" refers to information that stores a log of a series of operations performed by a user on the system.

[0507] "Screen interface update history" refers to data that records the history of changes to the screen displayed to the user.

[0508] "Analysis" is the process of thoroughly examining data and information to clarify its structure and relationships.

[0509] An "operating procedure" is a set of steps or actions performed to complete a specific task.

[0510] "Emotional state" refers to information that indicates the user's psychological and emotional state.

[0511] "Facial expression data" refers to information captured from a user's facial expressions and is used to analyze their emotions.

[0512] "Natural language generation technology" is a technology that uses computers to automatically generate meaningful sentences in human language.

[0513] "Feedback" refers to user evaluations and opinions regarding a system or service.

[0514] This invention is a system that uses a server connected to a business processing device to collect user operation record data and screen interface update history, and generates an operation manual based on this data. The system incorporates an emotion engine that recognizes the user's emotional state and can adaptively adjust the content of the generated manual.

[0515] The server first automatically collects user operation record data and screen interface update history from the business processing unit. During this process, data is obtained in real time via an API through the client application. The collected data is stored in a database management system (e.g., MySQL or PostgreSQL) and then analyzed using Python's pandas library, among others.

[0516] The server is equipped with an emotion engine that recognizes the user's emotional state by analyzing facial expression data and input information. This process utilizes video data acquired from a camera and performs analysis using facial recognition technology (e.g., OpenCV or Amazon Rekognition). The recognized emotional state is then considered during the document generation process.

[0517] Next, the server utilizes a generative AI model (e.g., GPT-3 or BERT) to generate an operation manual using natural language generation technology based on the analysis results. This operation manual is composed of appropriate content and tone according to the user's emotional state and the functions being used. For example, by inputting a prompt sentence such as "The user is confused about using the new function, please provide an easy-to-understand guide" into the generative AI model, a specific and user-friendly manual is generated.

[0518] The generated manual is displayed on the terminal and made available to users in an easily accessible format. The terminal presents the manual in PDF or interactive HTML format using a web browser or dedicated application.

[0519] Users can use this manual to check operating procedures and contribute to system improvements through feedback. This feedback information is sent to the server and will be applied to future manual generation processes, thereby improving the overall user experience of the system.

[0520] In this way, the system aims to effectively solve operational problems faced by users and significantly improve the user experience.

[0521] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0522] Step 1:

[0523] The server collects user operation record data and screen interface update history from the business processing unit. It uses an API via the client application as input to retrieve the record data and update history in real time. This data is stored in a database management system for subsequent analysis. Specifically, the data is stored in MySQL or PostgreSQL.

[0524] Step 2:

[0525] The server analyzes the collected record data and update history. The input here is the record data and update history obtained in step 1. The data is processed using an analysis tool (e.g., the pandas library in Python) to extract frequently performed operations and user usage patterns. The output is the operation patterns for use in the next process as analysis results. Specific actions include data cleaning, filtering, and statistical analysis.

[0526] Step 3:

[0527] The server uses an emotion engine to recognize the user's emotional state. User facial expression data and input information are used as input. This data is acquired from a camera or input device and analyzed by facial recognition software (e.g., OpenCV or Amazon Rekognition). The output is information indicating the user's current emotional state, which is considered when generating manuals. Specific operations include facial expression recognition and emotion categorization.

[0528] Step 4:

[0529] The server uses a generative AI model to generate an operation manual based on the analysis results and emotional state. The inputs are the operation patterns extracted in step 2 and the emotional state obtained in step 3. Using natural language generation technology, the server creates a manual tailored to the user based on this information. The output is an operation manual that is adapted to the user's situation. Specifically, the output is text generated using GPT-3 or BERT.

[0530] Step 5:

[0531] The terminal displays the user the operation manual received from the server. The input is the operation manual generated in step 4. The output is a document formatted for user viewing and can be displayed in a web browser or dedicated application. Specific operations include displaying the manual in PDF or HTML format.

[0532] Step 6:

[0533] Users refer to the operation manual and provide feedback on the system. Input consists of user evaluations of the operations performed and their results. Output is user feedback information sent to the server. Specific actions include submitting feedback forms and completing questionnaires. This information is used to improve the system.

[0534] (Application Example 2)

[0535] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0536] Providing appropriate guidance and support to customers who are confused or lost in physical stores can be challenging. While store staff need to directly assist in such situations, staff shortages and a lack of specialized knowledge can prevent them from providing adequate support. Furthermore, there is a need to provide personalized support tailored to each customer's feelings and level of understanding. Therefore, improving customer satisfaction and streamlining store operations are key challenges.

[0537] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0538] In this invention, the server is connected to a business processing device and includes means for collecting user operation record information and screen interface update information, means for analyzing the collected record information and update information and extracting operation procedures, and means for applying natural language generation technology to generate documents based on the extracted operation procedures. This makes it possible to analyze the customer's emotional state in real time and automatically provide appropriate guidance and product suggestions according to that emotion.

[0539] A "business processing device" is a device used to collect user operation records and screen interface update information.

[0540] "Recorded information" refers to data obtained when a user performs an operation, and it shows the history of that operation.

[0541] "Update information" refers to the history of changes to the screen interface, tracking how users interacted with the interface.

[0542] "Analysis means" refers to techniques for analyzing recorded information and updated information to extract meaningful patterns and operating procedures.

[0543] "Natural language generation technology" is a technology that automatically creates documents that are easy for humans to understand based on extracted operating procedures.

[0544] "Means for recognizing emotional states" refers to technologies that analyze a user's facial expression data and input information to identify their emotions.

[0545] "Means of receiving feedback" refers to the process of receiving opinions and evaluations from users and improving the generated documents.

[0546] "Adaptive adjustment mechanisms" refer to functions that modify generated documents and guidance content based on recognized user emotions to provide more effective support.

[0547] A system implementing this invention can be realized with the following main components: The server is connected to the business processing device and has the function of collecting user operation record information and screen interface update information. This information is used to track the user's operation history and analyze what operations were performed.

[0548] Next, the server analyzes the collected record and update information to extract operation procedures. Advanced data analysis techniques are used for the analysis to identify frequently occurring operation patterns. Furthermore, emotion recognition technology is incorporated to analyze facial expression data and input information to recognize the user's emotional state. This utilizes software known as emotion analysis libraries (e.g., Microsoft Azure Face API, Google Cloud Vision API, etc.).

[0549] The documents displayed by the device are created using natural language generation technology based on extracted operating procedures. The natural language generation model translates the operating procedures into clear and easily understandable language for humans. Furthermore, the device receives user feedback and revises the generated documents to improve the system. By adaptively adjusting the content of the documents according to the user's emotional state, it is possible to provide more effective guidance.

[0550] As a concrete example, in a physical store, if a customer shows a confused expression, this system recognizes the customer's emotions in real time and provides a detailed product description based on the customer's emotional state. This can improve customer satisfaction and reduce the burden on store staff.

[0551] An example of a prompt message is: "Send to the generating AI model: 'Adjust the guidance appropriately to match the product category selected by the customer, and display promotions for related products when the customer's facial expression is smiling.'"

[0552] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0553] Step 1:

[0554] The server connects to the business processing unit and collects user operation records and screen interface update information in real time. It receives operation log data and update history from the business processing unit as input and stores this data in a database. This makes it possible to systematically record what operations were performed.

[0555] Step 2:

[0556] The server analyzes the collected record information and update information. It takes operation logs and update history stored in the database as input and extracts operation patterns using advanced data analysis algorithms. As output, it identifies frequently performed operation procedures and abnormal operation patterns and formats them into a report format. This allows for an understanding of operation trends and problems.

[0557] Step 3:

[0558] The server analyzes facial expression data acquired from the device's camera and user input information to recognize the user's emotional state. It receives real-time facial image and text input as input and generates emotional data using an emotion analysis library (e.g., Microsoft Azure Face API). As output, it identifies the user's emotional state (e.g., confused, satisfied) and outputs it as a digital signal. This allows the server to determine the next action based on the user's emotions.

[0559] Step 4:

[0560] The server uses natural language generation technology to create documents based on analyzed operating procedures and sentiment data. It receives identified operating procedures and the user's emotional state as input, and utilizes a generative AI model to generate customized guidance documents. The output is easy-to-understand documents and explanations, which are then sent to the user's terminal. This allows users to receive support tailored to their specific situation.

[0561] Step 5:

[0562] The terminal displays documents sent from the server to the user and also receives feedback. It receives the displayed document and user feedback as input, and sends the collected feedback to the server as output. This feedback information can then be used to further improve the document content.

[0563] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0564] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0565] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0566] [Fourth Embodiment]

[0567] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0568] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0569] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0570] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0571] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0572] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0573] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0574] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0575] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0576] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0577] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0578] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0579] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0580] The system of the present invention is centered around a server connected to a business processing device, and automatically collects user operation logs and interface update history, and generates and updates operation manuals based on them. This system can quickly reflect changes in procedures that occur when companies or individuals perform their work, and ensures that users always have access to the latest operating procedures.

[0581] Specifically, the server periodically retrieves operation logs from the business processing unit. This includes detailed information such as which user performed which operation and which screens were displayed. Furthermore, by obtaining the update history of the screen interface, it is possible to understand which functions have been changed.

[0582] The server then analyzes the collected data to extract typical operating procedures and new operating methods. Using natural language generation technology, these procedures are documented in an easy-to-understand format to create a manual. This manual also integrates relevant screen captures, making it easier for users to visually understand the procedures.

[0583] Users can view this manual through their devices and provide feedback, which will help continuously improve the manual's content. This feedback is analyzed by the server, and the manual is revised as needed to provide better operational support.

[0584] As a concrete example, consider the case where an audit function is added. The server automatically extracts the operational steps for this new function and updates the manual in the format of "Step 1: Open the settings menu" and "Step 2: Select the audit settings." This allows users to quickly adapt when new functions are introduced, supporting smooth business operations.

[0585] The following describes the processing flow.

[0586] Step 1:

[0587] The server automatically collects user operation logs and screen interface update history for the business processing unit at regular intervals. This includes the date and time of the operation, the user ID, and details of the operation performed. The collected data is stored in a database on the server.

[0588] Step 2:

[0589] The server analyzes the collected operation logs and update history. Here, it extracts frequently used operation patterns and procedures related to newly added or modified features. This analysis allows for understanding trends in how each feature is being used.

[0590] Step 3:

[0591] The server uses natural language generation technology based on the analysis results to create a manual document that includes new operating procedures. This document will describe the operating procedures in easy-to-understand language and will include relevant screen captures and videos.

[0592] Step 4:

[0593] The terminal displays the generated manual documents in a format accessible to the user. Users can access these manuals at any time through the terminal and check the necessary operating procedures for their work.

[0594] Step 5:

[0595] Users can provide feedback on any unclear points or areas for improvement they find while using the manual. This feedback is submitted from their device via a simple form.

[0596] Step 6:

[0597] The server receives and analyzes user feedback to identify areas that need improvement. If necessary, it revises the manual documents to improve their accuracy. The revised content is then provided to the user again via the terminal.

[0598] (Example 1)

[0599] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0600] Traditional business systems often suffered from delays in updating manuals in response to changes in operating procedures or the addition of new features, making it difficult for users to access the latest procedures. Furthermore, the lack of mechanisms to appropriately incorporate user feedback and consistently provide the most up-to-date and optimal operating procedures sometimes led to decreased operational efficiency.

[0601] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0602] In this invention, the server is connected to a business processing device and includes means for collecting user operation logs and display interface update history, means for analyzing the collected operation logs and update history to extract typical operation procedures, and means for applying a generation AI model to create documents based on the extracted operation procedures. This enables quick response to changes in operation procedures and the addition of new functions, and allows users to always access the latest and most visually easy-to-understand operation manual.

[0603] A "business processing device" is an electronic device used to perform operations necessary for carrying out business tasks, and it is a device that generates user operation logs and screen interface update history.

[0604] "User" refers to an individual or organization that uses the business processing device to perform business operations, and is the entity that directly utilizes the information provided by the system.

[0605] An "operation log" is an electronic record of a series of operations performed by a user on a business processing device, and includes data such as the date and time and the details of the operations performed.

[0606] "Display interface" refers to the overall structure and elements on a screen that a business processing device uses to provide information to a user, enabling interaction with the user.

[0607] "Update history" refers to data that shows the history of changes and revisions to the display interface and the functions of the business processing device, and it holds information about past changes to the system.

[0608] A "generative AI model" is a mathematical model that uses artificial intelligence to generate natural language and data, and has the function of creating a desired output for a specific input.

[0609] A "typical operating procedure" refers to a series of frequently performed steps extracted from user operation logs, representing an optimized method of operation for smooth business operations.

[0610] A "document" refers to a collection of information generated based on extracted operating procedures, and is provided in a format that users can understand, such as an instruction manual or guide.

[0611] "Visual data" refers to digital information such as images and diagrams that users can visually refer to, and it plays a role in complementing operating procedures when integrated into documents.

[0612] The system of the present invention is centered around a server connected to a business processing device, and automatically collects user operation logs and display interface update history, and generates and updates operation manuals based on them.

[0613] The server periodically retrieves operation logs from the business processing unit. These operation logs contain detailed information such as which user performed which operation and which screen was displayed. The collected operation logs and update history are stored in a database on the server. The server then uses machine learning algorithms to analyze the collected data and extract typical operation procedures. This analysis utilizes Python's data analysis library Pandas and pattern recognition techniques.

[0614] Based on the extracted operating procedures, the server uses a generative AI model to create an operation manual. The generative AI model employed is an artificial intelligence model widely used to realize natural language generation technology. In this process, the operating procedures are documented in a user-friendly format, and related video data is also integrated.

[0615] Users can view operation manuals generated through their terminals. Furthermore, users can provide feedback on the manuals, which are then analyzed by the server, and the manuals are revised as needed. This process ensures users always have access to the latest information, improving work efficiency.

[0616] As a concrete example, consider the case where a new auditing function is added. The server automatically extracts the operational steps related to this new function and updates the manual with steps such as "Open the settings menu" and "Select audit settings." Users can then use this manual to smoothly operate the new function.

[0617] An example of a prompt message is as follows: "Please provide instructions for an AI program that automatically updates the manual so that users can easily understand how to use the system when new features are added to an enterprise system. Specifically, please provide a clear explanation, including the steps to take when a new auditing function is added." Based on this prompt, the AI ​​model generates an appropriate operation manual.

[0618] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0619] Step 1:

[0620] The server connects to the business processing unit and retrieves user operation logs and display interface update history. Inputs include real-time operation data and update information sent from the business processing unit. This information is sent to the server via a communication protocol and stored in a database. Specifically, it uses a particular API to retrieve data and organize log information in storage.

[0621] Step 2:

[0622] The server analyzes the collected operation logs and update history. The input for this step is the operation logs and update history stored in the database. Based on this data, the server applies machine learning algorithms to clean the data and extract frequently occurring operation patterns. As output, the server generates a list of the extracted typical operation procedures. Specifically, it constructs a dataframe using Python's Pandas and analyzes the frequency of operations.

[0623] Step 3:

[0624] The server utilizes a generative AI model to generate an operation manual based on the analysis results. The input is a list of typical operating procedures. The server uses natural language generation technology to convert the procedures into an easy-to-understand document format. The output is an operation manual in a user-friendly format. Specifically, prompts are input to the generative AI model, and the generated text is saved in HTML or PDF format.

[0625] Step 4:

[0626] Users view the operation manual generated through the terminal. The terminal downloads the latest manual from the server and displays it on the user interface. Specifically, users can access the manual using a web browser and check the procedures.

[0627] Step 5:

[0628] Users provide feedback on the operation manual. Input includes user opinions and suggestions for improvement. The terminal sends this information to the server via a dedicated feedback form. The server analyzes the received feedback and generates new improvement suggestions. The output is a revised operation manual based on the feedback results. The server analyzes this information and incorporates it into the next manual generation.

[0629] (Application Example 1)

[0630] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0631] In business operations, frequent updates to operating procedures and systems can make it difficult for employees to keep up with the latest procedures, leading to decreased work efficiency. Furthermore, existing manuals are static, making it difficult to quickly adapt to screen changes or new features, resulting in longer learning and adaptation times for users.

[0632] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0633] In this invention, the server is connected to a business processing terminal and includes means for collecting user operation record data and screen interface update information, means for analyzing the collected record data and update information and extracting usage procedures, and means for displaying the generated document on an augmented reality display device and providing it to the user. As a result, users can always perform their tasks based on the latest information, enabling improved work efficiency and enhanced adaptability.

[0634] A "business processing terminal" is a computer device used by users when performing business tasks, and is capable of recording operation logs and interface update information.

[0635] "User" refers to an individual or organization that operates a business processing terminal and performs business through the system.

[0636] "Recorded data" refers to data that shows detailed information about the operations performed by the user on the business processing terminal.

[0637] "Screen interface update information" refers to data about the history of changes to the screen configuration and functions within the system.

[0638] "Analysis" is the process of analyzing collected data and deriving meaning and patterns from it.

[0639] "Usage instructions" refers to a procedure that includes step-by-step operating instructions for performing a task.

[0640] "Natural language generation technology" is a technology that converts machine-generated data into a form of natural language that is easy for the reader to understand.

[0641] "Image information" refers to visual data related to a document, used to visually illustrate the usage procedure.

[0642] "Evaluation" refers to feedback provided by users, which is information used to improve the system.

[0643] An "augmented reality display device" is a device that overlays digital information onto physical space and is used to provide visual support for work.

[0644] This invention is a system that utilizes a business processing terminal and a server to collect and analyze user operation data and screen interface update information, and to display the latest operating procedures. The hardware used includes a business processing terminal, a server, and an augmented reality display device (e.g., smart glasses). The software includes a data analysis platform and a natural language generation tool (e.g., OpenAI GPT-3).

[0645] The server instantly acquires operation log data and screen interface update information from business processing terminals via the network. This data is processed through the server's analysis platform to extract frequently used operation patterns and procedures for using newly added functions. Using natural language generation technology, these procedures are converted into user-friendly document formats, and relevant image information is added.

[0646] The generated documents are sent to an augmented reality display device, allowing users to perform their tasks while visually confirming the latest operating procedures in real time. For example, when new equipment is introduced, the usage procedures are automatically updated, and employees can understand "what to do next" in real time through smart glasses.

[0647] Using a generative AI model, natural language generation is performed with prompts like the following:

[0648] "We need new manual content. Please generate an easy-to-understand operation guide for manufacturing line workers, based on the most efficient procedures collected from operation logs. Please include specific steps and relevant screen captures."

[0649] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0650] Step 1:

[0651] The server instantly acquires operation log data and screen interface update information from business processing terminals via the network. Inputs are operation logs and update history from the terminals, and output is a dataset stored on the server. Specifically, monitoring software on the terminal captures logs and sends them to the server.

[0652] Step 2:

[0653] The system analyzes the dataset collected by the server to extract frequently used operation patterns and modified functions. The input is the accumulated operation logs, and the output is a list of analyzed usage procedures. As a data processing step, data mining techniques are used to statistically analyze the frequency of operations and extract patterns.

[0654] Step 3:

[0655] The server uses a generative AI model to convert the analyzed usage instructions into a natural language-based document format. The input is a list of operating procedures, and the output is an operation manual in text format. Specifically, the process involves providing prompts to the generative AI model, which then outputs an easily understandable document based on those prompts.

[0656] Step 4:

[0657] The server integrates image information related to the document, visually enhancing it. The input is the generated operation manual and associated image data, and the output is a comprehensive operation guide incorporating the images. Specifically, the system searches for relevant images from a database and automatically places them within the document.

[0658] Step 5:

[0659] The server sends a generated comprehensive operation guide to an augmented reality display device, allowing the user to view the latest instructions in real time. The input is an image-integrated operation guide, and the output is a guide displayed in the user's field of view. Specifically, wireless communication technology is used to transmit data to the display device, enabling the user to deepen their visual understanding.

[0660] Step 6:

[0661] The user performs tasks based on the operation guide and then sends their evaluation of the guide as feedback to the server. The input is the user's evaluation information, and the output is stored on the server as feedback data. Specifically, the user evaluates the quality of the displayed guide, and the evaluation system analyzes this information.

[0662] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0663] This invention combines a system that uses a server connected to a business processing device to collect user operation records and screen interface update information, and generates operation manuals based on this information, with an emotion engine. This system can recognize the user's emotional state and adaptively adjust the content of the generated documents accordingly.

[0664] Specifically, the server automatically collects user operation logs and screen interface update history from the business processing unit. This data is analyzed to understand how users are using the system and what changes have been made.

[0665] Furthermore, this invention incorporates an emotion engine that analyzes the user's facial expression data and input information to recognize their emotional state. Based on this, the server creates an operation manual in a format that is easy for the user to understand. For example, if the system detects that the user is confused, it can adjust the manual to provide more detailed explanations or additional guidance.

[0666] The terminal displays the generated manual in a user-friendly format. Users can check the operating procedures through this manual and also contribute to system improvements through feedback.

[0667] To give a specific example, if a user expresses dissatisfaction while using a certain function, the server detects this emotion and generates a manual that clearly restructures the procedure for that function. In this process, the server receives negative feedback from the user and suggests appropriate improvements, thereby enhancing the user experience. As a result, users can always use the system comfortably, and it also contributes to reducing support requests.

[0668] The following describes the processing flow.

[0669] Step 1:

[0670] The server automatically collects user operation logs and screen interface update history from the business processing unit. This data will include information about all operations performed by the user and any changes to the interface.

[0671] Step 2:

[0672] The server analyzes the collected data to extract frequently used user patterns and operating procedures related to newly introduced features. This analysis identifies characteristics such as the frequency of frequently occurring operations and the usage frequency of new features.

[0673] Step 3:

[0674] The server uses an emotion engine to analyze the user's emotions during operation. This analysis is based on facial expression data captured by the camera, as well as operation data such as keystrokes and mouse movements, and evaluates the user's stress level and satisfaction level.

[0675] Step 4:

[0676] The server adjusts the content of the generated operation manual based on the recognized emotional state. For example, if a negative emotion is detected, the procedure is made simpler and additional explanations are added to help the user understand it better.

[0677] Step 5:

[0678] The terminal displays the generated operation manual on the user's screen. The user can review the operating procedures through this manual and revisit them if necessary.

[0679] Step 6:

[0680] Users provide feedback on the manual's content. This feedback includes their understanding of the operation and their opinions on the manual, and is sent from the terminal to the server.

[0681] Step 7:

[0682] The server analyzes user feedback and revises the manual as needed. By reviewing and updating the manual content to address the issues indicated by the feedback, a better user experience is provided.

[0683] (Example 2)

[0684] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0685] Conventional user manual generation systems lack information that takes into account the user's emotional state, sometimes leading to confusion and frustration when using the system. This results in a degraded user experience and an increase in support requests. These problems need to be solved by providing customized support that takes into account the user's emotional state.

[0686] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0687] In this invention, the server is connected to a business processing device and includes means for collecting user operation record data and screen interface update history, means for analyzing user facial expression data and input information to recognize emotional states, and means for generating documents based on extracted operation procedures and emotional states by applying natural language generation technology. This makes it possible to quickly and appropriately resolve operational problems faced by the user and improve the user experience.

[0688] A "business processing system" is a computer system used to efficiently process and manage the business operations of a company or organization.

[0689] "User operation log data" refers to information that stores a log of a series of operations performed by a user on the system.

[0690] "Screen interface update history" refers to data that records the history of changes to the screen displayed to the user.

[0691] "Analysis" is the process of thoroughly examining data and information to clarify its structure and relationships.

[0692] An "operating procedure" is a set of steps or actions performed to complete a specific task.

[0693] "Emotional state" refers to information that indicates the user's psychological and emotional state.

[0694] "Facial expression data" refers to information captured from a user's facial expressions and is used to analyze their emotions.

[0695] "Natural language generation technology" is a technology that uses computers to automatically generate meaningful sentences in human language.

[0696] "Feedback" refers to user evaluations and opinions regarding a system or service.

[0697] This invention is a system that uses a server connected to a business processing device to collect user operation record data and screen interface update history, and generates an operation manual based on this data. The system incorporates an emotion engine that recognizes the user's emotional state and can adaptively adjust the content of the generated manual.

[0698] The server first automatically collects user operation record data and screen interface update history from the business processing unit. During this process, data is obtained in real time via an API through the client application. The collected data is stored in a database management system (e.g., MySQL or PostgreSQL) and then analyzed using Python's pandas library, among others.

[0699] The server is equipped with an emotion engine that recognizes the user's emotional state by analyzing facial expression data and input information. This process utilizes video data acquired from a camera and performs analysis using facial recognition technology (e.g., OpenCV or Amazon Rekognition). The recognized emotional state is then considered during the document generation process.

[0700] Next, the server utilizes a generative AI model (e.g., GPT-3 or BERT) to generate an operation manual using natural language generation technology based on the analysis results. This operation manual is composed of appropriate content and tone according to the user's emotional state and the functions being used. For example, by inputting a prompt sentence such as "The user is confused about using the new function, please provide an easy-to-understand guide" into the generative AI model, a specific and user-friendly manual is generated.

[0701] The generated manual is displayed on the terminal and made available to users in an easily accessible format. The terminal presents the manual in PDF or interactive HTML format using a web browser or dedicated application.

[0702] Users can use this manual to check operating procedures and contribute to system improvements through feedback. This feedback information is sent to the server and will be applied to future manual generation processes, thereby improving the overall user experience of the system.

[0703] In this way, the system aims to effectively solve operational problems faced by users and significantly improve the user experience.

[0704] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0705] Step 1:

[0706] The server collects user operation record data and screen interface update history from the business processing unit. It uses an API via the client application as input to retrieve the record data and update history in real time. This data is stored in a database management system for subsequent analysis. Specifically, the data is stored in MySQL or PostgreSQL.

[0707] Step 2:

[0708] The server analyzes the collected record data and update history. The input here is the record data and update history obtained in step 1. The data is processed using an analysis tool (e.g., the pandas library in Python) to extract frequently performed operations and user usage patterns. The output is the operation patterns for use in the next process as analysis results. Specific actions include data cleaning, filtering, and statistical analysis.

[0709] Step 3:

[0710] The server uses an emotion engine to recognize the user's emotional state. User facial expression data and input information are used as input. This data is acquired from a camera or input device and analyzed by facial recognition software (e.g., OpenCV or Amazon Rekognition). The output is information indicating the user's current emotional state, which is considered when generating manuals. Specific operations include facial expression recognition and emotion categorization.

[0711] Step 4:

[0712] The server uses a generative AI model to generate an operation manual based on the analysis results and emotional state. The inputs are the operation patterns extracted in step 2 and the emotional state obtained in step 3. Using natural language generation technology, the server creates a manual tailored to the user based on this information. The output is an operation manual that is adapted to the user's situation. Specifically, the output is text generated using GPT-3 or BERT.

[0713] Step 5:

[0714] The terminal displays the user the operation manual received from the server. The input is the operation manual generated in step 4. The output is a document formatted for user viewing and can be displayed in a web browser or dedicated application. Specific operations include displaying the manual in PDF or HTML format.

[0715] Step 6:

[0716] Users refer to the operation manual and provide feedback on the system. Input consists of user evaluations of the operations performed and their results. Output is user feedback information sent to the server. Specific actions include submitting feedback forms and completing questionnaires. This information is used to improve the system.

[0717] (Application Example 2)

[0718] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0719] Providing appropriate guidance and support to customers who are confused or lost in physical stores can be challenging. While store staff need to directly assist in such situations, staff shortages and a lack of specialized knowledge can prevent them from providing adequate support. Furthermore, there is a need to provide personalized support tailored to each customer's feelings and level of understanding. Therefore, improving customer satisfaction and streamlining store operations are key challenges.

[0720] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0721] In this invention, the server is connected to a business processing device and includes means for collecting user operation record information and screen interface update information, means for analyzing the collected record information and update information and extracting operation procedures, and means for applying natural language generation technology to generate documents based on the extracted operation procedures. This makes it possible to analyze the customer's emotional state in real time and automatically provide appropriate guidance and product suggestions according to that emotion.

[0722] A "business processing device" is a device used to collect user operation records and screen interface update information.

[0723] "Recorded information" refers to data obtained when a user performs an operation, and it shows the history of that operation.

[0724] "Update information" refers to the history of changes to the screen interface, tracking how users interacted with the interface.

[0725] "Analysis means" refers to techniques for analyzing recorded information and updated information to extract meaningful patterns and operating procedures.

[0726] "Natural language generation technology" is a technology that automatically creates documents that are easy for humans to understand based on extracted operating procedures.

[0727] "Means for recognizing emotional states" refers to technologies that analyze a user's facial expression data and input information to identify their emotions.

[0728] "Means of receiving feedback" refers to the process of receiving opinions and evaluations from users and improving the generated documents.

[0729] "Adaptive adjustment mechanisms" refer to functions that modify generated documents and guidance content based on recognized user emotions to provide more effective support.

[0730] A system implementing this invention can be realized with the following main components: The server is connected to the business processing device and has the function of collecting user operation record information and screen interface update information. This information is used to track the user's operation history and analyze what operations were performed.

[0731] Next, the server analyzes the collected record and update information to extract operation procedures. Advanced data analysis techniques are used for the analysis to identify frequently occurring operation patterns. Furthermore, emotion recognition technology is incorporated to analyze facial expression data and input information to recognize the user's emotional state. This utilizes software known as emotion analysis libraries (e.g., Microsoft Azure Face API, Google Cloud Vision API, etc.).

[0732] The documents displayed by the device are created using natural language generation technology based on extracted operating procedures. The natural language generation model translates the operating procedures into clear and easily understandable language for humans. Furthermore, the device receives user feedback and revises the generated documents to improve the system. By adaptively adjusting the content of the documents according to the user's emotional state, it is possible to provide more effective guidance.

[0733] As a concrete example, in a physical store, if a customer shows a confused expression, this system recognizes the customer's emotions in real time and provides a detailed product description based on the customer's emotional state. This can improve customer satisfaction and reduce the burden on store staff.

[0734] An example of a prompt message is: "Send to the generating AI model: 'Adjust the guidance appropriately to match the product category selected by the customer, and display promotions for related products when the customer's facial expression is smiling.'"

[0735] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0736] Step 1:

[0737] The server connects to the business processing unit and collects user operation records and screen interface update information in real time. It receives operation log data and update history from the business processing unit as input and stores this data in a database. This makes it possible to systematically record what operations were performed.

[0738] Step 2:

[0739] The server analyzes the collected record information and update information. It takes operation logs and update history stored in the database as input and extracts operation patterns using advanced data analysis algorithms. As output, it identifies frequently performed operation procedures and abnormal operation patterns and formats them into a report format. This allows for an understanding of operation trends and problems.

[0740] Step 3:

[0741] The server analyzes facial expression data acquired from the device's camera and user input information to recognize the user's emotional state. It receives real-time facial image and text input as input and generates emotional data using an emotion analysis library (e.g., Microsoft Azure Face API). As output, it identifies the user's emotional state (e.g., confused, satisfied) and outputs it as a digital signal. This allows the server to determine the next action based on the user's emotions.

[0742] Step 4:

[0743] The server uses natural language generation technology to create documents based on analyzed operating procedures and sentiment data. It receives identified operating procedures and the user's emotional state as input, and utilizes a generative AI model to generate customized guidance documents. The output is easy-to-understand documents and explanations, which are then sent to the user's terminal. This allows users to receive support tailored to their specific situation.

[0744] Step 5:

[0745] The terminal displays documents sent from the server to the user and also receives feedback. It receives the displayed document and user feedback as input, and sends the collected feedback to the server as output. This feedback information can then be used to further improve the document content.

[0746] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0747] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0748] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0749] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0750] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0751] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0752] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0753] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0754] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0755] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0756] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0757] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0758] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0759] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0760] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0761] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0762] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0763] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0764] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0765] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0766] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0767] The following is further disclosed regarding the embodiments described above.

[0768] (Claim 1)

[0769] A means connected to a business processing device for collecting user operation record data and screen interface update history,

[0770] A means for analyzing the collected record data and update history and extracting the operating procedure,

[0771] A means for generating documents based on extracted operating procedures by applying natural language generation technology,

[0772] Means for acquiring image data related to the said document and integrating it into the said document,

[0773] A means of receiving user feedback and revising the aforementioned document,

[0774] A system that includes this.

[0775] (Claim 2)

[0776] The system according to claim 1, wherein the collection means acquires recorded data and update history in real time via a network of business processing devices.

[0777] (Claim 3)

[0778] The system according to claim 1, wherein the analysis means detects frequently occurring operation patterns from recorded data.

[0779] "Example 1"

[0780] (Claim 1)

[0781] A means connected to a business processing device for collecting user operation logs and update history of the display interface,

[0782] A means for analyzing the collected operation logs and update history and extracting typical operation procedures,

[0783] A means of creating a document based on extracted operating procedures by applying a generative AI model,

[0784] A means for acquiring video data related to the aforementioned document and integrating it into the document,

[0785] A means of receiving feedback from users and improving the aforementioned document,

[0786] A system that includes this.

[0787] (Claim 2)

[0788] The system according to claim 1, wherein the collection means instantly acquires operation logs and update history via the communication network of the business processing device.

[0789] (Claim 3)

[0790] The system according to claim 1, wherein the analysis means detects frequently occurring processing patterns from the operation log.

[0791] "Application Example 1"

[0792] (Claim 1)

[0793] A means connected to a business processing terminal to collect user operation record data and screen interface update information,

[0794] A means for analyzing the collected record data and update information and extracting the usage procedure,

[0795] A means for generating documents based on extracted usage procedures by applying natural language generation technology,

[0796] Means for acquiring image information related to the said document and integrating it into the said document,

[0797] A means of receiving feedback from users and revising the aforementioned document,

[0798] A means of displaying the generated document on an augmented reality display device and providing it to the user,

[0799] A system that includes this.

[0800] (Claim 2)

[0801] The system according to claim 1, wherein the collection means acquires recorded data and update information immediately via the information network of the business processing terminal.

[0802] (Claim 3)

[0803] The system according to claim 1, wherein the analysis means detects frequently occurring usage patterns from recorded data.

[0804] "Example 2 of combining an emotion engine"

[0805] (Claim 1)

[0806] A means connected to a business processing device for collecting user operation record data and screen interface update history,

[0807] A means for analyzing the collected record data and update history and extracting the operating procedure,

[0808] A means for analyzing user facial expression data and input information in order to recognize emotional states,

[0809] A means for generating documents based on extracted operating procedures and emotional states by applying natural language generation technology,

[0810] A means of displaying the generated document in a way that is easy for the user to understand,

[0811] A means of receiving user feedback and revising the aforementioned document,

[0812] A system that includes this.

[0813] (Claim 2)

[0814] The system according to claim 1, wherein the collection means acquires recorded data and update history in real time via a network of business processing devices.

[0815] (Claim 3)

[0816] The system according to claim 1, wherein the analysis means detects frequently occurring operation patterns from recorded data and generates a manual adapted to the emotional state.

[0817] "Application example 2 when combining with an emotional engine"

[0818] (Claim 1)

[0819] A means connected to a business processing device for collecting user operation record information and screen interface update information,

[0820] A means for analyzing the collected record information and update information and extracting the operating procedure,

[0821] A means for generating documents based on extracted operating procedures by applying natural language generation technology,

[0822] Means for acquiring image information related to the said document and integrating it into the said document,

[0823] Means for recognizing the user's emotional state and adaptively adjusting the generated document according to the user's emotional state,

[0824] A means of receiving user feedback and revising the aforementioned document,

[0825] A system that includes this.

[0826] (Claim 2)

[0827] The system according to claim 1, wherein the collection means acquires recorded information and updated information in real time via the information network of the business processing device.

[0828] (Claim 3)

[0829] The system according to claim 1, wherein the analysis means detects frequently occurring operation patterns from recorded information, and further analyzes the user's emotional information using emotion analysis technology. [Explanation of symbols]

[0830] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means connected to a business processing device for collecting user operation record data and screen interface update history, A means for analyzing the collected record data and update history and extracting the operating procedure, A means for generating documents based on extracted operating procedures by applying natural language generation technology, Means for acquiring image data related to the said document and integrating it into the said document, A means of receiving user feedback and revising the aforementioned document, A system that includes this.

2. The system according to claim 1, wherein the collection means acquires recorded data and update history in real time via a network of business processing devices.

3. The system according to claim 1, wherein the analysis means detects frequently occurring operation patterns from recorded data.

Images