system

Abstract

We provide the system. [Solution] A means to identify error-prone areas based on past work results and highlight them using a specific display means, A means for automatically displaying relevant work manuals and text templates, A means of performing authentication using user authentication information and adjusting the displayed content according to the user's permissions, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance as a response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In conventional in-house operation tasks, for tasks that cannot be fully automated, in many cases, employees need to separately refer to manuals or email templates, often resulting in a decrease in work efficiency. In addition, there are insufficient means to alert employees by making use of mistakes or near-miss information that occurred in similar tasks in the past, and there was a high possibility that the same mistakes would be repeated. As a result, the productivity and quality of the business have been a problem. An object of the present invention is to solve these problems and provide a system for preventing operation mistakes and improving work efficiency.

Means for Solving the Problems

[0005] The present invention solves the above problem with a system that includes means for identifying error-prone areas based on past work results and highlighting them using specific display means, means for automatically displaying relevant work manuals and text templates, and means for performing authentication using user authentication information and adjusting the displayed content according to the user's authority. When a user accesses a specific operation screen, this system can automatically acquire and display instruction information related to the relevant work content, and further learns precautions for error-prone areas from past records and highlights that content in red.

[0006] "Past work results" refers to records and results of operations and tasks performed in the past.

[0007] "Error-prone areas" are parts that have been identified, based on past work results, as being highly likely to cause errors in users.

[0008] "Specific display means" refers to a method or device used to visually highlight and present to the user areas that are prone to errors.

[0009] "Highlighting" refers to various methods used to visually make specific information stand out (e.g., displaying text in red, changing the background color, etc.).

[0010] A "related work manual" is a document that contains the procedures and information necessary to perform a specific operation or task.

[0011] A "text template" is a document that contains pre-prepared, standardized phrases tailored to a specific purpose.

[0012] "User authentication information" refers to information such as user IDs and passwords that users provide to access the system.

[0013] "Authentication" refers to the act of verifying whether a user has legitimate access rights based on the authentication information they have provided.

[0014] "Adjust according to authority" refers to changing the displayed information and functions based on the user's job position and authority level.

[0015] "Operation screen" refers to an interface for performing specific operations and work operations.

[0016] "Points for attention" refers to information on points and risks that should be particularly noted during work.

[0017] "Near-miss information" is a record of significant problems or events that could have led to mistakes that occurred during work in the past. [[ID=1\5]]

Brief description of the drawings

[0018] [Figure 1] It is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10]Shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Mode for Carrying Out the Invention

[0024] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0026] [First Embodiment]

[0027] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0028] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0031] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0034] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0038] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0039] This invention is a system for streamlining internal operations and preventing errors. Based on past work results, this system identifies error-prone areas, highlights them using specific display methods, and automatically displays relevant work manuals and text templates. It also has a function to authenticate users using user authentication information and adjust the displayed content according to their permissions.

[0040] User login authentication

[0041] To log in to the system, the user enters a username and password. The terminal sends this information to the server, which retrieves the user information from the database and performs authentication. If authentication is successful, the server generates a session ID and returns it to the terminal. If authentication fails, an error message is returned to the terminal.

[0042] Display manual templates

[0043] When a user opens a specific operation screen (e.g., inventory management), the terminal obtains the operation ID and sends a request to the server. The server retrieves the corresponding work manual and text template from the database and sends them to the terminal. The terminal displays the received manual and template in the upper right corner of the screen.

[0044] Display of near-miss information

[0045] The server identifies error-prone areas based on past work results and retrieves near-miss information from the database. The server sends this information to the terminal, which receives it and highlights the relevant fields or areas in red text.

[0046] Specific example

[0047] For example, consider the case where a user opens the "Inventory Management" screen. After logging in, when the user opens the inventory management screen, the terminal sends the "Inventory Management" operation ID to the server. The server retrieves the relevant manual (e.g., inventory check procedure) and text template (e.g., inventory shortage notification email template) from the database and sends them to the terminal. The terminal displays this information in the upper right corner of the screen.

[0048] Furthermore, the server retrieves near-miss information from past work results and sends relevant cautionary notes (e.g., points to be careful about when entering a specific inventory quantity) to the terminal. The terminal receives this and displays "This field requires attention" in red text in the relevant field (e.g., inventory quantity input field).

[0049] In this way, this system assists users in their operations, preventing operational errors and improving work efficiency.

[0050] The following describes the processing flow.

[0051] Step 1:

[0052] The user accesses the system and enters their username and password on the login screen.

[0053] Step 2:

[0054] The terminal sends the entered username and password to the server. This transmission is performed using HTTPS communication.

[0055] Step 3:

[0056] The server checks the database and verifies the submitted username and password. If the user information matches, authentication is successful; otherwise, authentication fails.

[0057] Step 4:

[0058] The server returns the authentication result to the terminal. If authentication is successful, it generates and returns a session ID; if authentication fails, it returns an error message.

[0059] Step 5:

[0060] The user opens a specific operation screen (e.g., inventory management).

[0061] Step 6:

[0062] The terminal obtains an operation ID and sends a request containing that ID to the server.

[0063] Step 7:

[0064] The server accesses the database and retrieves the relevant work manual and text template.

[0065] Step 8:

[0066] The server sends the acquired work manual and text template to the terminal.

[0067] Step 9:

[0068] The device analyzes the received manual and template and displays them in the upper right corner of the screen.

[0069] Step 10:

[0070] The server retrieves past near-miss information related to the operation ID from the database.

[0071] Step 11:

[0072] The server sends near-miss information it has acquired to the terminal.

[0073] Step 12:

[0074] The terminal analyzes near-miss information and highlights the relevant fields or areas in red text. For example, it might display a warning message such as "This field requires attention" in the "Inventory Quantity Input Field."

[0075] In this way, this system assists users in their operations, preventing operational errors and improving work efficiency.

[0076] (Example 1)

[0077] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0078] The conventional system fails to adequately streamline internal operations and prevent errors, and is inadequate at identifying and warning users about error-prone areas. Furthermore, insufficient user authentication and access control prevent the display of appropriate information. Additionally, the lack of an automatic display function for necessary work instructions and templates on the operation screen places a significant burden on users.

[0079] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0080] In this invention, the server includes means for identifying error-prone areas based on past work results and highlighting them using specific display means; means for automatically displaying relevant work instructions and text templates; means for authenticating using user authentication information and adjusting the display content according to the user's authority; and means for matching the authentication information entered by the user with a database and generating a session ID. This enables increased efficiency and error prevention in internal operations.

[0081] "Past work results" refers to the results and records of tasks performed in the past.

[0082] "Areas prone to errors" are those where mistakes frequently occur based on past work results, or where extra care is required.

[0083] "Specific display means" refers to a format or method used to highlight a specific area on the screen for the user.

[0084] A "work instruction sheet" is a document that details the procedures and instructions for a task.

[0085] A "text template" is a standardized template of phrases used for specific tasks or operations.

[0086] "User authentication information" refers to information such as the username and password that a user uses to log in to the system.

[0087] "Authentication" is the process of verifying whether the information entered by the user matches the information recorded in the database.

[0088] "Permissions" are settings used to restrict the operations a user can perform and the information they can access within a system.

[0089] A "session ID" is an identifier generated to uniquely identify a user who is logged into the system.

[0090] An "operation screen" is the system interface that a user uses to perform a specific task or operation.

[0091] A "machine learning algorithm" is a computational method used to recognize and analyze data patterns, and is a model designed to efficiently perform a specific task.

[0092] A "database" is a system that systematically stores data and makes it easily accessible and manageable.

[0093] The system of this invention is designed to streamline internal operations and prevent errors. This system consists of three components—a server, a terminal, and a user—that interact with each other.

[0094] System Configuration

[0095] Specific examples of hardware and software

[0096] Server: Apache® Tomcat will be used. Database: MySQL® will be used.

[0097] Terminal: Any computer terminal, using either Google Chrome® or Mozilla Firefox as the browser.

[0098] User: An employee within a company who uses the system.

[0099] Specific operation of the system

[0100] User Authentication

[0101] The user enters their username and password on the login page. This information is sent from the device to the server. The server retrieves the user information from the MySQL database and performs authentication. If authentication is successful, the server generates a session ID and returns it to the device. If authentication fails, an error message is displayed on the device.

[0102] Display manual templates

[0103] The user opens a specific operation screen (e.g., inventory management). The terminal obtains an operation ID and sends a request to the server. Based on this, the server retrieves the corresponding work order and text template from the MySQL database and returns them to the terminal. The terminal displays the received information in the upper right corner of the screen.

[0104] Display of near-miss information

[0105] The server analyzes past work results using machine learning algorithms (e.g., decision trees, random forests) to identify areas prone to errors. It retrieves this information from a MySQL database and sends it to the terminal. The terminal highlights the received information in red text in the corresponding fields or areas.

[0106] Specific example

[0107] For example, when a user opens the "Inventory Management" screen, it works as follows:

[0108] 1. When a user logs in and opens the inventory management screen, the terminal sends the "Inventory Management" operation ID to the server.

[0109] 2. The server retrieves the relevant work order (e.g., "Inventory Check Procedure") and text template (e.g., "Inventory Shortage Notification Email Template") from the MySQL database and returns them to the terminal.

[0110] 3. The device will display this information in the upper right corner of the screen.

[0111] 4. Furthermore, the server retrieves near-miss information from past work results and sends relevant cautionary notes (e.g., "Points to note when entering a specific inventory quantity") to the terminal.

[0112] 5. The terminal displays the acquired information in red text in the target field (e.g., the inventory quantity input field) with the message "This field requires attention."

[0113] System Usage Procedures

[0114] This system is designed to be easy for users to operate and to prevent errors. By using generative AI models, it is possible to analyze and predict data and optimize user interaction using specific prompts.

[0115] For example, the following are examples of prompt statements to be input to a generative AI model.

[0116] "Please explain the system's processing procedure for displaying near-miss information based on past work results when the inventory management screen is opened, and displaying related work instructions and text templates in the upper right corner of the screen."

[0117] Thus, the present invention is a system that dramatically simplifies user operation and significantly improves work efficiency.

[0118] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0119] Step 1:

[0120] The user enters their username and password on the login page. The device obtains this authentication information and sends it to the server using the HTTPS protocol.

[0121] Input: Username, Password

[0122] Data processing: Encrypt authentication information using the HTTPS protocol.

[0123] Output: Encrypted credentials

[0124] Step 2:

[0125] The server attempts authentication by comparing the received authentication information with the MySQL database. If the comparison is successful, it generates a session ID and returns it to the terminal. If the comparison fails, it generates an error message and sends it to the terminal.

[0126] Input: Encrypted credentials

[0127] Data processing: Database matching of authentication information, session ID generation, or error message generation.

[0128] Output: Session ID or error message

[0129] Step 3:

[0130] The user opens a specific operation screen (e.g., inventory management). The terminal obtains an operation ID and sends it to the server as a request.

[0131] Input: Operation ID

[0132] Data processing: Generating HTTP requests

[0133] Output: HTTP request

[0134] Step 4:

[0135] The server queries the MySQL database for the relevant work orders and text templates and returns them to the terminal.

[0136] Input: HTTP Request

[0137] Data Calculation: Database queries for work instructions and text templates

[0138] Output: Work instructions and text templates

[0139] Step 5:

[0140] The terminal analyzes the received work instructions and text templates and displays them in the upper right corner of the screen. This is done using JavaScript (registered trademark) and front-end frameworks such as React.

[0141] Input: Work instructions and text templates

[0142] Data processing: Data analysis using front-end frameworks

[0143] Output: Screen display

[0144] Step 6:

[0145] The server analyzes past work results using machine learning algorithms (e.g., decision trees, random forests) to identify areas prone to errors. It then retrieves this information from a database and sends it to the terminal.

[0146] Input: Past work results

[0147] Data processing: Analysis using machine learning algorithms and identification of error-prone areas.

[0148] Output: Information on common errors

[0149] Step 7:

[0150] The terminal analyzes the information it receives, identifying areas prone to errors, and highlights the corresponding fields or areas in red text.

[0151] Input: Information on areas prone to errors

[0152] Data Processing: Data analysis and highlighting using a front-end framework

[0153] Output: Screen display (highlighted fields and areas)

[0154] Step 8:

[0155] By following the steps described above, users can check the necessary information (e.g., work instructions, text templates, near-miss information) on the inventory management screen and perform their tasks safely and efficiently.

[0156] Input: Output information from each step

[0157] Data Calculation: Verification and manipulation of various data.

[0158] Output: Efficient business operations

[0159] (Application Example 1)

[0160] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0161] Operating machinery in factories is complex and prone to operator errors. These errors can lead to decreased work efficiency and safety concerns. Furthermore, maintaining consistency in work procedures becomes difficult when multiple operators are working simultaneously. To address these challenges and provide an efficient and safe working environment, a system is needed that prevents operational errors and provides appropriate work procedures and warning messages in real time.

[0162] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0163] In this invention, the server includes means for identifying error-prone areas based on past work results and highlighting them using specific display means, means for automatically displaying relevant work manuals and text templates, and means for authenticating using user authentication information and adjusting the display content according to the user's authority. This makes it possible to identify error-prone areas in real time when operating factory machinery and equipment and to display work procedures and warning messages via a smart terminal. Furthermore, by presenting warning messages regarding specific operations using a generative AI model, it is possible to improve operator attention and promote safe operation.

[0164] "Past work results" refers to records of operations and tasks performed previously.

[0165] "Areas prone to errors" refers to areas or items where mistakes are particularly likely to occur during operation or work.

[0166] "Display means" refers to devices or methods for visually presenting information.

[0167] A "work manual" refers to a document or digital file that contains procedures and instructions for performing a specific task or operation.

[0168] A "text template" refers to a set of pre-written phrases designed for a specific purpose or operation.

[0169] "User authentication information" refers to data used to identify a user and verify their permissions (e.g., username and password).

[0170] "Permissions" refer to access rights to operations and information held by a specific user.

[0171] "Factory machinery and equipment" refers to devices and systems used for the production and processing of industrial products.

[0172] "Real-time" refers to the processing and display of information and data instantly, without delay.

[0173] "Smart devices" refer to devices with advanced functions, such as smartphones, tablets, and smart glasses.

[0174] "Work procedure" refers to a series of instructions or methods for performing a specific operation or task.

[0175] A "warning message" refers to information or notifications displayed to draw attention to something.

[0176] A "generative AI model" refers to a model that uses artificial intelligence technology to generate messages or predictions for specific tasks.

[0177] An "operator" refers to a person who operates and manages machinery or systems.

[0178] This invention provides a system that identifies common error points in factory machinery operation in real time and displays work procedures and warning messages via a smart terminal. The system is configured as follows:

[0179] Authentication Steps

[0180] To log in to the system, the user enters a username and password. The terminal sends this information to the server, which retrieves the user information from the database and performs authentication. If authentication is successful, the server generates a session ID and returns it to the terminal.

[0181] Display of work manuals and text templates

[0182] When a user opens a specific operation screen, the terminal obtains its operation ID and sends a request to the server. The server retrieves the corresponding operation manual and text template from the database and sends them to the terminal. The terminal displays the received manual and template in the upper right corner of the screen.

[0183] Display of near-miss information

[0184] The server identifies error-prone areas based on past work results and retrieves near-miss information from the database. The server sends this information to the terminal, which receives it and highlights the relevant fields or areas in red text.

[0185] Hardware and software to be used

[0186] The hardware of this system consists of smart devices such as smartphones, tablets, and smart glasses. The software uses Flask (a Python web framework), and data is exchanged in JSON format. Furthermore, a generative AI model is used to provide warning messages for specific operations.

[0187] Specific example

[0188] For example, when performing "Operation 1" on a robot used in a factory, the user logs in and opens a specific operation screen. Then, the work procedure manual and operation template text are displayed in the upper right corner of the terminal screen. In addition, warning messages such as "Pay attention to adjusting the sensor on the left" and "Do not apply force when pressing the machine stop button" are highlighted in red based on past operation results.

[0189] Example of a prompt

[0190] The following are examples of prompts to input into a generative AI model:

[0191] "Develop an application that, when opening the screen for Operation 1 performed on a factory robot, highlights common errors and points to note based on past operation history, and displays appropriate work instructions and template text. Write the code for this using Flask."

[0192] In this way, this system reduces operational errors in the factory and improves operator attentiveness, thereby promoting safe and efficient work.

[0193] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0194] Step 1:

[0195] To log in to the system, the user enters a username and password. The input data (username and password) is collected by the terminal and sent to the server. The server uses this information to retrieve user information from the database and performs authentication. If authentication is successful, the server generates a session ID and returns it to the terminal. If authentication fails, the server returns an error message to the terminal.

[0196] Input: Username, Password

[0197] Output: Session ID on success, error message on failure.

[0198] Step 2:

[0199] When a user opens a specific operation screen, the terminal obtains an operation ID and sends it to the server. The server retrieves the corresponding operation manual and text template from its database and sends that data to the terminal. The terminal displays these in the upper right corner of the screen.

[0200] Input: Operation ID

[0201] Output: Operation manual, text template

[0202] Step 3:

[0203] The server identifies error-prone areas based on past work results and retrieves near-miss information from the database. It sends this information to the terminal, which highlights the relevant fields or areas in red. This includes prompts for the server to generate warning messages about specific operations using an AI model and provide them to the user.

[0204] Input: Past work results

[0205] Output: Highlighted near-miss information, red text warnings

[0206] Step 4:

[0207] The device displays warning messages generated using a generation AI model, along with highlighted near-miss information. This allows users to recognize operational pitfalls in real time and perform operations safely.

[0208] Input: Near-miss incident information, generated AI model

[0209] Output: Display of warning message

[0210] In this way, a system has been built that supports actual operations by performing appropriate data input and output at each step.

[0211] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0212] This invention is a system designed to streamline internal operations and prevent errors, and incorporates an emotion engine that recognizes user emotions. Based on the user's emotional information recognized by the emotion engine, this system adjusts the displayed content and presentation method, and further provides operational support and displays warnings.

[0213] User login authentication

[0214] To log in to the system, the user enters a username and password. The terminal sends the entered information to the server, which retrieves user information from the database and performs authentication. If authentication is successful, a session ID is generated and returned to the terminal. If authentication fails, an error message is returned.

[0215] Display manual templates

[0216] The user opens a specific operation screen (e.g., inventory management). The terminal obtains the operation ID and sends a request to the server. The server retrieves the corresponding work manual and text template from the database and sends them to the terminal. The terminal displays the received manual and template in the upper right corner of the screen.

[0217] Display of near-miss information

[0218] The server identifies error-prone areas based on past work results and retrieves near-miss information from the database. The server sends this information to the terminal, which receives it and highlights the relevant fields or areas in red text.

[0219] Introducing an emotional engine

[0220] To recognize the user's emotions in real time, the device activates an emotion engine. The emotion engine analyzes the user's facial expressions, voice, keyboard input speed, etc., to identify their emotional state. The recognized emotional information is then sent to the server.

[0221] Adjustment based on emotional information

[0222] The server dynamically adjusts the displayed content and presentation method based on the emotional information it receives. For example, if the user is stressed, the displayed manuals and templates can be simplified, and important notes can be further emphasized. Conversely, if the user is relaxed, detailed information can be provided to support efficient work.

[0223] Specific example

[0224] For example, consider the case where a user opens the "Inventory Management" screen. After logging in, when the user opens the inventory management screen, the terminal sends the "Inventory Management" operation ID to the server. The server retrieves the relevant manual (e.g., inventory check procedure) and text template (e.g., inventory shortage notification email template) from the database and sends them to the terminal. The terminal displays this information in the upper right corner of the screen.

[0225] Simultaneously, the terminal activates an emotion engine to analyze the user's emotions in real time. For example, if the server detects that the user is stressed, it simplifies the displayed content and highlights important points (e.g., points to note when entering specific inventory quantities) in red. Conversely, if the user is relaxed, it displays a detailed manual to improve work efficiency.

[0226] Thus, the present invention not only assists user operation, prevents operational errors, and improves work efficiency, but also dynamically adjusts the displayed content according to the user's emotional state, providing an even more comfortable and efficient work environment.

[0227] The following describes the processing flow.

[0228] Step 1:

[0229] The user accesses the system and enters their username and password on the login screen.

[0230] Step 2:

[0231] The terminal sends the entered username and password to the server. This transmission is performed using HTTPS communication.

[0232] Step 3:

[0233] The server checks the database and verifies the submitted username and password. If the user information matches, authentication is successful; otherwise, authentication fails.

[0234] Step 4:

[0235] The server returns the authentication result to the terminal. If authentication is successful, it generates and returns a session ID; if authentication fails, it returns an error message.

[0236] Step 5:

[0237] The user opens a specific operation screen (e.g., inventory management).

[0238] Step 6:

[0239] The terminal obtains an operation ID and sends a request containing that ID to the server.

[0240] Step 7:

[0241] The server accesses the database and retrieves the relevant work manual and text template.

[0242] Step 8:

[0243] The server sends the acquired work manual and text template to the terminal.

[0244] Step 9:

[0245] The device analyzes the received manual and template and displays them in the upper right corner of the screen.

[0246] Step 10:

[0247] The device activates an emotion engine that analyzes the user's facial expressions, voice, and keyboard input speed in real time.

[0248] Step 11:

[0249] The emotion engine recognizes the user's emotional state and sends that information to the server.

[0250] Step 12:

[0251] The server retrieves past near-miss information related to the operation ID from the database.

[0252] Step 13:

[0253] The server sends near-miss information it has acquired to the terminal.

[0254] Step 14:

[0255] The server adjusts the displayed content and presentation method based on the emotional information it receives. For example, if the user is stressed, the displayed content is simplified, and warnings are further highlighted in red.

[0256] Step 15:

[0257] The terminal analyzes near-miss information and highlights a warning message in red text in the relevant field or area.

[0258] Step 16:

[0259] The terminal displays customized content received from the server to the user and supports their operation.

[0260] This entire process streamlines user operations, prevents operational errors, and dynamically adjusts displayed content based on the user's emotional state.

[0261] (Example 2)

[0262] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0263] Conventional in-house operating systems fail to adequately prevent operational errors and improve work efficiency. Furthermore, they often lack display adjustments based on the user's emotional state, creating a stressful environment for users. This can lead to decreased user efficiency and an increase in operational errors. Therefore, the present invention aims to solve these problems.

[0264] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0265] In this invention, the server includes means for identifying error-prone areas based on past work results and highlighting them using specific display means, means for automatically displaying relevant work manuals and text templates, means for authenticating using user authentication information and adjusting the display content according to the user's authority, and means for recognizing the user's emotions and dynamically adjusting the display content and presentation method based on the recognized emotions. This enables dynamic display adjustments according to the user's emotional state and work environment, thereby preventing operational errors and improving work efficiency.

[0266] "Past work results" refers to records of the progress and results of a series of operations or tasks performed in the past.

[0267] "Error-prone areas" are points where users are likely to frequently make mistakes during operation or work.

[0268] "Specific display means" refers to methods or devices that highlight designated information or warnings.

[0269] A "work manual" is a document that contains procedures and instructions for performing specific operations or tasks accurately and efficiently.

[0270] A "text template" is a text template created according to a specific format, which can be edited and used as needed.

[0271] "User authentication information" refers to information used to identify a user and verify their permissions, and primarily includes usernames and passwords.

[0272] "User emotions" refer to the psychological state a user is experiencing at a particular moment, and include things like stress and relaxation.

[0273] "Dynamic adjustment" means automatically making changes or modifications in real time according to the environment and conditions.

[0274] A "system" is a comprehensive mechanism in which multiple components work together to achieve a specific function.

[0275] This invention is a system designed to streamline internal operations and prevent errors. By incorporating an emotion engine that recognizes user emotions, this system enables dynamic adjustment of display content and presentation methods based on the user's emotional state.

[0276] First, to log in to the system, the user enters a username and password. The terminal sends the entered information to the server, which retrieves the user information from the database and performs authentication. If authentication is successful, the server generates a session ID and sends it back to the terminal. If authentication fails, the server returns an error message.

[0277] When a user opens a specific operation screen (e.g., inventory management), the terminal sends its operation ID to the server. The server retrieves the corresponding work manual and text template from the database and sends them to the terminal. The terminal displays the received manual and template in the upper right corner of the screen. At this time, based on past work results, it identifies areas prone to errors and retrieves near-miss information from the database. The server sends this information to the terminal, and the terminal highlights this information in red text in the corresponding fields or areas.

[0278] Furthermore, to recognize the user's emotions in real time, the device activates an emotion engine. The emotion engine analyzes the user's facial expressions, voice, and keyboard input speed to identify their emotional state. The recognized emotional information is sent to a server. Based on the received emotional information, the server dynamically adjusts the displayed content and presentation method. For example, if the user is stressed, the displayed manuals and templates are simplified, and important notes are highlighted. On the other hand, if the user is relaxed, detailed information is provided to support efficient work.

[0279] As a concrete example, let's explain what happens when a user opens the inventory management screen. After logging in, when a user opens the inventory management screen, the terminal sends the "Inventory Management" operation ID to the server. The server retrieves the relevant manual (e.g., inventory check procedure) and text template (e.g., inventory shortage notification email template) from the database and sends them to the terminal. The terminal displays this information in the upper right corner of the screen.

[0280] Simultaneously, the terminal activates an emotion engine to analyze the user's emotions in real time. For example, if the server detects that the user is stressed, it simplifies the displayed content and highlights important notes (e.g., points to note when entering specific inventory quantities) in red. Conversely, if the user is relaxed, it displays a detailed manual to improve work efficiency.

[0281] Examples of prompt sentences using the generated AI model include "Please teach me the appropriate manual display method when the user is in a stressed state."

[0282] This system not only realizes the efficiency improvement and error prevention of in-house operation work, but also provides a comfortable and efficient working environment according to the emotional state of the user.

[0283] The flow of the specific process in Example 2 will be described using FIG. 13.

[0284] Step 1: The user enters the username and password on the login screen

[0285] Input: Username, password

[0286] Specific operation: The user enters the username and password on the login screen of the terminal.

[0287] Step 2: The terminal sends the entered username and password to the server

[0288] Input: Username, password

[0289] Specific operation: The terminal encrypts the username and password entered by the user and sends them to the server.

[0290] Step 3: The server retrieves user information from the database and performs authentication

[0291] Input: Username, password (encrypted state)

[0292] Data processing: The server queries the received username and password in the database and performs verification.

[0293] Output: Authentication result (success or failure)

[0294] Specific operation: The server retrieves the corresponding user information from the database and verifies the encrypted password.

[0295] Step 4: Return the authentication result to the terminal

[0296] Input: Authentication result (success or failure)

[0297] Specific operation: The server returns the authentication result to the terminal. In case of success, it generates and returns a session ID. In case of failure, it returns an error message.

[0298] Output: Session ID (when successful), error message (when failed)

[0299] Step 5: The user opens a specific operation screen (e.g., inventory management)

[0300] Specific operation: The user selects a specific operation screen within the system and opens it on the terminal.

[0301] Step 6: The terminal sends the operation ID to the server

[0302] Input: Operation ID

[0303] Specific operation: The terminal sends the operation ID corresponding to the current operation screen to the server.

[0304] Step 7: The server retrieves the work manual and text template from the database

[0305] Input: Operation ID

[0306] Data processing: The server retrieves the corresponding work manual and text template from the database.

[0307] Output: Work manual, text template

[0308] Specific operation: The server queries the database and retrieves the necessary manuals and templates.

[0309] Step 8: The device displays the manual and template on the screen.

[0310] Input: Work manual, text template

[0311] Specific action: The device displays the received manual and template in the upper right corner of the screen.

[0312] Step 9: The server identifies near-miss information based on past work results.

[0313] Input: Past work data

[0314] Data processing: The server analyzes past work data to identify areas prone to errors.

[0315] Output: Near Miss Information

[0316] Specific operation: The server retrieves past work results from the database and identifies near-miss information.

[0317] Step 10: The terminal displays near-miss information.

[0318] Input: Near-miss information

[0319] Specific operation: The terminal receives near-miss information and highlights the relevant field or area in red text.

[0320] Step 11: The device activates the emotion engine.

[0321] Specific action: The terminal starts the emotion engine software.

[0322] Step 12: The emotion engine analyzes the user's facial expressions, voice, and keyboard input speed.

[0323] Input: User's facial expressions, voice, keyboard input speed

[0324] Data processing: The emotion engine analyzes this data to identify emotional states.

[0325] Output: Emotional information

[0326] Specific operation: The emotion engine analyzes data in real time and sends the user's emotional state to the server.

[0327] Step 13: The server dynamically adjusts the displayed content based on sentiment information.

[0328] Input: Emotional information

[0329] Data processing: The server analyzes emotional information and adjusts the displayed content and presentation method.

[0330] Output: Display content after adjustment

[0331] Specific operation: Based on emotional information, the server simplifies the displayed content and highlights warnings if the user is stressed. If the user is relaxed, it displays detailed information.

[0332] Step 14: The device reflects the adjusted display content on the screen.

[0333] Input: Adjusted display content

[0334] Specific operation: The terminal displays the adjusted content received from the server on the screen.

[0335] (Application Example 2)

[0336] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0337] There is a need to improve work efficiency and prevent errors on factory production lines. However, conventional systems do not take into account the mental state of workers, so stress and fatigue can lead to work errors. As a result, production efficiency decreases, and quality declines and safety problems occur. To solve these problems, it is important to recognize workers' emotions in real time and dynamically adjust work instructions and precautions based on that.

[0338] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0339] In this invention, the server includes means for identifying error-prone areas based on past work results and highlighting them using specific display means; means for automatically displaying relevant work manuals and text templates; means for authenticating using user authentication information and adjusting the display content according to the user's authority; means for recognizing the user's emotions in real time; and means for dynamically adjusting the display content and presentation method based on the recognized emotion information. This makes it possible to improve work efficiency and prevent errors by adjusting the display content according to the worker's mental state.

[0340] "Past work results" refer to the results and data from work performed previously.

[0341] "Error-prone areas" refer to parts or processes in a work where mistakes frequently occur.

[0342] "Display means" refers to devices or methods for presenting information visually.

[0343] A "work manual" is a document that details the procedures and methods for performing a specific task.

[0344] A "text template" is a pre-prepared document template that follows a specific format.

[0345] "User authentication information" refers to identification information used to identify a user and verify their permissions.

[0346] "Authentication" is the process of verifying whether a user is legitimate.

[0347] "Permissions" refer to the range of operations a user can perform and information they can access within a system.

[0348] "Emotion recognition technology" refers to technology for detecting and identifying human emotions in real time.

[0349] "Emotional information" refers to data about a user's emotional state detected by emotion recognition tools.

[0350] "Dynamic adjustment" means making flexible changes in real time according to the situation and conditions.

[0351] A "user" is a person, such as a worker or employee, who uses the system.

[0352] A "system" is a collection of technical components, such as hardware, software, and networks, that are combined to achieve a specific purpose.

[0353] This invention is a system aimed at improving work efficiency and preventing errors on a factory production line. Specifically, it has a function to identify error-prone areas based on past work results and highlight them using a display device, and a function to automatically display relevant work manuals and text templates. Furthermore, this system recognizes the user's emotions in real time and dynamically adjusts the display content and presentation method based on that emotion information.

[0354] System Configuration

[0355] Hardware:

[0356] Camera: Used to capture the facial expressions of the workers.

[0357] Computer: Used to perform emotion recognition and adjust the displayed content.

[0358] Robots: Provide work assistance and instructions.

[0359] software:

[0360] OpenCV: A library for image processing.

[0361] Dlib: A library for face detection and facial landmark extraction.

[0362] Keras: A deep learning framework for using emotion recognition models.

[0363] Server: Stores and processes data.

[0364] Terminal: The interface that the user interacts with.

[0365] Program processing

[0366] The server receives video data transmitted from the camera and detects faces using OpenCV and Dlib. A Keras-based emotion recognition model analyzes the detected face regions in real time, and the results are obtained. This emotion information is sent to the terminal and used to adjust the displayed content. If the user is stressed, the server simplifies the displayed content and highlights important notices. Conversely, if the user is relaxed, a detailed manual is displayed to improve work efficiency.

[0367] Specific example

[0368] For example, when a user opens the inventory management operation screen, the terminal sends the "Inventory Management" operation ID to the server. The server retrieves the work manual and text template from the database and sends them to the terminal. Simultaneously, the terminal performs emotion recognition and sends the user's emotional state to the server. If the server detects that the user is stressed, it simplifies the displayed content and highlights important notes (such as points to note when entering specific inventory quantities) in red. Conversely, if the user is relaxed, it displays a detailed manual to improve work efficiency.

[0369] This system provides a flexible work environment tailored to the mental state of the worker, enabling improved work efficiency and prevention of operational errors.

[0370] Example of a prompt

[0371] "Please propose a program that uses emotion recognition to dynamically adjust operational guidelines based on the fatigue level of factory workers. When fatigued, the instructions should be simplified and precautions should be emphasized."

[0372] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0373] Step 1:

[0374] Input: User login information (username, password)

[0375] Specific operation and data processing: The terminal receives the username and password entered by the user and sends them to the server.

[0376] Output: The server retrieves user information from the database, generates a session ID if authentication is successful, and returns it to the terminal. If authentication fails, it returns an error message.

[0377] Step 2:

[0378] Input: User selects an operation screen (e.g., inventory management)

[0379] Specific actions and data processing: The terminal obtains an operation ID and sends it to the server as a request.

[0380] Output: The server retrieves the relevant work manual and text template from the database and sends it to the terminal. The terminal displays this information in the upper right corner of the screen.

[0381] Step 3:

[0382] Input: Past work results

[0383] Specific operations and data processing: The server analyzes past work results and identifies areas prone to errors.

[0384] Output: The server retrieves near-miss information from the database and sends it to the terminal. The terminal highlights the relevant fields or areas in red text.

[0385] Step 4:

[0386] Input: Real-time user emotion data (facial expressions, voice, keyboard input speed, etc.)

[0387] Specific operation and data processing: The terminal activates an emotion engine and analyzes the user's emotions in real time. The user's facial expressions are captured by the camera, faces are detected using OpenCV and Dlib, and emotions are analyzed using an emotion recognition model based on Keras.

[0388] Output: Recognized emotion information is sent from the terminal to the server.

[0389] Step 5:

[0390] Input: Emotional information (e.g., stressed state, relaxed state)

[0391] Specific operations and data processing: The server analyzes emotional information and dynamically adjusts the displayed content and presentation method. For example, if the user is stressed, the displayed content is simplified and warnings are highlighted. If the user is relaxed, a detailed manual is provided.

[0392] Output: The adjusted display content is sent to the terminal and displayed to the user.

[0393] Step 6:

[0394] Input: User actions according to the situation

[0395] Specific actions and data processing: The system performs operations based on the information provided by the user. The results of the operations are sent from the terminal to the server.

[0396] Output: The server records the operation results and adjusts the displayed content and instructions as needed.

[0397] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0398] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0399] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0400] [Second Embodiment]

[0401] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0402] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0403] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0404] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0405] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0406] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0407] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0408] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0409] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0410] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0411] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0412] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0413] This invention is a system for streamlining internal operations and preventing errors. Based on past work results, this system identifies error-prone areas, highlights them using specific display methods, and automatically displays relevant work manuals and text templates. It also has a function to authenticate users using user authentication information and adjust the displayed content according to their permissions.

[0414] User login authentication

[0415] To log in to the system, the user enters a username and password. The terminal sends this information to the server, which retrieves the user information from the database and performs authentication. If authentication is successful, the server generates a session ID and returns it to the terminal. If authentication fails, an error message is returned to the terminal.

[0416] Display manual templates

[0417] When a user opens a specific operation screen (e.g., inventory management), the terminal obtains the operation ID and sends a request to the server. The server retrieves the corresponding work manual and text template from the database and sends them to the terminal. The terminal displays the received manual and template in the upper right corner of the screen.

[0418] Display of near-miss information

[0419] The server identifies error-prone areas based on past work results and retrieves near-miss information from the database. The server sends this information to the terminal, which receives it and highlights the relevant fields or areas in red text.

[0420] Specific example

[0421] For example, consider the case where a user opens the "Inventory Management" screen. After logging in, when the user opens the inventory management screen, the terminal sends the "Inventory Management" operation ID to the server. The server retrieves the relevant manual (e.g., inventory check procedure) and text template (e.g., inventory shortage notification email template) from the database and sends them to the terminal. The terminal displays this information in the upper right corner of the screen.

[0422] Furthermore, the server retrieves near-miss information from past work results and sends relevant cautionary notes (e.g., points to be careful about when entering a specific inventory quantity) to the terminal. The terminal receives this and displays "This field requires attention" in red text in the relevant field (e.g., inventory quantity input field).

[0423] In this way, this system assists users in their operations, preventing operational errors and improving work efficiency.

[0424] The following describes the processing flow.

[0425] Step 1:

[0426] The user accesses the system and enters their username and password on the login screen.

[0427] Step 2:

[0428] The terminal sends the entered username and password to the server. This transmission is performed using HTTPS communication.

[0429] Step 3:

[0430] The server checks the database and verifies the submitted username and password. If the user information matches, authentication is successful; otherwise, authentication fails.

[0431] Step 4:

[0432] The server returns the authentication result to the terminal. If authentication is successful, it generates and returns a session ID; if authentication fails, it returns an error message.

[0433] Step 5:

[0434] The user opens a specific operation screen (e.g., inventory management).

[0435] Step 6:

[0436] The terminal obtains an operation ID and sends a request containing that ID to the server.

[0437] Step 7:

[0438] The server accesses the database and retrieves the relevant work manual and text template.

[0439] Step 8:

[0440] The server sends the acquired work manual and text template to the terminal.

[0441] Step 9:

[0442] The device analyzes the received manual and template and displays them in the upper right corner of the screen.

[0443] Step 10:

[0444] The server retrieves past near-miss information related to the operation ID from the database.

[0445] Step 11:

[0446] The server sends near-miss information it has acquired to the terminal.

[0447] Step 12:

[0448] The terminal analyzes near-miss information and highlights the relevant fields or areas in red text. For example, it might display a warning message such as "This field requires attention" in the "Inventory Quantity Input Field."

[0449] In this way, this system assists users in their operations, preventing operational errors and improving work efficiency.

[0450] (Example 1)

[0451] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0452] The conventional system fails to adequately streamline internal operations and prevent errors, and is inadequate at identifying and warning users about error-prone areas. Furthermore, insufficient user authentication and access control prevent the display of appropriate information. Additionally, the lack of an automatic display function for necessary work instructions and templates on the operation screen places a significant burden on users.

[0453] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0454] In this invention, the server includes means for identifying error-prone areas based on past work results and highlighting them using specific display means; means for automatically displaying relevant work instructions and text templates; means for authenticating using user authentication information and adjusting the display content according to the user's authority; and means for matching the authentication information entered by the user with a database and generating a session ID. This enables increased efficiency and error prevention in internal operations.

[0455] "Past work results" refers to the results and records of tasks performed in the past.

[0456] "Areas prone to errors" are those where mistakes frequently occur based on past work results, or where extra care is required.

[0457] "Specific display means" refers to a format or method used to highlight a specific area on the screen for the user.

[0458] A "work instruction sheet" is a document that details the procedures and instructions for a task.

[0459] A "text template" is a standardized template of phrases used for specific tasks or operations.

[0460] "User authentication information" refers to information such as the username and password that a user uses to log in to the system.

[0461] "Authentication" is the process of verifying whether the information entered by the user matches the information recorded in the database.

[0462] "Permissions" are settings used to restrict the operations a user can perform and the information they can access within a system.

[0463] A "session ID" is an identifier generated to uniquely identify a user who is logged into the system.

[0464] An "operation screen" is the system interface that a user uses to perform a specific task or operation.

[0465] A "machine learning algorithm" is a computational method used to recognize and analyze data patterns, and is a model designed to efficiently perform a specific task.

[0466] A "database" is a system that systematically stores data and makes it easily accessible and manageable.

[0467] The system of this invention is designed to streamline internal operations and prevent errors. This system consists of three components—a server, a terminal, and a user—that function in conjunction with each other.

[0468] System Configuration

[0469] Specific examples of hardware and software

[0470] Server: Apache Tomcat will be used. Database: MySQL will be used.

[0471] Terminal: Any computer terminal, using either Google Chrome or Mozilla Firefox as the browser.

[0472] User: An employee within a company who uses the system.

[0473] Specific operation of the system

[0474] User Authentication

[0475] The user enters their username and password on the login page. This information is sent from the device to the server. The server retrieves the user information from the MySQL database and performs authentication. If authentication is successful, the server generates a session ID and returns it to the device. If authentication fails, an error message is displayed on the device.

[0476] Display manual templates

[0477] The user opens a specific operation screen (e.g., inventory management). The terminal obtains an operation ID and sends a request to the server. Based on this, the server retrieves the corresponding work order and text template from the MySQL database and returns them to the terminal. The terminal displays the received information in the upper right corner of the screen.

[0478] Display of near-miss information

[0479] The server analyzes past work results using machine learning algorithms (e.g., decision trees, random forests) to identify areas prone to errors. It retrieves this information from a MySQL database and sends it to the terminal. The terminal highlights the received information in red text in the corresponding fields or areas.

[0480] Specific example

[0481] For example, when a user opens the "Inventory Management" screen, it works as follows:

[0482] 1. When a user logs in and opens the inventory management screen, the terminal sends the "Inventory Management" operation ID to the server.

[0483] 2. The server retrieves the relevant work order (e.g., "Inventory Check Procedure") and text template (e.g., "Inventory Shortage Notification Email Template") from the MySQL database and returns them to the terminal.

[0484] 3. The device will display this information in the upper right corner of the screen.

[0485] 4. Furthermore, the server retrieves near-miss information from past work results and sends relevant cautionary notes (e.g., "Points to note when entering a specific inventory quantity") to the terminal.

[0486] 5. The terminal displays the acquired information in red text in the target field (e.g., the inventory quantity input field) with the message "This field requires attention."

[0487] System Usage Procedures

[0488] This system is designed to be easy for users to operate and to prevent errors. By using generative AI models, it is possible to analyze and predict data and optimize user interaction using specific prompts.

[0489] For example, the following are examples of prompt statements to be input to a generative AI model.

[0490] "Please explain the system's processing procedure for displaying near-miss information based on past work results when the inventory management screen is opened, and displaying related work instructions and text templates in the upper right corner of the screen."

[0491] Thus, the present invention is a system that dramatically simplifies user operation and significantly improves work efficiency.

[0492] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0493] Step 1:

[0494] The user enters their username and password on the login page. The device obtains this authentication information and sends it to the server using the HTTPS protocol.

[0495] Input: Username, Password

[0496] Data processing: Encrypt authentication information using the HTTPS protocol.

[0497] Output: Encrypted credentials

[0498] Step 2:

[0499] The server attempts authentication by comparing the received authentication information with the MySQL database. If the comparison is successful, it generates a session ID and returns it to the terminal. If the comparison fails, it generates an error message and sends it to the terminal.

[0500] Input: Encrypted credentials

[0501] Data processing: Database matching of authentication information, session ID generation, or error message generation.

[0502] Output: Session ID or error message

[0503] Step 3:

[0504] The user opens a specific operation screen (e.g., inventory management). The terminal obtains an operation ID and sends it to the server as a request.

[0505] Input: Operation ID

[0506] Data processing: Generating HTTP requests

[0507] Output: HTTP request

[0508] Step 4:

[0509] The server queries the MySQL database for the relevant work orders and text templates and returns them to the terminal.

[0510] Input: HTTP Request

[0511] Data Calculation: Database queries for work instructions and text templates

[0512] Output: Work instructions and text templates

[0513] Step 5:

[0514] The terminal analyzes the received work instructions and text templates and displays them in the upper right corner of the screen. This is done using JavaScript or front-end frameworks such as React.

[0515] Input: Work instructions and text templates

[0516] Data processing: Data analysis using a front-end framework

[0517] Output: Screen display

[0518] Step 6:

[0519] The server analyzes past work results using machine learning algorithms (e.g., decision trees, random forests) to identify areas prone to errors. It then retrieves this information from a database and sends it to the terminal.

[0520] Input: Past work results

[0521] Data processing: Analysis using machine learning algorithms and identification of error-prone areas.

[0522] Output: Information on common errors

[0523] Step 7:

[0524] The terminal analyzes the information it receives, identifying areas prone to errors, and highlights the corresponding fields or areas in red text.

[0525] Input: Information on areas prone to errors

[0526] Data Processing: Data analysis and highlighting using a front-end framework

[0527] Output: Screen display (highlighted fields and areas)

[0528] Step 8:

[0529] By following the steps described above, users can check the necessary information (e.g., work instructions, text templates, near-miss information) on the inventory management screen and perform their tasks safely and efficiently.

[0530] Input: Output information from each step

[0531] Data Calculation: Verification and manipulation of various data.

[0532] Output: Efficient business operations

[0533] (Application Example 1)

[0534] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0535] Operating machinery in factories is complex and prone to operator errors. These errors can lead to decreased work efficiency and safety concerns. Furthermore, maintaining consistency in work procedures becomes difficult when multiple operators are working simultaneously. To address these challenges and provide an efficient and safe working environment, a system is needed that prevents operational errors and provides appropriate work procedures and warning messages in real time.

[0536] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0537] In this invention, the server includes means for identifying error-prone areas based on past work results and highlighting them using specific display means, means for automatically displaying relevant work manuals and text templates, and means for authenticating using user authentication information and adjusting the display content according to the user's authority. This makes it possible to identify error-prone areas in real time when operating factory machinery and equipment and to display work procedures and warning messages via a smart terminal. Furthermore, by presenting warning messages regarding specific operations using a generative AI model, it is possible to improve operator attention and promote safe operation.

[0538] "Past work results" refers to records of operations and tasks performed previously.

[0539] "Areas prone to errors" refers to areas or items where mistakes are particularly likely to occur during operation or work.

[0540] "Display means" refers to devices or methods for visually presenting information.

[0541] A "work manual" refers to a document or digital file that contains procedures and instructions for performing a specific task or operation.

[0542] A "text template" refers to a set of pre-written phrases designed for a specific purpose or operation.

[0543] "User authentication information" refers to data used to identify a user and verify their permissions (e.g., username and password).

[0544] "Permissions" refer to access rights to operations and information held by a specific user.

[0545] "Factory machinery and equipment" refers to devices and systems used for the production and processing of industrial products.

[0546] "Real-time" refers to the processing and display of information and data instantly, without delay.

[0547] "Smart devices" refer to devices with advanced functions such as smartphones, tablets, and smart glasses.

[0548] "Work procedure" refers to a series of instructions or methods for performing a specific operation or task.

[0549] A "warning message" refers to information or notifications displayed to draw attention.

[0550] A "generative AI model" refers to a model that uses artificial intelligence technology to generate messages or predictions for specific tasks.

[0551] An "operator" refers to a person who operates and manages machinery or systems.

[0552] This invention provides a system that identifies common error points in factory machinery operation in real time and displays work procedures and warning messages via a smart terminal. The system is configured as follows:

[0553] Authentication Steps

[0554] To log in to the system, the user enters a username and password. The terminal sends this information to the server, which retrieves the user information from the database and performs authentication. If authentication is successful, the server generates a session ID and returns it to the terminal.

[0555] Display of work manuals and text templates

[0556] When a user opens a specific operation screen, the terminal obtains its operation ID and sends a request to the server. The server retrieves the corresponding operation manual and text template from the database and sends them to the terminal. The terminal displays the received manual and template in the upper right corner of the screen.

[0557] Display of near-miss information

[0558] The server identifies error-prone areas based on past work results and retrieves near-miss information from the database. The server sends this information to the terminal, which receives it and highlights the relevant fields or areas in red text.

[0559] Hardware and software to be used

[0560] The hardware of this system consists of smart devices such as smartphones, tablets, and smart glasses. The software uses Flask (a Python web framework), and data is exchanged in JSON format. Furthermore, a generative AI model is used to provide warning messages for specific operations.

[0561] Specific example

[0562] For example, when performing "Operation 1" on a robot used in a factory, the user logs in and opens a specific operation screen. Then, the work procedure manual and operation template text are displayed in the upper right corner of the terminal screen. In addition, warning messages such as "Pay attention to adjusting the sensor on the left" and "Do not apply force when pressing the machine stop button" are highlighted in red based on past operation results.

[0563] Example of a prompt

[0564] The following are examples of prompts to input into a generative AI model:

[0565] "Develop an application that, when opening the screen for Operation 1 performed on a factory robot, highlights common errors and points to note based on past operation history, and displays appropriate work instructions and template text. Write the code for this using Flask."

[0566] In this way, the system reduces operational errors in the factory and improves operator attentiveness, thereby promoting safe and efficient work.

[0567] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0568] Step 1:

[0569] To log in to the system, the user enters a username and password. The input data (username and password) is collected by the terminal and sent to the server. The server uses this information to retrieve user information from the database and performs authentication. If authentication is successful, the server generates a session ID and returns it to the terminal. If authentication fails, the server returns an error message to the terminal.

[0570] Input: Username, Password

[0571] Output: Session ID on success, error message on failure.

[0572] Step 2:

[0573] When a user opens a specific operation screen, the terminal obtains an operation ID and sends it to the server. The server retrieves the corresponding operation manual and text template from its database and sends that data to the terminal. The terminal displays these in the upper right corner of the screen.

[0574] Input: Operation ID

[0575] Output: Operation manual, text template

[0576] Step 3:

[0577] The server identifies error-prone areas based on past work results and retrieves near-miss information from the database. It sends this information to the terminal, which highlights the relevant fields or areas in red. This includes prompts for the server to generate warning messages about specific operations using an AI model and provide them to the user.

[0578] Input: Past work results

[0579] Output: Highlighted near-miss information, red text warnings

[0580] Step 4:

[0581] The device displays warning messages generated using a generation AI model, along with highlighted near-miss information. This allows users to recognize operational pitfalls in real time and perform operations safely.

[0582] Input: Near-miss incident information, generated AI model

[0583] Output: Display of warning message

[0584] In this way, a system has been built that supports actual operations by performing appropriate data input and output at each step.

[0585] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0586] This invention is a system designed to streamline internal operations and prevent errors, and incorporates an emotion engine that recognizes user emotions. Based on the user's emotional information recognized by the emotion engine, this system adjusts the displayed content and presentation method, and further provides operational support and displays warnings.

[0587] User login authentication

[0588] To log in to the system, the user enters a username and password. The terminal sends the entered information to the server, which retrieves user information from the database and performs authentication. If authentication is successful, a session ID is generated and returned to the terminal. If authentication fails, an error message is returned.

[0589] Display manual templates

[0590] The user opens a specific operation screen (e.g., inventory management). The terminal obtains the operation ID and sends a request to the server. The server retrieves the corresponding work manual and text template from the database and sends them to the terminal. The terminal displays the received manual and template in the upper right corner of the screen.

[0591] Display of near-miss information

[0592] The server identifies error-prone areas based on past work results and retrieves near-miss information from the database. The server sends this information to the terminal, which receives it and highlights the relevant fields or areas in red text.

[0593] Introducing an emotional engine

[0594] To recognize the user's emotions in real time, the device activates an emotion engine. The emotion engine analyzes the user's facial expressions, voice, keyboard input speed, etc., to identify their emotional state. The recognized emotional information is then sent to the server.

[0595] Adjustment based on emotional information

[0596] The server dynamically adjusts the displayed content and presentation method based on the emotional information it receives. For example, if the user is stressed, the displayed manuals and templates can be simplified, and important notes can be further emphasized. Conversely, if the user is relaxed, detailed information can be provided to support efficient work.

[0597] Specific example

[0598] For example, consider the case where a user opens the "Inventory Management" screen. After logging in, when the user opens the inventory management screen, the terminal sends the "Inventory Management" operation ID to the server. The server retrieves the relevant manual (e.g., inventory check procedure) and text template (e.g., inventory shortage notification email template) from the database and sends them to the terminal. The terminal displays this information in the upper right corner of the screen.

[0599] Simultaneously, the terminal activates an emotion engine to analyze the user's emotions in real time. For example, if the server detects that the user is stressed, it simplifies the displayed content and highlights important points (e.g., points to note when entering specific inventory quantities) in red. Conversely, if the user is relaxed, it displays a detailed manual to improve work efficiency.

[0600] Thus, the present invention not only assists user operation, prevents operational errors, and improves work efficiency, but also dynamically adjusts the displayed content according to the user's emotional state, providing an even more comfortable and efficient work environment.

[0601] The following describes the processing flow.

[0602] Step 1:

[0603] The user accesses the system and enters their username and password on the login screen.

[0604] Step 2:

[0605] The terminal sends the entered username and password to the server. This transmission is performed using HTTPS communication.

[0606] Step 3:

[0607] The server checks the database and verifies the submitted username and password. If the user information matches, authentication is successful; otherwise, authentication fails.

[0608] Step 4:

[0609] The server returns the authentication result to the terminal. If authentication is successful, it generates and returns a session ID; if authentication fails, it returns an error message.

[0610] Step 5:

[0611] The user opens a specific operation screen (e.g., inventory management).

[0612] Step 6:

[0613] The terminal obtains an operation ID and sends a request containing that ID to the server.

[0614] Step 7:

[0615] The server accesses the database and retrieves the relevant work manual and text template.

[0616] Step 8:

[0617] The server sends the acquired work manual and text template to the terminal.

[0618] Step 9:

[0619] The device analyzes the received manual and template and displays them in the upper right corner of the screen.

[0620] Step 10:

[0621] The device activates an emotion engine that analyzes the user's facial expressions, voice, and keyboard input speed in real time.

[0622] Step 11:

[0623] The emotion engine recognizes the user's emotional state and sends that information to the server.

[0624] Step 12:

[0625] The server retrieves past near-miss information related to the operation ID from the database.

[0626] Step 13:

[0627] The server sends near-miss information it has acquired to the terminal.

[0628] Step 14:

[0629] The server adjusts the displayed content and presentation method based on the emotional information it receives. For example, if the user is stressed, the displayed content is simplified, and warnings are further highlighted in red.

[0630] Step 15:

[0631] The terminal analyzes near-miss information and highlights a warning message in red text in the relevant field or area.

[0632] Step 16:

[0633] The terminal displays customized content received from the server to the user and supports their operation.

[0634] This entire process streamlines user operations, prevents operational errors, and dynamically adjusts displayed content based on the user's emotional state.

[0635] (Example 2)

[0636] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0637] Conventional in-house operating systems fail to adequately prevent operational errors and improve work efficiency. Furthermore, they often lack display adjustments based on the user's emotional state, creating a stressful environment for users. This can lead to decreased user efficiency and an increase in operational errors. Therefore, the present invention aims to solve these problems.

[0638] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0639] In this invention, the server includes means for identifying error-prone areas based on past work results and highlighting them using specific display means, means for automatically displaying relevant work manuals and text templates, means for authenticating using user authentication information and adjusting the display content according to the user's authority, and means for recognizing the user's emotions and dynamically adjusting the display content and presentation method based on the recognized emotions. This enables dynamic display adjustments according to the user's emotional state and work environment, thereby preventing operational errors and improving work efficiency.

[0640] "Past work results" refers to records of the progress and results of a series of operations or tasks performed in the past.

[0641] "Error-prone areas" are points where users are likely to frequently make mistakes during operation or work.

[0642] "Specific display means" refers to methods or devices that highlight designated information or warnings.

[0643] A "work manual" is a document that contains procedures and instructions for performing specific operations or tasks accurately and efficiently.

[0644] A "text template" is a text template created according to a specific format, which can be edited and used as needed.

[0645] "User authentication information" refers to information used to identify a user and verify their permissions, and primarily includes usernames and passwords.

[0646] "User emotions" refer to the psychological state a user is experiencing at a particular moment, and include things like stress and relaxation.

[0647] "Dynamic adjustment" means automatically making changes or modifications in real time according to the environment and conditions.

[0648] A "system" is a comprehensive mechanism in which multiple components work together to achieve a specific function.

[0649] This invention is a system designed to streamline internal operations and prevent errors. By incorporating an emotion engine that recognizes user emotions, this system enables dynamic adjustment of display content and presentation methods based on the user's emotional state.

[0650] First, to log in to the system, the user enters a username and password. The terminal sends the entered information to the server, which retrieves the user information from the database and performs authentication. If authentication is successful, the server generates a session ID and sends it back to the terminal. If authentication fails, the server returns an error message.

[0651] When a user opens a specific operation screen (e.g., inventory management), the terminal sends its operation ID to the server. The server retrieves the corresponding work manual and text template from the database and sends them to the terminal. The terminal displays the received manual and template in the upper right corner of the screen. At this time, based on past work results, it identifies areas prone to errors and retrieves near-miss information from the database. The server sends this information to the terminal, and the terminal highlights this information in red text in the corresponding fields or areas.

[0652] Furthermore, to recognize the user's emotions in real time, the device activates an emotion engine. The emotion engine analyzes the user's facial expressions, voice, and keyboard input speed to identify their emotional state. The recognized emotional information is sent to a server. Based on the received emotional information, the server dynamically adjusts the displayed content and presentation method. For example, if the user is stressed, the displayed manuals and templates are simplified, and important notes are highlighted. On the other hand, if the user is relaxed, detailed information is provided to support efficient work.

[0653] As a concrete example, let's explain what happens when a user opens the inventory management screen. After logging in, when a user opens the inventory management screen, the terminal sends the "Inventory Management" operation ID to the server. The server retrieves the relevant manual (e.g., inventory check procedure) and text template (e.g., inventory shortage notification email template) from the database and sends them to the terminal. The terminal displays this information in the upper right corner of the screen.

[0654] Simultaneously, the terminal activates an emotion engine to analyze the user's emotions in real time. For example, if the server detects that the user is stressed, it simplifies the displayed content and highlights important notes (e.g., points to note when entering specific inventory quantities) in red. Conversely, if the user is relaxed, it displays a detailed manual to improve work efficiency.

[0655] An example of a prompt using a generative AI model is, "Please tell me the appropriate way to display the manual when the user is stressed."

[0656] This system not only streamlines internal operations and prevents errors, but also provides a comfortable and efficient work environment tailored to the user's emotional state.

[0657] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0658] Step 1: The user enters their username and password on the login screen.

[0659] Input: Username, Password

[0660] Specific action: The user enters their username and password on the device's login screen.

[0661] Step 2: The terminal sends the entered username and password to the server.

[0662] Input: Username, Password

[0663] Specific operation: The terminal encrypts the username and password entered by the user and sends them to the server.

[0664] Step 3: The server retrieves user information from the database and performs authentication.

[0665] Input: Username, Password (encrypted)

[0666] Data processing: The server queries the database for the received username and password and performs a verification.

[0667] Output: Authentication result (success or failure)

[0668] Specific operation: The server retrieves the relevant user information from the database and verifies it against the encrypted password.

[0669] Step 4: Send the authentication result back to the device.

[0670] Input: Authentication result (success or failure)

[0671] Specific operation: The server returns the authentication result to the terminal. If successful, it generates and returns a session ID. If unsuccessful, it returns an error message.

[0672] Output: Session ID (on success), Error message (on failure)

[0673] Step 5: The user opens a specific operation screen (e.g., inventory management).

[0674] Specific action: The user selects a specific operation screen within the system and opens it on the terminal.

[0675] Step 6: The terminal sends the operation ID to the server.

[0676] Input: Operation ID

[0677] Specific action: The terminal sends the operation ID corresponding to the current operation screen to the server.

[0678] Step 7: The server retrieves the work manual and text template from the database.

[0679] Input: Operation ID

[0680] Data processing: The server retrieves the relevant work manual and text template from the database.

[0681] Output: Operation manual, text template

[0682] Specific operation: The server queries the database and retrieves the necessary manuals and templates.

[0683] Step 8: The device displays the manual and template on the screen.

[0684] Input: Work manual, text template

[0685] Specific action: The device displays the received manual and template in the upper right corner of the screen.

[0686] Step 9: The server identifies near-miss information based on past work results.

[0687] Input: Past work data

[0688] Data processing: The server analyzes past work data to identify areas prone to errors.

[0689] Output: Near Miss Information

[0690] Specific operation: The server retrieves past work results from the database and identifies near-miss information.

[0691] Step 10: The terminal displays near-miss information.

[0692] Input: Near-miss information

[0693] Specific operation: The terminal receives near-miss information and highlights the relevant field or area in red text.

[0694] Step 11: The device activates the emotion engine.

[0695] Specific action: The terminal starts the emotion engine software.

[0696] Step 12: The emotion engine analyzes the user's facial expressions, voice, and keyboard input speed.

[0697] Input: User's facial expressions, voice, keyboard input speed

[0698] Data processing: The emotion engine analyzes this data to identify emotional states.

[0699] Output: Emotional information

[0700] Specific operation: The emotion engine analyzes data in real time and sends the user's emotional state to the server.

[0701] Step 13: The server dynamically adjusts the displayed content based on sentiment information.

[0702] Input: Emotional information

[0703] Data processing: The server analyzes emotional information and adjusts the displayed content and presentation method.

[0704] Output: Display content after adjustment

[0705] Specific operation: Based on emotional information, the server simplifies the displayed content and highlights warnings if the user is stressed. If the user is relaxed, it displays detailed information.

[0706] Step 14: The device reflects the adjusted display content on the screen.

[0707] Input: Adjusted display content

[0708] Specific operation: The terminal displays the adjusted content received from the server on the screen.

[0709] (Application Example 2)

[0710] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0711] There is a need to improve work efficiency and prevent errors on factory production lines. However, conventional systems do not take into account the mental state of workers, so stress and fatigue can lead to work errors. As a result, production efficiency decreases, and quality declines and safety problems occur. To solve these problems, it is important to recognize workers' emotions in real time and dynamically adjust work instructions and precautions based on that.

[0712] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0713] In this invention, the server includes means for identifying error-prone areas based on past work results and highlighting them using specific display means; means for automatically displaying relevant work manuals and text templates; means for authenticating using user authentication information and adjusting the display content according to the user's authority; means for recognizing the user's emotions in real time; and means for dynamically adjusting the display content and presentation method based on the recognized emotion information. This makes it possible to improve work efficiency and prevent errors by adjusting the display content according to the worker's mental state.

[0714] "Past work results" refer to the results and data from work performed previously.

[0715] "Error-prone areas" refer to parts or processes in a work where mistakes frequently occur.

[0716] "Display means" refers to devices or methods for presenting information visually.

[0717] A "work manual" is a document that details the procedures and methods for performing a specific task.

[0718] A "text template" is a pre-prepared document template that follows a specific format.

[0719] "User authentication information" refers to identification information used to identify a user and verify their permissions.

[0720] "Authentication" is the process of verifying whether a user is legitimate.

[0721] "Permissions" refer to the range of operations a user can perform and information they can access within a system.

[0722] "Emotion recognition technology" refers to technology for detecting and identifying human emotions in real time.

[0723] "Emotional information" refers to data about a user's emotional state detected by emotion recognition tools.

[0724] "Dynamic adjustment" means making flexible changes in real time according to the situation and conditions.

[0725] A "user" is a person, such as a worker or employee, who uses the system.

[0726] A "system" is a collection of technical components, such as hardware, software, and networks, that are combined to achieve a specific purpose.

[0727] This invention is a system aimed at improving work efficiency and preventing errors on a factory production line. Specifically, it has a function to identify error-prone areas based on past work results and highlight them using a display device, and a function to automatically display relevant work manuals and text templates. Furthermore, this system recognizes the user's emotions in real time and dynamically adjusts the display content and presentation method based on that emotion information.

[0728] System Configuration

[0729] Hardware:

[0730] Camera: Used to capture the facial expressions of the workers.

[0731] Computer: Used to perform emotion recognition and adjust the displayed content.

[0732] Robots: Provide work assistance and instructions.

[0733] software:

[0734] OpenCV: A library for image processing.

[0735] Dlib: A library for face detection and facial landmark extraction.

[0736] Keras: A deep learning framework for using emotion recognition models.

[0737] Server: Stores and processes data.

[0738] Terminal: The interface that the user interacts with.

[0739] Program processing

[0740] The server receives video data transmitted from the camera and detects faces using OpenCV and Dlib. A Keras-based emotion recognition model analyzes the detected face regions in real time, and the results are obtained. This emotion information is sent to the terminal and used to adjust the displayed content. If the user is stressed, the server simplifies the displayed content and highlights important notices. Conversely, if the user is relaxed, a detailed manual is displayed to improve work efficiency.

[0741] Specific example

[0742] For example, when a user opens the inventory management operation screen, the terminal sends the "Inventory Management" operation ID to the server. The server retrieves the work manual and text template from the database and sends them to the terminal. Simultaneously, the terminal performs emotion recognition and sends the user's emotional state to the server. If the server detects that the user is stressed, it simplifies the displayed content and highlights important notes (such as points to note when entering specific inventory quantities) in red. Conversely, if the user is relaxed, it displays a detailed manual to improve work efficiency.

[0743] This system provides a flexible work environment tailored to the mental state of the worker, enabling improved work efficiency and prevention of operational errors.

[0744] Example of a prompt

[0745] "Please propose a program that uses emotion recognition to dynamically adjust operational guidelines based on the fatigue level of factory workers. When fatigued, the instructions should be simplified and precautions should be emphasized."

[0746] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0747] Step 1:

[0748] Input: User login information (username, password)

[0749] Specific operation and data processing: The terminal receives the username and password entered by the user and sends them to the server.

[0750] Output: The server retrieves user information from the database, generates a session ID if authentication is successful, and returns it to the terminal. If authentication fails, it returns an error message.

[0751] Step 2:

[0752] Input: User selects an operation screen (e.g., inventory management)

[0753] Specific actions and data processing: The terminal obtains an operation ID and sends it to the server as a request.

[0754] Output: The server retrieves the relevant work manual and text template from the database and sends it to the terminal. The terminal displays this information in the upper right corner of the screen.

[0755] Step 3:

[0756] Input: Past work results

[0757] Specific operations and data processing: The server analyzes past work results and identifies areas prone to errors.

[0758] Output: The server retrieves near-miss information from the database and sends it to the terminal. The terminal highlights the relevant fields or areas in red text.

[0759] Step 4:

[0760] Input: Real-time user emotion data (facial expressions, voice, keyboard input speed, etc.)

[0761] Specific operation and data processing: The terminal activates an emotion engine and analyzes the user's emotions in real time. The user's facial expressions are captured by the camera, faces are detected using OpenCV and Dlib, and emotions are analyzed using an emotion recognition model based on Keras.

[0762] Output: Recognized emotion information is sent from the terminal to the server.

[0763] Step 5:

[0764] Input: Emotional information (e.g., stressed state, relaxed state)

[0765] Specific operations and data processing: The server analyzes emotional information and dynamically adjusts the displayed content and presentation method. For example, if the user is stressed, the displayed content is simplified and warnings are highlighted. If the user is relaxed, a detailed manual is provided.

[0766] Output: The adjusted display content is sent to the terminal and displayed to the user.

[0767] Step 6:

[0768] Input: User actions according to the situation

[0769] Specific actions and data processing: The system performs operations based on the information provided by the user. The results of the operations are sent from the terminal to the server.

[0770] Output: The server records the operation results and adjusts the displayed content and instructions as needed.

[0771] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0772] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0773] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0774] [Third Embodiment]

[0775] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0776] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0777] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0778] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0779] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0780] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0781] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0782] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0783] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0784] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0785] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0786] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0787] This invention is a system for streamlining internal operations and preventing errors. Based on past work results, this system identifies error-prone areas, highlights them using specific display methods, and automatically displays relevant work manuals and text templates. It also has a function to authenticate users using user authentication information and adjust the displayed content according to their permissions.

[0788] User login authentication

[0789] To log in to the system, the user enters a username and password. The terminal sends this information to the server, which retrieves the user information from the database and performs authentication. If authentication is successful, the server generates a session ID and returns it to the terminal. If authentication fails, an error message is returned to the terminal.

[0790] Display manual templates

[0791] When a user opens a specific operation screen (e.g., inventory management), the terminal obtains the operation ID and sends a request to the server. The server retrieves the corresponding work manual and text template from the database and sends them to the terminal. The terminal displays the received manual and template in the upper right corner of the screen.

[0792] Display of near-miss information

[0793] The server identifies error-prone areas based on past work results and retrieves near-miss information from the database. The server sends this information to the terminal, which receives it and highlights the relevant fields or areas in red text.

[0794] Specific example

[0795] For example, consider the case where a user opens the "Inventory Management" screen. After logging in, when the user opens the inventory management screen, the terminal sends the "Inventory Management" operation ID to the server. The server retrieves the relevant manual (e.g., inventory check procedure) and text template (e.g., inventory shortage notification email template) from the database and sends them to the terminal. The terminal displays this information in the upper right corner of the screen.

[0796] Furthermore, the server retrieves near-miss information from past work results and sends relevant cautionary notes (e.g., points to be careful about when entering a specific inventory quantity) to the terminal. The terminal receives this and displays "This field requires attention" in red text in the relevant field (e.g., inventory quantity input field).

[0797] In this way, this system assists users in their operations, preventing operational errors and improving work efficiency.

[0798] The following describes the processing flow.

[0799] Step 1:

[0800] The user accesses the system and enters their username and password on the login screen.

[0801] Step 2:

[0802] The terminal sends the entered username and password to the server. This transmission is performed using HTTPS communication.

[0803] Step 3:

[0804] The server checks the database and verifies the submitted username and password. If the user information matches, authentication is successful; otherwise, authentication fails.

[0805] Step 4:

[0806] The server returns the authentication result to the terminal. If authentication is successful, it generates and returns a session ID; if authentication fails, it returns an error message.

[0807] Step 5:

[0808] The user opens a specific operation screen (e.g., inventory management).

[0809] Step 6:

[0810] The terminal obtains an operation ID and sends a request containing that ID to the server.

[0811] Step 7:

[0812] The server accesses the database and retrieves the relevant work manual and text template.

[0813] Step 8:

[0814] The server sends the acquired work manual and text template to the terminal.

[0815] Step 9:

[0816] The device analyzes the received manual and template and displays them in the upper right corner of the screen.

[0817] Step 10:

[0818] The server retrieves past near-miss information related to the operation ID from the database.

[0819] Step 11:

[0820] The server sends near-miss information it has acquired to the terminal.

[0821] Step 12:

[0822] The terminal analyzes near-miss information and highlights the relevant fields or areas in red text. For example, it might display a warning message such as "This field requires attention" in the "Inventory Quantity Input Field."

[0823] In this way, this system assists users in their operations, preventing operational errors and improving work efficiency.

[0824] (Example 1)

[0825] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0826] The conventional system fails to adequately streamline internal operations and prevent errors, and is inadequate at identifying and warning users about error-prone areas. Furthermore, insufficient user authentication and access control prevent the display of appropriate information. Additionally, the lack of an automatic display function for necessary work instructions and templates on the operation screen places a significant burden on users.

[0827] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0828] In this invention, the server includes means for identifying error-prone areas based on past work results and highlighting them using specific display means; means for automatically displaying relevant work instructions and text templates; means for authenticating using user authentication information and adjusting the display content according to the user's authority; and means for matching the authentication information entered by the user with a database and generating a session ID. This enables increased efficiency and error prevention in internal operations.

[0829] "Past work results" refers to the results and records of tasks performed in the past.

[0830] "Areas prone to errors" are those where mistakes frequently occur based on past work results, or where extra care is required.

[0831] "Specific display means" refers to a format or method used to highlight a specific area on the screen for the user.

[0832] A "work instruction sheet" is a document that details the procedures and instructions for a task.

[0833] A "text template" is a standardized template of phrases used for specific tasks or operations.

[0834] "User authentication information" refers to information such as the username and password that a user uses to log in to the system.

[0835] "Authentication" is the process of verifying whether the information entered by the user matches the information recorded in the database.

[0836] "Permissions" are settings used to restrict the operations a user can perform and the information they can access within a system.

[0837] A "session ID" is an identifier generated to uniquely identify a user who is logged into the system.

[0838] An "operation screen" is the system interface that a user uses to perform a specific task or operation.

[0839] A "machine learning algorithm" is a computational method used to recognize and analyze data patterns, and is a model designed to efficiently perform a specific task.

[0840] A "database" is a system that systematically stores data and makes it easily accessible and manageable.

[0841] The system of this invention is designed to streamline internal operations and prevent errors. This system consists of three components—a server, a terminal, and a user—that function in conjunction with each other.

[0842] System Configuration

[0843] Specific examples of hardware and software

[0844] Server: Apache Tomcat will be used. Database: MySQL will be used.

[0845] Terminal: Any computer terminal, using either Google Chrome or Mozilla Firefox as the browser.

[0846] User: An employee within a company who uses the system.

[0847] Specific operation of the system

[0848] User Authentication

[0849] The user enters their username and password on the login page. This information is sent from the device to the server. The server retrieves the user information from the MySQL database and performs authentication. If authentication is successful, the server generates a session ID and returns it to the device. If authentication fails, an error message is displayed on the device.

[0850] Display manual templates

[0851] The user opens a specific operation screen (e.g., inventory management). The terminal obtains an operation ID and sends a request to the server. Based on this, the server retrieves the corresponding work order and text template from the MySQL database and returns them to the terminal. The terminal displays the received information in the upper right corner of the screen.

[0852] Display of near-miss information

[0853] The server analyzes past work results using machine learning algorithms (e.g., decision trees, random forests) to identify areas prone to errors. It retrieves this information from a MySQL database and sends it to the terminal. The terminal highlights the received information in red text in the corresponding fields or areas.

[0854] Specific example

[0855] For example, when a user opens the "Inventory Management" screen, it works as follows:

[0856] 1. When a user logs in and opens the inventory management screen, the terminal sends the "Inventory Management" operation ID to the server.

[0857] 2. The server retrieves the relevant work order (e.g., "Inventory Check Procedure") and text template (e.g., "Inventory Shortage Notification Email Template") from the MySQL database and returns them to the terminal.

[0858] 3. The device will display this information in the upper right corner of the screen.

[0859] 4. Furthermore, the server retrieves near-miss information from past work results and sends relevant cautionary notes (e.g., "Points to note when entering a specific inventory quantity") to the terminal.

[0860] 5. The terminal displays the acquired information in red text in the target field (e.g., the inventory quantity input field) with the message "This field requires attention."

[0861] System Usage Procedures

[0862] This system is designed to be easy for users to operate and to prevent errors. By using generative AI models, it is possible to analyze and predict data and optimize user interaction using specific prompts.

[0863] For example, the following are examples of prompt statements to be input to a generative AI model.

[0864] "Please explain the system's processing procedure for displaying near-miss information based on past work results when the inventory management screen is opened, and displaying related work instructions and text templates in the upper right corner of the screen."

[0865] Thus, the present invention is a system that dramatically simplifies user operation and significantly improves work efficiency.

[0866] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0867] Step 1:

[0868] The user enters their username and password on the login page. The device obtains this authentication information and sends it to the server using the HTTPS protocol.

[0869] Input: Username, Password

[0870] Data processing: Encrypt authentication information using the HTTPS protocol.

[0871] Output: Encrypted credentials

[0872] Step 2:

[0873] The server attempts authentication by comparing the received authentication information with the MySQL database. If the comparison is successful, it generates a session ID and returns it to the terminal. If the comparison fails, it generates an error message and sends it to the terminal.

[0874] Input: Encrypted credentials

[0875] Data processing: Database matching of authentication information, session ID generation, or error message generation.

[0876] Output: Session ID or error message

[0877] Step 3:

[0878] The user opens a specific operation screen (e.g., inventory management). The terminal obtains an operation ID and sends it to the server as a request.

[0879] Input: Operation ID

[0880] Data processing: Generating HTTP requests

[0881] Output: HTTP request

[0882] Step 4:

[0883] The server queries the MySQL database for the relevant work orders and text templates and returns them to the terminal.

[0884] Input: HTTP Request

[0885] Data Calculation: Database queries for work instructions and text templates

[0886] Output: Work instructions and text templates

[0887] Step 5:

[0888] The terminal analyzes the received work instructions and text templates and displays them in the upper right corner of the screen. This is done using JavaScript or front-end frameworks such as React.

[0889] Input: Work instructions and text templates

[0890] Data processing: Data analysis using a front-end framework

[0891] Output: Screen display

[0892] Step 6:

[0893] The server analyzes past work results using machine learning algorithms (e.g., decision trees, random forests) to identify areas prone to errors. It then retrieves this information from a database and sends it to the terminal.

[0894] Input: Past work results

[0895] Data processing: Analysis using machine learning algorithms and identification of error-prone areas.

[0896] Output: Information on common errors

[0897] Step 7:

[0898] The terminal analyzes the information it receives, identifying areas prone to errors, and highlights the corresponding fields or areas in red text.

[0899] Input: Information on areas prone to errors

[0900] Data Processing: Data analysis and highlighting using a front-end framework

[0901] Output: Screen display (highlighted fields and areas)

[0902] Step 8:

[0903] By following the steps described above, users can check the necessary information (e.g., work instructions, text templates, near-miss information) on the inventory management screen and perform their tasks safely and efficiently.

[0904] Input: Output information from each step

[0905] Data Calculation: Verification and manipulation of various data.

[0906] Output: Efficient business operations

[0907] (Application Example 1)

[0908] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0909] Operating machinery in factories is complex and prone to operator errors. These errors can lead to decreased work efficiency and safety concerns. Furthermore, maintaining consistency in work procedures becomes difficult when multiple operators are working simultaneously. To address these challenges and provide an efficient and safe working environment, a system is needed that prevents operational errors and provides appropriate work procedures and warning messages in real time.

[0910] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0911] In this invention, the server includes means for identifying error-prone areas based on past work results and highlighting them using specific display means, means for automatically displaying relevant work manuals and text templates, and means for authenticating using user authentication information and adjusting the display content according to the user's authority. This makes it possible to identify error-prone areas in real time when operating factory machinery and equipment and to display work procedures and warning messages via a smart terminal. Furthermore, by presenting warning messages regarding specific operations using a generative AI model, it is possible to improve operator attention and promote safe operation.

[0912] "Past work results" refers to records of operations and tasks performed previously.

[0913] "Areas prone to errors" refers to areas or items where mistakes are particularly likely to occur during operation or work.

[0914] "Display means" refers to devices or methods for visually presenting information.

[0915] A "work manual" refers to a document or digital file that contains procedures and instructions for performing a specific task or operation.

[0916] A "text template" refers to a set of pre-written phrases designed for a specific purpose or operation.

[0917] "User authentication information" refers to data used to identify a user and verify their permissions (e.g., username and password).

[0918] "Permissions" refer to access rights to operations and information held by a specific user.

[0919] "Factory machinery and equipment" refers to devices and systems used for the production and processing of industrial products.

[0920] "Real-time" refers to the processing and display of information and data instantly, without delay.

[0921] "Smart devices" refer to devices with advanced functions such as smartphones, tablets, and smart glasses.

[0922] "Work procedure" refers to a series of instructions or methods for performing a specific operation or task.

[0923] A "warning message" refers to information or notifications displayed to draw attention.

[0924] A "generative AI model" refers to a model that uses artificial intelligence technology to generate messages or predictions for specific tasks.

[0925] An "operator" refers to a person who operates and manages machinery or systems.

[0926] This invention provides a system that identifies common error points in factory machinery operation in real time and displays work procedures and warning messages via a smart terminal. The system is configured as follows:

[0927] Authentication Steps

[0928] To log in to the system, the user enters a username and password. The terminal sends this information to the server, which retrieves the user information from the database and performs authentication. If authentication is successful, the server generates a session ID and returns it to the terminal.

[0929] Display of work manuals and text templates

[0930] When a user opens a specific operation screen, the terminal obtains its operation ID and sends a request to the server. The server retrieves the corresponding operation manual and text template from the database and sends them to the terminal. The terminal displays the received manual and template in the upper right corner of the screen.

[0931] Display of near-miss information

[0932] The server identifies error-prone areas based on past work results and retrieves near-miss information from the database. The server sends this information to the terminal, which receives it and highlights the relevant fields or areas in red text.

[0933] Hardware and software to be used

[0934] The hardware of this system consists of smart devices such as smartphones, tablets, and smart glasses. The software uses Flask (a Python web framework), and data is exchanged in JSON format. Furthermore, a generative AI model is used to provide warning messages for specific operations.

[0935] Specific example

[0936] For example, when performing "Operation 1" on a robot used in a factory, the user logs in and opens a specific operation screen. Then, the work procedure manual and operation template text are displayed in the upper right corner of the terminal screen. In addition, warning messages such as "Pay attention to adjusting the sensor on the left" and "Do not apply force when pressing the machine stop button" are highlighted in red based on past operation results.

[0937] Example of a prompt

[0938] The following are examples of prompts to input into a generative AI model:

[0939] "Develop an application that, when opening the screen for Operation 1 performed on a factory robot, highlights common errors and points to note based on past operation history, and displays appropriate work instructions and template text. Write the code for this using Flask."

[0940] In this way, the system reduces operational errors in the factory and improves operator attentiveness, thereby promoting safe and efficient work.

[0941] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0942] Step 1:

[0943] To log in to the system, the user enters a username and password. The input data (username and password) is collected by the terminal and sent to the server. The server uses this information to retrieve user information from the database and performs authentication. If authentication is successful, the server generates a session ID and returns it to the terminal. If authentication fails, the server returns an error message to the terminal.

[0944] Input: Username, Password

[0945] Output: Session ID on success, error message on failure.

[0946] Step 2:

[0947] When a user opens a specific operation screen, the terminal obtains an operation ID and sends it to the server. The server retrieves the corresponding operation manual and text template from its database and sends that data to the terminal. The terminal displays these in the upper right corner of the screen.

[0948] Input: Operation ID

[0949] Output: Operation manual, text template

[0950] Step 3:

[0951] The server identifies error-prone areas based on past work results and retrieves near-miss information from the database. It sends this information to the terminal, which highlights the relevant fields or areas in red. This includes prompts for the server to generate warning messages about specific operations using an AI model and provide them to the user.

[0952] Input: Past work results

[0953] Output: Highlighted near-miss information, red text warnings

[0954] Step 4:

[0955] The device displays warning messages generated using a generation AI model, along with highlighted near-miss information. This allows users to recognize operational pitfalls in real time and perform operations safely.

[0956] Input: Near-miss incident information, generated AI model

[0957] Output: Display of warning message

[0958] In this way, a system has been built that supports actual operations by performing appropriate data input and output at each step.

[0959] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0960] This invention is a system designed to streamline internal operations and prevent errors, and incorporates an emotion engine that recognizes user emotions. Based on the user's emotional information recognized by the emotion engine, this system adjusts the displayed content and presentation method, and further provides operational support and displays warnings.

[0961] User login authentication

[0962] To log in to the system, the user enters a username and password. The terminal sends the entered information to the server, which retrieves user information from the database and performs authentication. If authentication is successful, a session ID is generated and returned to the terminal. If authentication fails, an error message is returned.

[0963] Display manual templates

[0964] The user opens a specific operation screen (e.g., inventory management). The terminal obtains the operation ID and sends a request to the server. The server retrieves the corresponding work manual and text template from the database and sends them to the terminal. The terminal displays the received manual and template in the upper right corner of the screen.

[0965] Display of near-miss information

[0966] The server identifies error-prone areas based on past work results and retrieves near-miss information from the database. The server sends this information to the terminal, which receives it and highlights the relevant fields or areas in red text.

[0967] Introducing an emotional engine

[0968] To recognize the user's emotions in real time, the device activates an emotion engine. The emotion engine analyzes the user's facial expressions, voice, keyboard input speed, etc., to identify their emotional state. The recognized emotional information is then sent to the server.

[0969] Adjustment based on emotional information

[0970] The server dynamically adjusts the displayed content and presentation method based on the emotional information it receives. For example, if the user is stressed, the displayed manuals and templates can be simplified, and important notes can be further emphasized. Conversely, if the user is relaxed, detailed information can be provided to support efficient work.

[0971] Specific example

[0972] For example, consider the case where a user opens the "Inventory Management" screen. After logging in, when the user opens the inventory management screen, the terminal sends the "Inventory Management" operation ID to the server. The server retrieves the relevant manual (e.g., inventory check procedure) and text template (e.g., inventory shortage notification email template) from the database and sends them to the terminal. The terminal displays this information in the upper right corner of the screen.

[0973] Simultaneously, the terminal activates an emotion engine to analyze the user's emotions in real time. For example, if the server detects that the user is stressed, it simplifies the displayed content and highlights important points (e.g., points to note when entering specific inventory quantities) in red. Conversely, if the user is relaxed, it displays a detailed manual to improve work efficiency.

[0974] Thus, the present invention not only assists user operation, prevents operational errors, and improves work efficiency, but also dynamically adjusts the displayed content according to the user's emotional state, providing an even more comfortable and efficient work environment.

[0975] The following describes the processing flow.

[0976] Step 1:

[0977] The user accesses the system and enters their username and password on the login screen.

[0978] Step 2:

[0979] The terminal sends the entered username and password to the server. This transmission is performed using HTTPS communication.

[0980] Step 3:

[0981] The server checks the database and verifies the submitted username and password. If the user information matches, authentication is successful; otherwise, authentication fails.

[0982] Step 4:

[0983] The server returns the authentication result to the terminal. If authentication is successful, it generates and returns a session ID; if authentication fails, it returns an error message.

[0984] Step 5:

[0985] The user opens a specific operation screen (e.g., inventory management).

[0986] Step 6:

[0987] The terminal obtains an operation ID and sends a request containing that ID to the server.

[0988] Step 7:

[0989] The server accesses the database and retrieves the relevant work manual and text template.

[0990] Step 8:

[0991] The server sends the acquired work manual and text template to the terminal.

[0992] Step 9:

[0993] The device analyzes the received manual and template and displays them in the upper right corner of the screen.

[0994] Step 10:

[0995] The device activates an emotion engine that analyzes the user's facial expressions, voice, and keyboard input speed in real time.

[0996] Step 11:

[0997] The emotion engine recognizes the user's emotional state and sends that information to the server.

[0998] Step 12:

[0999] The server retrieves past near-miss information related to the operation ID from the database.

[1000] Step 13:

[1001] The server sends near-miss information it has acquired to the terminal.

[1002] Step 14:

[1003] The server adjusts the displayed content and presentation method based on the emotional information it receives. For example, if the user is stressed, the displayed content is simplified, and warnings are further highlighted in red.

[1004] Step 15:

[1005] The terminal analyzes near-miss information and highlights a warning message in red text in the relevant field or area.

[1006] Step 16:

[1007] The terminal displays customized content received from the server to the user and supports their operation.

[1008] This entire process streamlines user operations, prevents operational errors, and dynamically adjusts displayed content based on the user's emotional state.

[1009] (Example 2)

[1010] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1011] Conventional in-house operating systems fail to adequately prevent operational errors and improve work efficiency. Furthermore, they often lack display adjustments based on the user's emotional state, creating a stressful environment for users. This can lead to decreased user efficiency and an increase in operational errors. Therefore, the present invention aims to solve these problems.

[1012] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1013] In this invention, the server includes means for identifying error-prone areas based on past work results and highlighting them using specific display means, means for automatically displaying relevant work manuals and text templates, means for authenticating using user authentication information and adjusting the display content according to the user's authority, and means for recognizing the user's emotions and dynamically adjusting the display content and presentation method based on the recognized emotions. This enables dynamic display adjustments according to the user's emotional state and work environment, thereby preventing operational errors and improving work efficiency.

[1014] "Past work results" refers to records of the progress and results of a series of operations or tasks performed in the past.

[1015] "Error-prone areas" are points where users are likely to frequently make mistakes during operation or work.

[1016] "Specific display means" refers to methods or devices that highlight designated information or warnings.

[1017] A "work manual" is a document that contains procedures and instructions for performing specific operations or tasks accurately and efficiently.

[1018] A "text template" is a text template created according to a specific format, which can be edited and used as needed.

[1019] "User authentication information" refers to information used to identify a user and verify their permissions, and primarily includes usernames and passwords.

[1020] "User emotions" refer to the psychological state a user is experiencing at a particular moment, and include things like stress and relaxation.

[1021] "Dynamic adjustment" means automatically making changes or modifications in real time according to the environment and conditions.

[1022] A "system" is a comprehensive mechanism in which multiple components work together to achieve a specific function.

[1023] This invention is a system designed to streamline internal operations and prevent errors. By incorporating an emotion engine that recognizes user emotions, this system enables dynamic adjustment of display content and presentation methods based on the user's emotional state.

[1024] First, to log in to the system, the user enters a username and password. The terminal sends the entered information to the server, which retrieves the user information from the database and performs authentication. If authentication is successful, the server generates a session ID and sends it back to the terminal. If authentication fails, the server returns an error message.

[1025] When a user opens a specific operation screen (e.g., inventory management), the terminal sends its operation ID to the server. The server retrieves the corresponding work manual and text template from the database and sends them to the terminal. The terminal displays the received manual and template in the upper right corner of the screen. At this time, based on past work results, it identifies areas prone to errors and retrieves near-miss information from the database. The server sends this information to the terminal, and the terminal highlights this information in red text in the corresponding fields or areas.

[1026] Furthermore, to recognize the user's emotions in real time, the device activates an emotion engine. The emotion engine analyzes the user's facial expressions, voice, and keyboard input speed to identify their emotional state. The recognized emotional information is sent to a server. Based on the received emotional information, the server dynamically adjusts the displayed content and presentation method. For example, if the user is stressed, the displayed manuals and templates are simplified, and important notes are highlighted. On the other hand, if the user is relaxed, detailed information is provided to support efficient work.

[1027] As a concrete example, let's explain what happens when a user opens the inventory management screen. After logging in, when a user opens the inventory management screen, the terminal sends the "Inventory Management" operation ID to the server. The server retrieves the relevant manual (e.g., inventory check procedure) and text template (e.g., inventory shortage notification email template) from the database and sends them to the terminal. The terminal displays this information in the upper right corner of the screen.

[1028] Simultaneously, the terminal activates an emotion engine to analyze the user's emotions in real time. For example, if the server detects that the user is stressed, it simplifies the displayed content and highlights important notes (e.g., points to note when entering specific inventory quantities) in red. Conversely, if the user is relaxed, it displays a detailed manual to improve work efficiency.

[1029] An example of a prompt using a generative AI model is, "Please tell me the appropriate way to display the manual when the user is stressed."

[1030] This system not only streamlines internal operations and prevents errors, but also provides a comfortable and efficient work environment tailored to the user's emotional state.

[1031] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1032] Step 1: The user enters their username and password on the login screen.

[1033] Input: Username, Password

[1034] Specific action: The user enters their username and password on the device's login screen.

[1035] Step 2: The terminal sends the entered username and password to the server.

[1036] Input: Username, Password

[1037] Specific operation: The terminal encrypts the username and password entered by the user and sends them to the server.

[1038] Step 3: The server retrieves user information from the database and performs authentication.

[1039] Input: Username, Password (encrypted)

[1040] Data processing: The server queries the database for the received username and password and performs a verification.

[1041] Output: Authentication result (success or failure)

[1042] Specific operation: The server retrieves the relevant user information from the database and verifies it against the encrypted password.

[1043] Step 4: Send the authentication result back to the device.

[1044] Input: Authentication result (success or failure)

[1045] Specific operation: The server returns the authentication result to the terminal. If successful, it generates and returns a session ID. If unsuccessful, it returns an error message.

[1046] Output: Session ID (on success), Error message (on failure)

[1047] Step 5: The user opens a specific operation screen (e.g., inventory management).

[1048] Specific action: The user selects a specific operation screen within the system and opens it on the terminal.

[1049] Step 6: The terminal sends the operation ID to the server.

[1050] Input: Operation ID

[1051] Specific action: The terminal sends the operation ID corresponding to the current operation screen to the server.

[1052] Step 7: The server retrieves the work manual and text template from the database.

[1053] Input: Operation ID

[1054] Data processing: The server retrieves the relevant work manual and text template from the database.

[1055] Output: Operation manual, text template

[1056] Specific operation: The server queries the database and retrieves the necessary manuals and templates.

[1057] Step 8: The device displays the manual and template on the screen.

[1058] Input: Work manual, text template

[1059] Specific action: The device displays the received manual and template in the upper right corner of the screen.

[1060] Step 9: The server identifies near-miss information based on past work results.

[1061] Input: Past work data

[1062] Data processing: The server analyzes past work data to identify areas prone to errors.

[1063] Output: Near Miss Information

[1064] Specific operation: The server retrieves past work results from the database and identifies near-miss information.

[1065] Step 10: The terminal displays near-miss information.

[1066] Input: Near-miss information

[1067] Specific operation: The terminal receives near-miss information and highlights the relevant field or area in red text.

[1068] Step 11: The device activates the emotion engine.

[1069] Specific action: The terminal starts the emotion engine software.

[1070] Step 12: The emotion engine analyzes the user's facial expressions, voice, and keyboard input speed.

[1071] Input: User's facial expressions, voice, keyboard input speed

[1072] Data processing: The emotion engine analyzes this data to identify emotional states.

[1073] Output: Emotional information

[1074] Specific operation: The emotion engine analyzes data in real time and sends the user's emotional state to the server.

[1075] Step 13: The server dynamically adjusts the displayed content based on sentiment information.

[1076] Input: Emotional information

[1077] Data processing: The server analyzes emotional information and adjusts the displayed content and presentation method.

[1078] Output: Display content after adjustment

[1079] Specific operation: Based on emotional information, the server simplifies the displayed content and highlights warnings if the user is stressed. If the user is relaxed, it displays detailed information.

[1080] Step 14: The device reflects the adjusted display content on the screen.

[1081] Input: Adjusted display content

[1082] Specific operation: The terminal displays the adjusted content received from the server on the screen.

[1083] (Application Example 2)

[1084] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1085] There is a need to improve work efficiency and prevent errors on factory production lines. However, conventional systems do not take into account the mental state of workers, so stress and fatigue can lead to work errors. As a result, production efficiency decreases, and quality declines and safety problems occur. To solve these problems, it is important to recognize workers' emotions in real time and dynamically adjust work instructions and precautions based on that.

[1086] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1087] In this invention, the server includes means for identifying error-prone areas based on past work results and highlighting them using specific display means; means for automatically displaying relevant work manuals and text templates; means for authenticating using user authentication information and adjusting the display content according to the user's authority; means for recognizing the user's emotions in real time; and means for dynamically adjusting the display content and presentation method based on the recognized emotion information. This makes it possible to improve work efficiency and prevent errors by adjusting the display content according to the worker's mental state.

[1088] "Past work results" refer to the results and data from work performed previously.

[1089] "Error-prone areas" refer to parts or processes in a work where mistakes frequently occur.

[1090] "Display means" refers to devices or methods for presenting information visually.

[1091] A "work manual" is a document that details the procedures and methods for performing a specific task.

[1092] A "text template" is a pre-prepared document template that follows a specific format.

[1093] "User authentication information" refers to identification information used to identify a user and verify their permissions.

[1094] "Authentication" is the process of verifying whether a user is legitimate.

[1095] "Permissions" refer to the range of operations a user can perform and information they can access within a system.

[1096] "Emotion recognition technology" refers to technology for detecting and identifying human emotions in real time.

[1097] "Emotional information" refers to data about a user's emotional state detected by emotion recognition tools.

[1098] "Dynamic adjustment" means making flexible changes in real time according to the situation and conditions.

[1099] A "user" is a person, such as a worker or employee, who uses the system.

[1100] A "system" is a collection of technical components, such as hardware, software, and networks, that are combined to achieve a specific purpose.

[1101] This invention is a system aimed at improving work efficiency and preventing errors on a factory production line. Specifically, it has a function to identify error-prone areas based on past work results and highlight them using a display device, and a function to automatically display relevant work manuals and text templates. Furthermore, this system recognizes the user's emotions in real time and dynamically adjusts the display content and presentation method based on that emotion information.

[1102] System Configuration

[1103] Hardware:

[1104] Camera: Used to capture the facial expressions of the workers.

[1105] Computer: Used to perform emotion recognition and adjust the displayed content.

[1106] Robots: Provide work assistance and instructions.

[1107] software:

[1108] OpenCV: A library for image processing.

[1109] Dlib: A library for face detection and facial landmark extraction.

[1110] Keras: A deep learning framework for using emotion recognition models.

[1111] Server: Stores and processes data.

[1112] Terminal: The interface that the user interacts with.

[1113] Program processing

[1114] The server receives video data transmitted from the camera and detects faces using OpenCV and Dlib. A Keras-based emotion recognition model analyzes the detected face regions in real time, and the results are obtained. This emotion information is sent to the terminal and used to adjust the displayed content. If the user is stressed, the server simplifies the displayed content and highlights important notices. Conversely, if the user is relaxed, a detailed manual is displayed to improve work efficiency.

[1115] Specific example

[1116] For example, when a user opens the inventory management operation screen, the terminal sends the "Inventory Management" operation ID to the server. The server retrieves the work manual and text template from the database and sends them to the terminal. Simultaneously, the terminal performs emotion recognition and sends the user's emotional state to the server. If the server detects that the user is stressed, it simplifies the displayed content and highlights important notes (such as points to note when entering specific inventory quantities) in red. Conversely, if the user is relaxed, it displays a detailed manual to improve work efficiency.

[1117] This system provides a flexible work environment tailored to the mental state of the worker, enabling improved work efficiency and prevention of operational errors.

[1118] Example of a prompt

[1119] "Please propose a program that uses emotion recognition to dynamically adjust operational guidelines based on the fatigue level of factory workers. When fatigued, the instructions should be simplified and precautions should be emphasized."

[1120] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1121] Step 1:

[1122] Input: User login information (username, password)

[1123] Specific operation and data processing: The terminal receives the username and password entered by the user and sends them to the server.

[1124] Output: The server retrieves user information from the database, generates a session ID if authentication is successful, and returns it to the terminal. If authentication fails, it returns an error message.

[1125] Step 2:

[1126] Input: User selects an operation screen (e.g., inventory management)

[1127] Specific actions and data processing: The terminal obtains an operation ID and sends it to the server as a request.

[1128] Output: The server retrieves the relevant work manual and text template from the database and sends it to the terminal. The terminal displays this information in the upper right corner of the screen.

[1129] Step 3:

[1130] Input: Past work results

[1131] Specific operations and data processing: The server analyzes past work results and identifies areas prone to errors.

[1132] Output: The server retrieves near-miss information from the database and sends it to the terminal. The terminal highlights the relevant fields or areas in red text.

[1133] Step 4:

[1134] Input: Real-time user emotion data (facial expressions, voice, keyboard input speed, etc.)

[1135] Specific operation and data processing: The terminal activates an emotion engine and analyzes the user's emotions in real time. The user's facial expressions are captured by the camera, faces are detected using OpenCV and Dlib, and emotions are analyzed using an emotion recognition model based on Keras.

[1136] Output: Recognized emotion information is sent from the terminal to the server.

[1137] Step 5:

[1138] Input: Emotional information (e.g., stressed state, relaxed state)

[1139] Specific operations and data processing: The server analyzes emotional information and dynamically adjusts the displayed content and presentation method. For example, if the user is stressed, the displayed content is simplified and warnings are highlighted. If the user is relaxed, a detailed manual is provided.

[1140] Output: The adjusted display content is sent to the terminal and displayed to the user.

[1141] Step 6:

[1142] Input: User actions according to the situation

[1143] Specific actions and data processing: The system performs operations based on the information provided by the user. The results of the operations are sent from the terminal to the server.

[1144] Output: The server records the operation results and adjusts the displayed content and instructions as needed.

[1145] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1146] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1147] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1148] [Fourth Embodiment]

[1149] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1150] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1151] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1152] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1153] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1154] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1155] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1156] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1157] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1158] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1159] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1160] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1161] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1162] This invention is a system for streamlining internal operations and preventing errors. Based on past work results, this system identifies error-prone areas, highlights them using specific display methods, and automatically displays relevant work manuals and text templates. It also has a function to authenticate users using user authentication information and adjust the displayed content according to their permissions.

[1163] User login authentication

[1164] To log in to the system, the user enters a username and password. The terminal sends this information to the server, which retrieves the user information from the database and performs authentication. If authentication is successful, the server generates a session ID and returns it to the terminal. If authentication fails, an error message is returned to the terminal.

[1165] Display manual templates

[1166] When a user opens a specific operation screen (e.g., inventory management), the terminal obtains the operation ID and sends a request to the server. The server retrieves the corresponding work manual and text template from the database and sends them to the terminal. The terminal displays the received manual and template in the upper right corner of the screen.

[1167] Display of near-miss information

[1168] The server identifies error-prone areas based on past work results and retrieves near-miss information from the database. The server sends this information to the terminal, which receives it and highlights the relevant fields or areas in red text.

[1169] Specific example

[1170] For example, consider the case where a user opens the "Inventory Management" screen. After logging in, when the user opens the inventory management screen, the terminal sends the "Inventory Management" operation ID to the server. The server retrieves the relevant manual (e.g., inventory check procedure) and text template (e.g., inventory shortage notification email template) from the database and sends them to the terminal. The terminal displays this information in the upper right corner of the screen.

[1171] Furthermore, the server retrieves near-miss information from past work results and sends relevant cautionary notes (e.g., points to be careful about when entering a specific inventory quantity) to the terminal. The terminal receives this and displays "This field requires attention" in red text in the relevant field (e.g., inventory quantity input field).

[1172] In this way, this system assists users in their operations, preventing operational errors and improving work efficiency.

[1173] The following describes the processing flow.

[1174] Step 1:

[1175] The user accesses the system and enters their username and password on the login screen.

[1176] Step 2:

[1177] The terminal sends the entered username and password to the server. This transmission is performed using HTTPS communication.

[1178] Step 3:

[1179] The server checks the database and verifies the submitted username and password. If the user information matches, authentication is successful; otherwise, authentication fails.

[1180] Step 4:

[1181] The server returns the authentication result to the terminal. If authentication is successful, it generates and returns a session ID; if authentication fails, it returns an error message.

[1182] Step 5:

[1183] The user opens a specific operation screen (e.g., inventory management).

[1184] Step 6:

[1185] The terminal obtains an operation ID and sends a request containing that ID to the server.

[1186] Step 7:

[1187] The server accesses the database and retrieves the relevant work manual and text template.

[1188] Step 8:

[1189] The server sends the acquired work manual and text template to the terminal.

[1190] Step 9:

[1191] The device analyzes the received manual and template and displays them in the upper right corner of the screen.

[1192] Step 10:

[1193] The server retrieves past near-miss information related to the operation ID from the database.

[1194] Step 11:

[1195] The server sends near-miss information it has acquired to the terminal.

[1196] Step 12:

[1197] The terminal analyzes near-miss information and highlights the relevant fields or areas in red text. For example, it might display a warning message such as "This field requires attention" in the "Inventory Quantity Input Field."

[1198] In this way, this system assists users in their operations, preventing operational errors and improving work efficiency.

[1199] (Example 1)

[1200] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1201] The conventional system fails to adequately streamline internal operations and prevent errors, and is inadequate at identifying and warning users about error-prone areas. Furthermore, insufficient user authentication and access control prevent the display of appropriate information. Additionally, the lack of an automatic display function for necessary work instructions and templates on the operation screen places a significant burden on users.

[1202] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1203] In this invention, the server includes means for identifying error-prone areas based on past work results and highlighting them using specific display means; means for automatically displaying relevant work instructions and text templates; means for authenticating using user authentication information and adjusting the display content according to the user's authority; and means for matching the authentication information entered by the user with a database and generating a session ID. This enables increased efficiency and error prevention in internal operations.

[1204] "Past work results" refers to the results and records of tasks performed in the past.

[1205] "Areas prone to errors" are those where mistakes frequently occur based on past work results, or where extra care is required.

[1206] "Specific display means" refers to a format or method used to highlight a specific area on the screen for the user.

[1207] A "work instruction sheet" is a document that details the procedures and instructions for a task.

[1208] A "text template" is a standardized template of phrases used for specific tasks or operations.

[1209] "User authentication information" refers to information such as the username and password that a user uses to log in to the system.

[1210] "Authentication" is the process of verifying whether the information entered by the user matches the information recorded in the database.

[1211] "Permissions" are settings used to restrict the operations a user can perform and the information they can access within a system.

[1212] A "session ID" is an identifier generated to uniquely identify a user who is logged into the system.

[1213] An "operation screen" is the system interface that a user uses to perform a specific task or operation.

[1214] A "machine learning algorithm" is a computational method used to recognize and analyze data patterns, and is a model designed to efficiently perform a specific task.

[1215] A "database" is a system that systematically stores data and makes it easily accessible and manageable.

[1216] The system of this invention is designed to streamline internal operations and prevent errors. This system consists of three components—a server, a terminal, and a user—that function in conjunction with each other.

[1217] System Configuration

[1218] Specific examples of hardware and software

[1219] Server: Apache Tomcat will be used. Database: MySQL will be used.

[1220] Terminal: Any computer terminal, using either Google Chrome or Mozilla Firefox as the browser.

[1221] User: An employee within a company who uses the system.

[1222] Specific operation of the system

[1223] User Authentication

[1224] The user enters their username and password on the login page. This information is sent from the device to the server. The server retrieves the user information from the MySQL database and performs authentication. If authentication is successful, the server generates a session ID and returns it to the device. If authentication fails, an error message is displayed on the device.

[1225] Display manual templates

[1226] The user opens a specific operation screen (e.g., inventory management). The terminal obtains an operation ID and sends a request to the server. Based on this, the server retrieves the corresponding work order and text template from the MySQL database and returns them to the terminal. The terminal displays the received information in the upper right corner of the screen.

[1227] Display of near-miss information

[1228] The server analyzes past work results using machine learning algorithms (e.g., decision trees, random forests) to identify areas prone to errors. It retrieves this information from a MySQL database and sends it to the terminal. The terminal highlights the received information in red text in the corresponding fields or areas.

[1229] Specific example

[1230] For example, when a user opens the "Inventory Management" screen, it works as follows:

[1231] 1. When a user logs in and opens the inventory management screen, the terminal sends the "Inventory Management" operation ID to the server.

[1232] 2. The server retrieves the relevant work order (e.g., "Inventory Check Procedure") and text template (e.g., "Inventory Shortage Notification Email Template") from the MySQL database and returns them to the terminal.

[1233] 3. The device will display this information in the upper right corner of the screen.

[1234] 4. Furthermore, the server retrieves near-miss information from past work results and sends relevant cautionary notes (e.g., "Points to note when entering a specific inventory quantity") to the terminal.

[1235] 5. The terminal displays the acquired information in red text in the target field (e.g., the inventory quantity input field) with the message "This field requires attention."

[1236] System Usage Procedures

[1237] This system is designed to be easy for users to operate and to prevent errors. By using generative AI models, it is possible to analyze and predict data and optimize user interaction using specific prompts.

[1238] For example, the following are examples of prompt statements to be input to a generative AI model.

[1239] "Please explain the system's processing procedure for displaying near-miss information based on past work results when the inventory management screen is opened, and displaying related work instructions and text templates in the upper right corner of the screen."

[1240] Thus, the present invention is a system that dramatically simplifies user operation and significantly improves work efficiency.

[1241] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1242] Step 1:

[1243] The user enters their username and password on the login page. The device obtains this authentication information and sends it to the server using the HTTPS protocol.

[1244] Input: Username, Password

[1245] Data processing: Encrypt authentication information using the HTTPS protocol.

[1246] Output: Encrypted credentials

[1247] Step 2:

[1248] The server attempts authentication by comparing the received authentication information with the MySQL database. If the comparison is successful, it generates a session ID and returns it to the terminal. If the comparison fails, it generates an error message and sends it to the terminal.

[1249] Input: Encrypted credentials

[1250] Data processing: Database matching of authentication information, session ID generation, or error message generation.

[1251] Output: Session ID or error message

[1252] Step 3:

[1253] The user opens a specific operation screen (e.g., inventory management). The terminal obtains an operation ID and sends it to the server as a request.

[1254] Input: Operation ID

[1255] Data processing: Generating HTTP requests

[1256] Output: HTTP request

[1257] Step 4:

[1258] The server queries the MySQL database for the relevant work orders and text templates and returns them to the terminal.

[1259] Input: HTTP Request

[1260] Data Calculation: Database queries for work instructions and text templates

[1261] Output: Work instructions and text templates

[1262] Step 5:

[1263] The terminal analyzes the received work instructions and text templates and displays them in the upper right corner of the screen. This is done using JavaScript or front-end frameworks such as React.

[1264] Input: Work instructions and text templates

[1265] Data processing: Data analysis using a front-end framework

[1266] Output: Screen display

[1267] Step 6:

[1268] The server analyzes past work results using machine learning algorithms (e.g., decision trees, random forests) to identify areas prone to errors. It then retrieves this information from a database and sends it to the terminal.

[1269] Input: Past work results

[1270] Data processing: Analysis using machine learning algorithms and identification of error-prone areas.

[1271] Output: Information on common errors

[1272] Step 7:

[1273] The terminal analyzes the information it receives, identifying areas prone to errors, and highlights the corresponding fields or areas in red text.

[1274] Input: Information on areas prone to errors

[1275] Data Processing: Data analysis and highlighting using a front-end framework

[1276] Output: Screen display (highlighted fields and areas)

[1277] Step 8:

[1278] By following the steps described above, users can check the necessary information (e.g., work instructions, text templates, near-miss information) on the inventory management screen and perform their tasks safely and efficiently.

[1279] Input: Output information from each step

[1280] Data Calculation: Verification and manipulation of various data.

[1281] Output: Efficient business operations

[1282] (Application Example 1)

[1283] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1284] Operating machinery in factories is complex and prone to operator errors. These errors can lead to decreased work efficiency and safety concerns. Furthermore, maintaining consistency in work procedures becomes difficult when multiple operators are working simultaneously. To address these challenges and provide an efficient and safe working environment, a system is needed that prevents operational errors and provides appropriate work procedures and warning messages in real time.

[1285] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1286] In this invention, the server includes means for identifying error-prone areas based on past work results and highlighting them using specific display means, means for automatically displaying relevant work manuals and text templates, and means for authenticating using user authentication information and adjusting the display content according to the user's authority. This makes it possible to identify error-prone areas in real time when operating factory machinery and equipment and to display work procedures and warning messages via a smart terminal. Furthermore, by presenting warning messages regarding specific operations using a generative AI model, it is possible to improve operator attention and promote safe operation.

[1287] "Past work results" refers to records of operations and tasks performed previously.

[1288] "Areas prone to errors" refers to areas or items where mistakes are particularly likely to occur during operation or work.

[1289] "Display means" refers to devices or methods for visually presenting information.

[1290] A "work manual" refers to a document or digital file that contains procedures and instructions for performing a specific task or operation.

[1291] A "text template" refers to a set of pre-written phrases designed for a specific purpose or operation.

[1292] "User authentication information" refers to data used to identify a user and verify their permissions (e.g., username and password).

[1293] "Permissions" refer to access rights to operations and information held by a specific user.

[1294] "Factory machinery and equipment" refers to devices and systems used for the production and processing of industrial products.

[1295] "Real-time" refers to the processing and display of information and data instantly, without delay.

[1296] "Smart devices" refer to devices with advanced functions such as smartphones, tablets, and smart glasses.

[1297] "Work procedure" refers to a series of instructions or methods for performing a specific operation or task.

[1298] A "warning message" refers to information or notifications displayed to draw attention.

[1299] A "generative AI model" refers to a model that uses artificial intelligence technology to generate messages or predictions for specific tasks.

[1300] An "operator" refers to a person who operates and manages machinery or systems.

[1301] This invention provides a system that identifies common error points in factory machinery operation in real time and displays work procedures and warning messages via a smart terminal. The system is configured as follows:

[1302] Authentication Steps

[1303] To log in to the system, the user enters a username and password. The terminal sends this information to the server, which retrieves the user information from the database and performs authentication. If authentication is successful, the server generates a session ID and returns it to the terminal.

[1304] Display of work manuals and text templates

[1305] When a user opens a specific operation screen, the terminal obtains its operation ID and sends a request to the server. The server retrieves the corresponding operation manual and text template from the database and sends them to the terminal. The terminal displays the received manual and template in the upper right corner of the screen.

[1306] Display of near-miss information

[1307] The server identifies error-prone areas based on past work results and retrieves near-miss information from the database. The server sends this information to the terminal, which receives it and highlights the relevant fields or areas in red text.

[1308] Hardware and software to be used

[1309] The hardware of this system consists of smart devices such as smartphones, tablets, and smart glasses. The software uses Flask (a Python web framework), and data is exchanged in JSON format. Furthermore, a generative AI model is used to provide warning messages for specific operations.

[1310] Specific example

[1311] For example, when performing "Operation 1" on a robot used in a factory, the user logs in and opens a specific operation screen. Then, the work procedure manual and operation template text are displayed in the upper right corner of the terminal screen. In addition, warning messages such as "Pay attention to adjusting the sensor on the left" and "Do not apply force when pressing the machine stop button" are highlighted in red based on past operation results.

[1312] Example of a prompt

[1313] The following are examples of prompts to input into a generative AI model:

[1314] "Develop an application that, when opening the screen for Operation 1 performed on a factory robot, highlights common errors and points to note based on past operation history, and displays appropriate work instructions and template text. Write the code for this using Flask."

[1315] In this way, the system reduces operational errors in the factory and improves operator attentiveness, thereby promoting safe and efficient work.

[1316] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1317] Step 1:

[1318] To log in to the system, the user enters a username and password. The input data (username and password) is collected by the terminal and sent to the server. The server uses this information to retrieve user information from the database and performs authentication. If authentication is successful, the server generates a session ID and returns it to the terminal. If authentication fails, the server returns an error message to the terminal.

[1319] Input: Username, Password

[1320] Output: Session ID on success, error message on failure.

[1321] Step 2:

[1322] When a user opens a specific operation screen, the terminal obtains an operation ID and sends it to the server. The server retrieves the corresponding operation manual and text template from its database and sends that data to the terminal. The terminal displays these in the upper right corner of the screen.

[1323] Input: Operation ID

[1324] Output: Operation manual, text template

[1325] Step 3:

[1326] The server identifies error-prone areas based on past work results and retrieves near-miss information from the database. It sends this information to the terminal, which highlights the relevant fields or areas in red. This includes prompts for the server to generate warning messages about specific operations using an AI model and provide them to the user.

[1327] Input: Past work results

[1328] Output: Highlighted near-miss information, red text warnings

[1329] Step 4:

[1330] The device displays warning messages generated using a generation AI model, along with highlighted near-miss information. This allows users to recognize operational pitfalls in real time and perform operations safely.

[1331] Input: Near-miss incident information, generated AI model

[1332] Output: Display of warning message

[1333] In this way, a system has been built that supports actual operations by performing appropriate data input and output at each step.

[1334] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1335] This invention is a system designed to streamline internal operations and prevent errors, and incorporates an emotion engine that recognizes user emotions. Based on the user's emotional information recognized by the emotion engine, this system adjusts the displayed content and presentation method, and further provides operational support and displays warnings.

[1336] User login authentication

[1337] To log in to the system, the user enters a username and password. The terminal sends the entered information to the server, which retrieves user information from the database and performs authentication. If authentication is successful, a session ID is generated and returned to the terminal. If authentication fails, an error message is returned.

[1338] Display manual templates

[1339] The user opens a specific operation screen (e.g., inventory management). The terminal obtains the operation ID and sends a request to the server. The server retrieves the corresponding work manual and text template from the database and sends them to the terminal. The terminal displays the received manual and template in the upper right corner of the screen.

[1340] Display of near-miss information

[1341] The server identifies error-prone areas based on past work results and retrieves near-miss information from the database. The server sends this information to the terminal, which receives it and highlights the relevant fields or areas in red text.

[1342] Introducing an emotional engine

[1343] To recognize the user's emotions in real time, the device activates an emotion engine. The emotion engine analyzes the user's facial expressions, voice, keyboard input speed, etc., to identify their emotional state. The recognized emotional information is then sent to the server.

[1344] Adjustment based on emotional information

[1345] The server dynamically adjusts the displayed content and presentation method based on the emotional information it receives. For example, if the user is stressed, the displayed manuals and templates can be simplified, and important notes can be further emphasized. Conversely, if the user is relaxed, detailed information can be provided to support efficient work.

[1346] Specific example

[1347] For example, consider the case where a user opens the "Inventory Management" screen. After logging in, when the user opens the inventory management screen, the terminal sends the "Inventory Management" operation ID to the server. The server retrieves the relevant manual (e.g., inventory check procedure) and text template (e.g., inventory shortage notification email template) from the database and sends them to the terminal. The terminal displays this information in the upper right corner of the screen.

[1348] Simultaneously, the terminal activates an emotion engine to analyze the user's emotions in real time. For example, if the server detects that the user is stressed, it simplifies the displayed content and highlights important points (e.g., points to note when entering specific inventory quantities) in red. Conversely, if the user is relaxed, it displays a detailed manual to improve work efficiency.

[1349] Thus, the present invention not only assists user operation, prevents operational errors, and improves work efficiency, but also dynamically adjusts the displayed content according to the user's emotional state, providing an even more comfortable and efficient work environment.

[1350] The following describes the processing flow.

[1351] Step 1:

[1352] The user accesses the system and enters their username and password on the login screen.

[1353] Step 2:

[1354] The terminal sends the entered username and password to the server. This transmission is performed using HTTPS communication.

[1355] Step 3:

[1356] The server checks the database and verifies the submitted username and password. If the user information matches, authentication is successful; otherwise, authentication fails.

[1357] Step 4:

[1358] The server returns the authentication result to the terminal. If authentication is successful, it generates and returns a session ID; if authentication fails, it returns an error message.

[1359] Step 5:

[1360] The user opens a specific operation screen (e.g., inventory management).

[1361] Step 6:

[1362] The terminal obtains an operation ID and sends a request containing that ID to the server.

[1363] Step 7:

[1364] The server accesses the database and retrieves the relevant work manual and text template.

[1365] Step 8:

[1366] The server sends the acquired work manual and text template to the terminal.

[1367] Step 9:

[1368] The device analyzes the received manual and template and displays them in the upper right corner of the screen.

[1369] Step 10:

[1370] The device activates an emotion engine that analyzes the user's facial expressions, voice, and keyboard input speed in real time.

[1371] Step 11:

[1372] The emotion engine recognizes the user's emotional state and sends that information to the server.

[1373] Step 12:

[1374] The server retrieves past near-miss information related to the operation ID from the database.

[1375] Step 13:

[1376] The server sends near-miss information it has acquired to the terminal.

[1377] Step 14:

[1378] The server adjusts the displayed content and presentation method based on the emotional information it receives. For example, if the user is stressed, the displayed content is simplified, and warnings are further highlighted in red.

[1379] Step 15:

[1380] The terminal analyzes near-miss information and highlights a warning message in red text in the relevant field or area.

[1381] Step 16:

[1382] The terminal displays customized content received from the server to the user and supports their operation.

[1383] This entire process streamlines user operations, prevents operational errors, and dynamically adjusts displayed content based on the user's emotional state.

[1384] (Example 2)

[1385] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1386] Conventional in-house operating systems fail to adequately prevent operational errors and improve work efficiency. Furthermore, they often lack display adjustments based on the user's emotional state, creating a stressful environment for users. This can lead to decreased user efficiency and an increase in operational errors. Therefore, the present invention aims to solve these problems.

[1387] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1388] In this invention, the server includes means for identifying error-prone areas based on past work results and highlighting them using specific display means, means for automatically displaying relevant work manuals and text templates, means for authenticating using user authentication information and adjusting the display content according to the user's authority, and means for recognizing the user's emotions and dynamically adjusting the display content and presentation method based on the recognized emotions. This enables dynamic display adjustments according to the user's emotional state and work environment, thereby preventing operational errors and improving work efficiency.

[1389] "Past work results" refers to records of the progress and results of a series of operations or tasks performed in the past.

[1390] "Error-prone areas" are points where users are likely to frequently make mistakes during operation or work.

[1391] "Specific display means" refers to methods or devices that highlight designated information or warnings.

[1392] A "work manual" is a document that contains procedures and instructions for performing specific operations or tasks accurately and efficiently.

[1393] A "text template" is a text template created according to a specific format, which can be edited and used as needed.

[1394] "User authentication information" refers to information used to identify a user and verify their permissions, and primarily includes usernames and passwords.

[1395] "User emotions" refer to the psychological state a user is experiencing at a particular moment, and include things like stress and relaxation.

[1396] "Dynamic adjustment" means automatically making changes or modifications in real time according to the environment and conditions.

[1397] A "system" is a comprehensive mechanism in which multiple components work together to achieve a specific function.

[1398] This invention is a system designed to streamline internal operations and prevent errors. By incorporating an emotion engine that recognizes user emotions, this system enables dynamic adjustment of display content and presentation methods based on the user's emotional state.

[1399] First, to log in to the system, the user enters a username and password. The terminal sends the entered information to the server, which retrieves the user information from the database and performs authentication. If authentication is successful, the server generates a session ID and sends it back to the terminal. If authentication fails, the server returns an error message.

[1400] When a user opens a specific operation screen (e.g., inventory management), the terminal sends its operation ID to the server. The server retrieves the corresponding work manual and text template from the database and sends them to the terminal. The terminal displays the received manual and template in the upper right corner of the screen. At this time, based on past work results, it identifies areas prone to errors and retrieves near-miss information from the database. The server sends this information to the terminal, and the terminal highlights this information in red text in the corresponding fields or areas.

[1401] Furthermore, to recognize the user's emotions in real time, the device activates an emotion engine. The emotion engine analyzes the user's facial expressions, voice, and keyboard input speed to identify their emotional state. The recognized emotional information is sent to a server. Based on the received emotional information, the server dynamically adjusts the displayed content and presentation method. For example, if the user is stressed, the displayed manuals and templates are simplified, and important notes are highlighted. On the other hand, if the user is relaxed, detailed information is provided to support efficient work.

[1402] As a concrete example, let's explain what happens when a user opens the inventory management screen. After logging in, when a user opens the inventory management screen, the terminal sends the "Inventory Management" operation ID to the server. The server retrieves the relevant manual (e.g., inventory check procedure) and text template (e.g., inventory shortage notification email template) from the database and sends them to the terminal. The terminal displays this information in the upper right corner of the screen.

[1403] Simultaneously, the terminal activates an emotion engine to analyze the user's emotions in real time. For example, if the server detects that the user is stressed, it simplifies the displayed content and highlights important notes (e.g., points to note when entering specific inventory quantities) in red. Conversely, if the user is relaxed, it displays a detailed manual to improve work efficiency.

[1404] An example of a prompt using a generative AI model is, "Please tell me the appropriate way to display the manual when the user is stressed."

[1405] This system not only streamlines internal operations and prevents errors, but also provides a comfortable and efficient work environment tailored to the user's emotional state.

[1406] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1407] Step 1: The user enters their username and password on the login screen.

[1408] Input: Username, Password

[1409] Specific action: The user enters their username and password on the device's login screen.

[1410] Step 2: The terminal sends the entered username and password to the server.

[1411] Input: Username, Password

[1412] Specific operation: The terminal encrypts the username and password entered by the user and sends them to the server.

[1413] Step 3: The server retrieves user information from the database and performs authentication.

[1414] Input: Username, Password (encrypted)

[1415] Data processing: The server queries the database for the received username and password and performs a verification.

[1416] Output: Authentication result (success or failure)

[1417] Specific operation: The server retrieves the relevant user information from the database and verifies it against the encrypted password.

[1418] Step 4: Send the authentication result back to the device.

[1419] Input: Authentication result (success or failure)

[1420] Specific operation: The server returns the authentication result to the terminal. If successful, it generates and returns a session ID. If unsuccessful, it returns an error message.

[1421] Output: Session ID (on success), Error message (on failure)

[1422] Step 5: The user opens a specific operation screen (e.g., inventory management).

[1423] Specific action: The user selects a specific operation screen within the system and opens it on the terminal.

[1424] Step 6: The terminal sends the operation ID to the server.

[1425] Input: Operation ID

[1426] Specific action: The terminal sends the operation ID corresponding to the current operation screen to the server.

[1427] Step 7: The server retrieves the work manual and text template from the database.

[1428] Input: Operation ID

[1429] Data processing: The server retrieves the relevant work manual and text template from the database.

[1430] Output: Operation manual, text template

[1431] Specific operation: The server queries the database and retrieves the necessary manuals and templates.

[1432] Step 8: The device displays the manual and template on the screen.

[1433] Input: Work manual, text template

[1434] Specific action: The device displays the received manual and template in the upper right corner of the screen.

[1435] Step 9: The server identifies near-miss information based on past work results.

[1436] Input: Past work data

[1437] Data processing: The server analyzes past work data to identify areas prone to errors.

[1438] Output: Near Miss Information

[1439] Specific operation: The server retrieves past work results from the database and identifies near-miss information.

[1440] Step 10: The terminal displays near-miss information.

[1441] Input: Near-miss information

[1442] Specific operation: The terminal receives near-miss information and highlights the relevant field or area in red text.

[1443] Step 11: The device activates the emotion engine.

[1444] Specific action: The terminal starts the emotion engine software.

[1445] Step 12: The emotion engine analyzes the user's facial expressions, voice, and keyboard input speed.

[1446] Input: User's facial expressions, voice, keyboard input speed

[1447] Data processing: The emotion engine analyzes this data to identify emotional states.

[1448] Output: Emotional information

[1449] Specific operation: The emotion engine analyzes data in real time and sends the user's emotional state to the server.

[1450] Step 13: The server dynamically adjusts the displayed content based on sentiment information.

[1451] Input: Emotional information

[1452] Data processing: The server analyzes emotional information and adjusts the displayed content and presentation method.

[1453] Output: Display content after adjustment

[1454] Specific operation: Based on emotional information, the server simplifies the displayed content and highlights warnings if the user is stressed. If the user is relaxed, it displays detailed information.

[1455] Step 14: The device reflects the adjusted display content on the screen.

[1456] Input: Adjusted display content

[1457] Specific operation: The terminal displays the adjusted content received from the server on the screen.

[1458] (Application Example 2)

[1459] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1460] There is a need to improve work efficiency and prevent errors on factory production lines. However, conventional systems do not take into account the mental state of workers, so stress and fatigue can lead to work errors. As a result, production efficiency decreases, and quality declines and safety problems occur. To solve these problems, it is important to recognize workers' emotions in real time and dynamically adjust work instructions and precautions based on that.

[1461] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1462] In this invention, the server includes means for identifying error-prone areas based on past work results and highlighting them using specific display means; means for automatically displaying relevant work manuals and text templates; means for authenticating using user authentication information and adjusting the display content according to the user's authority; means for recognizing the user's emotions in real time; and means for dynamically adjusting the display content and presentation method based on the recognized emotion information. This makes it possible to improve work efficiency and prevent errors by adjusting the display content according to the worker's mental state.

[1463] "Past work results" refer to the results and data from work performed previously.

[1464] "Error-prone areas" refer to parts or processes in a work where mistakes frequently occur.

[1465] "Display means" refers to devices or methods for presenting information visually.

[1466] A "work manual" is a document that details the procedures and methods for performing a specific task.

[1467] A "text template" is a pre-prepared document template that follows a specific format.

[1468] "User authentication information" refers to identification information used to identify a user and verify their permissions.

[1469] "Authentication" is the process of verifying whether a user is legitimate.

[1470] "Permissions" refer to the range of operations a user can perform and information they can access within a system.

[1471] "Emotion recognition technology" refers to technology for detecting and identifying human emotions in real time.

[1472] "Emotional information" refers to data about a user's emotional state detected by emotion recognition tools.

[1473] "Dynamic adjustment" means making flexible changes in real time according to the situation and conditions.

[1474] A "user" is a person, such as a worker or employee, who uses the system.

[1475] A "system" is a collection of technical components, such as hardware, software, and networks, that are combined to achieve a specific purpose.

[1476] This invention is a system aimed at improving work efficiency and preventing errors on a factory production line. Specifically, it has a function to identify error-prone areas based on past work results and highlight them using a display device, and a function to automatically display relevant work manuals and text templates. Furthermore, this system recognizes the user's emotions in real time and dynamically adjusts the display content and presentation method based on that emotion information.

[1477] System Configuration

[1478] Hardware:

[1479] Camera: Used to capture the facial expressions of the workers.

[1480] Computer: Used to perform emotion recognition and adjust the displayed content.

[1481] Robots: Provide work assistance and instructions.

[1482] software:

[1483] OpenCV: A library for image processing.

[1484] Dlib: A library for face detection and facial landmark extraction.

[1485] Keras: A deep learning framework for using emotion recognition models.

[1486] Server: Stores and processes data.

[1487] Terminal: The interface that the user interacts with.

[1488] Program processing

[1489] The server receives video data transmitted from the camera and detects faces using OpenCV and Dlib. A Keras-based emotion recognition model analyzes the detected face regions in real time, and the results are obtained. This emotion information is sent to the terminal and used to adjust the displayed content. If the user is stressed, the server simplifies the displayed content and highlights important notices. Conversely, if the user is relaxed, a detailed manual is displayed to improve work efficiency.

[1490] Specific example

[1491] For example, when a user opens the inventory management operation screen, the terminal sends the "Inventory Management" operation ID to the server. The server retrieves the work manual and text template from the database and sends them to the terminal. Simultaneously, the terminal performs emotion recognition and sends the user's emotional state to the server. If the server detects that the user is stressed, it simplifies the displayed content and highlights important notes (such as points to note when entering specific inventory quantities) in red. Conversely, if the user is relaxed, it displays a detailed manual to improve work efficiency.

[1492] This system provides a flexible work environment tailored to the mental state of the worker, enabling improved work efficiency and prevention of operational errors.

[1493] Example of a prompt

[1494] "Please propose a program that uses emotion recognition to dynamically adjust operational guidelines based on the fatigue level of factory workers. When fatigued, the instructions should be simplified and precautions should be emphasized."

[1495] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1496] Step 1:

[1497] Input: User login information (username, password)

[1498] Specific operation and data processing: The terminal receives the username and password entered by the user and sends them to the server.

[1499] Output: The server retrieves user information from the database, generates a session ID if authentication is successful, and returns it to the terminal. If authentication fails, it returns an error message.

[1500] Step 2:

[1501] Input: User selects an operation screen (e.g., inventory management)

[1502] Specific actions and data processing: The terminal obtains an operation ID and sends it to the server as a request.

[1503] Output: The server retrieves the relevant work manual and text template from the database and sends it to the terminal. The terminal displays this information in the upper right corner of the screen.

[1504] Step 3:

[1505] Input: Past work results

[1506] Specific operations and data processing: The server analyzes past work results and identifies areas prone to errors.

[1507] Output: The server retrieves near-miss information from the database and sends it to the terminal. The terminal highlights the relevant fields or areas in red text.

[1508] Step 4:

[1509] Input: Real-time user emotion data (facial expressions, voice, keyboard input speed, etc.)

[1510] Specific operation and data processing: The terminal activates an emotion engine and analyzes the user's emotions in real time. The user's facial expressions are captured by the camera, faces are detected using OpenCV and Dlib, and emotions are analyzed using an emotion recognition model based on Keras.

[1511] Output: Recognized emotion information is sent from the terminal to the server.

[1512] Step 5:

[1513] Input: Emotional information (e.g., stressed state, relaxed state)

[1514] Specific operations and data processing: The server analyzes emotional information and dynamically adjusts the displayed content and presentation method. For example, if the user is stressed, the displayed content is simplified and warnings are highlighted. If the user is relaxed, a detailed manual is provided.

[1515] Output: The adjusted display content is sent to the terminal and displayed to the user.

[1516] Step 6:

[1517] Input: User actions according to the situation

[1518] Specific actions and data processing: The system performs operations based on the information provided by the user. The results of the operations are sent from the terminal to the server.

[1519] Output: The server records the operation results and adjusts the displayed content and instructions as needed.

[1520] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1521] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1522] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[1523] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1524] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[1525] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[1526] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[1527] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[1528] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[1529] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[1530] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[1531] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[1532] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[1533] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1534] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[1535] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[1536] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[1537] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[1538] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[1539] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[1540] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted as being incorporated by reference.

[1541] The following is further disclosed regarding the embodiments described above.

[1542] (Claim 1)

[1543] A means for identifying error-prone areas based on past work results and highlighting them using a specific display means,

[1544] A means for automatically displaying relevant work manuals and text templates,

[1545] A means of performing authentication using user authentication information and adjusting the displayed content according to the user's permissions,

[1546] A system that includes this.

[1547] (Claim 2)

[1548] The system according to claim 1, further comprising means for automatically acquiring and displaying instruction information related to the relevant work content when a user accesses a specific operation screen.

[1549] (Claim 3)

[1550] The system according to claim 1, comprising means for learning from past records to provide warnings about areas prone to errors and highlighting those contents in red.

[1551] "Example 1"

[1552] (Claim 1)

[1553] A means for identifying error-prone areas based on past work results and highlighting them using a specific display means,

[1554] A means for automatically displaying relevant work instructions and text templates,

[1555] A means of performing authentication using user authentication information and adjusting the displayed content according to the user's permissions,

[1556] A means of verifying the authentication information entered by the user against a database and generating a session ID,

[1557] A system that includes this.

[1558] (Claim 2)

[1559] A means to automatically acquire and display instruction information related to the relevant task when a user accesses a specific operation screen,

[1560] A means by which the terminal sends a request to the server based on the operation ID and obtains the corresponding work order and text template,

[1561] The system according to claim 1.

[1562] (Claim 3)

[1563] A method to learn from past records about areas prone to errors and highlight that information in red,

[1564] The server uses machine learning algorithms to analyze past work results and identify areas prone to errors,

[1565] The system according to claim 1.

[1566] "Application Example 1"

[1567] (Claim 1)

[1568] A means for identifying error-prone areas based on past work results and highlighting them using a specific display means,

[1569] A means for automatically displaying relevant work manuals and text templates,

[1570] A means of performing authentication using user authentication information and adjusting the displayed content according to the user's permissions,

[1571] A means to identify in real time areas prone to errors when operating factory machinery and equipment, and to display work procedures and warning messages via a smart terminal,

[1572] A means of presenting warning messages regarding specific operations using a generative AI model,

[1573] A system that includes this.

[1574] (Claim 2)

[1575] The system according to claim 1, further comprising means for automatically acquiring and displaying instruction information related to the relevant work content when a user accesses a specific operation screen.

[1576] (Claim 3)

[1577] The system according to claim 1, comprising means for learning from past records to provide warnings about areas prone to errors and highlighting those contents in red.

[1578] "Example 2 of combining an emotion engine"

[1579] (Claim 1)

[1580] A means for identifying error-prone areas based on past work results and highlighting them using a specific display means,

[1581] A means for automatically displaying relevant work manuals and text templates,

[1582] A means of performing authentication using user authentication information and adjusting the displayed content according to the user's permissions,

[1583] A means for recognizing user emotions and dynamically adjusting the displayed content and presentation method based on the recognized emotions,

[1584] A system that includes this.

[1585] (Claim 2)

[1586] The system according to claim 1, further comprising means for automatically acquiring and displaying instruction information related to the relevant work content when a user accesses a specific operation screen.

[1587] (Claim 3)

[1588] The system according to claim 1, comprising means for learning from past records to provide warnings about areas prone to errors and highlighting those contents in red.

[1589] "Application example 2 when combining with an emotional engine"

[1590] (Claim 1)

[1591] A means for identifying error-prone areas based on past work results and highlighting them using a specific display means,

[1592] A means for automatically displaying relevant work manuals and text templates,

[1593] A means of performing authentication using user authentication information and adjusting the displayed content according to the user's permissions,

[1594] A means of recognizing user emotions in real time,

[1595] A means for dynamically adjusting the display content and presentation method based on recognized emotional information,

[1596] A system that includes this.

[1597] (Claim 2)

[1598] The system according to claim 1, further comprising means for automatically acquiring and displaying instruction information related to the relevant work content when a user accesses a specific operation screen.

[1599] (Claim 3)

[1600] The system according to claim 1, comprising means for learning from past records to provide warnings about areas prone to errors and highlighting those contents in red, and means for adjusting the displayed content according to the user's emotions. [Explanation of Symbols]

[1601] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for identifying error-prone areas based on past work results and highlighting them using a specific display means, A means for automatically displaying relevant work manuals and text templates, A means of performing authentication using user authentication information and adjusting the displayed content according to the user's permissions, A system that includes this.

2. The system according to claim 1, further comprising means for automatically acquiring and displaying instruction information related to the relevant work content when a user accesses a specific operation screen.

3. The system according to claim 1, comprising means for learning from past records what to watch out for regarding areas prone to errors and highlighting that content in red.

Images