System

Abstract

To provide a system.SOLUTION: A system comprises: first means for displaying, on smart glasses worn by a worker, information including work procedures, equipment images, and past accident cases; second means for receiving voice input related to the work from the worker; third means for generating a prompt sentence to output the next work procedure on the basis of the content of the voice input by the worker; fourth means for inputting the generated prompt sentence into a generative AI model to generate the next work procedure; and fifth means for displaying the generated next work procedure on the display of the smart glasses.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Currently, when performing any work on equipment, one worker and one inspector (whether on-site or remote) are required, which has led to problems with labor shortages and operational errors. [Means for solving the problem]

[0005] The system learns the information required for work (procedures, equipment images, past accident cases, etc.) in advance, and when performing work, inputs the operation screen and procedures to be carried out by voice through smart glasses, and the system judges whether the work is OK or NG and displays the answer on the smart glasses. This reduces the number of people required to check and prevents operational errors. [Brief explanation of the drawings]

[0006] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 2 is a sequence diagram showing a flow of processing in the data processing system according to the first embodiment of the first form example. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Embodiment 1. [Figure 13] FIG. 10 is a sequence diagram showing a processing flow of a data processing system in a second embodiment of the second form example. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Embodiment Example 2. [Figure 15] FIG. 10 is a sequence diagram showing the flow of processing in a data processing system according to a third embodiment of the third embodiment. [Figure 16]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Embodiment 3. [Figure 17] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the first embodiment of the first form example when an emotion engine is combined. [Figure 18] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Form Example 1 when an emotion engine is combined. [Figure 19] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the second embodiment of the second form example when an emotion engine is combined. [Figure 20] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Form Example 2 when an emotion engine is combined. [Figure 21] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the third embodiment of the third form example when an emotion engine is combined. [Figure 22] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Form Example 3 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0007] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0008] First, the terms used in the following description will be explained.

[0009] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be one arithmetic device or a combination of multiple arithmetic devices. Also, the processor may be one type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit),

[0010] Examples include a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)).

[0011] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0012] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g.,

[0013] hard disk), or magnetic tape.

[0014] In the following embodiments, the coded communication I / F (Interface) refers to a communication processor and

[0015] The communication I / F is an interface that includes a network interface (NIC) and an antenna. The communication I / F controls communication between multiple computers. Examples of communication standards that can be applied to the communication I / F include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), and Bluetooth (registered trademark).

[0016] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0017] [First embodiment]

[0018] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0019] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0020] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0021] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0022] The reception device 38 is equipped with a touch panel 38A and a microphone 38B.

[0023] The touch panel 38A receives user input by touch of a pointer (for example, a pen or a finger) by detecting contact with the pointer. The microphone 38B receives user input by voice by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0024] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 includes an optical system including a lens, an aperture, and a shutter, and a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or CC

[0025] A compact digital camera equipped with an imaging element such as a D (Charge Coupled Device) image sensor.

[0026] It's a camera.

[0027] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0028] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0029] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0030] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0031] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0032] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0033] "Example 1"

[0034] One embodiment of the present invention is a system in which workers wear smart glasses and learn information such as work procedures, facility images, and past accident cases in advance. The workers can view the operation screen through the smart glasses and input instructions by voice. This system judges whether the work is OK or NG based on the worker's voice input and displays the result on the smart glasses' display.

[0035] "Example 2"

[0036] As a specific example, consider the case where a worker is performing maintenance work on equipment. The worker wears smart glasses and inputs the maintenance work procedures by voice. The system determines whether the input procedures are correct and displays the results on the smart glasses' display. In this way, the worker can proceed with the work while checking whether their work is correct.

[0037] "Example 3"

[0038] The system of the present invention can also learn work information using artificial intelligence. The artificial intelligence learns patterns from past work data and accident cases, and judges whether a work is OK or not based on that. This allows the system's judgment accuracy to improve over time, further improving the work efficiency and safety of workers.

[0039] The processing flow of each embodiment will be described below.

[0040] "Example 1"

[0041] Step 1: The worker puts on the smart glasses.

[0042] Step 2: The smart glasses learn the information necessary for the work (procedures, equipment images, past accident cases, etc.) in advance.

[0043] Step 3: The worker can view the operation screen through the smart glasses and input instructions by voice.

[0044] Step 4: The system judges whether the task is OK or NG based on the worker's voice input and displays the result on the smart glasses display.

[0045] "Example 2"

[0046] Step 1: When a worker performs maintenance work on equipment, he or she first puts on the smart glasses.

[0047] Step 2: The worker inputs the maintenance procedure by voice.

[0048] Step 3: The system determines whether the entered steps are correct and displays the results on the smart glasses display.

[0049] Step 4: The worker performs the task based on the system's feedback.

[0050] "Example 3"

[0051] Step 1: The system of the present invention uses artificial intelligence to learn task information.

[0052] Step 2: The artificial intelligence learns patterns from past work data and accident cases.

[0053] Step 3: Based on the learned patterns, the system determines whether the work is OK or not.

[0054] Step 4: The results of the assessment are displayed on the smart glasses display, and the worker proceeds with the work based on that feedback.

[0055] Example 1

[0056] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0057] With conventional work support systems, it was difficult for workers to efficiently learn information such as procedures, equipment images, and past accident cases, and receive appropriate instructions while working. There were also issues with the accuracy of voice-input instructions and the speed with which work was judged to be OK / NG. This could potentially lead to a decline in work efficiency and safety.

[0058] The identification process by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means. In this invention, the server includes a means for learning information required for the work (procedures, equipment images, past accident cases, etc.) in advance, a means for inputting the operation screen and procedures to be performed by voice through the smart glasses when performing the work, a means for determining whether the work is OK / NG and displaying the answer on the smart glasses, a means for analyzing the voice data and converting it into text, and a means for generating prompt sentences using a generative AI model and determining whether the work is OK / NG. This allows the worker to learn information efficiently, improves the accuracy of instructions input by voice, and enables quick OK / NG determination of the work.

[0059] "Information necessary for the work" refers to the knowledge and data necessary to carry out the work, such as procedures, equipment images, and past accident cases.

[0060] "Means for learning" refers to a method or device that allows a worker to learn the information necessary for the work in advance.

[0061] "Smart glasses" are wearable devices worn by workers to display operation screens and information.

[0062] An "operation screen" is a screen that displays information such as work procedures and instructions on smart glasses.

[0063] "Voice input means" refers to a method or device that allows a worker to input instructions or information using voice.

[0064] "Means for determining OK / NG" refers to a method or device for evaluating the progress and results of work and determining whether they are appropriate.

[0065] "Means for analyzing voice data and converting it into text" refers to a method or device for converting voice data into text data using voice recognition technology.

[0066] A "generative AI model" is a model that uses artificial intelligence to generate prompt sentences and determine whether a task is OK or not.

[0067] A "prompt sentence" is an instruction sentence to be input into a generative AI model, and is the sentence that serves as the basis for determining whether a task is OK or not.

[0068] MODE FOR CARRYING OUT THE INVENTION

[0069] This invention is a system in which workers wear smart glasses and learn information such as work procedures, equipment images, and past accident cases in advance. Workers can view the operation screen through the smart glasses and input instructions by voice. The system judges whether the work is OK or NG based on the worker's voice input and displays the result on the smart glasses' display.

[0070] 1. Program Generation

[0071] The server generates a program that allows workers to wear smart glasses and learn in advance information such as work procedures, facility images, and past accident cases. This program includes functions that allow workers to view the operation screen through the smart glasses and input instructions by voice.

[0072] 2. Program processing explanation

[0073] The server installs the generated program into the smart glasses. When a worker wears the smart glasses, they display information such as work procedures, images of equipment, and past accident cases. The worker can check this information through the smart glasses' display.

[0074] When a worker inputs instructions by voice, the smart glasses send the voice data to a server, which then uses voice recognition software (e.g., voice recognition technology) to convert the voice data into text. The server then uses a generative AI model (e.g., artificial intelligence model) to generate prompts to determine whether the input instructions are acceptable or not.

[0075] The server inputs the generated prompt sentences into the AI ​​model and obtains the results that determine whether the task is OK or not. The results are then sent to the smart glasses and displayed on the smart glasses' display.

[0076] 3. Examples of concrete examples and prompts

[0077] As a concrete example, consider the case where a worker inputs a command by voice, such as "Please tell me the next step in the work." In this case, the server generates the following prompt:

[0078] Example prompt sentence:

[0079] "The worker wants to know the next step. What's the next step?"

[0080] The server inputs this prompt into the generative AI model to obtain the next steps, which are then displayed on the smart glasses screen.

[0081] In this way, workers can check information such as work procedures, equipment images, and past accident cases through smart glasses, and determine whether the work is OK or not by inputting instructions via voice.

[0082] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0083] Step 1:

[0084] Starting up and connecting to smart glasses

[0085] The user puts on the smart glasses and turns them on.

[0086] The device (smart glasses) connects to the server via Wi-Fi.

[0087] Input: Smart glasses power on, Wi-Fi connection information

[0088] Output: Connection established with server

[0089] Step 2:

[0090] Displaying work procedures and information

[0091] The server recognizes the worker's ID and sends information such as relevant work procedures, equipment images, and past accident cases to the smart glasses.

[0092] The terminal displays the received information on its screen, such as a list of work procedures or an image of the equipment.

[0093] Input: Worker ID, related information

[0094] Output: Work procedures and equipment images displayed on smart glasses

[0095] Step 3:

[0096] Entering voice commands

[0097] Users can speak into the smart glasses to input instructions, such as "Please tell me the next step."

[0098] Input: User's voice commands

[0099] Output: Audio data

[0100] Step 4:

[0101] Sending and analyzing voice data

[0102] The terminal transmits the user's voice data to the server.

[0103] The server converts the voice data into text using voice recognition software (eg, speech recognition technology).

[0104] Input: Audio data

[0105] Output: Text data

[0106] Step 5:

[0107] Judging whether work is OK or NG

[0108] The server generates a prompt based on the converted text, for example, "The worker wants to know the next step in the work. Please tell me the next step."

[0109] The server inputs this prompt into a generative AI model (e.g., an artificial intelligence model) and obtains the next work procedure and a pass / fail judgment for the work.

[0110] Input: Text data, prompt

[0111] Output: Work procedure and OK / NG judgment

[0112] Step 6:

[0113] Display of judgment results

[0114] The server sends the obtained results to the smart glasses.

[0115] The terminal displays the received results on its display, such as "The next step is to install part A" or "The work is OK."

[0116] Input: Work procedure and OK / NG judgment

[0117] Output: The results displayed on the smart glasses

[0118] (Application example 1)

[0119] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0120] In conventional factory work, workers spend a lot of time and effort checking procedures and the status of equipment. It is also difficult to refer to past accident cases, which means that work safety is not adequately ensured. Furthermore, there is a lack of means to report and properly evaluate work progress in real time, which reduces work efficiency. To solve these issues, a system is needed that allows workers to work efficiently and safely using smart glasses.

[0121] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0122] In this invention, the server includes means for learning information required for work (procedures, equipment images, past accident cases, etc.) in advance, means for inputting the operation screen and procedures to be performed by voice through the smart glasses when performing work, means for determining whether the work is OK / NG and displaying the answer on the smart glasses, means for displaying the real-time status of the equipment, means for displaying past accident cases and issuing warnings, means for reporting the progress of the work based on the voice input, means for determining whether the work is OK / NG based on the progress of the work, and means for displaying the result on the display of the smart glasses. This enables workers to obtain the necessary information in real time and perform their work efficiently and safely.

[0123] "Information necessary for work" refers to information necessary to perform work, such as work procedures, equipment images, and past accident cases.

[0124] "Means of learning in advance" refers to the means of obtaining and understanding the information necessary for the work in advance.

[0125] "Smart glasses" are eyeglass-type devices worn by workers to display information.

[0126] The "operation screen" is the interface that the worker can see through the smart glasses.

[0127] "Voice input means" refers to a means by which a worker inputs instructions or information using voice.

[0128] "Means for determining whether something is OK / NG" refers to a means for evaluating the progress and results of work and determining whether they are appropriate.

[0129] The "means for displaying the answer on the smart glasses" refers to a means for displaying the judgment result on the display of the smart glasses.

[0130] The "means for displaying the real-time status of the equipment" is a means for displaying the current status of the equipment in real time.

[0131] The "means for displaying past accident cases and calling attention" is a means for displaying past accident cases and calling attention to workers.

[0132] The "means for reporting the progress of work based on voice input" is a means for reporting the progress of work based on voice input by the worker.

[0133] "Means for determining whether work is OK or NG based on the progress of the work" refers to a means for evaluating the progress of the work and determining whether the work is being carried out appropriately.

[0134] "Means for displaying on the display of the smart glasses" refers to means for displaying the determination results and other information on the display of the smart glasses.

[0135] In the system for implementing this invention, workers wear smart glasses and can carry out their work while checking information in real time, such as work procedures, equipment status, past accident cases, etc. Specific embodiments of this system are described below.

[0136] Hardware and software used

[0137] Smart glasses: Eyeglass-type devices worn by workers to display information.

[0138] Microphone: A device for capturing voice input from the worker.

[0139] Server: A computer system that manages the information required for work and processes voice input.

[0140] speech_recognition library: Software for converting voice input into text.

[0141] smart_Glass® es_display module: Custom software for displaying information on the smart glasses display.

[0142] factory_system module: Custom software that obtains work procedures, equipment status, and past accident cases from the factory system and determines whether the work is OK or not.

[0143] System Operation

[0144] 1. Display of work procedures: The server displays the necessary steps for the work on the smart glasses display, allowing the worker to check the next step in real time.

[0145] 2. Checking equipment status: The server acquires the real-time status of the equipment and displays it on the smart glasses, allowing workers to continue their work while understanding the current status of the equipment.

[0146] 3. Display of past accident cases: The server displays past accident cases on the smart glasses and alerts the worker, allowing the worker to learn from past mistakes and work safely.

[0147] 4. Acquire voice input: The worker inputs instructions by voice through a microphone. The server converts the voice into text using the speech_recognition library.

[0148] 5. Work progress reporting: Based on the worker's voice input, the server reports the work progress, allowing the worker to report the progress without using their hands.

[0149] 6. Judgement of work pass / fail: The server uses the factory_system module to judge whether the work is pass / fail based on the voice input.

[0150] 7. Display of results: The results are displayed on the smart glasses screen, allowing workers to check the evaluation of their work in real time.

[0151] Specific examples

[0152] When a worker wears smart glasses and performs maintenance work on equipment, they use the application following the steps below.

[0153] 1. Voice command to the smart glasses: "Show me the next steps."

[0154] 2. Voice command to the smart glasses to "check the equipment status."

[0155] 3. Voice command to the smart glasses: "Show me past accident cases."

[0156] 4. When the task is complete, say "The task is complete."

[0157] 5. The server determines whether the work is OK or not, and the result is displayed on the smart glasses.

[0158] Prompt Sentence Examples

[0159] "Show me the next step"

[0160] Check the condition of the equipment

[0161] "Show me past accident cases"

[0162] "The work is complete."

[0163] In this way, the use of smart glasses can significantly improve work efficiency and safety within factories.

[0164] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0165] Step 1:

[0166] The server learns the information required for the work (procedures, equipment images, past accident cases, etc.) in advance. This involves acquiring data from the factory system and organizing and analyzing the information using artificial intelligence. The input is data from the factory system, and the output is organized and analyzed work procedures, equipment images, and past accident cases.

[0167] Step 2:

[0168] The user puts on the smart glasses and starts working. The server displays the work procedure on the smart glasses display. The input is the work procedure data from the server, and the output is the work procedure displayed on the smart glasses display.

[0169] Step 3:

[0170] The user inputs voice instructions through a microphone. The server converts the speech to text using the speech_recognition library. The input is the user's voice instructions and the output is the text instructions.

[0171] Step 4:

[0172] The server obtains the real-time status of the equipment based on the textual instructions and displays it on the smart glasses. The input is the textual instructions, and the output is the real-time status of the equipment displayed on the smart glasses display.

[0173] Step 5:

[0174] The server retrieves past accident cases and displays them on the smart glasses. The input is the user's voice command, and the output is the past accident cases displayed on the smart glasses display.

[0175] Step 6:

[0176] The user reports the progress of the work by voice. The server converts the voice to text using the speech_recognition library and records the progress of the work. The input is the user's voice report and the output is the progress report in text format.

[0177] Step 7:

[0178] The server judges whether the work is OK or NG based on the progress report in text format. The input is the progress report in text format, and the output is the OK / NG judgment result of the work.

[0179] Step 8:

[0180] The server displays the judgment result on the smart glasses display. The input is the OK / NG judgment result of the work, and the output is the judgment result displayed on the smart glasses display.

[0181] Example 2

[0182] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0183] Conventional maintenance work requires workers to accurately understand and execute procedures, but it can be difficult to confirm procedures and determine their accuracy. Furthermore, workers must use their hands to confirm procedures while working, which reduces work efficiency. Even when voice input is used, it is difficult to accurately analyze the voice data and determine the procedure, so real-time feedback may not be obtained. To solve these issues, a system that uses voice input to confirm procedures and provide real-time feedback is needed.

[0184] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0185] In this invention, the server includes means for learning information required for work (procedures, equipment images, past accident cases, etc.) in advance, means for inputting the operation screen and procedures to be performed by voice through the smart glasses when performing work, means for determining whether the work is OK / NG and displaying the response on the smart glasses, means for converting voice data into text data, means for analyzing the text data and determining the accuracy of the procedures, means for transmitting the determination result to the smart glasses, and means for displaying the determination result on the display of the smart glasses. This enables workers to confirm procedures using voice input and accurately proceed with maintenance work while receiving feedback in real time.

[0186] "Information necessary for the work" refers to information such as procedures required to perform maintenance work, equipment images, and past accident cases.

[0187] "Means of learning" are methods and techniques for acquiring and understanding the information necessary for a task in advance.

[0188] "Smart glasses" are wearable devices that have built-in displays and microphones and allow for voice input and information display.

[0189] The "operation screen" is a screen that appears on the smart glasses display and is used to check work procedures and other information.

[0190] "Voice input means" refers to a method or technology that allows a worker to input information about an operation screen or procedures to be performed using voice.

[0191] "Means for determining whether the procedure entered is correct" refers to the method or technology used to determine whether the procedure entered is correct.

[0192] "Means for displaying the answer" refers to the method or technology for displaying the judgment result on the smart glasses display.

[0193] A "means for converting voice data to text data" is a method or technique for converting voice input into text form.

[0194] The "means for analyzing text data to determine the accuracy of procedures" refers to a method or technology for analyzing text data and determining whether the entered procedures are correct.

[0195] "Means for transmitting the determination result to the smart glasses" refers to a method or technology for transmitting the determination result of the correctness of the procedure to the smart glasses.

[0196] "Means for displaying the determination result on the display of the smart glasses" refers to a method or technology for displaying the determination result on the display of the smart glasses.

[0197] MODE FOR CARRYING OUT THE INVENTION

[0198] This invention provides a system that, when a worker performs maintenance work on equipment, uses smart glasses to input procedures by voice and determines the accuracy of the procedures in real time. Specific embodiments of this system are described below.

[0199] Hardware and software used

[0200] Smart glasses: A wearable device with a built-in display and microphone that allows voice input and information display.

[0201] Server: A computer system that analyzes voice data, determines procedures, and transmits results.

[0202] Speech recognition technology for converting voice data into text data.

[0203] Generative AI model (e.g., OpenAI's GPT-4®): An artificial intelligence model for analyzing text data and determining the accuracy of procedures.

[0204] System Operation

[0205] 1. Voice input:

[0206] The user puts on the smart glasses and inputs maintenance instructions by voice, such as "remove the filter."

[0207] 2. Sending audio data:

[0208] The device (smart glasses) captures audio data through a built-in microphone and transmits the data to a server via Wi-Fi (registered trademark) or Bluetooth (registered trademark).

[0209] 3. Audio data conversion:

[0210] The server converts the received voice data into text data using a predefined API, which provides highly accurate voice recognition.

[0211] 4. Procedure determination:

[0212] The server uses a generative AI model (e.g., OpenAI's GPT-4) to determine whether the maintenance procedures received as text data are correct, which involves checking them against a database of predefined correct procedures.

[0213] 5. Sending the results:

[0214] The server determines the accuracy of the procedure and sends the results to the smart glasses, again via Wi-Fi or Bluetooth.

[0215] 6. Displaying the results:

[0216] The device (smart glasses) displays the received judgment result on its display. For example, it may display a message such as "The procedure is correct. Please proceed to the next step."

[0217] Specific examples

[0218] As a specific example, consider the case where a worker changes an air conditioner filter.

[0219] 1. The user puts on the smart glasses and says, "Remove filter."

[0220] 2. The device (smart glasses) sends this voice data to the server.

[0221] 3. The server uses a specified API to convert the voice data into text data that says "Remove filter."

[0222] 4. The server uses a generative AI model to determine whether this text data is a valid instruction, for example, by checking whether the instruction "remove filter" exists in the database.

[0223] 5. The server sends the result of the judgment that "the procedure is correct" to the smart glasses.

[0224] 6. The device (smart glasses) displays the result of this judgment on the screen and informs the user, "The procedure is correct. Please proceed to the next step."

[0225] Prompt Sentence Examples

[0226] User: "Remove filter"

[0227] Server: "Your steps are correct. Please proceed to the next step."

[0228] User: "Install a new filter"

[0229] Server: "The procedure was correct. The maintenance work has been completed."

[0230] In this way, the user can receive real-time feedback and perform maintenance tasks accurately.

[0231] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0232] Program processing flow

[0233] Step 1: Voice Input

[0234] The user puts on the smart glasses and inputs maintenance instructions by voice, such as "remove the filter."

[0235] Input: User's voice command

[0236] Output: Audio data captured by the smart glasses microphone

[0237] Step 2: Sending audio data

[0238] The device (smart glasses) captures audio data through a built-in microphone and transmits it to a server via Wi-Fi or Bluetooth.

[0239] Input: Audio data captured by the smart glasses microphone

[0240] Output: Audio data sent to the server

[0241] Step 3: Convert the audio data

[0242] The server converts the received voice data into text data using a predefined API, which provides highly accurate voice recognition.

[0243] Input: Audio data sent to the server

[0244] Output: Text data (e.g. "Remove filter")

[0245] Step 4: Determine the procedure

[0246] The server uses a generative AI model (e.g., OpenAI's GPT-4) to determine whether the maintenance procedures received as text data are correct, which involves checking them against a database of predefined correct procedures.

[0247] Input: Text data (e.g., "Remove filter")

[0248] Output: Verification result (e.g. "The procedure is correct")

[0249] Step 5: Sending the results

[0250] The server determines the accuracy of the procedure and sends the results to the smart glasses, again via Wi-Fi or Bluetooth.

[0251] Input: Verification result (e.g., "The procedure is correct")

[0252] Output: The judgment result sent to the smart glasses.

[0253] Step 6: View the results

[0254] The device (smart glasses) displays the received judgment result on its display. For example, it may display a message such as "The procedure is correct. Please proceed to the next step."

[0255] Input: The judgment result sent to the smart glasses

[0256] Output: A message displayed on the smart glasses display (e.g., "Your steps are correct. Please proceed to the next step.")

[0257] Specific examples of operation

[0258] 1. User: Say "Remove filter."

[0259] 2. Device (smart glasses): Sends voice data to the server.

[0260] 3. Server: Converts the audio data into text data using a specified API.

[0261] 4. Server: Analyzes the text data using a generative AI model to determine the accuracy of the steps.

[0262] 5. Server: Sends the judgment results to the smart glasses.

[0263] 6. Device (smart glasses): Displays the judgment result on the screen and informs the user, "The procedure is correct. Please proceed to the next step."

[0264] In this way, the user can receive real-time feedback and perform maintenance tasks accurately.

[0265] (Application example 2)

[0266] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0267] In conventional maintenance work, workers are required to accurately understand and carry out procedures, but there is a lack of means to confirm procedures and the accuracy of work in real time, which can lead to work errors and reduced efficiency. There is also a need for a system that allows workers to input procedures by voice and instantly confirm their accuracy. This will improve the efficiency and accuracy of work.

[0268] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0269] In this invention, the server includes a means for learning information required for work (procedures, facility images, past accident cases, etc.) in advance, a means for inputting the operation screen and procedures to be performed by voice through the smart glasses when performing work, a means for determining whether the work is OK / NG and displaying the answer on the smart glasses, and a means for determining the procedures input by voice in real time and displaying the result on the display of the smart glasses. This enables workers to efficiently carry out maintenance work while checking the accuracy of the work in real time.

[0270] "Information necessary for the work" refers to information that is necessary when performing maintenance work, such as procedures, equipment images, and past accident cases.

[0271] "Means of learning" refers to a means of learning the information necessary for the work in advance, and is carried out using artificial intelligence.

[0272] "Smart glasses" are eyeglass-type devices that can be worn by workers to display operation screens and procedures to be carried out, and can also accept voice input.

[0273] The "operation screen" is the screen displayed on the smart glasses, which the worker refers to when performing maintenance work.

[0274] The "means for inputting by voice" refers to a means for an operator to input maintenance procedures by voice, and is carried out using voice recognition technology.

[0275] The "means for determining whether the maintenance procedure entered is correct" is a means for determining whether the maintenance procedure entered is correct.

[0276] The "means for displaying the answer" is a means for displaying the judgment result on the display of the smart glasses.

[0277] The "means for determining in real time" is a means for instantly determining the procedure input by voice and displaying the result in real time.

[0278] A "display" is a display device installed in smart glasses that displays the judgment results and operation screen.

[0279] The system for implementing this invention allows a worker to wear smart glasses and, when performing maintenance work, input procedures by voice, and the system determines in real time whether the procedures are correct and displays the results on the smart glasses' display.

[0280] System configuration

[0281] The system consists of the following main components:

[0282] 1. Smart glasses: A glasses-type device worn by workers, equipped with a display that shows the operation screen and judgment results, as well as a microphone that accepts voice input.

[0283] 2. Server: Uses voice recognition technology and artificial intelligence to process the input procedures.

[0284] 3. Speech recognition software: Software that converts voice input into text, such as Google's speech recognition API.

[0285] 4. Artificial intelligence model: A model that learns the information necessary for the task in advance and determines whether the input procedures are correct.

[0286] Program processing

[0287] The server works as follows:

[0288] 1. Acquiring voice input: Acquire the worker's voice through the microphone in the smart glasses.

[0289] 2. Speech recognition: Convert captured speech into text using speech recognition software.

[0290] 3. Procedure validation: The converted text is input into an artificial intelligence model to determine whether the procedure is correct.

[0291] 4. Displaying the results: The results are displayed on the smart glasses screen.

[0292] Hardware and software used

[0293] Hardware: Smart glasses, microphone

[0294] Software: Python (registered trademark), SpeechRecognition library, smart glasses SDK, artificial intelligence model (e.g., TENSORFLOW (registered trademark))

[0295] Specific examples

[0296] For example, when a technician performs maintenance on a robot, they can say, "Turn off the robot, remove the cover, and inspect the internal components." This speech is picked up by the microphone in the smart glasses and converted into text by speech recognition software. The converted text is sent to a server, where an artificial intelligence model determines the accuracy of the procedure. The result is displayed on the smart glasses' display, saying, "The procedure is correct."

[0297] Prompt Sentence Examples

[0298] Turn off the robot, remove the cover, and inspect the internal components.

[0299] In this way, workers can efficiently carry out maintenance work while checking the accuracy of the work in real time.

[0300] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0301] Step 1:

[0302] The user puts on the smart glasses and begins maintenance work. The user then speaks maintenance procedures into the microphone on the smart glasses. The voice input contains specific instructions, such as "Turn off the robot's power, remove the cover, and inspect the internal parts." Input: User's voice. Output: Voice data.

[0303] Step 2:

[0304] The smart glasses send the captured voice data to the server, which then uses voice recognition software (e.g., Google's voice recognition API) to convert the voice data into text data. Input: Voice data. Output: Text data.

[0305] Step 3:

[0306] The server inputs the converted text data into an AI model. The AI ​​model compares the input data with a database of previously learned maintenance procedures and determines whether the procedure is correct. Input: Text data. Output: Verification result (correct / incorrect).

[0307] Step 4:

[0308] The server sends the judgment result to the smart glasses. The smart glasses display the judgment result on the display. For example, a message such as "The procedure is correct" or "The procedure is incorrect. Please check again" is displayed. Input: Judgment result. Output: Message displayed on the display.

[0309] Step 5:

[0310] The user checks the judgment results displayed on the smart glasses display and corrects the procedure if necessary. If the correct procedure is displayed, the user continues working. Input: Message displayed on the display. Output: User action (correcting the procedure or continuing the work).

[0311] In this way, the user can proceed with the work while checking the accuracy of the maintenance procedure in real time.

[0312] Example 3

[0313] Next, a description will be given of a third embodiment of the third embodiment. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0314] Conventional work support systems require workers to manually judge the safety and efficiency of their work, which increases the risk of work errors and accidents. Furthermore, there was a lack of effective ways to utilize past accident cases and work data, which led to insufficient work improvements and safety measures. Furthermore, real-time work support was difficult, placing a heavy burden on workers.

[0315] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0316] In this invention, the server includes a means for acquiring past work data and accident cases from a database and performing preprocessing, a means for training an AI model using the preprocessed data, and a means for inputting new work data and using the AI ​​model to determine whether the work is OK or not. This makes it possible to effectively utilize past data and automatically determine the safety and efficiency of work in real time.

[0317] "Information necessary for the work" refers to the data and knowledge necessary to carry out the work, such as procedures, equipment images, and past accident cases.

[0318] "Smart glasses" refers to a wearable device that can be worn by workers to visually display the operation screen and procedures to be performed.

[0319] "Voice input means" refers to technology that allows workers to give operational instructions and input data using voice.

[0320] "Means for determining whether something is OK / NG" refers to the technology used to evaluate the safety and efficiency of work and determine the results as OK or NG.

[0321] A "database" refers to an information management system that systematically stores past work data and accident cases, and allows them to be searched and retrieved as needed.

[0322] "Preprocessing means" refers to technology that processes collected data by completing missing values, normalizing it, extracting features, etc., and converting it into a format suitable for an artificial intelligence model.

[0323] An "artificial intelligence model" refers to a model that uses machine learning algorithms to learn data and perform a specific task (in this case, determining whether a task is OK or not).

[0324] "Training means" refers to techniques for training an artificial intelligence model using preprocessed data to improve its performance.

[0325] "New work data" refers to information about current or future work, and the artificial intelligence model uses this information to determine whether the work is OK or not.

[0326] "Feedback means" refers to technology that notifies users of the results determined by the artificial intelligence model and encourages them to improve their work and take safety measures.

[0327] The present invention is a system for improving work safety and efficiency, and can be implemented as follows.

[0328] First, users use a development environment such as Python or TensorFlow to generate the system's program. The program includes code for building an artificial intelligence (AI) model and learning from past work data and accident cases.

[0329] The server retrieves past work data and accident case studies provided by the user from a database (e.g., MySQL (registered trademark)). Next, the server preprocesses this data and converts it into a format suitable for the AI ​​model. Specifically, the server performs data cleaning, normalization, feature extraction, etc.

[0330] The server then trains the AI ​​model using a machine learning library such as TensorFlow. Once training is complete, the server receives new task data as input and uses the AI ​​model to determine whether the task is OK or not.

[0331] As a concrete example, consider the case of using factory work data. Users input past work data (e.g., work hours, equipment used, years of experience of workers) and accident cases (e.g., type, cause, and impact of accidents) into the system. The server learns from this data, and when new work data is input, it determines whether the work is safe.

[0332] Example prompt sentence:

[0333] "Based on past work data and accident cases, determine whether your current work is safe."

[0334] This system is expected to improve the work efficiency and safety of workers. The flow of the identification process in the third embodiment will be described with reference to FIG.

[0335] Step 1: Collect data

[0336] Users upload past work data and accident cases to the system. Specifically, data is extracted from CSV files and databases. The input information includes work hours, equipment used, years of experience of workers, type of accident, cause, and impact. The output is that this data is saved on the server.

[0337] Step 2: Preprocessing the data

[0338] The server preprocesses the collected data. Specifically, it imputes missing values, normalizes data, and encodes categorical data. The input is the raw data collected in step 1. The output is a preprocessed dataset. For example, it imputes missing values ​​with the mean, standardizes work time, and one-hot encodes the equipment used.

[0339] Step 3: Training the AI ​​model

[0340] The server trains an AI model using the preprocessed data, specifically using a machine learning library such as TensorFlow. The input is the preprocessed dataset from step 2. The output is the trained AI model. Training can take several hours.

[0341] Step 4: Entering work data

[0342] Users input new work data into the system. Specifically, this involves real-time data entry and batch processing. The input includes information such as the start time of the new work, the equipment used, and the worker's years of experience. The output is that this data is stored on the server.

[0343] Step 5: Judging whether the work is OK or NG

[0344] The server passes the new work data that was input to the AI ​​model, which judges whether the work is OK or not. Specifically, the AI ​​model evaluates the safety and efficiency of the work. The input is the new work data that was input in step 4. The output is the OK / NG judgment result for the work. For example, a judgment may be made that "the work time is too long, so it is not OK."

[0345] Step 6: Feedback on results

[0346] The server feeds back the judgment results to the user. Specifically, it displays notifications from the system and a dashboard. The input is the judgment results obtained in step 5. The output is feedback information for the user. The user reviews the work plan based on these results. For example, they can take measures such as "shortening work time" or "changing the equipment used."

[0347] (Application example 3)

[0348] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0349] Conventional work support systems required workers to study procedures, equipment images, and past accident examples in advance, but they did not acquire work data in real time or judge whether a task passed or failed based on past data. This resulted in issues such as insufficient improvement in work efficiency and safety. Furthermore, there was a need to improve the accuracy of judgments when workers input operation screens and procedures to be performed by voice through smart glasses.

[0350] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0351] In this invention, the server includes a means for learning information required for work (procedures, facility images, past accident cases, etc.) in advance, a means for inputting the operation screen and procedures to be performed by voice through the smart glasses when performing work, a means for determining whether the work is OK or not and displaying the answer on the smart glasses, and a means using artificial intelligence for acquiring work data in real time and determining whether the work is OK or not based on past data and accident cases, which makes it possible to improve work efficiency and safety.

[0352] "Information necessary for the work" refers to information such as procedures required to perform the work, equipment images, and past accident cases.

[0353] "Means for learning in advance" refers to means for learning the information necessary for the work in advance.

[0354] "Smart glasses" are eyeglass-type devices that can be worn by workers to display operation screens and procedures to be carried out.

[0355] The "operation screen" is the screen that the worker can view through the smart glasses.

[0356] "Procedures to be implemented" are specific procedures that must be followed when performing a task.

[0357] "Voice input means" refers to a means by which a worker uses voice to input information about the operation screen and procedures to be carried out.

[0358] "Means for determining whether something is OK or not" refers to a means for determining whether the results of the work are appropriate or not.

[0359] The "means for acquiring task data in real time" refers to a means for acquiring data in real time while the task is being performed.

[0360] "Past data" refers to data on work that has been done in the past.

[0361] "Accident cases" are specific examples of accidents that have occurred in the past.

[0362] "Means using artificial intelligence" refers to means for determining whether a task is OK or not using artificial intelligence technology.

[0363] An embodiment of the present invention will be described.

[0364] First, the server has a means of learning the information required for the work (procedures, equipment images, past accident cases, etc.) in advance. This is done using artificial intelligence technology. Specifically, it uses machine learning libraries such as TensorFlow to learn from past work data and accident cases and model work patterns.

[0365] Next, when performing work, the user wears smart glasses. Smart glasses are devices that display operation screens and procedures to be performed, and have a means of inputting instructions by voice using voice recognition technology. When the user inputs operation instructions by voice, the voice data is sent to the server, which then converts it into text data using voice recognition technology.

[0366] The server has a means of acquiring work data in real time. Specifically, it monitors the progress of work in real time through cameras and sensors installed in the smart glasses. This data is sent to the server, which has a means of using artificial intelligence to judge whether the work is OK or not based on past data and accident cases. The AI ​​model is built using machine learning libraries such as TensorFlow, and judges whether the work is OK or not based on the data acquired in real time.

[0367] The results are displayed on the smart glasses, allowing users to check the progress of their work in real time and make corrections as necessary, improving work efficiency and safety.

[0368] As a concrete example, consider a robot assembling parts in a factory. In this case, cameras and sensors installed on the robot collect work data in real time and input that data into an AI model. The AI ​​model judges whether the work is OK or not based on past data and accident cases, and displays the result in real time on the smart glasses.

[0369] Examples of prompts to be input to a generative AI model include:

[0370] "Robots are assembling parts in a factory. Please build an AI system that will judge whether a task is OK or not based on real-time work data and past data and accident cases."

[0371] The above is an embodiment of the present invention.

[0372] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0373] Step 1:

[0374] The server learns in advance the information required for the work (procedures, equipment images, past accident cases, etc.). Specifically, it uses machine learning libraries such as TensorFlow to learn from past work data and accident cases and model work patterns. The input is past work data and accident cases, and the output is the trained AI model.

[0375] Step 2:

[0376] The user puts on the smart glasses and begins work. The smart glasses are a device that displays the operation screen and procedures to be performed. The input is the user's voice instructions, and the output is the operation screen and procedures displayed on the smart glasses.

[0377] Step 3:

[0378] The user inputs operation instructions by voice. The smart glasses use voice recognition technology to convert the voice into text data and send that data to the server. The input is the user's voice instructions, and the output is text data.

[0379] Step 4:

[0380] The server acquires work data in real time. The progress of work is monitored in real time through the cameras and sensors installed in the smart glasses. The input is real-time data from the cameras and sensors, and the output is work data sent to the server.

[0381] Step 5:

[0382] The server inputs the work data acquired in real time into the AI ​​model and judges whether the work is OK or not. The AI ​​model judges whether the work is OK or not based on past data and accident cases. The input is the work data acquired in real time, and the output is the OK or not judgment result of the work.

[0383] Step 6:

[0384] The server sends the judgment result to the smart glasses and displays it to the user. This allows the user to check the progress of the work in real time and make corrections as necessary. The input is the OK / NG judgment result of the work, and the output is the judgment result displayed on the smart glasses.

[0385] The above are the specific processing steps for carrying out the present invention.

[0386] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0387] "Example 1"

[0388] One embodiment of the present invention is a system that combines an emotion engine that recognizes the emotions of workers. In this system, a worker wears smart glasses and, as he or she works, the emotion engine recognizes the worker's emotions from the worker's tone of voice and facial expressions. For example, if the system senses that the worker is nervous, it can adjust work instructions according to the worker's emotions, such as slowing down the pace of work.

[0389] "Example 2"

[0390] The emotion engine can also take measures such as pausing work if a worker's emotions exceed a certain threshold. For example, if the system determines that a worker is feeling excessively stressed, it can pause work and instruct the worker to take a break.

[0391] Cut.

[0392] "Example 3"

[0393] Furthermore, the emotion engine can record changes in a worker's emotions and analyze the data to identify areas for improvement in the work. For example, if a worker tends to feel stressed during a particular task, it can suggest improvement measures such as reviewing the procedures for that task.

[0394] The processing flow of each embodiment will be described below.

[0395] "Example 1"

[0396] Step 1: The worker puts on the smart glasses.

[0397] Step 2: When you start working, the emotion engine recognizes your emotions from your tone of voice and facial expressions.

[0398] Step 3: The emotion engine adjusts the work instructions according to the worker's emotions. For example, if the worker feels nervous, the system will adjust the work pace, such as slowing it down. "Example 2"

[0399] Step 1: The worker puts on the smart glasses and begins work.

[0400] Step 2: The emotion engine recognizes the worker's emotions and pauses the work if the emotions exceed a certain threshold.

[0401] Step 3: After pausing work, the system prompts the worker to take a break.

[0402] "Example 3"

[0403] Step 1: The worker puts on the smart glasses and begins work.

[0404] Step 2: The emotion engine records changes in the worker's emotions.

[0405] Step 3: The system analyzes the recorded emotional data and identifies areas for improvement in the work. For example, if a worker tends to feel stressed during a particular task, it suggests improvements such as reviewing the procedures for that task.

[0406] Example 1

[0407] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0408] Conventional work support systems allow workers to learn information such as work procedures, equipment images, and past accident cases in advance, but they are unable to respond to emotional changes during work. As a result, it is difficult to provide appropriate instructions when workers feel nervous or stressed, which can lead to reduced work efficiency and safety.

[0409] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0410] In this invention, the server includes a means for learning information required for work (procedures, facility images, past accident cases, etc.) in advance, a means for inputting the operation screen and procedures to be performed by voice through the smart glasses when performing work, a means for determining whether the work is OK or NG and displaying the answer on the smart glasses, and a means for recognizing the worker's emotions and adjusting work instructions according to those emotions. This makes it possible to respond to changes in the worker's emotions and provide appropriate instructions.

[0411] "Information necessary for work" is a general term for data and materials necessary to carry out work, such as work procedures, equipment images, and past accident cases.

[0412] "Smart glasses" are a type of wearable device that has a built-in display, camera, microphone, etc. and is worn by workers.

[0413] The "operation screen" is an interface that is displayed on the smart glasses display and contains information such as work procedures and instructions.

[0414] "Voice input means" refers to a technique that allows a worker to input instructions or information through voice, and typically uses voice recognition technology.

[0415] The "means for determining whether something is OK or not" refers to an algorithm or program that analyzes the worker's voice input and the work content, and determines whether the results are correct.

[0416] "Means for recognizing emotions" refers to technology that analyzes the tone of a worker's voice and facial expressions to identify emotions such as tension or stress.

[0417] The "means for adjusting work instructions" refers to an algorithm or program for changing the pace of work or the content of instructions in response to the worker's recognized emotions.

[0418] MODE FOR CARRYING OUT THE INVENTION

[0419] This invention is a system in which workers wear smart glasses and learn information such as work procedures, facility images, and past accident cases in advance. Workers can check the operation screen through the smart glasses and input instructions by voice. In addition, by combining it with an emotion engine, it has the function of recognizing the worker's emotions and adjusting work instructions.

[0420] Hardware and software used

[0421] Server: A relational database such as MySQL is used to store information such as work procedures, equipment images, and past accident cases.

[0422] Device (smart glasses): Use wearable devices such as Google Glass and Microsoft(R) HoloLens(R).

[0423] Speech recognition software: Converts speech to text using a predefined API.

[0424] Emotion recognition software: Uses a system that combines Microsoft Azure (registered trademark) Emotion API, OpenCV (registered trademark), and deep learning models.

[0425] Specific operation of the system

[0426] 1. The server stores information such as work procedures, equipment images, and past accident cases in a database. For example, text data of work procedures and image files of equipment are stored in the database.

[0427] 2. When a worker wears the device (smart glasses), it connects to the server via Wi-Fi and requests the necessary information. In response to the request, the server sends work procedures and equipment images to the device. The device then displays the received information on its screen.

[0428] 3. The user (worker) checks the information displayed through the smart glasses and inputs voice instructions such as "Tell me the next step." The smart glasses capture the voice using a built-in microphone and convert the voice into text using a specified API.

[0429] 4. The server receives the text data of the voice instructions sent from the device and analyzes it using a script written in Python. For example, it determines whether the instruction, "Tell me the next step," is correct and generates a pass / fail result.

[0430] 5. The device displays the result of the judgment received from the server on the display, for example, a message such as "The next step is OK."

[0431] 6. The emotion engine analyzes the user's tone of voice and facial expressions. The smart glasses use a built-in camera and microphone to capture the user's facial expressions and tone of voice, and then use Microsoft Azure's Emotion API to recognize emotions.

[0432] 7. The server adjusts the work instructions based on the emotion data received from the emotion engine. For example, if the server detects that the user is nervous, it will send an additional instruction such as "Please proceed slowly" to the device. The device will then display this instruction on its screen.

[0433] Specific examples

[0434] For example, consider the case where a user wears smart glasses and is inspecting equipment. The user inputs a command by voice, such as "Tell me the next step." The server analyzes this command and displays the next step on the smart glasses' display. At the same time, if the emotion engine detects tension in the user's voice tone, the server will display an additional command such as "Please proceed slowly."

[0435] Prompt Sentence Examples

[0436] Examples of prompts to be input to a generative AI model include:

[0437] "A worker is wearing smart glasses and inspecting equipment. The worker inputs instructions for the next step by voice. The server analyzes this voice instruction and displays the next step on the smart glasses' display. At the same time, if the emotion engine detects tension in the worker's tone of voice, the server displays an additional instruction to slow down the pace of the work."

[0438] By inputting this prompt into a generative AI model, the system's behavior can be simulated.

[0439] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0440] Step 1:

[0441] The server stores the information in a database.

[0442] Input: Information such as work procedures, equipment images, and past accident cases.

[0443] Processing: The server stores this information in a relational database such as MySQL. Specifically, it stores the text data of work procedures and image files of the equipment in the database.

[0444] Output: Information stored in a database.

[0445] Step 2:

[0446] The terminal retrieves the information from the server and displays it on the screen.

[0447] Input: Information such as work procedures and equipment images stored on the server.

[0448] Processing: When the user wears the device (smart glasses), it connects to the server via Wi-Fi and requests the necessary information. The server then sends the requested information, such as work procedures and facility images, to the device.

[0449] Output: Work procedures and equipment images displayed on the terminal screen.

[0450] Step 3:

[0451] The user inputs instructions by voice.

[0452] Input: User's spoken command (e.g., "What's the next step?").

[0453] Processing: The smart glasses use the built-in microphone to capture audio and convert it to text using a predefined API.

[0454] Output: Spoken instructions converted to text.

[0455] Step 4:

[0456] The server analyzes the voice instructions and determines whether the task is OK or not.

[0457] Input: Spoken instructions converted to text.

[0458] Processing: The server receives the text data of the voice instructions sent from the device and analyzes it using a script written in Python. For example, it determines whether the instruction "tell me the next step" is correct and generates a OK / NG result.

[0459] Output: OK / NG result.

[0460] Step 5:

[0461] The terminal displays the judgment result on the display.

[0462] Input: OK / NG result received from the server.

[0463] Processing: The device displays the result of the judgment received from the server on the display. For example, it displays a message such as "The next step is OK."

[0464] Output: The result displayed on the device screen.

[0465] Step 6:

[0466] The emotion engine recognizes the user's emotions.

[0467] Input: The user's tone of voice and facial expressions.

[0468] Processing: The smart glasses use a built-in camera and microphone to capture the user's facial expressions and tone of voice, and then use Microsoft Azure's Emotion API to recognize emotions.

[0469] Output: Recognized emotion data.

[0470] Step 7:

[0471] The server adjusts the work instructions according to the emotion.

[0472] Input: Emotion data received from the emotion engine.

[0473] Processing: The server adjusts the work instructions based on the emotion data received from the emotion engine. For example, if the server detects that the user is nervous, it will send an additional instruction such as "Please proceed slowly" to the device.

[0474] Output: Adjusted work instructions displayed on the terminal display.

[0475] (Application example 1)

[0476] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0477] Conventional work support systems allow workers to learn information such as work procedures, equipment images, and past accident cases in advance, but lack support that takes into account the worker's emotional state while working. As a result, workers may not be able to respond appropriately if they feel nervous or stressed, which can lead to reduced work efficiency and safety. In addition, the OK / NG judgment of operations based on voice input does not take into account the worker's emotional state, which can increase the psychological burden on the worker.

[0478] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means. In this invention, the server includes a means for learning information required for the work (procedures, equipment images, past accident cases, etc.) in advance, a means for inputting the operation screen and procedures to be performed by voice through the smart glasses when performing the work, a means for determining whether the work is OK / NG and displaying the answer on the smart glasses, an emotion recognition means for recognizing the emotion of the worker, and a means for adjusting the pace of the work and instructions according to the emotion of the worker. This enables appropriate support that takes into account the emotional state of the worker, and is expected to improve work efficiency and safety.

[0479] "Information necessary for the work" refers to information necessary for the smooth and safe progress of work, such as work procedures, equipment images, and past accident cases.

[0480] "Means for learning in advance" refers to means by which workers can learn the information necessary for their work in advance, and includes learning systems that use artificial intelligence.

[0481] "Smart glasses" are wearable devices that can be worn by workers to visually display operation screens and procedures to be carried out.

[0482] The "means for inputting operation screens and procedures to be performed by voice" refers to a means for an operator to input operation screens and procedures to be performed by voice, and voice recognition technology can be used.

[0483] The "means for determining whether a task is OK / NG and displaying the answer on the smart glasses" is a means for determining whether a task is OK / NG based on the worker's voice input and displaying the result on the smart glasses display.

[0484] The "emotion recognition means" is a means for recognizing emotions from the tone of a worker's voice and facial expressions, and an emotion recognition engine can be used.

[0485] "Means for adjusting the pace of work and instructions" refers to means for adjusting the pace of work and instructions according to the worker's emotions.

[0486] As an embodiment of the present invention, an application installed on a factory robot is used. A specific embodiment of this application will be described below.

[0487] Hardware and software used

[0488] Hardware: smart glasses (e.g., wearable devices), factory robots (e.g., robotic arms)

[0489] Software: emotion recognition engine (e.g. emotion recognition software), voice recognition engine (e.g. voice recognition software), display software (e.g. display software)

[0490] Data processing and calculation

[0491] 1. Handling voice input:

[0492] The user inputs voice commands into the smart glasses.

[0493] Use a speech recognition engine to convert speech to text.

[0494] The converted text is analyzed and the operation is judged to be OK or not.

[0495] 2. Emotion recognition processing:

[0496] The smart glasses' camera and microphone are used to capture the user's facial expressions and tone of voice.

[0497] An emotion recognition engine is used to recognize the user's emotions.

[0498] Adjust work pace or instructions based on perceived emotions.

[0499] 3. Display processing:

[0500] The OK / NG results of the operation and adjusted instructions are displayed on the smart glasses display.

[0501] Use display software to present information in a visually understandable format.

[0502] Specific examples

[0503] For example, when a user performs maintenance on a robot arm, they wear smart glasses and perform the following operations.

[0504] 1. The user says "What's the next step?"

[0505] 2. The speech recognition engine converts the speech into text and displays the next steps.

[0506] 3. If the user is nervous, the emotion recognition engine will detect this and adjust instructions to slow down the pace of the task.

[0507] 4. The smart glasses display will say, "Please proceed slowly with the next step."

[0508] Prompt Sentence Examples

[0509] An example of a prompt sentence to input to the generative AI model is as follows:

[0510] "Design a smart glasses application to teach maintenance procedures to factory robots. The application should include a function where the user can input instructions by voice, and an emotion recognition engine can be used to recognize the user's emotions and adjust the pace of work and instructions accordingly. The hardware used will be a wearable device and a robotic arm, and the software will be speech recognition software, emotion recognition software, and display software."

[0511] In this way, smart glasses applications can be realized for efficient and safe operation and maintenance of factory robots.

[0512] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0513] Step 1:

[0514] The user inputs voice commands into the smart glasses, which are then captured by the smart glasses' microphone.

[0515] Step 2:

[0516] The server uses a speech recognition engine to convert the captured speech into text: the input is speech data and the output is text data.

[0517] Step 3:

[0518] The server analyzes the converted text and determines whether the operation is OK or NG. The input is text data, and the output is the OK / NG result.

[0519] Step 4:

[0520] The server uses the camera and microphone of the smart glasses to capture the user's facial expressions and tone of voice. The input is the camera image and audio data.

[0521] Step 5:

[0522] The server uses an emotion recognition engine to recognize the user's emotions from the captured facial expressions and tone of voice. The input is camera footage and audio data, and the output is emotion data.

[0523] Step 6:

[0524] The server adjusts the pace of work and instructions based on the recognized emotions. The input is emotion data, and the output is adjusted instructions.

[0525] Step 7:

[0526] The server displays the OK / NG result of the operation and adjusted instructions on the smart glasses display. The input is the OK / NG judgment result and adjusted instructions, and the output is the display.

[0527] Step 8:

[0528] The user checks the information displayed on the smart glasses display and proceeds to the next step. The input is the display, and the output is the user's next action.

[0529] Example 2

[0530] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0531] In conventional maintenance work, workers are required to accurately understand procedures and carry them out appropriately, but procedural errors and worker stress can lead to reduced work efficiency.In addition, there was no system that monitored workers' emotional state in real time and instructed them to take breaks at the appropriate time, so the health and safety of workers was not adequately ensured.

[0532] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0533] In this invention, the server includes a means for learning information required for work (procedures, facility images, past accident cases, etc.) in advance, a means for inputting the operation screen and procedures to be performed by voice through the smart glasses when performing work, a means for determining whether the work is OK or NG and displaying the answer on the smart glasses, and a means for monitoring the user's emotional state and pausing the work and instructing them to take a break if it exceeds a certain threshold. This allows the worker to proceed with the work while checking the accuracy of the procedures, and also allows them to take an appropriate break if they feel excessive stress.

[0534] "Information necessary for the work" refers to the data and knowledge necessary to carry out maintenance work accurately and efficiently, such as procedures, equipment images, and past accident cases.

[0535] "Smart glasses" are wearable devices equipped with a display and voice input functions, allowing workers to check and input information hands-free.

[0536] "Voice input means" refers to technology or devices that allow workers to use voice to input information on the operation screen or procedures to be performed, and includes voice recognition technology.

[0537] "Means for determining whether something is OK / NG" refers to the technology or device that evaluates whether the entered procedures or operations are correct and outputs the results.

[0538] "Means for monitoring emotional state" refers to technologies and devices that measure and assess workers' stress levels and fatigue in real time.

[0539] "Means to temporarily suspend work and instruct workers to take a break when a certain threshold is exceeded" refers to technology or devices that suspend work and encourage workers to take a break when the evaluation result of their emotional state exceeds a set standard.

[0540] "Artificial intelligence" refers to technologies and systems that use machine learning and data analysis to automatically learn the information necessary for work and make decisions.

[0541] "Speech recognition technology" refers to technology or systems for converting speech into text data, enabling voice input.

[0542] This invention is a system that uses smart glasses to input procedures by voice when a worker performs maintenance work, judges the accuracy of the procedures, and monitors the worker's emotional state. This system is implemented using the following hardware and software.

[0543] Hardware and software used

[0544] Hardware: Smart glasses

[0545] Software: Speech recognition system, emotion engine, generative AI model

[0546] Specific operation of the system

[0547] 1. The user puts on the smart glasses. The smart glasses start up and connect to the system.

[0548] 2. The user speaks the maintenance procedure, for example, "Change the engine oil."

[0549] 3. The device (smart glasses) activates the voice recognition system and converts the input voice into text data. Specifically, voice recognition software (e.g., a specified API) is used.

[0550] 4. The server receives the text data and uses a generative AI model to determine whether the input procedure is correct. For example, a generative AI model (e.g., GPT-4) evaluates whether the procedure "change the engine oil" is correct.

[0551] 5. The server sends the result to the smart glasses display. The device (smart glasses) displays the result on the display. For example, the result "Correct" is displayed.

[0552] 6. The device (smart glasses) uses built-in sensors to monitor the user's emotional state. The emotion engine evaluates the user's stress level and fatigue level.

[0553] 7. The server receives the emotion engine's evaluation results, and if the user's emotion exceeds a certain threshold, the work is paused. The device (smart glasses) displays a message on the display saying, "Please take a break." For example, if the system determines that the user is feeling excessively stressed, it will display the message, "Please take a break."

[0554] Specific examples

[0555] Example 1: Entering maintenance procedure

[0556] The user speaks, "Change the engine oil."

[0557] The smart glasses' voice recognition system converts this into text data: "Change the engine oil."

[0558] The server uses a generative AI model to determine whether this procedure is correct and determines that it is.

[0559] The server displays the "correct" result on the smart glasses display.

[0560] The user checks the display and continues working.

[0561] Example 2: Emotion engine stalls work

[0562] The emotion engine determines that the user is feeling excessive stress while working.

[0563] The emotion engine pauses the work and displays the message "Take a break" on the smartglasses display.

[0564] The user follows the instructions and takes a break.

[0565] Prompt Sentence Examples

[0566] "You can input the steps for changing your engine oil by voice. The system will assess the accuracy of the steps and show the results on the display. If you feel excessive stress while working, the system will pause your work and advise you to take a break."

[0567] In this way, users can work with peace of mind while checking whether their work is correct. If users feel excessive stress, the system will encourage them to take an appropriate break, thereby improving the safety and efficiency of their work.

[0568] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0569] Step 1:

[0570] The user puts on the smart glasses, which then start up and connect to the system.

[0571] Input: Wearing smart glasses

[0572] Output: System connection completed message

[0573] Specific operation: The smart glasses start up and connect to the server via Wi-Fi or Bluetooth. The message "System connection complete" appears on the display.

[0574] Step 2:

[0575] The user inputs the maintenance procedure by voice, for example, "Change the engine oil."

[0576] Input: Voice input (e.g. "Change engine oil")

[0577] Output: Audio data

[0578] How it works: The microphone in the smart glasses captures sound and stores it as audio data.

[0579] Step 3:

[0580] The device (smart glasses) activates a voice recognition system and converts the input voice into text data.

[0581] Input: Audio data

[0582] Output: Text data (e.g. "Change the engine oil")

[0583] What happens: Speech recognition software (e.g., a predefined API) analyzes the audio data and generates corresponding text data.

[0584] Step 4:

[0585] The server receives the text data and uses a generative AI model to determine whether the input steps are correct.

[0586] Input: Text data (e.g., "Change the engine oil")

[0587] Output: Verification result (e.g. "correct")

[0588] What it does: The server uses a generative AI model (e.g., GPT-4) to analyze the content of the text data and evaluate the accuracy of the steps.

[0589] Step 5:

[0590] The server sends the judgment result to the smart glasses display, and the device (smart glasses) displays the judgment result on the display.

[0591] Input: Verification result (e.g. "correct")

[0592] Output: Display (e.g. "Correct")

[0593] Specific operation: The server sends the judgment result to the smart glasses, which then display the result on the screen.

[0594] Step 6:

[0595] The device (smart glasses) uses built-in sensors to monitor the user's emotional state.

[0596] Input: Biometric data (e.g., heart rate, galvanic skin response)

[0597] Output: Evaluation of emotional state (e.g., stress level)

[0598] How it works: The smart glasses' sensors collect the user's biometric data, and the emotion engine analyzes the data to assess their emotional state.

[0599] Step 7:

[0600] The server receives the evaluation results of the emotion engine, and if the user's emotion exceeds a certain threshold, the work is paused. The device (smart glasses) displays a message on the display saying, "Please take a break."

[0601] Input: Emotional state assessment (e.g., high stress)

[0602] Output: Break prompt (e.g., "Please take a break")

[0603] Specific operation: The server receives the evaluation results of the emotion engine, and if it determines that the stress level is high, it sends a message to the smart glasses instructing them to take a break and displays it on the display.

[0604] (Application example 2)

[0605] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0606] In conventional maintenance work, workers are required to accurately understand procedures and execute them appropriately, but accidents and mistakes can occur due to procedural errors or worker stress. Furthermore, there is a lack of a system to monitor workers' emotional state in real time and encourage them to take breaks at appropriate times, which leads to problems with reduced work efficiency and safety. To solve these issues, a system is needed to check the accuracy of work procedures and monitor workers' emotional state.

[0607] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0608] In this invention, the server includes a means for learning information required for work (procedures, facility images, past accident cases, etc.) in advance, a means for inputting the operation screen and procedures to be performed by voice through the smart glasses when performing work, a means for determining whether the work is OK or NG and displaying the answer on the smart glasses, a means for monitoring the emotional state of the worker and pausing the work if excessive stress is felt, and a means for instructing the worker to take a break. This allows the work to be carried out while confirming the accuracy of the work procedures, and makes it possible to monitor the emotional state of the worker in real time and encourage breaks at appropriate times.

[0609] "Information necessary for work" refers to information such as procedures, equipment images, and past accident cases that are necessary when performing maintenance work.

[0610] "Means of learning" refers to a means for learning the information necessary for the work in advance, and is carried out using artificial intelligence.

[0611] "Voice input means" refers to a means by which workers can input operation screens and procedures to be performed by voice through smart glasses.

[0612] The "means for determining whether the work procedure entered is correct" is a means for determining whether the procedure is correct or not and displaying the results on the smart glasses.

[0613] The "means for monitoring emotional state" is a means for monitoring the emotional state of the worker in real time and for temporarily suspending work if excessive stress is felt.

[0614] The "means for instructing a worker to take a break" refers to a means for instructing a worker to temporarily stop work and take a break when the worker feels excessive stress.

[0615] A system for implementing this invention has the following configuration: First, a worker wears smart glasses and performs maintenance work. The smart glasses are equipped with a microphone and a camera, allowing audio and video input.

[0616] Hardware and software used

[0617] Hardware: smart glasses (e.g., Google Glass), microphone, camera

[0618] Software: Predefined API, Microsoft Azure Emotion API

[0619] Data processing and calculation

[0620] Voice input and recognition

[0621] 1. Voice input: The worker inputs maintenance procedures by voice through the microphone in the smart glasses.

[0622] 2. Speech Recognition: The server uses a predefined API to convert speech into text, which is used to verify the accuracy of the work instructions.

[0623] Procedure Judgment

[0624] 3. Procedure determination: The server determines whether the converted text is a correct procedure by comparing it with a pre-trained database of work procedures.

[0625] Emotional state monitoring

[0626] 4. Emotion analysis: The smart glasses capture facial images of workers and use the Microsoft Azure Emotion API to analyze their emotional state, focusing in particular on when their stress level exceeds a certain threshold.

[0627] Results display and break instructions

[0628] 5. Display of results: The judgement result and emotional state are displayed on the smart glasses display. If the procedure is correct, the message "Correct procedure" is displayed, and if it is incorrect, the message "Incorrect procedure" is displayed.

[0629] 6. Break instruction: If a worker feels excessive stress, the system will pause the work and instruct the worker to take a break.

[0630] Specific examples

[0631] Voice input: "Next, remove the screws."

[0632] Procedure result: "This is the correct procedure."

[0633] Sentiment analysis result: "The worker is under too much stress. Please ask them to pause their work and take a break."

[0634] Prompt Sentence Examples

[0635] Voice input: "Next, remove the screws."

[0636] Procedure result: "This is the correct procedure."

[0637] Sentiment analysis result: "The worker is under too much stress. Please ask them to pause their work and take a break."

[0638] In this way, the accuracy of work procedures can be confirmed while work is being carried out, and the emotional state of workers can be monitored in real time to encourage them to take breaks at appropriate times, thereby improving work efficiency and safety.

[0639] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0640] Step 1:

[0641] The user puts on the smart glasses and starts the maintenance work. The user inputs the work procedure by voice through the microphone of the smart glasses. The input voice data is captured by the microphone of the smart glasses.

[0642] Step 2:

[0643] The server receives the voice data sent from the smart glasses. It sends the received voice data to a specified API and converts the voice into text data. The input is voice data and the output is text data.

[0644] Step 3:

[0645] The server compares the converted text data with a pre-trained database of work procedures. Based on the comparison results, it determines whether the input procedure is correct. The input is text data, and the output is a judgment result regarding the accuracy of the procedure.

[0646] Step 4:

[0647] The server receives the user's facial image captured through the smart glasses camera. The received facial image is sent to the Microsoft Azure Emotion API to analyze the user's emotional state. The input is facial image data, and the output is the analysis result of the emotional state.

[0648] Step 5:

[0649] The server integrates the procedure judgment results and the emotional state analysis results and displays them on the smart glasses' display. If the procedure is correct, it displays "This is the correct procedure," and if it is incorrect, it displays "There is an error in the procedure." If the user is feeling excessive stress, it displays "Please pause the work and ask the user to take a break." The input is the judgment results and analysis results, and the output is the message displayed on the display.

[0650] Step 6:

[0651] The user chooses whether to continue working or take a break according to the message displayed on the smart glasses display. If the user chooses to take a break, the system pauses the work and prompts the user to take a break. The input is the message displayed on the display, and the output is the user's behavior.

[0652] Example 3

[0653] Next, a description will be given of a third embodiment of the third embodiment. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0654] Conventional work support systems have the ability to assess the accuracy of a worker's actions and procedures, but they are unable to provide feedback that takes into account the worker's emotions and stress levels. This can increase the worker's mental burden and reduce work efficiency and safety. Furthermore, the lack of real-time pass / fail judgments and the collection and analysis of emotional data makes it difficult to quickly identify areas for improvement.

[0655] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0656] In this invention, the server includes a means for learning information required for work (procedures, facility images, past accident cases, etc.) in advance, a means for inputting the operation screen and procedures to be performed by voice through a visual device when performing work, a means for determining whether the work is OK or NG and displaying the answer on the visual device, a means for collecting and analyzing emotional data of the worker, and a means for suggesting improvements to the work based on the collected emotional data. This makes it possible to provide feedback that takes into account the worker's emotions and stress level, thereby improving work efficiency and safety.

[0657] "Information necessary for the work" refers to data necessary to carry out the work accurately and safely, such as procedures, equipment images, and past accident cases.

[0658] A "visual device" is a device worn by a worker that allows them to visually check the operation screen and the procedures that need to be carried out, and includes smart glasses and the like.

[0659] "Voice input means" refers to a technology that allows a worker to input information about an operation screen or procedures to be performed using voice, and includes voice recognition technology.

[0660] "OK / NG judgment means" is a technology for evaluating the accuracy and appropriateness of work and providing feedback to the worker on the results.

[0661] "Emotion data collection means" is a technology for detecting the emotions and stress levels of workers and collecting that data.

[0662] "Emotion data analysis means" is a technology for analyzing collected emotion data and evaluating changes in the emotions and stress levels of workers.

[0663] The "means for suggesting work improvements" is a technology for proposing improvements to work procedures and environments based on the results of analyzing emotional data.

[0664] This invention is a system for improving the work efficiency and safety of workers, and is implemented by clearly defining the roles of the server, terminal, and user.

[0665] Server Roles

[0666] The server provides a means for advance learning of the information required for the work (procedures, equipment images, past accident cases, etc.). Specifically, the server uses a database management system such as MySQL to collect and store past work data and accident cases. Next, it trains an AI model based on the collected data. A machine learning framework such as TensorFlow is used for the AI ​​model.

[0667] The server does the following:

[0668] 1. Obtain past work data and accident cases from the database.

[0669] 2. Preprocess the acquired data and convert it into a format suitable for the AI ​​model.

[0670] 3. Train the AI ​​model using the preprocessed data.

[0671] 4. Save the trained AI model and use it to judge whether the work is OK or not.

[0672] Device Role

[0673] The terminal collects data on the work performed by the worker in real time and sends it to a server. The terminal is equipped with hardware such as a camera and sensors, which are used to collect work data. The collected data is temporarily stored in the terminal and periodically uploaded to the server.

[0674] The terminal does the following:

[0675] 1. Collect work data using cameras and sensors.

[0676] 2. Temporarily store the collected data.

[0677] 3. The stored data is periodically sent to the server.

[0678] User Roles

[0679] Users can receive feedback from the system and use it to improve their work. For example, if the system judges a particular task to be unsuccessful, the user can check the reason and review the work procedure. Furthermore, if the emotion engine detects stress in the worker, the user can use that information to improve the work environment.

[0680] The user does the following:

[0681] 1. Check the feedback from the system.

[0682] 2. Review work procedures and environments based on feedback.

[0683] 3. Implement the improvement measures and receive feedback from the system again.

[0684] Specific examples

[0685] For example, consider a factory worker assembling parts. The server trains an AI model based on past assembly work data and accident cases, while the device collects data by capturing images of the worker's movements with a camera. The AI ​​model judges in real time whether the work is OK or not, and if it is not OK, notifies the user of the reason. The emotion engine analyzes the worker's facial expressions and provides information to the user if the worker is feeling stressed.

[0686] Prompt Sentence Examples

[0687] "Please design a system that trains an AI model based on past work data and accident cases, and judges whether a task is OK or not in real time. Also, please add a function that analyzes the emotions of workers and provides information if they are feeling stressed."

[0688] In this way, the roles of the server, terminal, and user are clarified, and the flow of data processing using specific hardware and software is explained, making it easier to understand the overall system. The flow of specific processing in the third embodiment will be explained with reference to FIG.

[0689] Step 1:

[0690] Data collection

[0691] The server collects past work data and accident cases. The input is data from each work station in the factory. This includes worker activity logs, sensor data from the work environment, and past accident reports. The server stores this data in a database such as MySQL. Specifically, the server connects to the database and saves the collected data.

[0692] Step 2:

[0693] Data Preprocessing

[0694] The server preprocesses the collected data. The input is the raw data collected in step 1. Specific data processing includes filling in missing values, removing outliers, and normalizing the data. The output is a dataset in a format suitable for AI models. Specifically, the server cleans the data using Python libraries and saves the preprocessed data in a new table.

[0695] Step 3:

[0696] Training an AI model

[0697] The server trains the AI ​​model using the preprocessed data. The input is the dataset preprocessed in step 2. The specific data processing involves building and training a model using a machine learning framework such as TensorFlow. The output is a trained AI model. Specifically, the server inputs the training data into TensorFlow, trains the model, and saves the trained model.

[0698] Step 4:

[0699] Real-time data collection and transmission

[0700] The terminal collects data on the work performed by the worker in real time and sends it to the server. Inputs include data on the worker's movements and the work environment. Specifically, data is collected using cameras and sensors and temporarily stored. Outputs include sending the collected data to the server. Specifically, the terminal uses cameras to capture the movements of the worker and sensors to collect data on the work environment, which is then periodically sent to the server.

[0701] Step 5:

[0702] Judging whether work is OK or NG

[0703] The server inputs the data received in real time into the AI ​​model and judges whether the work is OK or NG. The input is the real-time data collected in step 4. The specific data calculation involves inputting the data into the AI ​​model and obtaining a judgment result. The output is an OK / NG judgment result. The specific operation is that the server inputs the received data into the AI ​​model, analyzes the model's output result, judges whether the work is OK or NG, and sends the judgment result to the terminal.

[0704] Step 6:

[0705] Collecting and analyzing emotional data

[0706] The terminal collects the worker's emotional data and sends it to the server. Inputs include the worker's facial expressions and voice data. Specifically, data is collected using a camera and microphone, and then analyzed using an emotion engine. The analyzed emotional data is sent to the server as output. Specifically, the terminal takes pictures of the worker's facial expressions with a camera, records the worker's voice with a microphone, analyzes the data using an emotion engine, detects changes in emotion, and sends the data to the server.

[0707] Step 7:

[0708] Providing feedback

[0709] The user receives feedback provided by the system and uses it to improve their work. The inputs are the OK / NG judgment result from step 5 and the emotional data from step 6. Specific feedback includes suggestions for reviewing work procedures and improving the environment. The output is the implementation of improvement measures, and the system provides feedback again. The specific operation is for the user to check the feedback on the device screen, review the work procedures and environment based on the feedback, implement improvement measures, and receive feedback from the system again.

[0710] (Application example 3)

[0711] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0712] Conventional work support systems lacked sufficient accuracy in determining whether a worker was performing a task efficiently or safely, and did not provide suggestions for improvement that took into account the worker's emotions and stress. This resulted in the accumulation of stress among workers and a decline in work efficiency.

[0713] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means. In this invention, the server includes a means for learning information required for work (procedures, equipment images, past accident cases, etc.) in advance, a means for inputting the operation screen and procedures to be performed by voice through the smart glasses when performing the work, a means for determining whether the work is OK / NG and displaying the answer on the smart glasses, a means for collecting emotional data of the worker and recording and analyzing changes in emotion, and a means for suggesting improvements to the work procedures based on the emotional data and work data. This not only improves the efficiency and safety of work, but also reduces worker stress and makes it possible to improve work procedures.

[0714] "Information necessary for the work" refers to the data and knowledge necessary to carry out the work, such as work procedures, equipment images, and past accident cases.

[0715] "Means for learning" refers to a method or device for using artificial intelligence to learn in advance the information necessary for a task.

[0716] "Smart glasses" refers to eyeglass-type devices that can be worn by workers to visually display operation screens and procedures to be carried out.

[0717] "Voice input means" refers to a method or device that allows a worker to input information about the operation screen or procedures to be performed using voice.

[0718] "Means for determining whether a task is OK / NG" refers to a method or device for determining the appropriateness of the task and displaying the results on the smart glasses.

[0719] "Emotional data" refers to data that indicates the emotional state of a worker, such as the worker's heart rate or galvanic skin response.

[0720] "Means for recording and analyzing changes in emotions" refers to methods and devices for collecting emotional data from workers, recording changes in those data, and analyzing them.

[0721] The "means for proposing improvements to work procedures" refers to a method or device for finding and proposing improvements to work procedures based on emotion data and work data.

[0722] A system for implementing this invention has the following configuration: First, the server uses artificial intelligence (AI) to learn in advance the information required for the work (procedures, equipment images, past accident cases, etc.). Specifically, it learns patterns based on past work data and accident cases, and builds a model for determining whether the work is OK or not.

[0723] Next, smart glasses are used as the terminal. When worn by a worker, the smart glasses visually display the operation screen and the procedures to be performed. Using voice recognition technology, the worker can input the operation screen and the procedures to be performed by voice. This allows the worker to operate the system without using their hands, improving work efficiency.

[0724] In addition, the server collects the worker's emotional data, recording and analyzing changes in their emotions. Emotional data includes heart rate and galvanic skin response. This data is collected in real time using an emotion engine. The emotion engine analyzes the worker's emotional state and issues an alert if stress levels are increasing.

[0725] Finally, the server proposes improvements to the work process based on the emotion data and work data, thereby reducing worker stress and improving the efficiency of the work process.

[0726] As a concrete example, consider the case of assembling parts in a factory. The server learns from past assembly work data and accident cases, and builds a model to determine whether a task is OK or not. The worker wears smart glasses and inputs operation procedures by voice. The smart glasses visually display the operation screen and procedures, allowing the worker to perform operations without using their hands. The emotion engine collects the worker's heart rate and galvanic skin response in real time, and issues an alert if stress levels are rising. The server suggests improvements to the work procedures based on the emotion data and work data.

[0727] An example of a prompt sentence might be:

[0728] "Create a program that takes current task data and emotion data as input, judges whether the task is OK or NG, and issues an alert if necessary. Task data includes location data and speed data, and emotion data includes heart rate and galvanic skin response. The AI ​​model will use a random forest classifier, and the emotion engine will use the EmotionEngine class."

[0729] In this way, the mode for carrying out the invention can be concretely shown.

[0730] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0731] Step 1:

[0732] The server collects the information necessary for the work (procedures, equipment images, past accident cases, etc.) and uses artificial intelligence (AI) to learn from it. Specifically, it uses past work data and accident cases as input to learn patterns and builds a model that determines whether the work is OK or not. The output is a trained AI model.

[0733] Step 2:

[0734] The user puts on the smart glasses and starts working. The smart glasses visually display the operation screen and the procedures to be performed. As input, they receive work procedure data provided by the server, and as output, they provide visual instructions to the user.

[0735] Step 3:

[0736] The user uses voice recognition technology to input operation procedures by voice. The smart glasses convert the voice input into text data and send it to the server. The smart glasses receive the user's voice data as input and generate text data as output.

[0737] Step 4:

[0738] The server receives the text data sent from the smart glasses and uses the trained AI model to determine whether the work is OK or not.The server uses the text data and the trained AI model as input and generates a OK / NG judgment result for the work as output.

[0739] Step 5:

[0740] The server sends the OK / NG result of the work to the smart glasses and displays it visually to the user. It receives the OK / NG result as input and provides visual feedback to the user as output.

[0741] Step 6:

[0742] The server uses an emotion engine to collect the worker's emotion data (e.g., heart rate and galvanic skin response) in real time. It receives data from the emotion sensor as input and generates emotion data as output.

[0743] Step 7:

[0744] The server analyzes the collected emotion data and evaluates the worker's stress level. It uses the emotion data as input and generates a stress level evaluation result as output.

[0745] Step 8:

[0746] The server issues an alert and notifies the smart glasses if the stress level is high. The server receives the stress level assessment result as input and generates an alert notification as output.

[0747] Step 9:

[0748] The server proposes improvements to the work procedure based on the emotion data and task data. It uses the emotion data and task data as input and generates improvement proposals as output.

[0749] Step 10:

[0750] The server sends the improvement suggestions for the work procedure to the smart glasses and displays them visually to the user. The server receives the improvement suggestions as input and provides visual feedback to the user as output.

[0751] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0752] The data generation model 58 is a so-called generative AI (Artificial Intelligence).

[0753] An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generative AIs. Data generation models 58

[0754] , obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in the data format of voice data, text data, etc. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0755] Another example of generative AI is Gemini (registered trademark) (Internet search engine). <url: https: gemini.google.com ?hl="ja">) are mentioned.

[0756] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0757] [Second embodiment]

[0758] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0759] .

[0760] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0761] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0762] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0763] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0764] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0765] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0766] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0767] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0768] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0769] In the smart glasses 214, the processor 46 performs the reception and output process.

[0770] A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0771] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0772] "Example 1"

[0773] One embodiment of the present invention is a system in which workers wear smart glasses and learn information such as work procedures, facility images, and past accident cases in advance. The workers can view the operation screen through the smart glasses and input instructions by voice. This system judges whether the work is OK or NG based on the worker's voice input and displays the result on the smart glasses' display.

[0774] "Example 2"

[0775] As a specific example, consider the case where a worker is performing maintenance work on equipment. The worker wears smart glasses and inputs the maintenance work procedures by voice. The system determines whether the input procedures are correct and displays the results on the smart glasses' display. In this way, the worker can proceed with the work while checking whether their work is correct.

[0776] "Example 3"

[0777] The system of the present invention can also learn work information using artificial intelligence. The artificial intelligence learns patterns from past work data and accident cases, and judges whether a work is OK or not based on that. This allows the system's judgment accuracy to improve over time, further improving the work efficiency and safety of workers.

[0778] The processing flow of each embodiment will be described below.

[0779] "Example 1"

[0780] Step 1: The worker puts on the smart glasses.

[0781] Step 2: The smart glasses learn the information necessary for the work (procedures, equipment images, past accident cases, etc.) in advance.

[0782] Step 3: The worker can view the operation screen through the smart glasses and input instructions by voice.

[0783] Step 4: The system judges whether the task is OK or NG based on the worker's voice input and displays the result on the smart glasses display.

[0784] "Example 2"

[0785] Step 1: When a worker performs maintenance work on equipment, he or she first puts on the smart glasses.

[0786] Step 2: The worker inputs the maintenance procedure by voice.

[0787] Step 3: The system determines whether the entered steps are correct and displays the results on the smart glasses display.

[0788] Step 4: The worker performs the task based on the system's feedback.

[0789] "Example 3"

[0790] Step 1: The system of the present invention uses artificial intelligence to learn task information.

[0791] Step 2: The artificial intelligence learns patterns from past work data and accident cases.

[0792] Step 3: Based on the learned patterns, the system determines whether the work is OK or not.

[0793] Step 4: The results of the assessment are displayed on the smart glasses display, and the worker proceeds with the work based on that feedback.

[0794] Example 1

[0795] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0796] With conventional work support systems, it was difficult for workers to efficiently learn information such as procedures, equipment images, and past accident cases, and receive appropriate instructions while working. There were also issues with the accuracy of voice-input instructions and the speed with which work was judged to be OK / NG. This could potentially lead to a decline in work efficiency and safety.

[0797] The identification process by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means. In this invention, the server includes a means for learning information required for the work (procedures, equipment images, past accident cases, etc.) in advance, a means for inputting the operation screen and procedures to be performed by voice through the smart glasses when performing the work, a means for determining whether the work is OK / NG and displaying the answer on the smart glasses, a means for analyzing the voice data and converting it into text, and a means for generating prompt sentences using a generative AI model and determining whether the work is OK / NG. This allows the worker to learn information efficiently, improves the accuracy of instructions input by voice, and enables quick OK / NG determination of the work.

[0798] "Information necessary for the work" refers to the knowledge and data necessary to carry out the work, such as procedures, equipment images, and past accident cases.

[0799] "Means for learning" refers to a method or device that allows a worker to learn the information necessary for the work in advance.

[0800] "Smart glasses" are wearable devices worn by workers to display operation screens and information.

[0801] An "operation screen" is a screen that displays information such as work procedures and instructions on smart glasses.

[0802] "Voice input means" refers to a method or device that allows a worker to input instructions or information using voice.

[0803] "Means for determining OK / NG" refers to a method or device for evaluating the progress and results of work and determining whether they are appropriate.

[0804] "Means for analyzing voice data and converting it into text" refers to a method or device for converting voice data into text data using voice recognition technology.

[0805] A "generative AI model" is a model that uses artificial intelligence to generate prompt sentences and determine whether a task is OK or not.

[0806] A "prompt sentence" is an instruction sentence to be input into a generative AI model, and is the sentence that serves as the basis for determining whether a task is OK or not.

[0807] MODE FOR CARRYING OUT THE INVENTION

[0808] This invention is a system in which workers wear smart glasses and learn information such as work procedures, equipment images, and past accident cases in advance. Workers can view the operation screen through the smart glasses and input instructions by voice. The system judges whether the work is OK or NG based on the worker's voice input and displays the result on the smart glasses' display.

[0809] 1. Program Generation

[0810] The server generates a program that allows workers to wear smart glasses and learn in advance information such as work procedures, facility images, and past accident cases. This program includes functions that allow workers to view the operation screen through the smart glasses and input instructions by voice.

[0811] 2. Program processing explanation

[0812] The server installs the generated program into the smart glasses. When a worker wears the smart glasses, they display information such as work procedures, images of equipment, and past accident cases. The worker can check this information through the smart glasses' display.

[0813] When a worker inputs instructions by voice, the smart glasses send the voice data to a server, which then uses voice recognition software (e.g., voice recognition technology) to convert the voice data into text. The server then uses a generative AI model (e.g., artificial intelligence model) to generate prompts to determine whether the input instructions are acceptable or not.

[0814] The server inputs the generated prompt sentences into the AI ​​model and obtains the results that determine whether the task is OK or not. The results are then sent to the smart glasses and displayed on the smart glasses' display.

[0815] 3. Examples of concrete examples and prompts

[0816] As a concrete example, consider the case where a worker inputs a command by voice, such as "Please tell me the next step in the work." In this case, the server generates the following prompt:

[0817] Example prompt sentence:

[0818] "The worker wants to know the next step. What's the next step?"

[0819] The server inputs this prompt into the generative AI model to obtain the next steps, which are then displayed on the smart glasses screen.

[0820] In this way, workers can check information such as work procedures, equipment images, and past accident cases through smart glasses, and determine whether the work is OK or not by inputting instructions via voice.

[0821] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0822] Step 1:

[0823] Starting up and connecting to smart glasses

[0824] The user puts on the smart glasses and turns them on.

[0825] The device (smart glasses) connects to the server via Wi-Fi.

[0826] Input: Smart glasses power on, Wi-Fi connection information

[0827] Output: Connection established with server

[0828] Step 2:

[0829] Displaying work procedures and information

[0830] The server recognizes the worker's ID and sends information such as relevant work procedures, equipment images, and past accident cases to the smart glasses.

[0831] The terminal displays the received information on its screen, such as a list of work procedures or an image of the equipment.

[0832] Input: Worker ID, related information

[0833] Output: Work procedures and equipment images displayed on smart glasses

[0834] Step 3:

[0835] Entering voice commands

[0836] Users can speak into the smart glasses to input instructions, such as "Please tell me the next step."

[0837] Input: User's voice commands

[0838] Output: Audio data

[0839] Step 4:

[0840] Sending and analyzing voice data

[0841] The terminal transmits the user's voice data to the server.

[0842] The server converts the voice data into text using voice recognition software (eg, speech recognition technology).

[0843] Input: Audio data

[0844] Output: Text data

[0845] Step 5:

[0846] Judging whether work is OK or NG

[0847] The server generates a prompt based on the converted text, for example, "The worker wants to know the next step in the work. Please tell me the next step."

[0848] The server inputs this prompt into a generative AI model (e.g., an artificial intelligence model) and obtains the next work procedure and a pass / fail judgment for the work.

[0849] Input: Text data, prompt

[0850] Output: Work procedure and OK / NG judgment

[0851] Step 6:

[0852] Display of judgment results

[0853] The server sends the obtained results to the smart glasses.

[0854] The terminal displays the received results on its display, such as "The next step is to install part A" or "The work is OK."

[0855] Input: Work procedure and OK / NG judgment

[0856] Output: The results displayed on the smart glasses

[0857] (Application example 1)

[0858] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0859] In conventional factory work, workers spend a lot of time and effort checking procedures and the status of equipment. It is also difficult to refer to past accident cases, which means that work safety is not adequately ensured. Furthermore, there is a lack of means to report and properly evaluate work progress in real time, which reduces work efficiency. To solve these issues, a system is needed that allows workers to work efficiently and safely using smart glasses.

[0860] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0861] In this invention, the server includes means for learning information required for work (procedures, equipment images, past accident cases, etc.) in advance, means for inputting the operation screen and procedures to be performed by voice through the smart glasses when performing work, means for determining whether the work is OK / NG and displaying the answer on the smart glasses, means for displaying the real-time status of the equipment, means for displaying past accident cases and issuing warnings, means for reporting the progress of the work based on the voice input, means for determining whether the work is OK / NG based on the progress of the work, and means for displaying the result on the display of the smart glasses. This enables workers to obtain the necessary information in real time and perform their work efficiently and safely.

[0862] "Information necessary for work" refers to information necessary to perform work, such as work procedures, equipment images, and past accident cases.

[0863] "Means of learning in advance" refers to the means of obtaining and understanding the information necessary for the work in advance.

[0864] "Smart glasses" are eyeglass-type devices worn by workers to display information.

[0865] The "operation screen" is the interface that the worker can see through the smart glasses.

[0866] "Voice input means" refers to a means by which a worker inputs instructions or information using voice.

[0867] "Means for determining whether something is OK / NG" refers to a means for evaluating the progress and results of work and determining whether they are appropriate.

[0868] The "means for displaying the answer on the smart glasses" refers to a means for displaying the judgment result on the display of the smart glasses.

[0869] The "means for displaying the real-time status of the equipment" is a means for displaying the current status of the equipment in real time.

[0870] The "means for displaying past accident cases and calling attention" is a means for displaying past accident cases and calling attention to workers.

[0871] The "means for reporting the progress of work based on voice input" is a means for reporting the progress of work based on voice input by the worker.

[0872] "Means for determining whether work is OK or NG based on the progress of the work" refers to a means for evaluating the progress of the work and determining whether the work is being carried out appropriately.

[0873] "Means for displaying on the display of the smart glasses" refers to means for displaying the determination results and other information on the display of the smart glasses.

[0874] In the system for implementing this invention, workers wear smart glasses and can carry out their work while checking information in real time, such as work procedures, equipment status, past accident cases, etc. Specific embodiments of this system are described below.

[0875] Hardware and software used

[0876] Smart glasses: Eyeglass-type devices worn by workers to display information.

[0877] Microphone: A device for capturing voice input from the worker.

[0878] Server: A computer system that manages the information required for work and processes voice input.

[0879] speech_recognition library: Software for converting voice input into text.

[0880] smart_glasses_display module: Custom software for displaying information on a smart glasses display.

[0881] factory_system module: Custom software that obtains work procedures, equipment status, and past accident cases from the factory system and determines whether the work is OK or not.

[0882] System Operation

[0883] 1. Display of work procedures: The server displays the necessary steps for the work on the smart glasses display, allowing the worker to check the next step in real time.

[0884] 2. Checking equipment status: The server acquires the real-time status of the equipment and displays it on the smart glasses, allowing workers to continue their work while understanding the current status of the equipment.

[0885] 3. Display of past accident cases: The server displays past accident cases on the smart glasses and alerts the worker, allowing the worker to learn from past mistakes and work safely.

[0886] 4. Acquire voice input: The worker inputs instructions by voice through a microphone. The server converts the voice into text using the speech_recognition library.

[0887] 5. Work progress reporting: Based on the worker's voice input, the server reports the work progress, allowing the worker to report the progress without using their hands.

[0888] 6. Judgement of work pass / fail: The server uses the factory_system module to judge whether the work is pass / fail based on the voice input.

[0889] 7. Display of results: The results are displayed on the smart glasses screen, allowing workers to check the evaluation of their work in real time.

[0890] Specific examples

[0891] When a worker wears smart glasses and performs maintenance work on equipment, they use the application following the steps below.

[0892] 1. Voice command to the smart glasses: "Show me the next steps."

[0893] 2. Voice command to the smart glasses to "check the equipment status."

[0894] 3. Voice command to the smart glasses: "Show me past accident cases."

[0895] 4. When the task is complete, say "The task is complete."

[0896] 5. The server determines whether the work is OK or not, and the result is displayed on the smart glasses.

[0897] Prompt Sentence Examples

[0898] "Show me the next step"

[0899] Check the condition of the equipment

[0900] "Show me past accident cases"

[0901] "The work is complete."

[0902] In this way, the use of smart glasses can significantly improve work efficiency and safety within factories.

[0903] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0904] Step 1:

[0905] The server learns the information required for the work (procedures, equipment images, past accident cases, etc.) in advance. This involves acquiring data from the factory system and organizing and analyzing the information using artificial intelligence. The input is data from the factory system, and the output is organized and analyzed work procedures, equipment images, and past accident cases.

[0906] Step 2:

[0907] The user puts on the smart glasses and starts working. The server displays the work procedure on the smart glasses display. The input is the work procedure data from the server, and the output is the work procedure displayed on the smart glasses display.

[0908] Step 3:

[0909] The user inputs voice instructions through a microphone. The server converts the speech to text using the speech_recognition library. The input is the user's voice instructions and the output is the text instructions.

[0910] Step 4:

[0911] The server obtains the real-time status of the equipment based on the textual instructions and displays it on the smart glasses. The input is the textual instructions, and the output is the real-time status of the equipment displayed on the smart glasses display.

[0912] Step 5:

[0913] The server retrieves past accident cases and displays them on the smart glasses. The input is the user's voice command, and the output is the past accident cases displayed on the smart glasses display.

[0914] Step 6:

[0915] The user reports the progress of the work by voice. The server converts the voice to text using the speech_recognition library and records the progress of the work. The input is the user's voice report and the output is the progress report in text format.

[0916] Step 7:

[0917] The server judges whether the work is OK or NG based on the progress report in text format. The input is the progress report in text format, and the output is the OK / NG judgment result of the work.

[0918] Step 8:

[0919] The server displays the judgment result on the smart glasses display. The input is the OK / NG judgment result of the work, and the output is the judgment result displayed on the smart glasses display.

[0920] Example 2

[0921] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0922] In conventional maintenance work, workers are required to accurately understand and execute procedures, but it can be difficult to confirm procedures and judge their accuracy. Furthermore, workers must use their hands to confirm procedures while working, which reduces work efficiency. Even when using voice input, it can be difficult to accurately analyze voice data and determine procedures, making it difficult to obtain real-time feedback. To solve these issues, a system is needed that can confirm procedures using voice input and provide real-time feedback.

[0923] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0924] In this invention, the server includes means for learning information required for work (procedures, equipment images, past accident cases, etc.) in advance, means for inputting the operation screen and procedures to be performed by voice through the smart glasses when performing work, means for determining whether the work is OK / NG and displaying the response on the smart glasses, means for converting voice data into text data, means for analyzing the text data and determining the accuracy of the procedures, means for transmitting the determination result to the smart glasses, and means for displaying the determination result on the display of the smart glasses. This enables workers to confirm procedures using voice input and accurately proceed with maintenance work while receiving feedback in real time.

[0925] "Information necessary for the work" refers to information such as procedures required to perform maintenance work, equipment images, and past accident cases.

[0926] "Means of learning" are methods and techniques for acquiring and understanding the information necessary for a task in advance.

[0927] "Smart glasses" are wearable devices that have built-in displays and microphones and allow for voice input and information display.

[0928] The "operation screen" is a screen that appears on the smart glasses display and is used to check work procedures and other information.

[0929] "Voice input means" refers to a method or technology that allows a worker to input information about an operation screen or procedures to be performed using voice.

[0930] "Means for determining whether the procedure entered is correct" refers to the method or technology used to determine whether the procedure entered is correct.

[0931] "Means for displaying the answer" refers to the method or technology for displaying the judgment result on the smart glasses display.

[0932] A "means for converting voice data to text data" is a method or technique for converting voice input into text form.

[0933] The "means for analyzing text data to determine the accuracy of procedures" refers to a method or technology for analyzing text data and determining whether the entered procedures are correct.

[0934] "Means for transmitting the determination result to the smart glasses" refers to a method or technology for transmitting the determination result of the correctness of the procedure to the smart glasses.

[0935] "Means for displaying the determination result on the display of the smart glasses" refers to a method or technology for displaying the determination result on the display of the smart glasses.

[0936] MODE FOR CARRYING OUT THE INVENTION

[0937] This invention provides a system that, when a worker performs maintenance work on equipment, uses smart glasses to input procedures by voice and determines the accuracy of the procedures in real time. Specific embodiments of this system are described below.

[0938] Hardware and software used

[0939] Smart glasses: A wearable device with a built-in display and microphone that allows voice input and information display.

[0940] Server: A computer system that analyzes voice data, determines procedures, and transmits results.

[0941] Speech recognition technology for converting voice data into text data.

[0942] Generative AI model (e.g., OpenAI's GPT-4): An artificial intelligence model that analyzes text data and determines the accuracy of procedures.

[0943] System Operation

[0944] 1. Voice input:

[0945] The user puts on the smart glasses and inputs maintenance instructions by voice, such as "remove the filter."

[0946] 2. Sending audio data:

[0947] The device (smart glasses) captures audio data through a built-in microphone and transmits the data to a server via Wi-Fi or Bluetooth.

[0948] 3. Audio data conversion:

[0949] The server converts the received voice data into text data using a predefined API, which provides highly accurate voice recognition.

[0950] 4. Procedure determination:

[0951] The server uses a generative AI model (e.g., OpenAI's GPT-4) to determine whether the maintenance procedures received as text data are correct, which involves checking them against a database of predefined correct procedures.

[0952] 5. Sending the results:

[0953] The server determines the accuracy of the procedure and sends the results to the smart glasses, again via Wi-Fi or Bluetooth.

[0954] 6. Displaying the results:

[0955] The device (smart glasses) displays the received judgment result on its display. For example, it may display a message such as "The procedure is correct. Please proceed to the next step."

[0956] Specific examples

[0957] As a specific example, consider the case where a worker changes an air conditioner filter.

[0958] 1. The user puts on the smart glasses and says, "Remove filter."

[0959] 2. The device (smart glasses) sends this voice data to the server.

[0960] 3. The server uses a specified API to convert the voice data into text data that says "Remove filter."

[0961] 4. The server uses a generative AI model to determine whether this text data is a valid instruction, for example, by checking whether the instruction "remove filter" exists in the database.

[0962] 5. The server sends the result of the judgment that "the procedure is correct" to the smart glasses.

[0963] 6. The device (smart glasses) displays the result of this judgment on the screen and informs the user, "The procedure is correct. Please proceed to the next step."

[0964] Prompt Sentence Examples

[0965] User: "Remove filter"

[0966] Server: "Your steps are correct. Please proceed to the next step."

[0967] User: "Install a new filter"

[0968] Server: "The procedure was correct. The maintenance work has been completed."

[0969] In this way, the user can receive real-time feedback and perform maintenance tasks accurately.

[0970] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0971] Program processing flow

[0972] Step 1: Voice Input

[0973] The user puts on the smart glasses and inputs maintenance instructions by voice, such as "remove the filter."

[0974] Input: User's voice command

[0975] Output: Audio data captured by the smart glasses microphone

[0976] Step 2: Sending audio data

[0977] The device (smart glasses) captures audio data through a built-in microphone and transmits it to a server via Wi-Fi or Bluetooth.

[0978] Input: Audio data captured by the smart glasses microphone

[0979] Output: Audio data sent to the server

[0980] Step 3: Convert the audio data

[0981] The server converts the received voice data into text data using a predefined API, which provides highly accurate voice recognition.

[0982] Input: Audio data sent to the server

[0983] Output: Text data (e.g. "Remove filter")

[0984] Step 4: Determine the procedure

[0985] The server uses a generative AI model (e.g., OpenAI's GPT-4) to determine whether the maintenance procedures received as text data are correct, which involves checking them against a database of predefined correct procedures.

[0986] Input: Text data (e.g., "Remove filter")

[0987] Output: Verification result (e.g. "The procedure is correct")

[0988] Step 5: Sending the results

[0989] The server determines the accuracy of the procedure and sends the results to the smart glasses, again via Wi-Fi or Bluetooth.

[0990] Input: Verification result (e.g., "The procedure is correct")

[0991] Output: The judgment result sent to the smart glasses.

[0992] Step 6: View the results

[0993] The device (smart glasses) displays the received judgment result on its display. For example, it may display a message such as "The procedure is correct. Please proceed to the next step."

[0994] Input: The judgment result sent to the smart glasses

[0995] Output: A message displayed on the smart glasses display (e.g., "Your steps are correct. Please proceed to the next step.")

[0996] Specific examples of operation

[0997] 1. User: Say "Remove filter."

[0998] 2. Device (smart glasses): Sends voice data to the server.

[0999] 3. Server: Converts the audio data into text data using a specified API.

[1000] 4. Server: Analyzes the text data using a generative AI model to determine the accuracy of the steps.

[1001] 5. Server: Sends the judgment results to the smart glasses.

[1002] 6. Device (smart glasses): Displays the judgment result on the screen and informs the user, "The procedure is correct. Please proceed to the next step."

[1003] In this way, the user can receive real-time feedback and perform maintenance tasks accurately.

[1004] (Application example 2)

[1005] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1006] In conventional maintenance work, workers are required to accurately understand and carry out procedures, but there is a lack of means to confirm procedures and the accuracy of work in real time, which can lead to work errors and reduced efficiency. There is also a need for a system that allows workers to input procedures by voice and instantly confirm their accuracy. This will improve the efficiency and accuracy of work.

[1007] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1008] In this invention, the server includes a means for learning information required for work (procedures, facility images, past accident cases, etc.) in advance, a means for inputting the operation screen and procedures to be performed by voice through the smart glasses when performing work, a means for determining whether the work is OK / NG and displaying the answer on the smart glasses, and a means for determining the procedures input by voice in real time and displaying the result on the display of the smart glasses. This enables workers to efficiently carry out maintenance work while checking the accuracy of the work in real time.

[1009] "Information necessary for the work" refers to information that is necessary when performing maintenance work, such as procedures, equipment images, and past accident cases.

[1010] "Means of learning" refers to a means of learning the information necessary for the work in advance, and is carried out using artificial intelligence.

[1011] "Smart glasses" are eyeglass-type devices that can be worn by workers to display operation screens and procedures to be carried out, and can also accept voice input.

[1012] The "operation screen" is the screen displayed on the smart glasses, which the worker refers to when performing maintenance work.

[1013] The "means for inputting by voice" refers to a means for an operator to input maintenance procedures by voice, and is carried out using voice recognition technology.

[1014] The "means for determining whether the maintenance procedure entered is correct" is a means for determining whether the maintenance procedure entered is correct.

[1015] The "means for displaying the answer" is a means for displaying the judgment result on the display of the smart glasses.

[1016] The "means for determining in real time" is a means for instantly determining the procedure input by voice and displaying the result in real time.

[1017] A "display" is a display device installed in smart glasses that displays the judgment results and operation screen.

[1018] The system for implementing this invention allows a worker to wear smart glasses and, when performing maintenance work, input procedures by voice, and the system determines in real time whether the procedures are correct and displays the results on the smart glasses' display.

[1019] System configuration

[1020] The system consists of the following main components:

[1021] 1. Smart glasses: A glasses-type device worn by workers, equipped with a display that shows the operation screen and judgment results, as well as a microphone that accepts voice input.

[1022] 2. Server: Uses voice recognition technology and artificial intelligence to process the input procedures.

[1023] 3. Speech recognition software: Software that converts voice input into text, such as Google's speech recognition API.

[1024] 4. Artificial intelligence model: A model that learns the information necessary for the task in advance and determines whether the input procedures are correct.

[1025] Program processing

[1026] The server works as follows:

[1027] 1. Acquiring voice input: Acquire the worker's voice through the microphone in the smart glasses.

[1028] 2. Speech recognition: Convert captured speech into text using speech recognition software.

[1029] 3. Procedure validation: The converted text is input into an artificial intelligence model to determine whether the procedure is correct.

[1030] 4. Displaying the results: The results are displayed on the smart glasses screen.

[1031] Hardware and software used

[1032] Hardware: Smart glasses, microphone

[1033] Software: Python, SpeechRecognition library, smart glasses SDK, artificial intelligence model (e.g., TensorFlow)

[1034] Specific examples

[1035] For example, when a technician performs maintenance on a robot, they can say, "Turn off the robot, remove the cover, and inspect the internal components." This speech is picked up by the microphone in the smart glasses and converted into text by speech recognition software. The converted text is sent to a server, where an artificial intelligence model determines the accuracy of the procedure. The result is displayed on the smart glasses' display, saying, "The procedure is correct."

[1036] Prompt Sentence Examples

[1037] Turn off the robot, remove the cover, and inspect the internal components.

[1038] In this way, workers can efficiently carry out maintenance work while checking the accuracy of the work in real time.

[1039] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1040] Step 1:

[1041] The user puts on the smart glasses and begins maintenance work. The user then speaks maintenance procedures into the microphone on the smart glasses. The voice input contains specific instructions, such as "Turn off the robot's power, remove the cover, and inspect the internal parts." Input: User's voice. Output: Voice data.

[1042] Step 2:

[1043] The smart glasses send the captured voice data to the server, which then uses voice recognition software (e.g., Google's voice recognition API) to convert the voice data into text data. Input: Voice data. Output: Text data.

[1044] Step 3:

[1045] The server inputs the converted text data into an AI model. The AI ​​model compares the input data with a database of previously learned maintenance procedures and determines whether the procedure is correct. Input: Text data. Output: Verification result (correct / incorrect).

[1046] Step 4:

[1047] The server sends the judgment result to the smart glasses. The smart glasses display the judgment result on the display. For example, a message such as "The procedure is correct" or "The procedure is incorrect. Please check again" is displayed. Input: Judgment result. Output: Message displayed on the display.

[1048] Step 5:

[1049] The user checks the judgment results displayed on the smart glasses display and corrects the procedure if necessary. If the correct procedure is displayed, the user continues working. Input: Message displayed on the display. Output: User action (correcting the procedure or continuing the work).

[1050] In this way, the user can proceed with the work while checking the accuracy of the maintenance procedure in real time.

[1051] Example 3

[1052] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1053] Conventional work support systems require workers to manually judge the safety and efficiency of their work, which increases the risk of work errors and accidents. Furthermore, there was a lack of effective ways to utilize past accident cases and work data, which led to insufficient work improvements and safety measures. Furthermore, real-time work support was difficult, placing a heavy burden on workers.

[1054] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1055] In this invention, the server includes a means for acquiring past work data and accident cases from a database and performing preprocessing, a means for training an AI model using the preprocessed data, and a means for inputting new work data and using the AI ​​model to determine whether the work is OK or not. This makes it possible to effectively utilize past data and automatically determine the safety and efficiency of work in real time.

[1056] "Information necessary for the work" refers to the data and knowledge necessary to carry out the work, such as procedures, equipment images, and past accident cases.

[1057] "Smart glasses" refers to a wearable device that can be worn by workers to visually display the operation screen and procedures to be performed.

[1058] "Voice input means" refers to technology that allows workers to give operational instructions and input data using voice.

[1059] "Means for determining whether something is OK / NG" refers to the technology used to evaluate the safety and efficiency of work and determine the results as OK or NG.

[1060] A "database" refers to an information management system that systematically stores past work data and accident cases, and allows them to be searched and retrieved as needed.

[1061] "Preprocessing means" refers to technology that processes collected data by completing missing values, normalizing it, extracting features, etc., and converting it into a format suitable for an artificial intelligence model.

[1062] An "artificial intelligence model" refers to a model that uses machine learning algorithms to learn data and perform a specific task (in this case, determining whether a task is OK or not).

[1063] "Training means" refers to techniques for training an artificial intelligence model using preprocessed data to improve its performance.

[1064] "New work data" refers to information about current or future work, and the artificial intelligence model uses this information to determine whether the work is OK or not.

[1065] "Feedback means" refers to technology that notifies users of the results determined by the artificial intelligence model and encourages them to improve their work and take safety measures.

[1066] The present invention is a system for improving work safety and efficiency, and can be implemented as follows.

[1067] First, users use a development environment such as Python or TensorFlow to generate the system's program. The program includes code for building an artificial intelligence (AI) model and learning from past work data and accident cases.

[1068] The server retrieves past work data and accident case studies provided by the user from a database (e.g., MySQL). The server then preprocesses this data and converts it into a format suitable for the AI ​​model. Specifically, the server performs data cleaning, normalization, feature extraction, etc.

[1069] The server then trains the AI ​​model using a machine learning library such as TensorFlow. Once training is complete, the server receives new task data as input and uses the AI ​​model to determine whether the task is OK or not.

[1070] As a concrete example, consider the case of using factory work data. Users input past work data (e.g., work hours, equipment used, years of experience of workers) and accident cases (e.g., type, cause, and impact of accidents) into the system. The server learns from this data, and when new work data is input, it determines whether the work is safe.

[1071] Example prompt sentence:

[1072] "Based on past work data and accident cases, determine whether your current work is safe."

[1073] This system is expected to improve the work efficiency and safety of workers. The flow of the identification process in the third embodiment will be described with reference to FIG.

[1074] Step 1: Collect data

[1075] Users upload past work data and accident cases to the system. Specifically, data is extracted from CSV files and databases. The input information includes work hours, equipment used, years of experience of workers, type of accident, cause, and impact. The output is that this data is saved on the server.

[1076] Step 2: Preprocessing the data

[1077] The server preprocesses the collected data. Specifically, it imputes missing values, normalizes data, and encodes categorical data. The input is the raw data collected in step 1. The output is a preprocessed dataset. For example, it imputes missing values ​​with the mean, standardizes work time, and one-hot encodes the equipment used.

[1078] Step 3: Training the AI ​​model

[1079] The server trains an AI model using the preprocessed data, specifically using a machine learning library such as TensorFlow. The input is the preprocessed dataset from step 2. The output is the trained AI model. Training can take several hours.

[1080] Step 4: Entering work data

[1081] Users input new work data into the system. Specifically, this involves real-time data entry and batch processing. The input includes information such as the start time of the new work, the equipment used, and the worker's years of experience. The output is that this data is stored on the server.

[1082] Step 5: Judging whether the work is OK or NG

[1083] The server passes the new work data that was input to the AI ​​model, which judges whether the work is OK or not. Specifically, the AI ​​model evaluates the safety and efficiency of the work. The input is the new work data that was input in step 4. The output is the OK / NG judgment result for the work. For example, a judgment may be made that "the work time is too long, so it is not OK."

[1084] Step 6: Feedback on results

[1085] The server feeds back the judgment results to the user. Specifically, it displays notifications from the system and a dashboard. The input is the judgment results obtained in step 5. The output is feedback information for the user. The user reviews the work plan based on these results. For example, they can take measures such as "shortening work time" or "changing the equipment used."

[1086] (Application example 3)

[1087] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1088] Conventional work support systems required workers to study procedures, equipment images, and past accident examples in advance, but they did not acquire work data in real time or judge whether a task passed or failed based on past data. This resulted in issues such as insufficient improvement in work efficiency and safety. Furthermore, there was a need to improve the accuracy of judgments when workers input operation screens and procedures to be performed by voice through smart glasses.

[1089] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1090] In this invention, the server includes a means for learning information required for work (procedures, facility images, past accident cases, etc.) in advance, a means for inputting the operation screen and procedures to be performed by voice through the smart glasses when performing work, a means for determining whether the work is OK or not and displaying the answer on the smart glasses, and a means using artificial intelligence for acquiring work data in real time and determining whether the work is OK or not based on past data and accident cases, which makes it possible to improve work efficiency and safety.

[1091] "Information necessary for the work" refers to information such as procedures required to perform the work, equipment images, and past accident cases.

[1092] "Means for learning in advance" refers to means for learning the information necessary for the work in advance.

[1093] "Smart glasses" are eyeglass-type devices that can be worn by workers to display operation screens and procedures to be carried out.

[1094] The "operation screen" is the screen that the worker can view through the smart glasses.

[1095] "Procedures to be implemented" are specific procedures that must be followed when performing a task.

[1096] "Voice input means" refers to a means by which a worker uses voice to input information about the operation screen and procedures to be carried out.

[1097] "Means for determining whether something is OK or not" refers to a means for determining whether the results of the work are appropriate or not.

[1098] The "means for acquiring task data in real time" refers to a means for acquiring data in real time while the task is being performed.

[1099] "Past data" refers to data on work that has been done in the past.

[1100] "Accident cases" are specific examples of accidents that have occurred in the past.

[1101] "Means using artificial intelligence" refers to means for determining whether a task is OK or not using artificial intelligence technology.

[1102] An embodiment of the present invention will be described.

[1103] First, the server has a means of learning the information required for the work (procedures, equipment images, past accident cases, etc.) in advance. This is done using artificial intelligence technology. Specifically, it uses machine learning libraries such as TensorFlow to learn from past work data and accident cases and model work patterns.

[1104] Next, when performing work, the user wears smart glasses. Smart glasses are devices that display operation screens and procedures to be performed, and have a means of inputting instructions by voice using voice recognition technology. When the user inputs operation instructions by voice, the voice data is sent to the server, which then converts it into text data using voice recognition technology.

[1105] The server has a means of acquiring work data in real time. Specifically, it monitors the progress of work in real time through cameras and sensors installed in the smart glasses. This data is sent to the server, which has a means of using artificial intelligence to judge whether the work is OK or not based on past data and accident cases. The AI ​​model is built using machine learning libraries such as TensorFlow, and judges whether the work is OK or not based on the data acquired in real time.

[1106] The results are displayed on the smart glasses, allowing users to check the progress of their work in real time and make corrections as necessary, improving work efficiency and safety.

[1107] As a concrete example, consider a robot assembling parts in a factory. In this case, cameras and sensors installed on the robot collect work data in real time and input that data into an AI model. The AI ​​model judges whether the work is OK or not based on past data and accident cases, and displays the result in real time on the smart glasses.

[1108] Examples of prompts to be input to a generative AI model include:

[1109] "Robots are assembling parts in a factory. Please build an AI system that will use real-time data on the work to determine whether the work is OK or not, by referring to past data and accident cases."

[1110] The above is an embodiment of the present invention.

[1111] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1112] Step 1:

[1113] The server learns in advance the information required for the work (procedures, equipment images, past accident cases, etc.). Specifically, it uses machine learning libraries such as TensorFlow to learn from past work data and accident cases and model work patterns. The input is past work data and accident cases, and the output is the trained AI model.

[1114] Step 2:

[1115] The user puts on the smart glasses and begins work. The smart glasses are a device that displays the operation screen and procedures to be performed. The input is the user's voice instructions, and the output is the operation screen and procedures displayed on the smart glasses.

[1116] Step 3:

[1117] The user inputs operation instructions by voice. The smart glasses use voice recognition technology to convert the voice into text data and send that data to the server. The input is the user's voice instructions, and the output is text data.

[1118] Step 4:

[1119] The server acquires work data in real time. The progress of work is monitored in real time through the cameras and sensors installed in the smart glasses. The input is real-time data from the cameras and sensors, and the output is work data sent to the server.

[1120] Step 5:

[1121] The server inputs the work data acquired in real time into the AI ​​model and judges whether the work is OK or not. The AI ​​model judges whether the work is OK or not based on past data and accident cases. The input is the work data acquired in real time, and the output is the OK or not judgment result of the work.

[1122] Step 6:

[1123] The server sends the judgment result to the smart glasses and displays it to the user. This allows the user to check the progress of the work in real time and make corrections as necessary. The input is the OK / NG judgment result of the work, and the output is the judgment result displayed on the smart glasses.

[1124] The above are the specific processing steps for carrying out the present invention.

[1125] Furthermore, an emotion engine that estimates the user's emotion may be combined.

[1126] The identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1127] "Example 1"

[1128] One embodiment of the present invention is a system that combines an emotion engine that recognizes the emotions of workers. In this system, a worker wears smart glasses and, as he or she works, the emotion engine recognizes the worker's emotions from the worker's tone of voice and facial expressions. For example, if the system senses that the worker is nervous, it can adjust work instructions according to the worker's emotions, such as slowing down the pace of work.

[1129] "Example 2"

[1130] The emotion engine can also take measures such as pausing work if a worker's emotions exceed a certain threshold. For example, if the system determines that a worker is feeling excessively stressed, it can pause work and instruct the worker to take a break.

[1131] "Example 3"

[1132] Furthermore, the emotion engine can record changes in a worker's emotions and analyze the data to identify areas for improvement in the work. For example, if a worker tends to feel stressed during a particular task, it can suggest improvement measures such as reviewing the procedures for that task.

[1133] The processing flow of each embodiment will be described below.

[1134] "Example 1"

[1135] Step 1: The worker puts on the smart glasses.

[1136] Step 2: When you start working, the emotion engine recognizes your emotions from your tone of voice and facial expressions.

[1137] Step 3: The emotion engine adjusts the work instructions according to the worker's emotions. For example, if the worker feels nervous, the system will adjust the work pace, such as slowing it down. "Example 2"

[1138] Step 1: The worker puts on the smart glasses and begins work.

[1139] Step 2: The emotion engine recognizes the worker's emotions and pauses the work if the emotions exceed a certain threshold.

[1140] Step 3: After pausing work, the system prompts the worker to take a break.

[1141] "Example 3"

[1142] Step 1: The worker puts on the smart glasses and begins work.

[1143] Step 2: The emotion engine records changes in the worker's emotions.

[1144] Step 3: The system analyzes the recorded emotional data and identifies areas for improvement in the work. For example, if a worker tends to feel stressed during a particular task, it suggests improvements such as reviewing the procedures for that task.

[1145] Example 1

[1146] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1147] Conventional work support systems allow workers to learn information such as work procedures, equipment images, and past accident cases in advance, but they are unable to respond to emotional changes during work. As a result, it is difficult to provide appropriate instructions when workers feel nervous or stressed, which can lead to reduced work efficiency and safety.

[1148] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1149] In this invention, the server includes a means for learning information required for work (procedures, facility images, past accident cases, etc.) in advance, a means for inputting the operation screen and procedures to be performed by voice through the smart glasses when performing work, a means for determining whether the work is OK or NG and displaying the answer on the smart glasses, and a means for recognizing the worker's emotions and adjusting work instructions according to those emotions. This makes it possible to respond to changes in the worker's emotions and provide appropriate instructions.

[1150] "Information necessary for work" is a general term for data and materials necessary to carry out work, such as work procedures, equipment images, and past accident cases.

[1151] "Smart glasses" are a type of wearable device that has a built-in display, camera, microphone, etc. and is worn by workers.

[1152] The "operation screen" is an interface that is displayed on the smart glasses display and contains information such as work procedures and instructions.

[1153] "Voice input means" refers to a technique that allows a worker to input instructions or information through voice, and typically uses voice recognition technology.

[1154] The "means for determining whether something is OK or not" refers to an algorithm or program that analyzes the worker's voice input and the work content, and determines whether the results are correct.

[1155] "Means for recognizing emotions" refers to technology that analyzes the tone of a worker's voice and facial expressions to identify emotions such as tension or stress.

[1156] The "means for adjusting work instructions" refers to an algorithm or program for changing the pace of work or the content of instructions in response to the worker's recognized emotions.

[1157] MODE FOR CARRYING OUT THE INVENTION

[1158] This invention is a system in which workers wear smart glasses and learn information such as work procedures, facility images, and past accident cases in advance. Workers can check the operation screen through the smart glasses and input instructions by voice. In addition, by combining it with an emotion engine, it has the function of recognizing the worker's emotions and adjusting work instructions.

[1159] Hardware and software used

[1160] Server: A relational database such as MySQL is used to store information such as work procedures, equipment images, and past accident cases.

[1161] Devices (smart glasses): Use wearable devices such as Google Glass and Microsoft HoloLens.

[1162] Speech recognition software: Converts speech to text using a predefined API.

[1163] Emotion recognition software: Uses a system that combines Microsoft Azure's Emotion API, OpenCV, and deep learning models.

[1164] Specific operation of the system

[1165] 1. The server stores information such as work procedures, equipment images, and past accident cases in a database. For example, text data of work procedures and image files of equipment are stored in the database.

[1166] 2. When a worker wears the device (smart glasses), it connects to the server via Wi-Fi and requests the necessary information. In response to the request, the server sends work procedures and equipment images to the device. The device then displays the received information on its screen.

[1167] 3. The user (worker) checks the information displayed through the smart glasses and inputs voice instructions such as "Tell me the next step." The smart glasses capture the voice using a built-in microphone and convert the voice into text using a specified API.

[1168] 4. The server receives the text data of the voice instructions sent from the device and analyzes it using a script written in Python. For example, it determines whether the instruction, "Tell me the next step," is correct and generates a pass / fail result.

[1169] 5. The device displays the result of the judgment received from the server on the display, for example, a message such as "The next step is OK."

[1170] 6. The emotion engine analyzes the user's tone of voice and facial expressions. The smart glasses use a built-in camera and microphone to capture the user's facial expressions and tone of voice, and then use Microsoft Azure's Emotion API to recognize emotions.

[1171] 7. The server adjusts the work instructions based on the emotion data received from the emotion engine. For example, if the server detects that the user is nervous, it will send an additional instruction such as "Please proceed slowly" to the device. The device will then display this instruction on its screen.

[1172] Specific examples

[1173] For example, consider the case where a user wears smart glasses and is inspecting equipment. The user inputs a command by voice, such as "Tell me the next step." The server analyzes this command and displays the next step on the smart glasses' display. At the same time, if the emotion engine detects tension in the user's voice tone, the server will display an additional command such as "Please proceed slowly."

[1174] Prompt Sentence Examples

[1175] Examples of prompts to be input to a generative AI model include:

[1176] "A worker is wearing smart glasses and inspecting equipment. The worker inputs instructions for the next step by voice. The server analyzes this voice instruction and displays the next step on the smart glasses' display. At the same time, if the emotion engine detects tension in the worker's tone of voice, the server displays an additional instruction to slow down the pace of the work."

[1177] By inputting this prompt into a generative AI model, the system's behavior can be simulated.

[1178] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1179] Step 1:

[1180] The server stores the information in a database.

[1181] Input: Information such as work procedures, equipment images, and past accident cases.

[1182] Processing: The server stores this information in a relational database such as MySQL. Specifically, it stores the text data of work procedures and image files of the equipment in the database.

[1183] Output: Information stored in a database.

[1184] Step 2:

[1185] The terminal retrieves the information from the server and displays it on the screen.

[1186] Input: Information such as work procedures and equipment images stored on the server.

[1187] Processing: When the user wears the device (smart glasses), it connects to the server via Wi-Fi and requests the necessary information. The server then sends the requested information, such as work procedures and facility images, to the device.

[1188] Output: Work procedures and equipment images displayed on the terminal screen.

[1189] Step 3:

[1190] The user inputs instructions by voice.

[1191] Input: User's spoken command (e.g., "What's the next step?").

[1192] Processing: The smart glasses use the built-in microphone to capture audio and convert it to text using a predefined API.

[1193] Output: Spoken instructions converted to text.

[1194] Step 4:

[1195] The server analyzes the voice instructions and determines whether the task is OK or not.

[1196] Input: Spoken instructions converted to text.

[1197] Processing: The server receives the text data of the voice instructions sent from the device and analyzes it using a script written in Python. For example, it determines whether the instruction "tell me the next step" is correct and generates a OK / NG result.

[1198] Output: OK / NG result.

[1199] Step 5:

[1200] The terminal displays the judgment result on the display.

[1201] Input: OK / NG result received from the server.

[1202] Processing: The device displays the result of the judgment received from the server on the display. For example, it displays a message such as "The next step is OK."

[1203] Output: The result displayed on the device screen.

[1204] Step 6:

[1205] The emotion engine recognizes the user's emotions.

[1206] Input: The user's tone of voice and facial expressions.

[1207] Processing: The smart glasses use a built-in camera and microphone to capture the user's facial expressions and tone of voice, and then use Microsoft Azure's Emotion API to recognize emotions.

[1208] Output: Recognized emotion data.

[1209] Step 7:

[1210] The server adjusts the work instructions according to the emotion.

[1211] Input: Emotion data received from the emotion engine.

[1212] Processing: The server adjusts the work instructions based on the emotion data received from the emotion engine. For example, if the server detects that the user is nervous, it will send an additional instruction such as "Please proceed slowly" to the device.

[1213] Output: Adjusted work instructions displayed on the terminal display.

[1214] (Application example 1)

[1215] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1216] Conventional work support systems allow workers to learn information such as work procedures, equipment images, and past accident cases in advance, but lack support that takes into account the worker's emotional state while working. As a result, workers may not be able to respond appropriately if they feel nervous or stressed, which can lead to reduced work efficiency and safety. In addition, the OK / NG judgment of operations based on voice input does not take into account the worker's emotional state, which can increase the psychological burden on the worker.

[1217] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means. In this invention, the server includes a means for learning information required for the work (procedures, equipment images, past accident cases, etc.) in advance, a means for inputting the operation screen and procedures to be performed by voice through the smart glasses when performing the work, a means for determining whether the work is OK / NG and displaying the answer on the smart glasses, an emotion recognition means for recognizing the emotion of the worker, and a means for adjusting the pace of the work and instructions according to the emotion of the worker. This enables appropriate support that takes into account the emotional state of the worker, and is expected to improve work efficiency and safety.

[1218] "Information necessary for the work" refers to information necessary for the smooth and safe progress of work, such as work procedures, equipment images, and past accident cases.

[1219] "Means for learning in advance" refers to means by which workers can learn the information necessary for their work in advance, and includes learning systems that use artificial intelligence.

[1220] "Smart glasses" are wearable devices that can be worn by workers to visually display operation screens and procedures to be carried out.

[1221] The "means for inputting operation screens and procedures to be performed by voice" refers to a means for an operator to input operation screens and procedures to be performed by voice, and voice recognition technology can be used.

[1222] The "means for determining whether a task is OK / NG and displaying the answer on the smart glasses" is a means for determining whether a task is OK / NG based on the worker's voice input and displaying the result on the smart glasses display.

[1223] The "emotion recognition means" is a means for recognizing emotions from the tone of a worker's voice and facial expressions, and an emotion recognition engine can be used.

[1224] "Means for adjusting the pace of work and instructions" refers to means for adjusting the pace of work and instructions according to the worker's emotions.

[1225] As an embodiment of the present invention, an application installed on a factory robot is used. A specific embodiment of this application will be described below.

[1226] Hardware and software used

[1227] Hardware: smart glasses (e.g., wearable devices), factory robots (e.g., robotic arms)

[1228] Software: emotion recognition engine (e.g. emotion recognition software), voice recognition engine (e.g. voice recognition software), display software (e.g. display software)

[1229] Data processing and calculation

[1230] 1. Handling voice input:

[1231] The user inputs voice commands into the smart glasses.

[1232] Use a speech recognition engine to convert speech to text.

[1233] The converted text is analyzed and the operation is judged to be OK or not.

[1234] 2. Emotion recognition processing:

[1235] The smart glasses' camera and microphone are used to capture the user's facial expressions and tone of voice.

[1236] An emotion recognition engine is used to recognize the user's emotions.

[1237] Adjust work pace or instructions based on perceived emotions.

[1238] 3. Display processing:

[1239] The OK / NG results of the operation and adjusted instructions are displayed on the smart glasses display.

[1240] Use display software to present information in a visually understandable format.

[1241] Specific examples

[1242] For example, when a user performs maintenance on a robot arm, they wear smart glasses and perform the following operations.

[1243] 1. The user says "What's the next step?"

[1244] 2. The speech recognition engine converts the speech into text and displays the next steps.

[1245] 3. If the user is nervous, the emotion recognition engine will detect this and adjust instructions to slow down the pace of the task.

[1246] 4. The smart glasses display will say, "Please proceed slowly with the next step."

[1247] Prompt Sentence Examples

[1248] An example of a prompt sentence to input to the generative AI model is as follows:

[1249] "Design a smart glasses application to teach maintenance procedures to factory robots. The application should include a function where the user can input instructions by voice, and an emotion recognition engine can be used to recognize the user's emotions and adjust the pace of work and instructions accordingly. The hardware used will be a wearable device and a robotic arm, and the software will be speech recognition software, emotion recognition software, and display software."

[1250] In this way, smart glasses applications can be realized for efficient and safe operation and maintenance of factory robots.

[1251] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1252] Step 1:

[1253] The user inputs voice commands into the smart glasses, which are then captured by the smart glasses' microphone.

[1254] Step 2:

[1255] The server uses a speech recognition engine to convert the captured speech into text: the input is speech data and the output is text data.

[1256] Step 3:

[1257] The server analyzes the converted text and determines whether the operation is OK or NG. The input is text data, and the output is the OK / NG result.

[1258] Step 4:

[1259] The server uses the camera and microphone of the smart glasses to capture the user's facial expressions and tone of voice. The input is the camera image and audio data.

[1260] Step 5:

[1261] The server uses an emotion recognition engine to recognize the user's emotions from the captured facial expressions and tone of voice. The input is camera footage and audio data, and the output is emotion data.

[1262] Step 6:

[1263] The server adjusts the pace of work and instructions based on the recognized emotions. The input is emotion data, and the output is adjusted instructions.

[1264] Step 7:

[1265] The server displays the OK / NG result of the operation and adjusted instructions on the smart glasses display. The input is the OK / NG judgment result and adjusted instructions, and the output is the display.

[1266] Step 8:

[1267] The user checks the information displayed on the smart glasses display and proceeds to the next step. The input is the display, and the output is the user's next action.

[1268] Example 2

[1269] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1270] In conventional maintenance work, workers are required to accurately understand procedures and carry them out appropriately, but procedural errors and worker stress can lead to reduced work efficiency.In addition, there was no system that monitored workers' emotional state in real time and instructed them to take breaks at the appropriate time, so the health and safety of workers was not adequately ensured.

[1271] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1272] In this invention, the server includes a means for learning information required for work (procedures, facility images, past accident cases, etc.) in advance, a means for inputting the operation screen and procedures to be performed by voice through the smart glasses when performing work, a means for determining whether the work is OK or NG and displaying the answer on the smart glasses, and a means for monitoring the user's emotional state and pausing the work and instructing them to take a break if it exceeds a certain threshold. This allows the worker to proceed with the work while checking the accuracy of the procedures, and also allows them to take an appropriate break if they feel excessive stress.

[1273] "Information necessary for the work" refers to the data and knowledge necessary to carry out maintenance work accurately and efficiently, such as procedures, equipment images, and past accident cases.

[1274] "Smart glasses" are wearable devices equipped with a display and voice input functions, allowing workers to check and input information hands-free.

[1275] "Voice input means" refers to technology or devices that allow workers to use voice to input information on the operation screen or procedures to be performed, and includes voice recognition technology.

[1276] "Means for determining whether something is OK / NG" refers to the technology or device that evaluates whether the entered procedures or operations are correct and outputs the results.

[1277] "Means for monitoring emotional state" refers to technologies and devices that measure and assess workers' stress levels and fatigue in real time.

[1278] "Means to temporarily suspend work and instruct workers to take a break when a certain threshold is exceeded" refers to technology or devices that suspend work and encourage workers to take a break when the evaluation result of their emotional state exceeds a set standard.

[1279] "Artificial intelligence" refers to technologies and systems that use machine learning and data analysis to automatically learn the information necessary for work and make decisions.

[1280] "Speech recognition technology" refers to technology or systems for converting speech into text data, enabling voice input.

[1281] This invention is a system that uses smart glasses to input procedures by voice when a worker performs maintenance work, judges the accuracy of the procedures, and monitors the worker's emotional state. This system is implemented using the following hardware and software.

[1282] Hardware and software used

[1283] Hardware: Smart glasses

[1284] Software: Speech recognition system, emotion engine, generative AI model

[1285] Specific operation of the system

[1286] 1. The user puts on the smart glasses. The smart glasses start up and connect to the system.

[1287] 2. The user speaks the maintenance procedure, for example, "Change the engine oil."

[1288] 3. The device (smart glasses) activates the voice recognition system and converts the input voice into text data. Specifically, voice recognition software (e.g., a specified API) is used.

[1289] 4. The server receives the text data and uses a generative AI model to determine whether the input procedure is correct. For example, a generative AI model (e.g., GPT-4) evaluates whether the procedure "change the engine oil" is correct.

[1290] 5. The server sends the result to the smart glasses display. The device (smart glasses) displays the result on the display. For example, the result "Correct" is displayed.

[1291] 6. The device (smart glasses) uses built-in sensors to monitor the user's emotional state. The emotion engine evaluates the user's stress level and fatigue level.

[1292] 7. The server receives the emotion engine's evaluation results, and if the user's emotion exceeds a certain threshold, the work is paused. The device (smart glasses) displays a message on the display saying, "Please take a break." For example, if the system determines that the user is feeling excessively stressed, it will display the message, "Please take a break."

[1293] Specific examples

[1294] Example 1: Entering maintenance procedure

[1295] The user speaks, "Change the engine oil."

[1296] The smart glasses' voice recognition system converts this into text data: "Change the engine oil."

[1297] The server uses a generative AI model to determine whether this procedure is correct and determines that it is.

[1298] The server displays the "correct" result on the smart glasses display.

[1299] The user checks the display and continues working.

[1300] Example 2: Emotion engine stalls work

[1301] The emotion engine determines that the user is feeling excessive stress while working.

[1302] The emotion engine pauses the work and displays the message "Take a break" on the smartglasses display.

[1303] The user follows the instructions and takes a break.

[1304] Prompt Sentence Examples

[1305] "You can input the steps for changing your engine oil by voice. The system will assess the accuracy of the steps and show the results on the display. If you feel excessive stress while working, the system will pause your work and advise you to take a break."

[1306] In this way, users can work with peace of mind while checking whether their work is correct. If users feel excessive stress, the system will encourage them to take an appropriate break, thereby improving the safety and efficiency of their work.

[1307] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1308] Step 1:

[1309] The user puts on the smart glasses, which then start up and connect to the system.

[1310] Input: Wearing smart glasses

[1311] Output: System connection completed message

[1312] Specific operation: The smart glasses start up and connect to the server via Wi-Fi or Bluetooth. The message "System connection complete" appears on the display.

[1313] Step 2:

[1314] The user inputs the maintenance procedure by voice, for example, "Change the engine oil."

[1315] Input: Voice input (e.g. "Change engine oil")

[1316] Output: Audio data

[1317] How it works: The microphone in the smart glasses captures sound and stores it as audio data.

[1318] Step 3:

[1319] The device (smart glasses) activates a voice recognition system and converts the input voice into text data.

[1320] Input: Audio data

[1321] Output: Text data (e.g. "Change the engine oil")

[1322] What happens: Speech recognition software (e.g., a predefined API) analyzes the audio data and generates corresponding text data.

[1323] Step 4:

[1324] The server receives the text data and uses a generative AI model to determine whether the input steps are correct.

[1325] Input: Text data (e.g., "Change the engine oil")

[1326] Output: Verification result (e.g. "correct")

[1327] What it does: The server uses a generative AI model (e.g., GPT-4) to analyze the content of the text data and evaluate the accuracy of the steps.

[1328] Step 5:

[1329] The server sends the judgment result to the smart glasses display, and the device (smart glasses) displays the judgment result on the display.

[1330] Input: Verification result (e.g. "correct")

[1331] Output: Display (e.g. "Correct")

[1332] Specific operation: The server sends the judgment result to the smart glasses, which then display the result on the screen.

[1333] Step 6:

[1334] The device (smart glasses) uses built-in sensors to monitor the user's emotional state.

[1335] Input: Biometric data (e.g., heart rate, galvanic skin response)

[1336] Output: Evaluation of emotional state (e.g., stress level)

[1337] How it works: The smart glasses' sensors collect the user's biometric data, and the emotion engine analyzes the data to assess their emotional state.

[1338] Step 7:

[1339] The server receives the evaluation results of the emotion engine, and if the user's emotion exceeds a certain threshold, the work is paused. The device (smart glasses) displays a message on the display saying, "Please take a break."

[1340] Input: Emotional state assessment (e.g., high stress)

[1341] Output: Break prompt (e.g., "Please take a break")

[1342] Specific operation: The server receives the evaluation results of the emotion engine, and if it determines that the stress level is high, it sends a message to the smart glasses instructing them to take a break and displays it on the display.

[1343] (Application example 2)

[1344] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1345] In conventional maintenance work, workers are required to accurately understand procedures and execute them appropriately, but accidents and mistakes can occur due to procedural errors or worker stress. Furthermore, there is a lack of a system to monitor workers' emotional state in real time and encourage them to take breaks at appropriate times, which leads to problems with reduced work efficiency and safety. To solve these issues, a system is needed to check the accuracy of work procedures and monitor workers' emotional state.

[1346] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1347] In this invention, the server includes a means for learning information required for work (procedures, facility images, past accident cases, etc.) in advance, a means for inputting the operation screen and procedures to be performed by voice through the smart glasses when performing work, a means for determining whether the work is OK or NG and displaying the answer on the smart glasses, a means for monitoring the emotional state of the worker and pausing the work if excessive stress is felt, and a means for instructing the worker to take a break. This allows the work to be carried out while confirming the accuracy of the work procedures, and makes it possible to monitor the emotional state of the worker in real time and encourage breaks at appropriate times.

[1348] "Information necessary for work" refers to information such as procedures, equipment images, and past accident cases that are necessary when performing maintenance work.

[1349] "Means of learning" refers to a means of learning the information necessary for the work in advance, and is carried out using artificial intelligence.

[1350] "Voice input means" refers to a means by which workers can input operation screens and procedures to be performed by voice through smart glasses.

[1351] The "means for determining whether the work procedure entered is correct" is a means for determining whether the procedure is correct or not and displaying the results on the smart glasses.

[1352] The "means for monitoring emotional state" is a means for monitoring the emotional state of the worker in real time and for temporarily suspending work if excessive stress is felt.

[1353] The "means for instructing a worker to take a break" refers to a means for instructing a worker to temporarily stop work and take a break when the worker feels excessive stress.

[1354] A system for implementing this invention has the following configuration: First, a worker wears smart glasses and performs maintenance work. The smart glasses are equipped with a microphone and a camera, allowing audio and video input.

[1355] Hardware and software used

[1356] Hardware: smart glasses (e.g., Google Glass), microphone, camera

[1357] Software: Predefined API, Microsoft Azure Emotion API

[1358] Data processing and calculation

[1359] Voice input and recognition

[1360] 1. Voice input: The worker inputs maintenance procedures by voice through the microphone in the smart glasses.

[1361] 2. Speech Recognition: The server uses a predefined API to convert speech into text, which is used to verify the accuracy of the work instructions.

[1362] Procedure Judgment

[1363] 3. Procedure determination: The server determines whether the converted text is a correct procedure by comparing it with a pre-trained database of work procedures.

[1364] Emotional state monitoring

[1365] 4. Emotion analysis: The smart glasses capture facial images of workers and use the Microsoft Azure Emotion API to analyze their emotional state, focusing in particular on when their stress level exceeds a certain threshold.

[1366] Results display and break instructions

[1367] 5. Display of results: The judgement result and emotional state are displayed on the smart glasses display. If the procedure is correct, the message "Correct procedure" is displayed, and if it is incorrect, the message "Incorrect procedure" is displayed.

[1368] 6. Break instruction: If a worker feels excessive stress, the system will pause the work and instruct the worker to take a break.

[1369] Specific examples

[1370] Voice input: "Next, remove the screws."

[1371] Procedure result: "This is the correct procedure."

[1372] Sentiment analysis result: "The worker is under too much stress. Please ask them to pause their work and take a break."

[1373] Prompt Sentence Examples

[1374] Voice input: "Next, remove the screws."

[1375] Procedure result: "This is the correct procedure."

[1376] Sentiment analysis result: "The worker is under too much stress. Please ask them to pause their work and take a break."

[1377] In this way, the accuracy of work procedures can be confirmed while work is being carried out, and the emotional state of workers can be monitored in real time to encourage them to take breaks at appropriate times, thereby improving work efficiency and safety.

[1378] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1379] Step 1:

[1380] The user puts on the smart glasses and starts the maintenance work. The user inputs the work procedure by voice through the microphone of the smart glasses. The input voice data is captured by the microphone of the smart glasses.

[1381] Step 2:

[1382] The server receives the voice data sent from the smart glasses. It sends the received voice data to a specified API and converts the voice into text data. The input is voice data and the output is text data.

[1383] Step 3:

[1384] The server compares the converted text data with a pre-trained database of work procedures. Based on the comparison results, it determines whether the input procedure is correct. The input is text data, and the output is a judgment result regarding the accuracy of the procedure.

[1385] Step 4:

[1386] The server receives the user's facial image captured through the smart glasses camera. The received facial image is sent to the Microsoft Azure Emotion API to analyze the user's emotional state. The input is facial image data, and the output is the analysis result of the emotional state.

[1387] Step 5:

[1388] The server integrates the procedure judgment results and the emotional state analysis results and displays them on the smart glasses' display. If the procedure is correct, it displays "This is the correct procedure," and if it is incorrect, it displays "There is an error in the procedure." If the user is feeling excessive stress, it displays "Please pause the work and ask the user to take a break." The input is the judgment results and analysis results, and the output is the message displayed on the display.

[1389] Step 6:

[1390] The user chooses whether to continue working or take a break according to the message displayed on the smart glasses display. If the user chooses to take a break, the system pauses the work and prompts the user to take a break. The input is the message displayed on the display, and the output is the user's behavior.

[1391] Example 3

[1392] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1393] Conventional work support systems have the ability to assess the accuracy of a worker's actions and procedures, but they are unable to provide feedback that takes into account the worker's emotions and stress levels. This can increase the worker's mental burden and reduce work efficiency and safety. Furthermore, the lack of real-time pass / fail judgments and the collection and analysis of emotional data makes it difficult to quickly identify areas for improvement.

[1394] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1395] In this invention, the server includes a means for learning information required for work (procedures, facility images, past accident cases, etc.) in advance, a means for inputting the operation screen and procedures to be performed by voice through a visual device when performing work, a means for determining whether the work is OK or NG and displaying the answer on the visual device, a means for collecting and analyzing emotional data of the worker, and a means for suggesting improvements to the work based on the collected emotional data. This makes it possible to provide feedback that takes into account the worker's emotions and stress level, thereby improving work efficiency and safety.

[1396] "Information necessary for the work" refers to data necessary to carry out the work accurately and safely, such as procedures, equipment images, and past accident cases.

[1397] A "visual device" is a device worn by a worker that allows them to visually check the operation screen and the procedures that need to be carried out, and includes smart glasses and the like.

[1398] "Voice input means" refers to a technology that allows a worker to input information about an operation screen or procedures to be performed using voice, and includes voice recognition technology.

[1399] "OK / NG judgment means" is a technology for evaluating the accuracy and appropriateness of work and providing feedback to the worker on the results.

[1400] "Emotion data collection means" is a technology for detecting the emotions and stress levels of workers and collecting that data.

[1401] "Emotion data analysis means" is a technology for analyzing collected emotion data and evaluating changes in the emotions and stress levels of workers.

[1402] The "means for suggesting work improvements" is a technology for proposing improvements to work procedures and environments based on the results of analyzing emotional data.

[1403] This invention is a system for improving the work efficiency and safety of workers, and is implemented by clearly defining the roles of the server, terminal, and user.

[1404] Server Roles

[1405] The server provides a means for advance learning of the information required for the work (procedures, equipment images, past accident cases, etc.). Specifically, the server uses a database management system such as MySQL to collect and store past work data and accident cases. Next, it trains an AI model based on the collected data. A machine learning framework such as TensorFlow is used for the AI ​​model.

[1406] The server does the following:

[1407] 1. Obtain past work data and accident cases from the database.

[1408] 2. Preprocess the acquired data and convert it into a format suitable for the AI ​​model.

[1409] 3. Train the AI ​​model using the preprocessed data.

[1410] 4. Save the trained AI model and use it to judge whether the work is OK or not.

[1411] Device Role

[1412] The terminal collects data on the work performed by the worker in real time and sends it to a server. The terminal is equipped with hardware such as a camera and sensors, which are used to collect work data. The collected data is temporarily stored in the terminal and periodically uploaded to the server.

[1413] The terminal does the following:

[1414] 1. Collect work data using cameras and sensors.

[1415] 2. Temporarily store the collected data.

[1416] 3. The stored data is periodically sent to the server.

[1417] User Roles

[1418] Users can receive feedback from the system and use it to improve their work. For example, if the system judges a particular task to be unsuccessful, the user can check the reason and review the work procedure. Furthermore, if the emotion engine detects stress in the worker, the user can use that information to improve the work environment.

[1419] The user does the following:

[1420] 1. Check the feedback from the system.

[1421] 2. Review work procedures and environments based on feedback.

[1422] 3. Implement the improvement measures and receive feedback from the system again.

[1423] Specific examples

[1424] For example, consider a factory worker assembling parts. The server trains an AI model based on past assembly work data and accident cases, while the device collects data by capturing images of the worker's movements with a camera. The AI ​​model judges in real time whether the work is OK or not, and if it is not OK, notifies the user of the reason. The emotion engine analyzes the worker's facial expressions and provides information to the user if the worker is feeling stressed.

[1425] Prompt Sentence Examples

[1426] "Please design a system that trains an AI model based on past work data and accident cases, and judges whether a task is OK or not in real time. Also, please add a function that analyzes the emotions of workers and provides information if they are feeling stressed."

[1427] In this way, the roles of the server, terminal, and user are clarified, and the flow of data processing using specific hardware and software is explained, making it easier to understand the overall system. The flow of specific processing in the third embodiment will be explained with reference to FIG.

[1428] Step 1:

[1429] Data collection

[1430] The server collects past work data and accident cases. The input is data from each work station in the factory. This includes worker activity logs, sensor data from the work environment, and past accident reports. The server stores this data in a database such as MySQL. Specifically, the server connects to the database and saves the collected data.

[1431] Step 2:

[1432] Data Preprocessing

[1433] The server preprocesses the collected data. The input is the raw data collected in step 1. Specific data processing includes filling in missing values, removing outliers, and normalizing the data. The output is a dataset in a format suitable for AI models. Specifically, the server cleans the data using Python libraries and saves the preprocessed data in a new table.

[1434] Step 3:

[1435] Training an AI model

[1436] The server trains the AI ​​model using the preprocessed data. The input is the dataset preprocessed in step 2. The specific data processing involves building and training a model using a machine learning framework such as TensorFlow. The output is a trained AI model. Specifically, the server inputs the training data into TensorFlow, trains the model, and saves the trained model.

[1437] Step 4:

[1438] Real-time data collection and transmission

[1439] The terminal collects data on the work performed by the worker in real time and sends it to the server. Inputs include data on the worker's movements and the work environment. Specifically, data is collected using cameras and sensors and temporarily stored. Outputs include sending the collected data to the server. Specifically, the terminal uses cameras to capture the movements of the worker and sensors to collect data on the work environment, which is then periodically sent to the server.

[1440] Step 5:

[1441] Judging whether work is OK or NG

[1442] The server inputs the data received in real time into the AI ​​model and judges whether the work is OK or NG. The input is the real-time data collected in step 4. The specific data calculation involves inputting the data into the AI ​​model and obtaining a judgment result. The output is an OK / NG judgment result. The specific operation is that the server inputs the received data into the AI ​​model, analyzes the model's output result, judges whether the work is OK or NG, and sends the judgment result to the terminal.

[1443] Step 6:

[1444] Collecting and analyzing emotional data

[1445] The terminal collects the worker's emotional data and sends it to the server. Inputs include the worker's facial expressions and voice data. Specifically, data is collected using a camera and microphone, and then analyzed using an emotion engine. The analyzed emotional data is sent to the server as output. Specifically, the terminal takes pictures of the worker's facial expressions with a camera, records the worker's voice with a microphone, analyzes the data using an emotion engine, detects changes in emotion, and sends the data to the server.

[1446] Step 7:

[1447] Providing feedback

[1448] The user receives feedback provided by the system and uses it to improve their work. The inputs are the OK / NG judgment result from step 5 and the emotional data from step 6. Specific feedback includes suggestions for reviewing work procedures and improving the environment. The output is the implementation of improvement measures, and the system provides feedback again. The specific operation is for the user to check the feedback on the device screen, review the work procedures and environment based on the feedback, implement improvement measures, and receive feedback from the system again.

[1449] (Application example 3)

[1450] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1451] Conventional work support systems lacked sufficient accuracy in determining whether a worker was performing a task efficiently or safely, and did not provide suggestions for improvement that took into account the worker's emotions and stress. This resulted in the accumulation of stress among workers and a decline in work efficiency.

[1452] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means. In this invention, the server includes a means for learning information required for work (procedures, equipment images, past accident cases, etc.) in advance, a means for inputting the operation screen and procedures to be performed by voice through the smart glasses when performing the work, a means for determining whether the work is OK / NG and displaying the answer on the smart glasses, a means for collecting emotional data of the worker and recording and analyzing changes in emotion, and a means for suggesting improvements to the work procedures based on the emotional data and work data. This not only improves the efficiency and safety of work, but also reduces worker stress and makes it possible to improve work procedures.

[1453] "Information necessary for the work" refers to the data and knowledge necessary to carry out the work, such as work procedures, equipment images, and past accident cases.

[1454] "Means for learning" refers to a method or device for using artificial intelligence to learn in advance the information necessary for a task.

[1455] "Smart glasses" refers to eyeglass-type devices that can be worn by workers to visually display operation screens and procedures to be carried out.

[1456] "Voice input means" refers to a method or device that allows a worker to input information about the operation screen or procedures to be performed using voice.

[1457] "Means for determining whether a task is OK / NG" refers to a method or device for determining the appropriateness of the task and displaying the results on the smart glasses.

[1458] "Emotional data" refers to data that indicates the emotional state of a worker, such as the worker's heart rate or galvanic skin response.

[1459] "Means for recording and analyzing changes in emotions" refers to methods and devices for collecting emotional data from workers, recording changes in those data, and analyzing them.

[1460] The "means for proposing improvements to work procedures" refers to a method or device for finding and proposing improvements to work procedures based on emotion data and work data.

[1461] A system for implementing this invention has the following configuration: First, the server uses artificial intelligence (AI) to learn in advance the information required for the work (procedures, equipment images, past accident cases, etc.). Specifically, it learns patterns based on past work data and accident cases, and builds a model for determining whether the work is OK or not.

[1462] Next, smart glasses are used as the terminal. When worn by a worker, the smart glasses visually display the operation screen and the procedures to be performed. Using voice recognition technology, the worker can input the operation screen and the procedures to be performed by voice. This allows the worker to operate the system without using their hands, improving work efficiency.

[1463] In addition, the server collects the worker's emotional data, recording and analyzing changes in their emotions. Emotional data includes heart rate and galvanic skin response. This data is collected in real time using an emotion engine. The emotion engine analyzes the worker's emotional state and issues an alert if stress levels are increasing.

[1464] Finally, the server proposes improvements to the work process based on the emotion data and work data, thereby reducing worker stress and improving the efficiency of the work process.

[1465] As a concrete example, consider the case of assembling parts in a factory. The server learns from past assembly work data and accident cases, and builds a model to determine whether a task is OK or not. The worker wears smart glasses and inputs operation procedures by voice. The smart glasses visually display the operation screen and procedures, allowing the worker to perform operations without using their hands. The emotion engine collects the worker's heart rate and galvanic skin response in real time, and issues an alert if stress levels are rising. The server suggests improvements to the work procedures based on the emotion data and work data.

[1466] An example of a prompt sentence might be:

[1467] "Create a program that takes current task data and emotion data as input, judges whether the task is OK or NG, and issues an alert if necessary. Task data includes location data and speed data, and emotion data includes heart rate and galvanic skin response. The AI ​​model will use a random forest classifier, and the emotion engine will use the EmotionEngine class."

[1468] In this way, the mode for carrying out the invention can be concretely shown.

[1469] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1470] Step 1:

[1471] The server collects the information necessary for the work (procedures, equipment images, past accident cases, etc.) and uses artificial intelligence (AI) to learn from it. Specifically, it uses past work data and accident cases as input to learn patterns and builds a model that determines whether the work is OK or not. The output is a trained AI model.

[1472] Step 2:

[1473] The user puts on the smart glasses and starts working. The smart glasses visually display the operation screen and the procedures to be performed. As input, they receive work procedure data provided by the server, and as output, they provide visual instructions to the user.

[1474] Step 3:

[1475] The user uses voice recognition technology to input operation procedures by voice. The smart glasses convert the voice input into text data and send it to the server. The smart glasses receive the user's voice data as input and generate text data as output.

[1476] Step 4:

[1477] The server receives the text data sent from the smart glasses and uses the trained AI model to determine whether the work is OK or not.The server uses the text data and the trained AI model as input and generates a OK / NG judgment result for the work as output.

[1478] Step 5:

[1479] The server sends the OK / NG result of the work to the smart glasses and displays it visually to the user. It receives the OK / NG result as input and provides visual feedback to the user as output.

[1480] Step 6:

[1481] The server uses an emotion engine to collect the worker's emotion data (e.g., heart rate and galvanic skin response) in real time. It receives data from the emotion sensor as input and generates emotion data as output.

[1482] Step 7:

[1483] The server analyzes the collected emotion data and evaluates the worker's stress level. It uses the emotion data as input and generates a stress level evaluation result as output.

[1484] Step 8:

[1485] The server issues an alert and notifies the smart glasses if the stress level is high. The server receives the stress level assessment result as input and generates an alert notification as output.

[1486] Step 9:

[1487] The server proposes improvements to the work procedure based on the emotion data and task data. It uses the emotion data and task data as input and generates improvement proposals as output.

[1488] Step 10:

[1489] The server sends the improvement suggestions for the work procedure to the smart glasses and displays them visually to the user. The server receives the improvement suggestions as input and provides visual feedback to the user as output.

[1490] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1491] The data generation model 58 is a so-called generative AI (Artificial Intelligence).

[1492] An example of a data generation model58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ) and other generative AIs. Data generation models 58

[1493] It is obtained by performing deep learning on a neural network.

[1494] A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in the form of data such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1495] Another example of generative AI is Gemini (internet search engine). <url: https: gemini.google.com ?hl="ja">) are mentioned.

[1496] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1497] [Third embodiment]

[1498] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1499] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1500] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1501] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1502] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1503] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1504] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1505] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1506] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1507] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1508] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1509] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1510] "Example 1"

[1511] One embodiment of the present invention is a system in which workers wear smart glasses and learn information such as work procedures, facility images, and past accident cases in advance. The workers can view the operation screen through the smart glasses and input instructions by voice. This system judges whether the work is OK or NG based on the worker's voice input and displays the result on the smart glasses' display.

[1512] "Example 2"

[1513] As a specific example, consider the case where a worker is performing maintenance work on equipment. The worker wears smart glasses and inputs the maintenance work procedures by voice. The system determines whether the input procedures are correct and displays the results on the smart glasses' display. In this way, the worker can proceed with the work while checking whether their work is correct.

[1514] "Example 3"

[1515] The system of the present invention can also learn work information using artificial intelligence. The artificial intelligence learns patterns from past work data and accident cases, and judges whether a work is OK or not based on that. This allows the system's judgment accuracy to improve over time, further improving the work efficiency and safety of workers.

[1516] The processing flow of each embodiment will be described below.

[1517] "Example 1"

[1518] Step 1: The worker puts on the smart glasses.

[1519] Step 2: The smart glasses learn the information necessary for the work (procedures, equipment images, past accident cases, etc.) in advance.

[1520] Step 3: The worker can view the operation screen through the smart glasses and input instructions by voice.

[1521] Step 4: The system judges whether the task is OK or NG based on the worker's voice input, and

[1522] The results are displayed on the smart glasses display.

[1523] "Example 2"

[1524] Step 1: When a worker performs maintenance work on equipment, he or she first puts on the smart glasses.

[1525] Step 2: The worker inputs the maintenance procedure by voice.

[1526] Step 3: The system determines whether the entered steps are correct and displays the results on the smart glasses display.

[1527] Step 4: The worker performs the task based on the system's feedback.

[1528] "Example 3"

[1529] Step 1: The system of the present invention uses artificial intelligence to learn task information.

[1530] Step 2: The artificial intelligence learns patterns from past work data and accident cases.

[1531] Step 3: Based on the learned patterns, the system determines whether the work is OK or not.

[1532] Step 4: The results of the assessment are displayed on the smart glasses display, and the worker proceeds with the work based on that feedback.

[1533] Example 1

[1534] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1535] With conventional work support systems, it was difficult for workers to efficiently learn information such as procedures, equipment images, and past accident cases, and receive appropriate instructions while working. There were also issues with the accuracy of voice-input instructions and the speed with which work was judged to be OK / NG. This could potentially lead to a decline in work efficiency and safety.

[1536] The identification process by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means. In this invention, the server includes a means for learning information required for the work (procedures, equipment images, past accident cases, etc.) in advance, a means for inputting the operation screen and procedures to be performed by voice through the smart glasses when performing the work, a means for determining whether the work is OK / NG and displaying the answer on the smart glasses, a means for analyzing the voice data and converting it into text, and a means for generating prompt sentences using a generative AI model and determining whether the work is OK / NG. This allows the worker to learn information efficiently, improves the accuracy of instructions input by voice, and enables quick OK / NG determination of the work.

[1537] "Information necessary for the work" refers to the knowledge and data necessary to carry out the work, such as procedures, equipment images, and past accident cases.

[1538] "Means for learning" refers to a method or device that allows a worker to learn the information necessary for the work in advance.

[1539] "Smart glasses" are wearable devices worn by workers to display operation screens and information.

[1540] An "operation screen" is a screen that displays information such as work procedures and instructions on smart glasses.

[1541] "Voice input means" refers to a method or device that allows a worker to input instructions or information using voice.

[1542] "Means for determining OK / NG" refers to a method or device for evaluating the progress and results of work and determining whether they are appropriate.

[1543] "Means for analyzing voice data and converting it into text" refers to a method or device for converting voice data into text data using voice recognition technology.

[1544] A "generative AI model" is a model that uses artificial intelligence to generate prompt sentences and determine whether a task is OK or not.

[1545] A "prompt sentence" is an instruction sentence to be input into a generative AI model, and is the sentence that serves as the basis for determining whether a task is OK or not.

[1546] MODE FOR CARRYING OUT THE INVENTION

[1547] This invention is a system in which workers wear smart glasses and learn information such as work procedures, equipment images, and past accident cases in advance. Workers can view the operation screen through the smart glasses and input instructions by voice. The system judges whether the work is OK or NG based on the worker's voice input and displays the result on the smart glasses' display.

[1548] 1. Program Generation

[1549] The server generates a program that allows workers to wear smart glasses and learn in advance information such as work procedures, facility images, and past accident cases. This program includes functions that allow workers to view the operation screen through the smart glasses and input instructions by voice.

[1550] 2. Program processing explanation

[1551] The server installs the generated program into the smart glasses. When a worker wears the smart glasses, they display information such as work procedures, images of equipment, and past accident cases. The worker can check this information through the smart glasses' display.

[1552] When a worker inputs instructions by voice, the smart glasses send the voice data to a server, which then uses voice recognition software (e.g., voice recognition technology) to convert the voice data into text. The server then uses a generative AI model (e.g., artificial intelligence model) to generate prompts to determine whether the input instructions are acceptable or not.

[1553] The server inputs the generated prompt sentences into the AI ​​model and obtains the results that determine whether the task is OK or not. The results are then sent to the smart glasses and displayed on the smart glasses' display.

[1554] 3. Examples of concrete examples and prompts

[1555] As a concrete example, consider the case where a worker inputs a command by voice, such as "Please tell me the next step in the work." In this case, the server generates the following prompt:

[1556] Example prompt sentence:

[1557] "The worker wants to know the next step. What's the next step?"

[1558] The server inputs this prompt into the generative AI model to obtain the next steps, which are then displayed on the smart glasses screen.

[1559] In this way, workers can check information such as work procedures, equipment images, and past accident cases through smart glasses, and determine whether the work is OK or not by inputting instructions via voice.

[1560] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1561] Step 1:

[1562] Starting up and connecting to smart glasses

[1563] The user puts on the smart glasses and turns them on.

[1564] The device (smart glasses) connects to the server via Wi-Fi.

[1565] Input: Smart glasses power on, Wi-Fi connection information

[1566] Output: Connection established with server

[1567] Step 2:

[1568] Displaying work procedures and information

[1569] The server recognizes the worker's ID and sends information such as relevant work procedures, equipment images, and past accident cases to the smart glasses.

[1570] The terminal displays the received information on its screen, such as a list of work procedures or an image of the equipment.

[1571] Input: Worker ID, related information

[1572] Output: Work procedures and equipment images displayed on smart glasses

[1573] Step 3:

[1574] Entering voice commands

[1575] Users can speak into the smart glasses to input instructions, such as "Please tell me the next step."

[1576] Input: User's voice commands

[1577] Output: Audio data

[1578] Step 4:

[1579] Sending and analyzing voice data

[1580] The terminal transmits the user's voice data to the server.

[1581] The server converts the voice data into text using voice recognition software (eg, speech recognition technology).

[1582] Input: Audio data

[1583] Output: Text data

[1584] Step 5:

[1585] Judging whether work is OK or NG

[1586] The server generates a prompt based on the converted text, for example, "The worker wants to know the next step in the work. Please tell me the next step."

[1587] The server inputs this prompt into a generative AI model (e.g., an artificial intelligence model) and obtains the next work procedure and a pass / fail judgment for the work.

[1588] Input: Text data, prompt

[1589] Output: Work procedure and OK / NG judgment

[1590] Step 6:

[1591] Display of judgment results

[1592] The server sends the obtained results to the smart glasses.

[1593] The terminal displays the received results on its display, such as "The next step is to install part A" or "The work is OK."

[1594] Input: Work procedure and OK / NG judgment

[1595] Output: The results displayed on the smart glasses

[1596] (Application example 1)

[1597] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1598] In conventional factory work, workers spend a lot of time and effort checking procedures and the status of equipment. It is also difficult to refer to past accident cases, which means that work safety is not adequately ensured. Furthermore, there is a lack of means to report and properly evaluate work progress in real time, which reduces work efficiency. To solve these issues, a system is needed that allows workers to work efficiently and safely using smart glasses.

[1599] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1600] In this invention, the server includes means for learning information required for work (procedures, equipment images, past accident cases, etc.) in advance, means for inputting the operation screen and procedures to be performed by voice through the smart glasses when performing work, means for determining whether the work is OK / NG and displaying the answer on the smart glasses, means for displaying the real-time status of the equipment, means for displaying past accident cases and issuing warnings, means for reporting the progress of the work based on the voice input, means for determining whether the work is OK / NG based on the progress of the work, and means for displaying the result on the display of the smart glasses. This enables workers to obtain the necessary information in real time and perform their work efficiently and safely.

[1601] "Information necessary for work" refers to information necessary to perform work, such as work procedures, equipment images, and past accident cases.

[1602] "Means of learning in advance" refers to the means of obtaining and understanding the information necessary for the work in advance.

[1603] "Smart glasses" are eyeglass-type devices worn by workers to display information.

[1604] The "operation screen" is the interface that the worker can see through the smart glasses.

[1605] "Voice input means" refers to a means by which a worker inputs instructions or information using voice.

[1606] "Means for determining whether something is OK / NG" refers to a means for evaluating the progress and results of work and determining whether they are appropriate.

[1607] The "means for displaying the answer on the smart glasses" refers to a means for displaying the judgment result on the display of the smart glasses.

[1608] The "means for displaying the real-time status of the equipment" is a means for displaying the current status of the equipment in real time.

[1609] The "means for displaying past accident cases and calling attention" is a means for displaying past accident cases and calling attention to workers.

[1610] The "means for reporting the progress of work based on voice input" is a means for reporting the progress of work based on voice input by the worker.

[1611] "Means for determining whether work is OK or NG based on the progress of the work" refers to a means for evaluating the progress of the work and determining whether the work is being carried out appropriately.

[1612] "Means for displaying on the display of the smart glasses" refers to means for displaying the determination results and other information on the display of the smart glasses.

[1613] In the system for implementing this invention, workers wear smart glasses and can carry out their work while checking information in real time, such as work procedures, equipment status, past accident cases, etc. Specific embodiments of this system are described below.

[1614] Hardware and software used

[1615] Smart glasses: Eyeglass-type devices worn by workers to display information.

[1616] Microphone: A device for capturing voice input from the worker.

[1617] Server: A computer system that manages the information required for work and processes voice input.

[1618] speech_recognition library: Software for converting voice input into text.

[1619] smart_glasses_display module: Custom software for displaying information on a smart glasses display.

[1620] factory_system module: Custom software that obtains work procedures, equipment status, and past accident cases from the factory system and determines whether the work is OK or not.

[1621] System Operation

[1622] 1. Display of work procedures: The server displays the necessary steps for the work on the smart glasses display, allowing the worker to check the next step in real time.

[1623] 2. Checking equipment status: The server acquires the real-time status of the equipment and displays it on the smart glasses, allowing workers to continue their work while understanding the current status of the equipment.

[1624] 3. Display of past accident cases: The server displays past accident cases on the smart glasses and alerts the worker, allowing the worker to learn from past mistakes and work safely.

[1625] 4. Acquire voice input: The worker inputs instructions by voice through a microphone. The server converts the voice into text using the speech_recognition library.

[1626] 5. Work progress reporting: Based on the worker's voice input, the server reports the work progress, allowing the worker to report the progress without using their hands.

[1627] 6. Judgement of work pass / fail: The server uses the factory_system module to judge whether the work is pass / fail based on the voice input.

[1628] 7. Display of results: The results are displayed on the smart glasses screen, allowing workers to check the evaluation of their work in real time.

[1629] Specific examples

[1630] When a worker wears smart glasses and performs maintenance work on equipment, they use the application following the steps below.

[1631] 1. Voice command to the smart glasses: "Show me the next steps."

[1632] 2. Voice command to the smart glasses to "check the equipment status."

[1633] 3. Voice command to the smart glasses: "Show me past accident cases."

[1634] 4. When the task is complete, say "The task is complete."

[1635] 5. The server determines whether the work is OK or not, and the result is displayed on the smart glasses.

[1636] Prompt Sentence Examples

[1637] "Show me the next step"

[1638] Check the condition of the equipment

[1639] "Show me past accident cases"

[1640] "The work is complete."

[1641] In this way, the use of smart glasses can significantly improve work efficiency and safety within factories.

[1642] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1643] Step 1:

[1644] The server learns the information required for the work (procedures, equipment images, past accident cases, etc.) in advance. This involves acquiring data from the factory system and organizing and analyzing the information using artificial intelligence. The input is data from the factory system, and the output is organized and analyzed work procedures, equipment images, and past accident cases.

[1645] Step 2:

[1646] The user puts on the smart glasses and starts working. The server displays the work procedure on the smart glasses display. The input is the work procedure data from the server, and the output is the work procedure displayed on the smart glasses display.

[1647] Step 3:

[1648] The user inputs voice instructions through a microphone. The server converts the speech to text using the speech_recognition library. The input is the user's voice instructions and the output is the text instructions.

[1649] Step 4:

[1650] The server obtains the real-time status of the equipment based on the textual instructions and displays it on the smart glasses. The input is the textual instructions, and the output is the real-time status of the equipment displayed on the smart glasses display.

[1651] Step 5:

[1652] The server retrieves past accident cases and displays them on the smart glasses. The input is the user's voice command, and the output is the past accident cases displayed on the smart glasses display.

[1653] Step 6:

[1654] The user reports the progress of the work by voice. The server converts the voice to text using the speech_recognition library and records the progress of the work. The input is the user's voice report and the output is the progress report in text format.

[1655] Step 7:

[1656] The server judges whether the work is OK or NG based on the progress report in text format. The input is the progress report in text format, and the output is the OK / NG judgment result of the work.

[1657] Step 8:

[1658] The server displays the judgment result on the smart glasses display. The input is the OK / NG judgment result of the work, and the output is the judgment result displayed on the smart glasses display.

[1659] Example 2

[1660] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1661] In conventional maintenance work, workers are required to accurately understand and execute procedures, but it can be difficult to confirm procedures and judge their accuracy. Furthermore, workers must use their hands to confirm procedures while working, which reduces work efficiency. Even when using voice input, it can be difficult to accurately analyze voice data and determine procedures, making it difficult to obtain real-time feedback. To solve these issues, a system is needed that can confirm procedures using voice input and provide real-time feedback.

[1662] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1663] In this invention, the server includes means for learning information required for work (procedures, equipment images, past accident cases, etc.) in advance, means for inputting the operation screen and procedures to be performed by voice through the smart glasses when performing work, means for determining whether the work is OK / NG and displaying the response on the smart glasses, means for converting voice data into text data, means for analyzing the text data and determining the accuracy of the procedures, means for transmitting the determination result to the smart glasses, and means for displaying the determination result on the display of the smart glasses. This enables workers to confirm procedures using voice input and accurately proceed with maintenance work while receiving feedback in real time.

[1664] "Information necessary for the work" refers to information such as procedures required to perform maintenance work, equipment images, and past accident cases.

[1665] "Means of learning" are methods and techniques for acquiring and understanding the information necessary for a task in advance.

[1666] "Smart glasses" are wearable devices that have built-in displays and microphones and allow for voice input and information display.

[1667] The "operation screen" is a screen that appears on the smart glasses display and is used to check work procedures and other information.

[1668] "Voice input means" refers to a method or technology that allows a worker to input information about an operation screen or procedures to be performed using voice.

[1669] "Means for determining whether the procedure entered is correct" refers to the method or technology used to determine whether the procedure entered is correct.

[1670] "Means for displaying the answer" refers to the method or technology for displaying the judgment result on the smart glasses display.

[1671] A "means for converting voice data to text data" is a method or technique for converting voice input into text form.

[1672] The "means for analyzing text data to determine the accuracy of procedures" refers to a method or technology for analyzing text data and determining whether the entered procedures are correct.

[1673] "Means for transmitting the determination result to the smart glasses" refers to a method or technology for transmitting the determination result of the correctness of the procedure to the smart glasses.

[1674] "Means for displaying the determination result on the display of the smart glasses" refers to a method or technology for displaying the determination result on the display of the smart glasses.

[1675] MODE FOR CARRYING OUT THE INVENTION

[1676] This invention provides a system that, when a worker performs maintenance work on equipment, uses smart glasses to input procedures by voice and determines the accuracy of the procedures in real time. Specific embodiments of this system are described below.

[1677] Hardware and software used

[1678] Smart glasses: A wearable device with a built-in display and microphone that allows voice input and information display.

[1679] Server: A computer system that analyzes voice data, determines procedures, and transmits results.

[1680] Speech recognition technology for converting voice data into text data.

[1681] Generative AI model (e.g., OpenAI's GPT-4): An artificial intelligence model that analyzes text data and determines the accuracy of procedures.

[1682] System Operation

[1683] 1. Voice input:

[1684] The user puts on the smart glasses and inputs maintenance instructions by voice, such as "remove the filter."

[1685] 2. Sending audio data:

[1686] The device (smart glasses) captures audio data through a built-in microphone and transmits the data to a server via Wi-Fi or Bluetooth.

[1687] 3. Audio data conversion:

[1688] The server converts the received voice data into text data using a predefined API, which provides highly accurate voice recognition.

[1689] 4. Procedure determination:

[1690] The server uses a generative AI model (e.g., OpenAI's GPT-4) to determine whether the maintenance procedures received as text data are correct, which involves checking them against a database of predefined correct procedures.

[1691] 5. Sending the results:

[1692] The server determines the accuracy of the procedure and sends the results to the smart glasses, again via Wi-Fi or Bluetooth.

[1693] 6. Displaying the results:

[1694] The device (smart glasses) displays the received judgment result on its display. For example, it may display a message such as "The procedure is correct. Please proceed to the next step."

[1695] Specific examples

[1696] As a specific example, consider the case where a worker changes an air conditioner filter.

[1697] 1. The user puts on the smart glasses and says, "Remove filter."

[1698] 2. The device (smart glasses) sends this voice data to the server.

[1699] 3. The server uses a specified API to convert the voice data into text data that says "Remove filter."

[1700] 4. The server uses a generative AI model to determine whether this text data is a valid instruction, for example, by checking whether the instruction "remove filter" exists in the database.

[1701] 5. The server sends the result of the judgment that "the procedure is correct" to the smart glasses.

[1702] 6. The device (smart glasses) displays the result of this judgment on the screen and informs the user, "The procedure is correct. Please proceed to the next step."

[1703] Prompt Sentence Examples

[1704] User: "Remove filter"

[1705] Server: "Your steps are correct. Please proceed to the next step."

[1706] User: "Install a new filter"

[1707] Server: "The procedure was correct. The maintenance work has been completed."

[1708] In this way, the user can receive real-time feedback and perform maintenance tasks accurately.

[1709] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1710] Program processing flow

[1711] Step 1: Voice Input

[1712] The user puts on the smart glasses and inputs maintenance instructions by voice, such as "remove the filter."

[1713] Input: User's voice command

[1714] Output: Audio data captured by the smart glasses microphone

[1715] Step 2: Sending audio data

[1716] The device (smart glasses) captures audio data through a built-in microphone and transmits it to a server via Wi-Fi or Bluetooth.

[1717] Input: Audio data captured by the smart glasses microphone

[1718] Output: Audio data sent to the server

[1719] Step 3: Convert the audio data

[1720] The server converts the received voice data into text data using a predefined API, which provides highly accurate voice recognition.

[1721] Input: Audio data sent to the server

[1722] Output: Text data (e.g. "Remove filter")

[1723] Step 4: Determine the procedure

[1724] The server uses a generative AI model (e.g., OpenAI's GPT-4) to determine whether the maintenance procedures received as text data are correct, which involves checking them against a database of predefined correct procedures.

[1725] Input: Text data (e.g., "Remove filter")

[1726] Output: Verification result (e.g. "The procedure is correct")

[1727] Step 5: Sending the results

[1728] The server determines the accuracy of the procedure and sends the results to the smart glasses, again via Wi-Fi or Bluetooth.

[1729] Input: Verification result (e.g., "The procedure is correct")

[1730] Output: The judgment result sent to the smart glasses.

[1731] Step 6: View the results

[1732] The device (smart glasses) displays the received judgment result on its display. For example, it may display a message such as "The procedure is correct. Please proceed to the next step."

[1733] Input: The judgment result sent to the smart glasses

[1734] Output: A message displayed on the smart glasses display (e.g., "Your steps are correct. Please proceed to the next step.")

[1735] Specific examples of operation

[1736] 1. User: Say "Remove filter."

[1737] 2. Device (smart glasses): Sends voice data to the server.

[1738] 3. Server: Converts the audio data into text data using a specified API.

[1739] 4. Server: Analyzes the text data using a generative AI model to determine the accuracy of the steps.

[1740] 5. Server: Sends the judgment results to the smart glasses.

[1741] 6. Device (smart glasses): Displays the judgment result on the screen and informs the user, "The procedure is correct. Please proceed to the next step."

[1742] In this way, the user can receive real-time feedback and perform maintenance tasks accurately.

[1743] (Application example 2)

[1744] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1745] In conventional maintenance work, workers are required to accurately understand and carry out procedures, but there is a lack of means to confirm procedures and the accuracy of work in real time, which can lead to work errors and reduced efficiency. There is also a need for a system that allows workers to input procedures by voice and instantly confirm their accuracy. This will improve the efficiency and accuracy of work.

[1746] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1747] In this invention, the server includes a means for learning information required for work (procedures, facility images, past accident cases, etc.) in advance, a means for inputting the operation screen and procedures to be performed by voice through the smart glasses when performing work, a means for determining whether the work is OK / NG and displaying the answer on the smart glasses, and a means for determining the procedures input by voice in real time and displaying the result on the display of the smart glasses. This enables workers to efficiently carry out maintenance work while checking the accuracy of the work in real time.

[1748] "Information necessary for the work" refers to information that is necessary when performing maintenance work, such as procedures, equipment images, and past accident cases.

[1749] "Means of learning" refers to a means of learning the information necessary for the work in advance, and is carried out using artificial intelligence.

[1750] "Smart glasses" are eyeglass-type devices that can be worn by workers to display operation screens and procedures to be carried out, and can also accept voice input.

[1751] The "operation screen" is the screen displayed on the smart glasses, which the worker refers to when performing maintenance work.

[1752] The "means for inputting by voice" refers to a means for an operator to input maintenance procedures by voice, and is carried out using voice recognition technology.

[1753] The "means for determining whether the maintenance procedure entered is correct" is a means for determining whether the maintenance procedure entered is correct.

[1754] The "means for displaying the answer" is a means for displaying the judgment result on the display of the smart glasses.

[1755] The "means for determining in real time" is a means for instantly determining the procedure input by voice and displaying the result in real time.

[1756] A "display" is a display device installed in smart glasses that displays the judgment results and operation screen.

[1757] The system for implementing this invention allows a worker to wear smart glasses and, when performing maintenance work, input procedures by voice, and the system determines in real time whether the procedures are correct and displays the results on the smart glasses' display.

[1758] System configuration

[1759] The system consists of the following main components:

[1760] 1. Smart glasses: A glasses-type device worn by workers, equipped with a display that shows the operation screen and judgment results, as well as a microphone that accepts voice input.

[1761] 2. Server: Uses voice recognition technology and artificial intelligence to process the input procedures.

[1762] 3. Speech recognition software: Software that converts voice input into text, such as Google's speech recognition API.

[1763] 4. Artificial intelligence model: A model that learns the information necessary for the task in advance and determines whether the input procedures are correct.

[1764] Program processing

[1765] The server works as follows:

[1766] 1. Acquiring voice input: Acquire the worker's voice through the microphone in the smart glasses.

[1767] 2. Speech recognition: Convert captured speech into text using speech recognition software.

[1768] 3. Procedure validation: The converted text is input into an artificial intelligence model to determine whether the procedure is correct.

[1769] 4. Displaying the results: The results are displayed on the smart glasses screen.

[1770] Hardware and software used

[1771] Hardware: Smart glasses, microphone

[1772] Software: Python, SpeechRecognition library, smart glasses SDK, artificial intelligence model (e.g., TensorFlow)

[1773] Specific examples

[1774] For example, when a technician performs maintenance on a robot, they can say, "Turn off the robot, remove the cover, and inspect the internal components." This speech is picked up by the microphone in the smart glasses and converted into text by speech recognition software. The converted text is sent to a server, where an artificial intelligence model determines the accuracy of the procedure. The result is displayed on the smart glasses' display, saying, "The procedure is correct."

[1775] Prompt Sentence Examples

[1776] Turn off the robot, remove the cover, and inspect the internal components.

[1777] In this way, workers can efficiently carry out maintenance work while checking the accuracy of the work in real time.

[1778] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1779] Step 1:

[1780] The user puts on the smart glasses and begins maintenance work. The user then speaks maintenance procedures into the microphone on the smart glasses. The voice input contains specific instructions, such as "Turn off the robot's power, remove the cover, and inspect the internal parts." Input: User's voice. Output: Voice data.

[1781] Step 2:

[1782] The smart glasses send the captured voice data to the server, which then uses voice recognition software (e.g., Google's voice recognition API) to convert the voice data into text data. Input: Voice data. Output: Text data.

[1783] Step 3:

[1784] The server inputs the converted text data into an AI model. The AI ​​model compares the input data with a database of previously learned maintenance procedures and determines whether the procedure is correct. Input: Text data. Output: Verification result (correct / incorrect).

[1785] Step 4:

[1786] The server sends the judgment result to the smart glasses. The smart glasses display the judgment result on the display. For example, a message such as "The procedure is correct" or "The procedure is incorrect. Please check again" is displayed. Input: Judgment result. Output: Message displayed on the display.

[1787] Step 5:

[1788] The user checks the judgment results displayed on the smart glasses display and corrects the procedure if necessary. If the correct procedure is displayed, the user continues working. Input: Message displayed on the display. Output: User action (correcting the procedure or continuing the work).

[1789] In this way, the user can proceed with the work while checking the accuracy of the maintenance procedure in real time.

[1790] Example 3

[1791] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1792] Conventional work support systems require workers to manually judge the safety and efficiency of their work, which increases the risk of work errors and accidents. Furthermore, there was a lack of effective ways to utilize past accident cases and work data, which led to insufficient work improvements and safety measures. Furthermore, real-time work support was difficult, placing a heavy burden on workers.

[1793] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1794] In this invention, the server includes a means for acquiring past work data and accident cases from a database and performing preprocessing, a means for training an AI model using the preprocessed data, and a means for inputting new work data and using the AI ​​model to determine whether the work is OK or not. This makes it possible to effectively utilize past data and automatically determine the safety and efficiency of work in real time.

[1795] "Information necessary for the work" refers to the data and knowledge necessary to carry out the work, such as procedures, equipment images, and past accident cases.

[1796] "Smart glasses" refers to a wearable device that can be worn by workers to visually display the operation screen and procedures to be performed.

[1797] "Voice input means" refers to technology that allows workers to give operational instructions and input data using voice.

[1798] "Means for determining whether something is OK / NG" refers to the technology used to evaluate the safety and efficiency of work and determine the results as OK or NG.

[1799] A "database" refers to an information management system that systematically stores past work data and accident cases, and allows them to be searched and retrieved as needed.

[1800] "Preprocessing means" refers to technology that processes collected data by completing missing values, normalizing it, extracting features, etc., and converting it into a format suitable for an artificial intelligence model.

[1801] An "artificial intelligence model" refers to a model that uses machine learning algorithms to learn data and perform a specific task (in this case, determining whether a task is OK or not).

[1802] "Training means" refers to techniques for training an artificial intelligence model using preprocessed data to improve its performance.

[1803] "New work data" refers to information about current or future work, and the artificial intelligence model uses this information to determine whether the work is OK or not.

[1804] "Feedback means" refers to technology that notifies users of the results determined by the artificial intelligence model and encourages them to improve their work and take safety measures.

[1805] The present invention is a system for improving work safety and efficiency, and can be implemented as follows.

[1806] First, users use a development environment such as Python or TensorFlow to generate the system's program. The program includes code for building an artificial intelligence (AI) model and learning from past work data and accident cases.

[1807] The server retrieves past work data and accident case studies provided by the user from a database (e.g., MySQL). The server then preprocesses this data and converts it into a format suitable for the AI ​​model. Specifically, the server performs data cleaning, normalization, feature extraction, etc.

[1808] The server then trains the AI ​​model using a machine learning library such as TensorFlow. Once training is complete, the server receives new task data as input and uses the AI ​​model to determine whether the task is OK or not.

[1809] As a concrete example, consider the case of using factory work data. Users input past work data (e.g., work hours, equipment used, years of experience of workers) and accident cases (e.g., type, cause, and impact of accidents) into the system. The server learns from this data, and when new work data is input, it determines whether the work is safe.

[1810] Example prompt sentence:

[1811] "Based on past work data and accident cases, determine whether your current work is safe."

[1812] This system is expected to improve the work efficiency and safety of workers. The flow of the identification process in the third embodiment will be described with reference to FIG.

[1813] Step 1: Collect data

[1814] Users upload past work data and accident cases to the system. Specifically, data is extracted from CSV files and databases. The input information includes work hours, equipment used, years of experience of workers, type of accident, cause, and impact. The output is that this data is saved on the server.

[1815] Step 2: Preprocessing the data

[1816] The server preprocesses the collected data. Specifically, it imputes missing values, normalizes data, and encodes categorical data. The input is the raw data collected in step 1. The output is a preprocessed dataset. For example, it imputes missing values ​​with the mean, standardizes work time, and one-hot encodes the equipment used.

[1817] Step 3: Training the AI ​​model

[1818] The server trains an AI model using the preprocessed data, specifically using a machine learning library such as TensorFlow. The input is the preprocessed dataset from step 2. The output is the trained AI model. Training can take several hours.

[1819] Step 4: Entering work data

[1820] Users input new work data into the system. Specifically, this involves real-time data entry and batch processing. The input includes information such as the start time of the new work, the equipment used, and the worker's years of experience. The output is that this data is stored on the server.

[1821] Step 5: Judging whether the work is OK or NG

[1822] The server passes the new work data that was input to the AI ​​model, which judges whether the work is OK or not. Specifically, the AI ​​model evaluates the safety and efficiency of the work. The input is the new work data that was input in step 4. The output is the OK / NG judgment result for the work. For example, a judgment may be made that "the work time is too long, so it is not OK."

[1823] Step 6: Feedback on results

[1824] The server feeds back the judgment results to the user. Specifically, it displays notifications from the system and a dashboard. The input is the judgment results obtained in step 5. The output is feedback information for the user. The user reviews the work plan based on these results. For example, they can take measures such as "shortening work time" or "changing the equipment used."

[1825] (Application example 3)

[1826] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1827] Conventional work support systems required workers to study procedures, equipment images, and past accident examples in advance, but they did not acquire work data in real time or judge whether a task passed or failed based on past data. This resulted in issues such as insufficient improvement in work efficiency and safety. Furthermore, there was a need to improve the accuracy of judgments when workers input operation screens and procedures to be performed by voice through smart glasses.

[1828] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1829] In this invention, the server includes a means for learning information required for work (procedures, facility images, past accident cases, etc.) in advance, a means for inputting the operation screen and procedures to be performed by voice through the smart glasses when performing work, a means for determining whether the work is OK or not and displaying the answer on the smart glasses, and a means using artificial intelligence for acquiring work data in real time and determining whether the work is OK or not based on past data and accident cases, which makes it possible to improve work efficiency and safety.

[1830] "Information necessary for the work" refers to information such as procedures required to perform the work, equipment images, and past accident cases.

[1831] "Means for learning in advance" refers to means for learning the information necessary for the work in advance.

[1832] "Smart glasses" are eyeglass-type devices that can be worn by workers to display operation screens and procedures to be carried out.

[1833] The "operation screen" is the screen that the worker can view through the smart glasses.

[1834] "Procedures to be implemented" are specific procedures that must be followed when performing a task.

[1835] "Voice input means" refers to a means by which a worker uses voice to input information about the operation screen and procedures to be carried out.

[1836] "Means for determining whether something is OK or not" refers to a means for determining whether the results of the work are appropriate or not.

[1837] The "means for acquiring task data in real time" refers to a means for acquiring data in real time while the task is being performed.

[1838] "Past data" refers to data on work that has been done in the past.

[1839] "Accident cases" are specific examples of accidents that have occurred in the past.

[1840] "Means using artificial intelligence" refers to means for determining whether a task is OK or not using artificial intelligence technology.

[1841] An embodiment of the present invention will be described.

[1842] First, the server has a means of learning the information required for the work (procedures, equipment images, past accident cases, etc.) in advance. This is done using artificial intelligence technology. Specifically, it uses machine learning libraries such as TensorFlow to learn from past work data and accident cases and model work patterns.

[1843] Next, when performing work, the user wears smart glasses. Smart glasses are devices that display operation screens and procedures to be performed, and have a means of inputting instructions by voice using voice recognition technology. When the user inputs operation instructions by voice, the voice data is sent to the server, which then converts it into text data using voice recognition technology.

[1844] The server has a means of acquiring work data in real time. Specifically, it monitors the progress of work in real time through cameras and sensors installed in the smart glasses. This data is sent to the server, which has a means of using artificial intelligence to judge whether the work is OK or not based on past data and accident cases. The AI ​​model is built using machine learning libraries such as TensorFlow, and judges whether the work is OK or not based on the data acquired in real time.

[1845] The results are displayed on the smart glasses, allowing users to check the progress of their work in real time and make corrections as necessary, improving work efficiency and safety.

[1846] As a concrete example, consider a robot assembling parts in a factory. In this case, cameras and sensors installed on the robot collect work data in real time and input that data into an AI model. The AI ​​model judges whether the work is OK or not based on past data and accident cases, and displays the result in real time on the smart glasses.

[1847] Examples of prompts to be input to a generative AI model include:

[1848] "Robots are assembling parts in a factory. Please build an AI system that will judge whether a task is OK or not based on real-time work data and past data and accident cases."

[1849] The above is an embodiment of the present invention.

[1850] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1851] Step 1:

[1852] The server learns in advance the information required for the work (procedures, equipment images, past accident cases, etc.). Specifically, it uses machine learning libraries such as TensorFlow to learn from past work data and accident cases and model work patterns. The input is past work data and accident cases, and the output is the trained AI model.

[1853] Step 2:

[1854] The user puts on the smart glasses and begins work. The smart glasses are a device that displays the operation screen and procedures to be performed. The input is the user's voice instructions, and the output is the operation screen and procedures displayed on the smart glasses.

[1855] Step 3:

[1856] The user inputs operation instructions by voice. The smart glasses use voice recognition technology to convert the voice into text data and send that data to the server. The input is the user's voice instructions, and the output is text data.

[1857] Step 4:

[1858] The server acquires work data in real time. The progress of work is monitored in real time through the cameras and sensors installed in the smart glasses. The input is real-time data from the cameras and sensors, and the output is work data sent to the server.

[1859] Step 5:

[1860] The server inputs the work data acquired in real time into the AI ​​model and judges whether the work is OK or not. The AI ​​model judges whether the work is OK or not based on past data and accident cases. The input is the work data acquired in real time, and the output is the OK or not judgment result of the work.

[1861] Step 6:

[1862] The server sends the judgment result to the smart glasses and displays it to the user. This allows the user to check the progress of the work in real time and make corrections as necessary. The input is the OK / NG judgment result of the work, and the output is the judgment result displayed on the smart glasses.

[1863] The above are the specific processing steps for carrying out the present invention.

[1864] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1865] "Example 1"

[1866] One embodiment of the present invention is a system that combines an emotion engine that recognizes the emotions of workers. In this system, a worker wears smart glasses and, as he or she works, the emotion engine recognizes the worker's emotions from the worker's tone of voice and facial expressions. For example, if the system senses that the worker is nervous, it can adjust work instructions according to the worker's emotions, such as slowing down the pace of work.

[1867] "Example 2"

[1868] The emotion engine can also take measures such as pausing work if a worker's emotions exceed a certain threshold. For example, if the system determines that a worker is feeling excessively stressed, it can pause work and instruct the worker to take a break.

[1869] "Example 3"

[1870] Furthermore, the emotion engine can record changes in a worker's emotions and analyze the data to identify areas for improvement in the work. For example, if a worker tends to feel stressed during a particular task, it can suggest improvement measures such as reviewing the procedures for that task.

[1871] The processing flow of each embodiment will be described below.

[1872] "Example 1"

[1873] Step 1: The worker puts on the smart glasses.

[1874] Step 2: When you start working, the emotion engine recognizes your emotions from your tone of voice and facial expressions.

[1875] Step 3: The emotion engine adjusts the work instructions according to the worker's emotions. For example, if the worker feels nervous, the system will adjust the work pace, such as slowing it down. "Example 2"

[1876] Step 1: The worker puts on the smart glasses and begins work.

[1877] Step 2: The emotion engine recognizes the worker's emotions and pauses the work if the emotions exceed a certain threshold.

[1878] Step 3: After pausing work, the system prompts the worker to take a break.

[1879] "Example 3"

[1880] Step 1: The worker puts on the smart glasses and begins work.

[1881] Step 2: The emotion engine records changes in the worker's emotions.

[1882] Step 3: The system analyzes the recorded emotional data and identifies areas for improvement in the work. For example, if a worker tends to feel stressed during a particular task, it suggests improvements such as reviewing the procedures for that task.

[1883] Example 1

[1884] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1885] Conventional work support systems allow workers to learn information such as work procedures, equipment images, and past accident cases in advance, but they are unable to respond to emotional changes during work. As a result, it is difficult to provide appropriate instructions when workers feel nervous or stressed, which can lead to reduced work efficiency and safety.

[1886] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1887] In this invention, the server includes a means for learning information required for work (procedures, facility images, past accident cases, etc.) in advance, a means for inputting the operation screen and procedures to be performed by voice through the smart glasses when performing work, a means for determining whether the work is OK or NG and displaying the answer on the smart glasses, and a means for recognizing the worker's emotions and adjusting work instructions according to those emotions. This makes it possible to respond to changes in the worker's emotions and provide appropriate instructions.

[1888] "Information necessary for work" is a general term for data and materials necessary to carry out work, such as work procedures, equipment images, and past accident cases.

[1889] "Smart glasses" are a type of wearable device that has a built-in display, camera, microphone, etc. and is worn by workers.

[1890] The "operation screen" is an interface that is displayed on the smart glasses display and contains information such as work procedures and instructions.

[1891] "Voice input means" refers to a technique that allows a worker to input instructions or information through voice, and typically uses voice recognition technology.

[1892] The "means for determining whether something is OK or not" refers to an algorithm or program that analyzes the worker's voice input and the work content, and determines whether the results are correct.

[1893] "Means for recognizing emotions" refers to technology that analyzes the tone of a worker's voice and facial expressions to identify emotions such as tension or stress.

[1894] The "means for adjusting work instructions" refers to an algorithm or program for changing the pace of work or the content of instructions in response to the worker's recognized emotions.

[1895] MODE FOR CARRYING OUT THE INVENTION

[1896] This invention is a system in which workers wear smart glasses and learn information such as work procedures, facility images, and past accident cases in advance. Workers can check the operation screen through the smart glasses and input instructions by voice. In addition, by combining it with an emotion engine, it has the function of recognizing the worker's emotions and adjusting work instructions.

[1897] Hardware and software used

[1898] Server: A relational database such as MySQL is used to store information such as work procedures, equipment images, and past accident cases.

[1899] Devices (smart glasses): Use wearable devices such as Google Glass and Microsoft HoloLens.

[1900] Speech recognition software: Converts speech to text using a predefined API.

[1901] Emotion recognition software: Uses a system that combines Microsoft Azure's Emotion API, OpenCV, and deep learning models.

[1902] Specific operation of the system

[1903] 1. The server stores information such as work procedures, equipment images, and past accident cases in a database. For example, text data of work procedures and image files of equipment are stored in the database.

[1904] 2. When a worker wears the device (smart glasses), it connects to the server via Wi-Fi and requests the necessary information. In response to the request, the server sends w...

Claims

1. A first means for displaying information including a work procedure, an equipment image, and past accident cases on smart glasses worn by a worker; A second means for receiving a voice input related to the work by the worker; a third means for generating a prompt sentence for outputting a next work procedure based on the contents of the voice input by the worker; A fourth means for inputting the generated prompt sentence into a generative AI model to generate the next work procedure; A fifth means for displaying the generated next work procedure on a display of the smart glasses; A system including:

2. a sixth means for recognizing the emotion of the worker from the tone of voice and facial expression of the worker using an emotion engine; A seventh means for adjusting a work instruction to the worker based on the feeling of the worker; The system of claim 1 , comprising:

3. and an eighth means for instructing the worker to stop the work and take a break when the evaluation result of the worker's emotion exceeds a set standard. The system of claim 2.

Images