system

Abstract

We provide the system. [Solution] In the on-site work environment, an acquisition means for collecting worker motion data and environmental data, An analysis means that analyzes the data transmitted from the acquisition means and generates the optimal work procedure, A presentation means for visually and audibly presenting the work procedure generated by the analysis means to the worker, A feedback means that receives the worker's feedback and applies it to the analysis means, A guide means that displays the mounting position and angle of a structure as a guideline, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance as a response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] At construction sites, due to the decrease in skilled workers and communication problems in multilingual work environments, the reduction of work efficiency and safety has become a problem. Under such circumstances, there is a need for a technology that enables even inexperienced workers to perform work with high quality and safety.

Means for Solving the Problems

[0005] This invention provides a system comprising means for acquiring worker motion data and environmental data, and means for analyzing the acquired data to generate an optimal work procedure. Furthermore, by combining means for visually and audibly presenting the generated work procedure with means for receiving feedback from workers and incorporating it into the analysis, even inexperienced workers can perform tasks at the level of skilled workers. The aim is to improve efficiency and safety at construction sites.

[0006] "Acquisition means" refers to a device or method for collecting worker motion data and environmental data in the on-site work environment.

[0007] "Analysis means" refers to an apparatus or method for generating an optimal work procedure based on data collected from acquisition means.

[0008] "Presentation means" refers to a device or method that presents the work procedure generated by the analysis means to the worker visually and audibly.

[0009] "Feedback means" refers to a device or method for receiving feedback from an operator and applying it to an analysis means.

[0010] An "augmented reality / virtual reality device" is a device that presents computer-generated visual information to a worker, overlaid onto the real environment.

[0011] "Motion data" refers to information that represents the physical movements and gestures of a worker.

[0012] "Environmental data" refers to information that describes the physical conditions and surrounding environment of the work site.

[0013] "Work procedures" refer to information that includes specific instructions on the actions and processes that workers should perform. [Brief explanation of the drawing]

[0014] [Figure 1]It is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Embodiments for Carrying Out the Invention

[0015] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, the labeled processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0018] In the following embodiments, the labeled RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0019] In the following embodiments, the labeled storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0020] In the following embodiments, the labeled communication I / F (Interface) is an interface including a communication processor and an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark), and the like.

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0022] [First Embodiment]

[0023] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0024] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0027] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0030] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0034] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0035] This invention provides a system designed to enable workers in the field to efficiently perform complex tasks. This system consists of three main elements: a server, a terminal, and a user.

[0036] First, the terminal uses an AR / VR device to collect motion data of the user working on-site and data about the surrounding environment. For example, it collects how the user is using a particular tool, including its movements and operation, and records this data digitally within the terminal. The terminal then transmits this data to a server.

[0037] The server receives and analyzes data sent from the terminal. Using state-of-the-art machine learning algorithms and computer vision technology, it recognizes user actions and generates the optimal procedure for the task. For example, in a specific welding process, it can automatically create a procedure that combines a 3D model with voice instructions to guide the user in making fine adjustments to the timing and angle of their actions.

[0038] The generated instructions are returned to the terminal. The terminal displays these as visual information overlaid on the user's field of view, and simultaneously provides specific guidance to the user using voice prompts. For example, if the welding position is inaccurate, visual guidelines and target marks are displayed on the AR display, allowing the user to correct it immediately.

[0039] Users perform tasks based on instructions received visually and audibly. This system allows even inexperienced individuals to achieve results comparable to those of skilled workers. If a user has questions or requires further instructions during the task, they can provide feedback using voice commands or gestures, which is sent to the server and incorporated into subsequent instructions.

[0040] In this way, the system achieves improved skills and optimized work efficiency in the workplace through close collaboration between servers, terminals, and users.

[0041] The following describes the processing flow.

[0042] Step 1:

[0043] The device uses AR / VR equipment to collect user motion data and surrounding environment data in real time. Specifically, it uses cameras and sensors to acquire data such as the user's hand movements, gaze, and the arrangement of objects around the user.

[0044] Step 2:

[0045] The terminal bundles the collected data into packets and sends them to the server via low-latency communication. This transmission process is designed to ensure data accuracy and timely delivery.

[0046] Step 3:

[0047] The server analyzes the received motion and environmental data. Using machine learning algorithms, it identifies the user's motion patterns and surrounding conditions from the data and generates optimal instructions necessary for the task. For example, it identifies the appropriate tool usage and operating angles.

[0048] Step 4:

[0049] The server reconstructs the generated instructions as data and returns it to the terminal. This data includes visual guidelines and audio instruction files.

[0050] Step 5:

[0051] The terminal displays the received instruction data on the user's AR display and provides supplementary information via audio output. This allows the user to proceed with their work while confirming the work procedure in real time.

[0052] Step 6:

[0053] The user performs the task according to the instructions provided. They perform or modify the instructions as needed, and provide feedback to the device via voice or gestures if they have any questions.

[0054] Step 7:

[0055] The terminal digitizes user feedback and sends it to the server. This data is used to improve instructions and is utilized for analysis in future sessions.

[0056] (Example 1)

[0057] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0058] In on-site work, the skill level and efficiency of workers directly impact the quality of the work. However, not all workers are skilled, and there is a need for support to enable less skilled workers to perform work efficiently and with high quality. Furthermore, since the work environment changes constantly, a system that can provide appropriate instructions in real time is necessary. Conventional methods have the problem of insufficient speed and accuracy in providing feedback to workers, making immediate response on-site difficult.

[0059] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0060] In this invention, the server includes means for collecting operational information and surrounding information in the work environment, means for transmitting the information using communication technology and analyzing it in a central processing unit, and means for generating work procedures using machine learning and visual recognition technology. This provides real-time feedback tailored to the worker's skills and environment, enabling even inexperienced workers to efficiently perform high-quality work.

[0061] "Work environment" refers to the physical or virtual space in which work is performed, and the place where the worker carries out their actions.

[0062] A "worker" refers to an individual who receives instructions through a system and actually performs the work.

[0063] A "visual device" refers to a device that displays digital information on a screen, and can overlay information onto the real-world environment.

[0064] "Motion information" refers to data related to the physical movements of workers and specific actions taken during their work.

[0065] "Ambient information" refers to data that indicates the conditions and circumstances of the work environment, including light intensity, temperature, and humidity.

[0066] "Communication technology" refers to wireless or wired technologies for efficiently and securely sending and receiving data.

[0067] A "central processing unit" refers to the main computer system that receives collected data and performs analysis and processing on it.

[0068] "Machine learning" refers to the technology that enables computers to learn patterns from data and make decisions and predictions.

[0069] "Visual recognition technology" refers to the technology of extracting and understanding specific information or patterns from images and videos.

[0070] A "work procedure" is a set of instructions outlining the steps necessary to perform a specific task efficiently and safely.

[0071] "Visual information" refers to images and video content displayed to the user.

[0072] "Audio information" refers to instructions and guidance provided to users via voice.

[0073] "Response" refers to the reaction that arises from the operator's actions or feedback.

[0074] This invention is a system that provides support for workers to perform their tasks efficiently. The system mainly consists of three components: a server, a terminal, and a user.

[0075] The terminal uses a visual device worn by the worker, specifically an AR / VR device, to collect information about the work environment, including movement and surroundings. This terminal captures the user's hand movements and tool usage in real time, and also measures ambient brightness and temperature. For example, when a user welds an object, the terminal meticulously records their hand movements and tool operation.

[0076] The server receives information sent from the terminal and performs advanced analysis. Specifically, it uses machine learning algorithms implemented in programming languages ​​such as Python and the open-source library OpenCV to analyze user actions in detail. Based on this data, the server generates the optimal work procedure. For example, a high-performance server equipped with an 856 processor will combine historical data with current environmental conditions to determine the most effective procedure for the user.

[0077] The generated instructions are sent to a terminal and presented to the user via a visual device. The terminal overlays visual information such as 3D models and arrows onto the worker's field of view, while simultaneously providing voice instructions. This allows the user to obtain information through multiple senses, improving work efficiency. For example, a user performing gardening tasks might receive voice guidance such as, "Next, cut the left branch at a 45-degree angle," while the terminal visually highlights the branch's position.

[0078] Users perform tasks by following explicit visual and auditory instructions. If necessary, they can send feedback to the terminal using voice commands or hand gestures. This feedback information is analyzed by the server and used to generate the next steps. For example, a possible prompt might be, "Suggest a new welding process step."

[0079] This system will enable improvements in technical skills and optimization of work efficiency at the work site.

[0080] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0081] Step 1:

[0082] The terminal collects motion and ambient information through the user's visual device. Inputs include the user's hand movements, tool usage, and ambient light levels and temperature. This data is converted into a digital format optimized for motion analysis and stored in a data buffer. Specifically, a camera captures the user using a tool, providing this footage as foundational data for real-time analysis.

[0083] Step 2:

[0084] The terminal transmits the collected data to the server using communication technology. The input consists of the operational information and surrounding information collected in step 1. The data is securely transmitted to the server via encryption technology such as TLS. The specific role of the terminal is to batch process the data at regular intervals and efficiently transmit it to the server over the network.

[0085] Step 3:

[0086] The server receives and analyzes data sent from the terminal. The input includes user action information and environmental information. The server processes this data using machine learning algorithms and visual recognition technology. Through this processing, it models the user's work actions and generates the optimal work procedure. Specifically, it uses Python and OpenCV to extract the characteristics of the user's movements.

[0087] Step 4:

[0088] The server sends the generated work procedure to the terminal. The input is the optimal procedure obtained through machine learning. The server uses this to construct work instructions in natural language and a 3D model. The output is visual and audio information presented to the user.

[0089] Step 5:

[0090] The terminal presents instructions received from the server to the user. It receives visual and audio instruction data from the server as input. The output consists of guidelines and audio guidance displayed overlaid on the visual device. Specific actions include displaying arrows on the AR display and playing instructions through the speaker.

[0091] Step 6:

[0092] The user performs the task according to the presented visual and audio instructions. The user utilizes instructions received from the terminal as input. As the user progresses, they send feedback to the terminal via voice commands or gestures if they have any questions. Specific actions include operating tools according to instructions and continuously providing feedback on the status to the terminal.

[0093] Step 7:

[0094] The terminal receives feedback from the user and sends it to the server. Input is the user's voice or gesture response. Output is data sent to the server prompting further analysis and instruction generation. The terminal's operation involves quickly detecting feedback and preparing data for the next cycle.

[0095] (Application Example 1)

[0096] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0097] In on-site work, even inexperienced workers are required to perform tasks accurately and efficiently. However, currently, there is a lack of specific and intuitive instruction, resulting in insufficient quality and safety of work. Furthermore, errors in installation position and angle frequently occur during the assembly and maintenance of structures, sometimes leading to rework and safety problems.

[0098] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0099] In this invention, the server includes acquisition means, analysis means, presentation means, feedback means, and guidance means. This makes it possible to present appropriate work procedures to the worker in real time based on motion data, and in particular to provide visual guidelines regarding the mounting position and angle of structures.

[0100] "Acquisition means" refers to a device or method for collecting worker motion data and environmental data during on-site work.

[0101] "Analysis means" refers to technology for analyzing data obtained from acquisition means and generating the optimal work procedure.

[0102] A "presentation means" is a device for visually and audibly presenting the work procedures generated by the analysis means to the worker.

[0103] A "feedback mechanism" is a function or method for receiving feedback from workers and applying it to an analysis mechanism.

[0104] "Guiding means" refers to a method and apparatus for displaying the mounting position and angle of a structure as a guideline.

[0105] The system that realizes this invention uses a combination of three main elements: a server, a terminal, and a user. The terminal uses an AR / VR device to collect worker motion data and environmental data in the field work environment and transmits this data to the server. The server uses advanced machine learning algorithms to analyze the acquired data and generate the optimal work procedure. Specifically, it analyzes the user's movements and generates guidelines to accurately indicate the installation position and angle of the structure.

[0106] The terminal receives instructions sent back from the server, presents them visually by overlaying them onto the user's field of view, and provides voice guidance. The user performs tasks based on the presented guidelines and voice instructions. Furthermore, the user sends voice input and gesture feedback to the server via the terminal, and this feedback is applied to the analysis system.

[0107] A concrete example of this system is the installation of window frames at a construction site. The user wears an AR headset, which visually displays the precise installation position and angle. Furthermore, if an incorrect procedure is performed during the installation process, immediate voice guidance and corrections are provided. This allows even inexperienced workers to perform the task with the same level of precision as experienced workers.

[0108] Example prompt for a generated AI model: "Design a system that uses an AR headset in construction work to display real-time guidelines for mounting position and angle."

[0109] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0110] Step 1:

[0111] The terminal uses AR / VR devices to collect motion and environmental data in the field work environment. The terminal's sensors capture the user's hand movements and position, recording them as digital data related to specific tasks. This data includes the user's current work state and information about the surrounding environment.

[0112] Step 2:

[0113] The device sends the collected data to the server. The transmitted data is raw data from the sensors, including information about the user's movements and location data. The data is securely transferred to the server via the network without any modifications.

[0114] Step 3:

[0115] The server analyzes the raw data received from the terminal and generates the optimal work procedure. Using a generative AI model and machine learning algorithms, the server characterizes user movements from the data and determines the steps necessary to optimize the task. This process involves pattern recognition of movements and analysis compared to past work data.

[0116] Step 4:

[0117] The server creates guidelines to present to the worker based on the analysis results. These guidelines include visual instructions indicating the installation location and angle of the structure, as well as audio instructions regarding work steps. This ensures that specific procedures are clearly communicated to the user.

[0118] Step 5:

[0119] The device receives guidelines sent from the server and displays them overlaid on the user's field of view. Audio instructions are also played simultaneously to inform the user of the next steps. Installation points and the progress of the work are visually represented on the AR display.

[0120] Step 6:

[0121] Users perform tasks according to the provided guidelines and voice instructions. Users can monitor their actions and modify the work procedure as needed.

[0122] Step 7:

[0123] Users send feedback to the server via their device using voice input or gestures. For example, they can provide feedback if they feel the current guidelines are not practical. The server receives this feedback and applies it to analysis to improve the accuracy of future procedures.

[0124] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0125] This invention provides a system that optimizes work procedures by comprehensively utilizing worker motion data, environmental data, and emotional state in the on-site work environment. This system is composed of three main components: a server, a terminal, and a user.

[0126] First, the device collects user motion data, surrounding environment data, and even the user's emotional state in real time through AR / VR devices and biosensors. Specifically, it acquires data such as the user's facial expressions, voice tone, and heart rate, and stores this data in digital format.

[0127] The device sends all collected data to the server. The server then comprehensively analyzes this data. In particular, an emotion engine operates, analyzing emotional data to identify the user's emotional state. For example, if anxiety or stress levels are high, the system adjusts the work procedures accordingly.

[0128] Based on the analysis, the server generates work procedures best suited to the user's current state and on-site conditions. For example, if the user is feeling anxious, it can generate a work guide that includes detailed instructions and additional safety checks.

[0129] The generated optimal work procedure is then sent back to the terminal. The terminal displays this instruction on the user's AR display and also provides audio guidance. This allows the user to receive emotionally sensitive work instructions in real time, enabling them to perform their tasks efficiently and safely.

[0130] Users follow instructions presented on the device and provide feedback using voice commands and gestures as needed. This feedback includes opinions on the user's emotional state, which is used to optimize future instructions.

[0131] Through this series of processes, this system, which incorporates an emotion engine, not only supports the user's skill improvement but also provides flexible work support that takes into account their emotional state.

[0132] The following describes the processing flow.

[0133] Step 1:

[0134] The device uses cameras and biosensors connected to the AR / VR device to collect user motion data, facial expressions, voice tone, and emotion-related data such as heart rate. For example, it simultaneously records the user's hand movements while using tools and their heart rate fluctuations during the task.

[0135] Step 2:

[0136] The device collects behavioral data, emotional data, and ambient environmental data, bundles them into data packets, and sends them to a server via the internet. This data transmission is configured to minimize latency.

[0137] Step 3:

[0138] The server analyzes the received data and evaluates the user's emotional state, particularly using an emotion engine. For example, it uses facial recognition algorithms to detect signs of stress and fatigue from the user's facial features.

[0139] Step 4:

[0140] Based on the analysis results, the server generates the optimal work procedure tailored to the user's current state. It takes emotional data into consideration and may include instructions to slow down if the user is working too fast, or to encourage a break if fatigue is detected.

[0141] Step 5:

[0142] The server sends the generated work procedure data to the terminal. This includes visual guidance (arrows and highlights) and voice instructions (specific methods of the work and points to pay attention to).

[0143] Step 6:

[0144] The device displays the received work instructions on the user's AR display and plays voice instructions at the appropriate time. This allows the user to understand in detail the specific actions they need to take next.

[0145] Step 7:

[0146] Users perform tasks according to the provided instructions and provide voice feedback on any questions or difficulties they encounter during the process. This feedback may include comments such as "It's too difficult" or "More detailed instructions are needed."

[0147] Step 8:

[0148] The device collects voice feedback data from the user and sends it back to the server. The server uses this feedback to adjust the emotion engine and analysis process to improve the accuracy and suitability of the next work procedure.

[0149] (Example 2)

[0150] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0151] In on-site work environments, it is necessary to appropriately understand the emotional state of workers and present them with the most appropriate work procedures. However, conventional systems do not adequately optimize work procedures while considering emotional states, resulting in insufficient improvements in work efficiency and safety.

[0152] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0153] In this invention, the server includes acquisition means for collecting worker motion data, environmental data, and emotional state data; analysis means for comprehensively analyzing the data transmitted from the acquisition means, identifying the worker's emotional state using an emotion engine, and generating an optimal work procedure; and presentation means for visually and audibly presenting the work procedure generated by the analysis means to the worker. This makes it possible to improve work efficiency and safety by providing appropriate work procedures while taking into account the worker's real-time emotional state.

[0154] A "worker" is an individual engaged in on-site work, whose emotional state and behavioral data are managed by the system.

[0155] "Motion data" refers to information about the physical movements and actions of workers, and is used to improve efficiency and safety in the work environment.

[0156] "Environmental data" includes information about the physical conditions of the work site, and data that indicates factors that affect workers, such as temperature and noise levels.

[0157] "Emotional state data" refers to information that indicates the psychological state of a worker, and includes analysis results from an emotion engine based on facial expressions, voice tone, heart rate, etc.

[0158] "Acquisition means" refers to a device or function in the system for collecting worker motion data, environmental data, and emotional state data.

[0159] "Analysis means" refers to a process and apparatus that uses acquired data to analyze the emotional state of workers using an emotion engine and generate the optimal work procedure.

[0160] "Presentation means" refers to a device or function for conveying visually and audibly generated work procedures to a worker, and includes augmented reality and virtual reality systems.

[0161] A "feedback mechanism" refers to a device or process that accepts responses from workers, such as voices or gestures, and applies them to the analysis mechanism to improve the accuracy of the system.

[0162] An "emotion engine" is an algorithm or technology used to evaluate and analyze a worker's emotional state, and it includes elements that identify emotions based on voice tone and facial expression data.

[0163] A "generative AI model" refers to an artificial intelligence model that learns from a large amount of historical data and generates the optimal work procedure according to the situation.

[0164] This invention provides a system that optimizes work procedures by comprehensively utilizing worker motion data, environmental data, and emotional state in the on-site work environment. This system is composed of three main components: a server, a terminal, and a user.

[0165] First, the device collects user motion data, surrounding environment data, and even the user's emotional state in real time through AR / VR devices and biosensors. Specifically, it captures the user's facial expressions with a camera, acquires voice tone with a microphone, and measures heart rate with a heart rate sensor. This data is stored in a formatted digital format.

[0166] The terminal transmits the collected data to the server via a secure protocol. Encryption technologies such as TLS (Transport Layer Security) are used to maintain the confidentiality and integrity of the data during transmission. The server comprehensively analyzes the data using various analytical software and an emotion engine. The emotion engine identifies the user's emotional state based on voice tone and facial expression data. For example, if the user is feeling anxious or stressed, the work procedures are adjusted accordingly.

[0167] Once the analysis is complete, the server uses a generative AI model to generate the most appropriate work procedure for the user's current state and the on-site conditions. This AI model has learned from past data and has the ability to infer what kind of work guide is best suited to a particular emotional state. For example, if the user is feeling anxious, it will generate a work guide that includes detailed instructions and additional safety checks.

[0168] The generated work procedures are sent to the terminal, and the instructions are displayed on the user's AR display. In addition, by providing voice guidance, users receive emotionally sensitive work instructions in real time, enabling them to perform their tasks efficiently and safely.

[0169] Users perform tasks according to instructions presented on the device and provide feedback using voice commands and gestures as needed. This feedback includes information about the worker's own emotional state, which is used to optimize future instructions.

[0170] One specific example is a scenario in which safety can be improved at a construction site when workers are performing tasks at height, by displaying a guide that includes a detailed safety check process on a terminal.

[0171] Examples of prompts for the generating AI model include: "Generate the optimal work procedure when the user is feeling anxious or stressed. Specifically, tell me how to provide a work guide that includes detailed instructions."

[0172] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0173] Step 1:

[0174] The device collects data using sensors and other devices. Inputs include the user's facial expressions, voice tone, heart rate, ambient temperature, and lighting levels. This data is acquired from cameras, microphones, heart rate sensors, etc., and converted into a digital format. The output is an integrated digital dataset of this data. Specifically, analog signals obtained from sensors are digitized, and filtering and format conversion are performed as needed.

[0175] Step 2:

[0176] The terminal sends the collected data to the server. The input is the digital dataset generated in step 1. The data is transferred through a secure channel using an encryption protocol such as TLS. The output is the secure data received by the server for analysis. This process involves packetizing the data and converting it to a format suitable for network transmission.

[0177] Step 3:

[0178] The server analyzes the received data. The input is an integrated dataset sent from the terminal. Internally, an emotion engine operates to identify the emotional state based on voice tone and facial expression data. Voice analysis algorithms and image processing technologies are used for the analysis. The output is worker state data that reflects the emotional state. For example, if the emotion identified by the analysis result is determined to be "anxiety," that result is used in the next step.

[0179] Step 4:

[0180] The server generates optimal work procedures using a generative AI model. Inputs include user emotional state data and field condition data. The generative AI model infers the optimal instructions by referencing past data and current inputs. The output is an appropriate work procedure, which includes detailed instructions and additional safety checks. For example, the generated work procedure is formatted as a guide text to be displayed on the user's AR display.

[0181] Step 5:

[0182] The terminal presents the user with work instructions received from the server. The input is the work instructions sent from the server. The terminal displays these as visual instructions on the AR display and provides additional explanations through audio guidance. The output is work instructions that the user can receive visually and audibly. This allows the user to proceed with the work while receiving information in real time.

[0183] Step 6:

[0184] The user performs the task based on the provided instructions while providing feedback. The input is the actual work environment and emotional state experienced by the user. The user sends feedback to the terminal using voice commands and gestures. The output is feedback data sent from the terminal to the server, which includes the worker's emotions and work-related requests. This feedback is used to optimize future work instructions.

[0185] (Application Example 2)

[0186] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0187] In on-site processing facilities, optimizing work procedures is necessary to improve the work efficiency and safety of operators. However, standard work procedures do not take into account the emotional state of operators, so when they feel anxious or fatigued, it can become a burden and hinder efficient work. To solve this problem, a system that can dynamically adjust according to the emotional state of operators is desired.

[0188] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0189] In this invention, the server includes an acquisition means for collecting operator motion data and environmental data at a field processing facility; an analysis means for analyzing the data transmitted from the acquisition means and generating an optimal processing procedure; a presentation means for visually and audibly presenting the processing procedure generated by the analysis means to the operator; and an emotion analysis means for analyzing the operator's emotional state and dynamically adjusting the generated processing procedure according to the operator's mental state. This enables flexible work support that takes into account the operator's emotional state.

[0190] A "site processing facility" is a place where manufacturing and assembly work takes place, and it is an environment where operators process goods using machinery and equipment.

[0191] An "operator" is a person who operates machinery and equipment in a processing facility, performs tasks, and is the entity that provides work data and feedback.

[0192] "Motion data" refers to information about the operator's physical movements, and is digital data collected by sensors.

[0193] "Environmental data" refers to digital data that includes information about the physical conditions of the on-site processing facility, such as temperature, humidity, and illuminance.

[0194] "Emotional state" refers to information indicating the operator's mental and psychological state, which can be inferred from facial expressions, tone of voice, heart rate, etc.

[0195] "Acquisition means" refers to devices and technologies that use sensors and other devices to acquire operator movement data and environmental data.

[0196] "Analysis means" refers to devices or processes that analyze data collected from acquisition means and generate the optimal processing procedure.

[0197] "Presentation means" refers to devices or methods for visually and audibly showing the generated processing procedure to an operator, and includes displays and speakers.

[0198] "Emotional analysis means" refers to devices or methods for analyzing the emotional state of an operator from data and dynamically adjusting the processing procedure.

[0199] "Feedback" refers to the opinions and reactions obtained from operators, which are used to improve the accuracy of future analysis methods.

[0200] This invention is a system that supports operators in on-site processing facilities in performing their work safely and efficiently. It is composed of three main components: a server, a terminal, and a user, each of which plays a specific role to ensure smooth operation.

[0201] First, the terminal collects motion data, environmental data, and emotional state in real time via smart glasses or head-mounted displays worn by the operator and biosensors. This data includes the operator's facial expressions, heart rate, ambient temperature, and humidity. The collected data is transmitted to a server in digital format.

[0202] The server uses software such as Python and TENSORFLOW® to analyze the operator's emotional state. An emotion engine is activated, and the mental state is identified based on the acquired emotional data. Then, based on the analysis results, the optimal processing procedure is generated. Specifically, if the operator is feeling anxious, adjustments are made to increase the level of detailed instructions and safety checks.

[0203] The generated work procedures are sent back to the terminal and displayed on the operator's smart glasses. The presentation system provides both visual and audio guidance, allowing the operator to confirm the procedures in real time. This system enables operators to work efficiently.

[0204] Users provide feedback using voice commands and gestures as they perform tasks. This feedback includes opinions on the user's emotions and is used to optimize instructions for future tasks.

[0205] For example, if an operator temporarily loses focus during assembly, the system detects this and the robot automatically adjusts its work pace. Additionally, smart glasses prompt the operator to check the parts needed for the next step.

[0206] An example of a prompt to the generating AI model might be presented in the form of, "The worker appears stressed; please suggest steps to adjust the work procedure for optimal results." This demonstrates how the system would make adjustments under specific conditions.

[0207] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0208] Step 1:

[0209] The terminal collects motion data, environmental data, and emotional state from the operator's smart glasses or head-mounted display. Inputs include the operator's facial expressions, heart rate, and ambient temperature and humidity, which are acquired as digital data by biosensors. Outputs are the collected biosensors and environmental data. Each data point is transmitted from the sensor to the smart glasses in real time.

[0210] Step 2:

[0211] The terminal sends the collected data to the server. The input consists of the motion data and environmental data collected in step 1, which are sent to the server via the network. The output is digital data stored on the server. This data forms the basis for analysis.

[0212] Step 3:

[0213] The server analyzes the data using software such as Python and TensorFlow to identify the operator's emotional state. The input is the digital data transmitted in step 2, and the emotion engine performs emotion analysis. The output is data indicating the operator's emotional state. Based on these analysis results, the optimal processing procedure is considered.

[0214] Step 4:

[0215] The server generates the optimal processing procedure based on the analysis results. The input is the emotional state data obtained in step 3, and the generated AI model adjusts the work procedure based on this data. The output is the adjusted work instruction data. Flexible and safe procedures are constructed according to the mental state of the operator.

[0216] Step 5:

[0217] The server resends the generated work instructions to the terminal. The input is the work instruction data generated in step 4, which is then sent to the terminal. The output is the visual and audio instruction data on the terminal. This data is displayed on the operator's smart glasses.

[0218] Step 6:

[0219] The user performs the task according to the work procedure presented on the terminal. The input is the work instruction data presented in step 5, and the user performs production activities based on it. The output is the actual work result. Provide feedback using voice commands or gestures as needed.

[0220] Step 7:

[0221] The terminal collects user feedback and sends it to the server. Input is feedback data provided by the operator, obtained through voice and gestures. Output is feedback data stored on the server. This data is used for analysis and instruction generation in subsequent instances.

[0222] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0223] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0224] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0225] [Second Embodiment]

[0226] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0227] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0228] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0229] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0230] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0231] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0232] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0233] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0234] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0235] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0236] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0237] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0238] This invention provides a system designed to enable workers in the field to efficiently perform complex tasks. This system consists of three main elements: a server, a terminal, and a user.

[0239] First, the terminal uses an AR / VR device to collect motion data of the user working on-site and data about the surrounding environment. For example, it collects how the user is using a particular tool, including its movements and operation, and records this data digitally within the terminal. The terminal then transmits this data to a server.

[0240] The server receives and analyzes data sent from the terminal. Using state-of-the-art machine learning algorithms and computer vision technology, it recognizes user actions and generates the optimal procedure for the task. For example, in a specific welding process, it can automatically create a procedure that combines a 3D model with voice instructions to guide the user in making fine adjustments to the timing and angle of their actions.

[0241] The generated instructions are returned to the terminal. The terminal displays these as visual information overlaid on the user's field of view, and simultaneously provides specific guidance to the user using voice prompts. For example, if the welding position is inaccurate, visual guidelines and target marks are displayed on the AR display, allowing the user to correct it immediately.

[0242] Users perform tasks based on instructions received visually and audibly. This system allows even inexperienced individuals to achieve results comparable to those of skilled workers. If a user has questions or requires further instructions during the task, they can provide feedback using voice commands or gestures, which is sent to the server and incorporated into subsequent instructions.

[0243] In this way, the system achieves improved skills and optimized work efficiency in the workplace through close collaboration between servers, terminals, and users.

[0244] The following describes the processing flow.

[0245] Step 1:

[0246] The device uses AR / VR equipment to collect user motion data and surrounding environment data in real time. Specifically, it uses cameras and sensors to acquire data such as the user's hand movements, gaze, and the arrangement of objects around the user.

[0247] Step 2:

[0248] The terminal bundles the collected data into packets and sends them to the server via low-latency communication. This transmission process is designed to ensure data accuracy and timely delivery.

[0249] Step 3:

[0250] The server analyzes the received motion and environmental data. Using machine learning algorithms, it identifies the user's motion patterns and surrounding conditions from the data and generates optimal instructions necessary for the task. For example, it identifies the appropriate tool usage and operating angles.

[0251] Step 4:

[0252] The server reconstructs the generated instructions as data and returns it to the terminal. This data includes visual guidelines and audio instruction files.

[0253] Step 5:

[0254] The terminal displays the received instruction data on the user's AR display and provides supplementary information via audio output. This allows the user to proceed with their work while confirming the work procedure in real time.

[0255] Step 6:

[0256] The user performs the task according to the instructions provided. They perform or modify the instructions as needed, and provide feedback to the device via voice or gestures if they have any questions.

[0257] Step 7:

[0258] The terminal digitizes user feedback and sends it to the server. This data is used to improve instructions and is utilized for analysis in future sessions.

[0259] (Example 1)

[0260] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0261] In on-site work, the skill level and efficiency of workers directly impact the quality of the work. However, not all workers are skilled, and there is a need for support to enable less skilled workers to perform work efficiently and with high quality. Furthermore, since the work environment changes constantly, a system that can provide appropriate instructions in real time is necessary. Conventional methods have the problem of insufficient speed and accuracy in providing feedback to workers, making immediate response on-site difficult.

[0262] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0263] In this invention, the server includes means for collecting operational information and surrounding information in the work environment, means for transmitting the information using communication technology and analyzing it in a central processing unit, and means for generating work procedures using machine learning and visual recognition technology. This provides real-time feedback tailored to the worker's skills and environment, enabling even inexperienced workers to efficiently perform high-quality work.

[0264] "Work environment" refers to the physical or virtual space in which work is performed, and the place where the worker carries out their actions.

[0265] A "worker" refers to an individual who receives instructions through a system and actually performs the work.

[0266] A "visual device" refers to a device that displays digital information on a screen, and can overlay information onto the real-world environment.

[0267] "Motion information" refers to data related to the physical movements of workers and specific actions taken during their work.

[0268] "Ambient information" refers to data that indicates the conditions and circumstances of the work environment, including light intensity, temperature, and humidity.

[0269] "Communication technology" refers to wireless or wired technologies for efficiently and securely sending and receiving data.

[0270] A "central processing unit" refers to the main computer system that receives collected data and performs analysis and processing on it.

[0271] "Machine learning" refers to the technology that enables computers to learn patterns from data and make decisions and predictions.

[0272] "Visual recognition technology" refers to the technology of extracting and understanding specific information or patterns from images and videos.

[0273] A "work procedure" is a set of instructions outlining the steps necessary to perform a specific task efficiently and safely.

[0274] "Visual information" refers to images and video content displayed to the user.

[0275] "Audio information" refers to instructions and guidance provided to users via voice.

[0276] "Response" refers to the reaction that arises from the operator's actions or feedback.

[0277] This invention is a system that provides support for workers to perform their tasks efficiently. The system mainly consists of three components: a server, a terminal, and a user.

[0278] The terminal uses a visual device worn by the worker, specifically an AR / VR device, to collect information about the work environment, including movement and surroundings. This terminal captures the user's hand movements and tool usage in real time, and also measures ambient brightness and temperature. For example, when a user welds an object, the terminal meticulously records their hand movements and tool operation.

[0279] The server receives the information sent from the terminal and performs advanced analysis. Specifically, by using a machine learning algorithm implemented in a programming language such as Python and OpenCV, an open-source library, it finely analyzes the user's actions. Based on these data, the server generates an optimal work procedure. For example, a high-performance server equipped with an 856 processor fuses past data and current environmental conditions to determine the most effective procedure for the user.

[0280] The generated procedure is sent to the terminal and presented to the user by a visual device. The terminal overlays visual information such as 3D models and arrows on the operator's field of vision and simultaneously provides voice instructions. As a result, the user can obtain information through various senses, improving work efficiency. As a specific example, for a user performing gardening work, the terminal visually highlights the position of the branch and provides a voice guide such as "Next, please cut the left branch at an angle of 45 degrees."

[0281] The user performs the work according to the explicit instructions by vision and voice. If necessary, feedback can be sent to the terminal using voice commands or hand gestures. This feedback information is analyzed by the server and reflected in the generation of the next procedure. For example, an example of a prompt sentence is "Please propose a procedure for the new welding process."

[0282] This system realizes the improvement of technical skills and the optimization of work efficiency at the work site.

[0283] The flow of the specific process in Example 1 will be described using FIG. 11.

[0284] Step 1:

[0285] The terminal collects operation information and surrounding information through a visual device worn by the user. As inputs, it obtains the movement of the user's hand, the usage status of tools, and the light amount and temperature of the environment. These data are converted into a digital format optimized for motion analysis and stored in a data buffer. As a specific operation, the camera captures the user using a tool and provides it as basic data for real-time analysis of the video.

[0286] Step 2:

[0287] The terminal transmits the collected data to the server using communication technology. The inputs are the operation information and surrounding information collected in Step 1. The data is securely transmitted to the server through encryption technology such as TLS. As a specific role of the terminal, it batch-processes the data at regular intervals and efficiently transmits it to the server via the network.

[0288] Step 3:

[0289] The server receives and analyzes the data transmitted from the terminal. The inputs are the user's operation information and environmental information. The server processes these data using machine learning algorithms and visual recognition technology. Through the processing, it models the user's work operations and generates an optimal work procedure. As a specific operation, it uses Python and OpenCV to extract the characteristics of the user's movements.

[0290] Step 4:

[0291] The server transmits the generated work procedure to the terminal. The input is the optimal procedure obtained by machine learning. The server uses this to construct a work instruction in natural language and a 3D model. The output is visual information and audio information presented to the user.

[0292] Step 5:

[0293] The terminal presents instructions received from the server to the user. It receives visual and audio instruction data from the server as input. The output consists of guidelines and audio guidance displayed overlaid on the visual device. Specific actions include displaying arrows on the AR display and playing instructions through the speaker.

[0294] Step 6:

[0295] The user performs the task according to the presented visual and audio instructions. The user utilizes instructions received from the terminal as input. As the user progresses, they send feedback to the terminal via voice commands or gestures if they have any questions. Specific actions include operating tools according to instructions and continuously providing feedback on the status to the terminal.

[0296] Step 7:

[0297] The terminal receives feedback from the user and sends it to the server. Input is the user's voice or gesture response. Output is data sent to the server prompting further analysis and instruction generation. The terminal's operation involves quickly detecting feedback and preparing data for the next cycle.

[0298] (Application Example 1)

[0299] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0300] In on-site work, even inexperienced workers are required to perform tasks accurately and efficiently. However, currently, there is a lack of specific and intuitive instruction, resulting in insufficient quality and safety of work. Furthermore, errors in installation position and angle frequently occur during the assembly and maintenance of structures, sometimes leading to rework and safety problems.

[0301] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means respectively.

[0302] In this invention, the server includes an acquisition means, an analysis means, a presentation means, a feedback means, and a guidance means. Thereby, it is possible to present an appropriate work procedure based on operation data to the operator in real time, and particularly to provide visual guidelines regarding the attachment position and angle of the structure.

[0303] The "acquisition means" is a device or method for collecting the operation data and environmental data of the operator in on-site work.

[0304] The "analysis means" is a technology for analyzing the data obtained from the acquisition means and generating an optimal work procedure.

[0305] The "presentation means" is a device for visually and audibly presenting the work procedure generated by the analysis means to the operator.

[0306] The "feedback means" is a function or method for receiving the feedback from the operator and applying it to the analysis means.

[0307] The "guidance means" is a method and device for displaying the attachment position and angle of the structure as guidelines.

[0308] In the system for realizing this invention, three main elements, namely the server, the terminal, and the user, are combined and used. The terminal uses an AR / VR device to collect the operation data and environmental data of the operator in the on-site work environment and transmits this to the server. The server uses an advanced machine learning algorithm to analyze the acquired data and generate an optimal work procedure. Specifically, it analyzes the operation of the user and generates guidelines for accurately indicating the attachment position and angle of the structure.

[0309] The terminal receives instructions sent back from the server, presents them visually by overlaying them onto the user's field of view, and provides voice guidance. The user performs tasks based on the presented guidelines and voice instructions. Furthermore, the user sends voice input and gesture feedback to the server via the terminal, and this feedback is applied to the analysis system.

[0310] A concrete example of this system is the installation of window frames at a construction site. The user wears an AR headset, which visually displays the precise installation position and angle. Furthermore, if an incorrect procedure is performed during the installation process, immediate voice guidance and corrections are provided. This allows even inexperienced workers to perform the task with the same level of precision as experienced workers.

[0311] Example prompt for a generated AI model: "Design a system that uses an AR headset in construction work to display real-time guidelines for mounting position and angle."

[0312] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0313] Step 1:

[0314] The terminal uses AR / VR devices to collect motion and environmental data in the field work environment. The terminal's sensors capture the user's hand movements and position, recording them as digital data related to specific tasks. This data includes the user's current work state and information about the surrounding environment.

[0315] Step 2:

[0316] The device sends the collected data to the server. The transmitted data is raw data from the sensors, including information about the user's movements and location data. The data is securely transferred to the server via the network without any modifications.

[0317] Step 3:

[0318] The server analyzes the raw data received from the terminal and generates the optimal work procedure. Using a generative AI model and machine learning algorithms, the server characterizes user movements from the data and determines the steps necessary to optimize the task. This process involves pattern recognition of movements and analysis compared to past work data.

[0319] Step 4:

[0320] The server creates guidelines to present to the worker based on the analysis results. These guidelines include visual instructions indicating the installation location and angle of the structure, as well as audio instructions regarding work steps. This ensures that specific procedures are clearly communicated to the user.

[0321] Step 5:

[0322] The device receives guidelines sent from the server and displays them overlaid on the user's field of view. Audio instructions are also played simultaneously to inform the user of the next steps. Installation points and the progress of the work are visually represented on the AR display.

[0323] Step 6:

[0324] Users perform tasks according to the provided guidelines and voice instructions. Users can monitor their actions and modify the work procedure as needed.

[0325] Step 7:

[0326] Users send feedback to the server via their device using voice input or gestures. For example, they can provide feedback if they feel the current guidelines are not practical. The server receives this feedback and applies it to analysis to improve the accuracy of future procedures.

[0327] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0328] This invention provides a system that optimizes work procedures by comprehensively utilizing worker motion data, environmental data, and emotional state in the on-site work environment. This system is composed of three main components: a server, a terminal, and a user.

[0329] First, the device collects user motion data, surrounding environment data, and even the user's emotional state in real time through AR / VR devices and biosensors. Specifically, it acquires data such as the user's facial expressions, voice tone, and heart rate, and stores this data in digital format.

[0330] The device sends all collected data to the server. The server then comprehensively analyzes this data. In particular, an emotion engine operates, analyzing emotional data to identify the user's emotional state. For example, if anxiety or stress levels are high, the system adjusts the work procedures accordingly.

[0331] Based on the analysis, the server generates work procedures best suited to the user's current state and on-site conditions. For example, if the user is feeling anxious, it can generate a work guide that includes detailed instructions and additional safety checks.

[0332] The generated optimal work procedure is then sent back to the terminal. The terminal displays this instruction on the user's AR display and also provides audio guidance. This allows the user to receive emotionally sensitive work instructions in real time, enabling them to perform their tasks efficiently and safely.

[0333] Users follow instructions presented on the device and provide feedback using voice commands and gestures as needed. This feedback includes opinions on the user's emotional state, which is used to optimize future instructions.

[0334] Through this series of processes, this system, which incorporates an emotion engine, not only supports the user's skill improvement but also provides flexible work support that takes into account their emotional state.

[0335] The following describes the processing flow.

[0336] Step 1:

[0337] The device uses cameras and biosensors connected to the AR / VR device to collect user motion data, facial expressions, voice tone, and emotion-related data such as heart rate. For example, it simultaneously records the user's hand movements while using tools and their heart rate fluctuations during the task.

[0338] Step 2:

[0339] The device collects behavioral data, emotional data, and ambient environmental data, bundles them into data packets, and sends them to a server via the internet. This data transmission is configured to minimize latency.

[0340] Step 3:

[0341] The server analyzes the received data and evaluates the user's emotional state, particularly using an emotion engine. For example, it uses facial recognition algorithms to detect signs of stress and fatigue from the user's facial features.

[0342] Step 4:

[0343] Based on the analysis results, the server generates the optimal work procedure tailored to the user's current state. It takes emotional data into consideration and may include instructions to slow down if the user is working too fast, or to encourage a break if fatigue is detected.

[0344] Step 5:

[0345] The server sends the generated work procedure data to the terminal. This includes visual guidance (arrows and highlights) and voice instructions (specific methods of the work and points to pay attention to).

[0346] Step 6:

[0347] The device displays the received work instructions on the user's AR display and plays voice instructions at the appropriate time. This allows the user to understand in detail the specific actions they need to take next.

[0348] Step 7:

[0349] Users perform tasks according to the provided instructions and provide voice feedback on any questions or difficulties they encounter during the process. This feedback may include comments such as "It's too difficult" or "More detailed instructions are needed."

[0350] Step 8:

[0351] The device collects voice feedback data from the user and sends it back to the server. The server uses this feedback to adjust the emotion engine and analysis process to improve the accuracy and suitability of the next work procedure.

[0352] (Example 2)

[0353] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0354] In on-site work environments, it is necessary to appropriately understand the emotional state of workers and present them with the most appropriate work procedures. However, conventional systems do not adequately optimize work procedures while considering emotional states, resulting in insufficient improvements in work efficiency and safety.

[0355] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0356] In this invention, the server includes acquisition means for collecting worker motion data, environmental data, and emotional state data; analysis means for comprehensively analyzing the data transmitted from the acquisition means, identifying the worker's emotional state using an emotion engine, and generating an optimal work procedure; and presentation means for visually and audibly presenting the work procedure generated by the analysis means to the worker. This makes it possible to improve work efficiency and safety by providing appropriate work procedures while taking into account the worker's real-time emotional state.

[0357] A "worker" is an individual engaged in on-site work, whose emotional state and behavioral data are managed by the system.

[0358] "Motion data" refers to information about the physical movements and actions of workers, and is used to improve efficiency and safety in the work environment.

[0359] "Environmental data" includes information about the physical conditions of the work site, and data that indicates factors that affect workers, such as temperature and noise levels.

[0360] "Emotional state data" refers to information that indicates the psychological state of a worker, and includes analysis results from an emotion engine based on facial expressions, voice tone, heart rate, etc.

[0361] "Acquisition means" refers to a device or function in the system for collecting worker motion data, environmental data, and emotional state data.

[0362] "Analysis means" refers to a process and apparatus that uses acquired data to analyze the emotional state of workers using an emotion engine and generate the optimal work procedure.

[0363] "Presentation means" refers to a device or function for conveying visually and audibly generated work procedures to a worker, and includes augmented reality and virtual reality systems.

[0364] A "feedback mechanism" refers to a device or process that accepts responses from workers, such as voices or gestures, and applies them to the analysis mechanism to improve the accuracy of the system.

[0365] An "emotion engine" is an algorithm or technology used to evaluate and analyze a worker's emotional state, and it includes elements that identify emotions based on voice tone and facial expression data.

[0366] A "generative AI model" refers to an artificial intelligence model that learns from a large amount of historical data and generates the optimal work procedure according to the situation.

[0367] This invention provides a system that optimizes work procedures by comprehensively utilizing worker motion data, environmental data, and emotional state in the on-site work environment. This system is composed of three main components: a server, a terminal, and a user.

[0368] First, the device collects user motion data, surrounding environment data, and even the user's emotional state in real time through AR / VR devices and biosensors. Specifically, it captures the user's facial expressions with a camera, acquires voice tone with a microphone, and measures heart rate with a heart rate sensor. This data is stored in a formatted digital format.

[0369] The terminal transmits the collected data to the server via a secure protocol. Encryption technologies such as TLS (Transport Layer Security) are used to maintain the confidentiality and integrity of the data during transmission. The server comprehensively analyzes the data using various analytical software and an emotion engine. The emotion engine identifies the user's emotional state based on voice tone and facial expression data. For example, if the user is feeling anxious or stressed, the work procedures are adjusted accordingly.

[0370] Once the analysis is complete, the server uses a generative AI model to generate the most appropriate work procedure for the user's current state and the on-site conditions. This AI model has learned from past data and has the ability to infer what kind of work guide is best suited to a particular emotional state. For example, if the user is feeling anxious, it will generate a work guide that includes detailed instructions and additional safety checks.

[0371] The generated work procedures are sent to the terminal, and the instructions are displayed on the user's AR display. In addition, by providing voice guidance, users receive emotionally sensitive work instructions in real time, enabling them to perform their tasks efficiently and safely.

[0372] Users perform tasks according to instructions presented on the device and provide feedback using voice commands and gestures as needed. This feedback includes information about the worker's own emotional state, which is used to optimize future instructions.

[0373] One specific example is a scenario in which safety can be improved at a construction site when workers are performing tasks at height, by displaying a guide that includes a detailed safety check process on a terminal.

[0374] Examples of prompts for the generating AI model include: "Generate the optimal work procedure when the user is feeling anxious or stressed. Specifically, tell me how to provide a work guide that includes detailed instructions."

[0375] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0376] Step 1:

[0377] The device collects data using sensors and other devices. Inputs include the user's facial expressions, voice tone, heart rate, ambient temperature, and lighting levels. This data is acquired from cameras, microphones, heart rate sensors, etc., and converted into a digital format. The output is an integrated digital dataset of this data. Specifically, analog signals obtained from sensors are digitized, and filtering and format conversion are performed as needed.

[0378] Step 2:

[0379] The terminal sends the collected data to the server. The input is the digital dataset generated in step 1. The data is transferred through a secure channel using an encryption protocol such as TLS. The output is the secure data received by the server for analysis. This process involves packetizing the data and converting it to a format suitable for network transmission.

[0380] Step 3:

[0381] The server analyzes the received data. The input is an integrated dataset sent from the terminal. Internally, an emotion engine operates to identify the emotional state based on voice tone and facial expression data. Voice analysis algorithms and image processing technologies are used for the analysis. The output is worker state data that reflects the emotional state. For example, if the emotion identified by the analysis result is determined to be "anxiety," that result is used in the next step.

[0382] Step 4:

[0383] The server generates optimal work procedures using a generative AI model. Inputs include user emotional state data and field condition data. The generative AI model infers the optimal instructions by referencing past data and current inputs. The output is an appropriate work procedure, which includes detailed instructions and additional safety checks. For example, the generated work procedure is formatted as a guide text to be displayed on the user's AR display.

[0384] Step 5:

[0385] The terminal presents the user with work instructions received from the server. The input is the work instructions sent from the server. The terminal displays these as visual instructions on the AR display and provides additional explanations through audio guidance. The output is work instructions that the user can receive visually and audibly. This allows the user to proceed with the work while receiving information in real time.

[0386] Step 6:

[0387] The user performs the task based on the provided instructions while providing feedback. The input is the actual work environment and emotional state experienced by the user. The user sends feedback to the terminal using voice commands and gestures. The output is feedback data sent from the terminal to the server, which includes the worker's emotions and work-related requests. This feedback is used to optimize future work instructions.

[0388] (Application Example 2)

[0389] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0390] In on-site processing facilities, optimizing work procedures is necessary to improve the work efficiency and safety of operators. However, standard work procedures do not take into account the emotional state of operators, so when they feel anxious or fatigued, it can become a burden and hinder efficient work. To solve this problem, a system that can dynamically adjust according to the emotional state of operators is desired.

[0391] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0392] In this invention, the server includes an acquisition means for collecting operator motion data and environmental data at a field processing facility; an analysis means for analyzing the data transmitted from the acquisition means and generating an optimal processing procedure; a presentation means for visually and audibly presenting the processing procedure generated by the analysis means to the operator; and an emotion analysis means for analyzing the operator's emotional state and dynamically adjusting the generated processing procedure according to the operator's mental state. This enables flexible work support that takes into account the operator's emotional state.

[0393] A "site processing facility" is a place where manufacturing and assembly work takes place, and it is an environment where operators process goods using machinery and equipment.

[0394] An "operator" is a person who operates machinery and equipment in a processing facility, performs tasks, and is the entity that provides work data and feedback.

[0395] "Motion data" refers to information about the operator's physical movements, and is digital data collected by sensors.

[0396] "Environmental data" refers to digital data that includes information about the physical conditions of the on-site processing facility, such as temperature, humidity, and illuminance.

[0397] "Emotional state" refers to information indicating the operator's mental and psychological state, which can be inferred from facial expressions, tone of voice, heart rate, etc.

[0398] "Acquisition means" refers to devices and technologies that use sensors and other devices to acquire operator movement data and environmental data.

[0399] "Analysis means" refers to devices or processes that analyze data collected from acquisition means and generate the optimal processing procedure.

[0400] "Presentation means" refers to devices or methods for visually and audibly showing the generated processing procedure to an operator, and includes displays and speakers.

[0401] "Emotional analysis means" refers to devices or methods for analyzing the emotional state of an operator from data and dynamically adjusting the processing procedure.

[0402] "Feedback" refers to the opinions and reactions obtained from operators, which are used to improve the accuracy of future analysis methods.

[0403] This invention is a system that supports operators in on-site processing facilities in performing their work safely and efficiently. It is composed of three main components: a server, a terminal, and a user, each of which plays a specific role to ensure smooth operation.

[0404] First, the terminal collects motion data, environmental data, and emotional state in real time via smart glasses or head-mounted displays worn by the operator and biosensors. This data includes the operator's facial expressions, heart rate, ambient temperature, and humidity. The collected data is transmitted to a server in digital format.

[0405] The server uses software such as Python and TensorFlow to analyze the operator's emotional state. An emotion engine is activated, and the mental state is identified based on the acquired emotional data. Then, based on the analysis results, the optimal processing procedure is generated. Specifically, if the operator is feeling anxious, adjustments are made to increase the level of detailed instructions and safety checks.

[0406] The generated work procedures are sent back to the terminal and displayed on the operator's smart glasses. The presentation system provides both visual and audio guidance, allowing the operator to confirm the procedures in real time. This system enables operators to work efficiently.

[0407] Users provide feedback using voice commands and gestures as they perform tasks. This feedback includes opinions on the user's emotions and is used to optimize instructions for future tasks.

[0408] For example, if an operator temporarily loses focus during assembly, the system detects this and the robot automatically adjusts its work pace. Additionally, smart glasses prompt the operator to check the parts needed for the next step.

[0409] An example of a prompt to the generating AI model might be presented in the form of, "The worker appears stressed; please suggest steps to adjust the work procedure for optimal results." This demonstrates how the system would make adjustments under specific conditions.

[0410] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0411] Step 1:

[0412] The terminal collects motion data, environmental data, and emotional state from the operator's smart glasses or head-mounted display. Inputs include the operator's facial expressions, heart rate, and ambient temperature and humidity, which are acquired as digital data by biosensors. Outputs are the collected biosensors and environmental data. Each data point is transmitted from the sensor to the smart glasses in real time.

[0413] Step 2:

[0414] The terminal sends the collected data to the server. The input consists of the motion data and environmental data collected in step 1, which are sent to the server via the network. The output is digital data stored on the server. This data forms the basis for analysis.

[0415] Step 3:

[0416] The server analyzes the data using software such as Python and TensorFlow to identify the operator's emotional state. The input is the digital data transmitted in step 2, and the emotion engine performs emotion analysis. The output is data indicating the operator's emotional state. Based on these analysis results, the optimal processing procedure is considered.

[0417] Step 4:

[0418] The server generates the optimal processing procedure based on the analysis results. The input is the emotional state data obtained in step 3, and the generated AI model adjusts the work procedure based on this data. The output is the adjusted work instruction data. Flexible and safe procedures are constructed according to the mental state of the operator.

[0419] Step 5:

[0420] The server resends the generated work instructions to the terminal. The input is the work instruction data generated in step 4, which is then sent to the terminal. The output is the visual and audio instruction data on the terminal. This data is displayed on the operator's smart glasses.

[0421] Step 6:

[0422] The user performs the task according to the work procedure presented on the terminal. The input is the work instruction data presented in step 5, and the user performs production activities based on it. The output is the actual work result. Provide feedback using voice commands or gestures as needed.

[0423] Step 7:

[0424] The terminal collects user feedback and sends it to the server. Input is feedback data provided by the operator, obtained through voice and gestures. Output is feedback data stored on the server. This data is used for analysis and instruction generation in subsequent instances.

[0425] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0426] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0427] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0428] [Third Embodiment]

[0429] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0430] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0431] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0432] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0433] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0434] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0435] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0436] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0437] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0438] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0439] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0440] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0441] This invention provides a system designed to enable workers in the field to efficiently perform complex tasks. This system consists of three main elements: a server, a terminal, and a user.

[0442] First, the terminal uses an AR / VR device to collect motion data of the user working on-site and data about the surrounding environment. For example, it collects how the user is using a particular tool, including its movements and operation, and records this data digitally within the terminal. The terminal then transmits this data to a server.

[0443] The server receives and analyzes data sent from the terminal. Using state-of-the-art machine learning algorithms and computer vision technology, it recognizes user actions and generates the optimal procedure for the task. For example, in a specific welding process, it can automatically create a procedure that combines a 3D model with voice instructions to guide the user in making fine adjustments to the timing and angle of their actions.

[0444] The generated instructions are returned to the terminal. The terminal displays these as visual information overlaid on the user's field of view, and simultaneously provides specific guidance to the user using voice prompts. For example, if the welding position is inaccurate, visual guidelines and target marks are displayed on the AR display, allowing the user to correct it immediately.

[0445] Users perform tasks based on instructions received visually and audibly. This system allows even inexperienced individuals to achieve results comparable to those of skilled workers. If a user has questions or requires further instructions during the task, they can provide feedback using voice commands or gestures, which is sent to the server and incorporated into subsequent instructions.

[0446] In this way, the system achieves improved skills and optimized work efficiency in the workplace through close collaboration between servers, terminals, and users.

[0447] The following describes the processing flow.

[0448] Step 1:

[0449] The device uses AR / VR equipment to collect user motion data and surrounding environment data in real time. Specifically, it uses cameras and sensors to acquire data such as the user's hand movements, gaze, and the arrangement of objects around the user.

[0450] Step 2:

[0451] The terminal bundles the collected data into packets and sends them to the server via low-latency communication. This transmission process is designed to ensure data accuracy and timely delivery.

[0452] Step 3:

[0453] The server analyzes the received motion and environmental data. Using machine learning algorithms, it identifies the user's motion patterns and surrounding conditions from the data and generates optimal instructions necessary for the task. For example, it identifies the appropriate tool usage and operating angles.

[0454] Step 4:

[0455] The server reconstructs the generated instructions as data and returns it to the terminal. This data includes visual guidelines and audio instruction files.

[0456] Step 5:

[0457] The terminal displays the received instruction data on the user's AR display and provides supplementary information via audio output. This allows the user to proceed with their work while confirming the work procedure in real time.

[0458] Step 6:

[0459] The user performs the task according to the instructions provided. They perform or modify the instructions as needed, and provide feedback to the device via voice or gestures if they have any questions.

[0460] Step 7:

[0461] The terminal digitizes user feedback and sends it to the server. This data is used to improve instructions and is utilized for analysis in future sessions.

[0462] (Example 1)

[0463] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0464] In on-site work, the skill level and efficiency of workers directly impact the quality of the work. However, not all workers are skilled, and there is a need for support to enable less skilled workers to perform work efficiently and with high quality. Furthermore, since the work environment changes constantly, a system that can provide appropriate instructions in real time is necessary. Conventional methods have the problem of insufficient speed and accuracy in providing feedback to workers, making immediate response on-site difficult.

[0465] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0466] In this invention, the server includes means for collecting operational information and surrounding information in the work environment, means for transmitting the information using communication technology and analyzing it in a central processing unit, and means for generating work procedures using machine learning and visual recognition technology. This provides real-time feedback tailored to the worker's skills and environment, enabling even inexperienced workers to efficiently perform high-quality work.

[0467] "Work environment" refers to the physical or virtual space in which work is performed, and the place where the worker carries out their actions.

[0468] A "worker" refers to an individual who receives instructions through a system and actually performs the work.

[0469] A "visual device" refers to a device that displays digital information on a screen, and can overlay information onto the real-world environment.

[0470] "Motion information" refers to data related to the physical movements of workers and specific actions taken during their work.

[0471] "Ambient information" refers to data that indicates the conditions and circumstances of the work environment, including light intensity, temperature, and humidity.

[0472] "Communication technology" refers to wireless or wired technologies for efficiently and securely sending and receiving data.

[0473] A "central processing unit" refers to the main computer system that receives collected data and performs analysis and processing on it.

[0474] "Machine learning" refers to the technology that enables computers to learn patterns from data and make decisions and predictions.

[0475] "Visual recognition technology" refers to the technology of extracting and understanding specific information or patterns from images and videos.

[0476] A "work procedure" is a set of instructions outlining the steps necessary to perform a specific task efficiently and safely.

[0477] "Visual information" refers to images and video content displayed to the user.

[0478] "Audio information" refers to instructions and guidance provided to users via voice.

[0479] "Response" refers to the reaction that arises from the operator's actions or feedback.

[0480] This invention is a system that provides support for workers to perform their tasks efficiently. The system mainly consists of three components: a server, a terminal, and a user.

[0481] The terminal uses a visual device worn by the worker, specifically an AR / VR device, to collect information about the work environment, including movement and surroundings. This terminal captures the user's hand movements and tool usage in real time, and also measures ambient brightness and temperature. For example, when a user welds an object, the terminal meticulously records their hand movements and tool operation.

[0482] The server receives information sent from the terminal and performs advanced analysis. Specifically, it uses machine learning algorithms implemented in programming languages ​​such as Python and the open-source library OpenCV to analyze user actions in detail. Based on this data, the server generates the optimal work procedure. For example, a high-performance server equipped with an 856 processor will combine historical data with current environmental conditions to determine the most effective procedure for the user.

[0483] The generated instructions are sent to a terminal and presented to the user via a visual device. The terminal overlays visual information such as 3D models and arrows onto the worker's field of view, while simultaneously providing voice instructions. This allows the user to obtain information through multiple senses, improving work efficiency. For example, a user performing gardening tasks might receive voice guidance such as, "Next, cut the left branch at a 45-degree angle," while the terminal visually highlights the branch's position.

[0484] Users perform tasks by following explicit visual and auditory instructions. If necessary, they can send feedback to the terminal using voice commands or hand gestures. This feedback information is analyzed by the server and used to generate the next steps. For example, a possible prompt might be, "Suggest a new welding process step."

[0485] This system will enable improvements in technical skills and optimization of work efficiency at the work site.

[0486] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0487] Step 1:

[0488] The terminal collects motion and ambient information through the user's visual device. Inputs include the user's hand movements, tool usage, and ambient light levels and temperature. This data is converted into a digital format optimized for motion analysis and stored in a data buffer. Specifically, a camera captures the user using a tool, providing this footage as foundational data for real-time analysis.

[0489] Step 2:

[0490] The terminal transmits the collected data to the server using communication technology. The input consists of the operational information and surrounding information collected in step 1. The data is securely transmitted to the server via encryption technology such as TLS. The specific role of the terminal is to batch process the data at regular intervals and efficiently transmit it to the server over the network.

[0491] Step 3:

[0492] The server receives and analyzes data sent from the terminal. The input includes user action information and environmental information. The server processes this data using machine learning algorithms and visual recognition technology. Through this processing, it models the user's work actions and generates the optimal work procedure. Specifically, it uses Python and OpenCV to extract the characteristics of the user's movements.

[0493] Step 4:

[0494] The server sends the generated work procedure to the terminal. The input is the optimal procedure obtained through machine learning. The server uses this to construct work instructions in natural language and a 3D model. The output is visual and audio information presented to the user.

[0495] Step 5:

[0496] The terminal presents instructions received from the server to the user. It receives visual and audio instruction data from the server as input. The output consists of guidelines and audio guidance displayed overlaid on the visual device. Specific actions include displaying arrows on the AR display and playing instructions through the speaker.

[0497] Step 6:

[0498] The user performs the task according to the presented visual and audio instructions. The user utilizes instructions received from the terminal as input. As the user progresses, they send feedback to the terminal via voice commands or gestures if they have any questions. Specific actions include operating tools according to instructions and continuously providing feedback on the status to the terminal.

[0499] Step 7:

[0500] The terminal receives feedback from the user and sends it to the server. Input is the user's voice or gesture response. Output is data sent to the server prompting further analysis and instruction generation. The terminal's operation involves quickly detecting feedback and preparing data for the next cycle.

[0501] (Application Example 1)

[0502] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0503] In on-site work, even inexperienced workers are required to perform tasks accurately and efficiently. However, currently, there is a lack of specific and intuitive instruction, resulting in insufficient quality and safety of work. Furthermore, errors in installation position and angle frequently occur during the assembly and maintenance of structures, sometimes leading to rework and safety problems.

[0504] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0505] In this invention, the server includes acquisition means, analysis means, presentation means, feedback means, and guidance means. This makes it possible to present appropriate work procedures to the worker in real time based on motion data, and in particular to provide visual guidelines regarding the mounting position and angle of structures.

[0506] "Acquisition means" refers to a device or method for collecting worker motion data and environmental data during on-site work.

[0507] "Analysis means" refers to technology for analyzing data obtained from acquisition means and generating the optimal work procedure.

[0508] A "presentation means" is a device for visually and audibly presenting the work procedures generated by the analysis means to the worker.

[0509] A "feedback mechanism" is a function or method for receiving feedback from workers and applying it to an analysis mechanism.

[0510] "Guiding means" refers to a method and apparatus for displaying the mounting position and angle of a structure as a guideline.

[0511] The system that realizes this invention uses a combination of three main elements: a server, a terminal, and a user. The terminal uses an AR / VR device to collect worker motion data and environmental data in the field work environment and transmits this data to the server. The server uses advanced machine learning algorithms to analyze the acquired data and generate the optimal work procedure. Specifically, it analyzes the user's movements and generates guidelines to accurately indicate the installation position and angle of the structure.

[0512] The terminal receives instructions sent back from the server, presents them visually by overlaying them onto the user's field of view, and provides voice guidance. The user performs tasks based on the presented guidelines and voice instructions. Furthermore, the user sends voice input and gesture feedback to the server via the terminal, and this feedback is applied to the analysis system.

[0513] A concrete example of this system is the installation of window frames at a construction site. The user wears an AR headset, which visually displays the precise installation position and angle. Furthermore, if an incorrect procedure is performed during the installation process, immediate voice guidance and corrections are provided. This allows even inexperienced workers to perform the task with the same level of precision as experienced workers.

[0514] Example prompt for a generated AI model: "Design a system that uses an AR headset in construction work to display real-time guidelines for mounting position and angle."

[0515] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0516] Step 1:

[0517] The terminal uses AR / VR devices to collect motion and environmental data in the field work environment. The terminal's sensors capture the user's hand movements and position, recording them as digital data related to specific tasks. This data includes the user's current work state and information about the surrounding environment.

[0518] Step 2:

[0519] The device sends the collected data to the server. The transmitted data is raw data from the sensors, including information about the user's movements and location data. The data is securely transferred to the server via the network without any modifications.

[0520] Step 3:

[0521] The server analyzes the raw data received from the terminal and generates the optimal work procedure. Using a generative AI model and machine learning algorithms, the server characterizes user movements from the data and determines the steps necessary to optimize the task. This process involves pattern recognition of movements and analysis compared to past work data.

[0522] Step 4:

[0523] The server creates guidelines to present to the worker based on the analysis results. These guidelines include visual instructions indicating the installation location and angle of the structure, as well as audio instructions regarding work steps. This ensures that specific procedures are clearly communicated to the user.

[0524] Step 5:

[0525] The device receives guidelines sent from the server and displays them overlaid on the user's field of view. Audio instructions are also played simultaneously to inform the user of the next steps. Installation points and the progress of the work are visually represented on the AR display.

[0526] Step 6:

[0527] Users perform tasks according to the provided guidelines and voice instructions. Users can monitor their actions and modify the work procedure as needed.

[0528] Step 7:

[0529] Users send feedback to the server via their device using voice input or gestures. For example, they can provide feedback if they feel the current guidelines are not practical. The server receives this feedback and applies it to analysis to improve the accuracy of future procedures.

[0530] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0531] This invention provides a system that optimizes work procedures by comprehensively utilizing worker motion data, environmental data, and emotional state in the on-site work environment. This system is composed of three main components: a server, a terminal, and a user.

[0532] First, the device collects user motion data, surrounding environment data, and even the user's emotional state in real time through AR / VR devices and biosensors. Specifically, it acquires data such as the user's facial expressions, voice tone, and heart rate, and stores this data in digital format.

[0533] The device sends all collected data to the server. The server then comprehensively analyzes this data. In particular, an emotion engine operates, analyzing emotional data to identify the user's emotional state. For example, if anxiety or stress levels are high, the system adjusts the work procedures accordingly.

[0534] Based on the analysis, the server generates work procedures best suited to the user's current state and on-site conditions. For example, if the user is feeling anxious, it can generate a work guide that includes detailed instructions and additional safety checks.

[0535] The generated optimal work procedure is then sent back to the terminal. The terminal displays this instruction on the user's AR display and also provides audio guidance. This allows the user to receive emotionally sensitive work instructions in real time, enabling them to perform their tasks efficiently and safely.

[0536] Users follow instructions presented on the device and provide feedback using voice commands and gestures as needed. This feedback includes opinions on the user's emotional state, which is used to optimize future instructions.

[0537] Through this series of processes, this system, which incorporates an emotion engine, not only supports the user's skill improvement but also provides flexible work support that takes into account their emotional state.

[0538] The following describes the processing flow.

[0539] Step 1:

[0540] The device uses cameras and biosensors connected to the AR / VR device to collect user motion data, facial expressions, voice tone, and emotion-related data such as heart rate. For example, it simultaneously records the user's hand movements while using tools and their heart rate fluctuations during the task.

[0541] Step 2:

[0542] The device collects behavioral data, emotional data, and ambient environmental data, bundles them into data packets, and sends them to a server via the internet. This data transmission is configured to minimize latency.

[0543] Step 3:

[0544] The server analyzes the received data and evaluates the user's emotional state, particularly using an emotion engine. For example, it uses facial recognition algorithms to detect signs of stress and fatigue from the user's facial features.

[0545] Step 4:

[0546] Based on the analysis results, the server generates the optimal work procedure tailored to the user's current state. It takes emotional data into consideration and may include instructions to slow down if the user is working too fast, or to encourage a break if fatigue is detected.

[0547] Step 5:

[0548] The server sends the generated work procedure data to the terminal. This includes visual guidance (arrows and highlights) and voice instructions (specific methods of the work and points to pay attention to).

[0549] Step 6:

[0550] The device displays the received work instructions on the user's AR display and plays voice instructions at the appropriate time. This allows the user to understand in detail the specific actions they need to take next.

[0551] Step 7:

[0552] Users perform tasks according to the provided instructions and provide voice feedback on any questions or difficulties they encounter during the process. This feedback may include comments such as "It's too difficult" or "More detailed instructions are needed."

[0553] Step 8:

[0554] The device collects voice feedback data from the user and sends it back to the server. The server uses this feedback to adjust the emotion engine and analysis process to improve the accuracy and suitability of the next work procedure.

[0555] (Example 2)

[0556] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0557] In on-site work environments, it is necessary to appropriately understand the emotional state of workers and present them with the most appropriate work procedures. However, conventional systems do not adequately optimize work procedures while considering emotional states, resulting in insufficient improvements in work efficiency and safety.

[0558] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0559] In this invention, the server includes acquisition means for collecting worker motion data, environmental data, and emotional state data; analysis means for comprehensively analyzing the data transmitted from the acquisition means, identifying the worker's emotional state using an emotion engine, and generating an optimal work procedure; and presentation means for visually and audibly presenting the work procedure generated by the analysis means to the worker. This makes it possible to improve work efficiency and safety by providing appropriate work procedures while taking into account the worker's real-time emotional state.

[0560] A "worker" is an individual engaged in on-site work, whose emotional state and behavioral data are managed by the system.

[0561] "Motion data" refers to information about the physical movements and actions of workers, and is used to improve efficiency and safety in the work environment.

[0562] "Environmental data" includes information about the physical conditions of the work site, and data that indicates factors that affect workers, such as temperature and noise levels.

[0563] "Emotional state data" refers to information that indicates the psychological state of a worker, and includes analysis results from an emotion engine based on facial expressions, voice tone, heart rate, etc.

[0564] "Acquisition means" refers to a device or function in the system for collecting worker motion data, environmental data, and emotional state data.

[0565] "Analysis means" refers to a process and apparatus that uses acquired data to analyze the emotional state of workers using an emotion engine and generate the optimal work procedure.

[0566] "Presentation means" refers to a device or function for conveying visually and audibly generated work procedures to a worker, and includes augmented reality and virtual reality systems.

[0567] A "feedback mechanism" refers to a device or process that accepts responses from workers, such as voices or gestures, and applies them to the analysis mechanism to improve the accuracy of the system.

[0568] An "emotion engine" is an algorithm or technology used to evaluate and analyze a worker's emotional state, and it includes elements that identify emotions based on voice tone and facial expression data.

[0569] A "generative AI model" refers to an artificial intelligence model that learns from a large amount of historical data and generates the optimal work procedure according to the situation.

[0570] This invention provides a system that optimizes work procedures by comprehensively utilizing worker motion data, environmental data, and emotional state in the on-site work environment. This system is composed of three main components: a server, a terminal, and a user.

[0571] First, the device collects user motion data, surrounding environment data, and even the user's emotional state in real time through AR / VR devices and biosensors. Specifically, it captures the user's facial expressions with a camera, acquires voice tone with a microphone, and measures heart rate with a heart rate sensor. This data is stored in a formatted digital format.

[0572] The terminal transmits the collected data to the server via a secure protocol. Encryption technologies such as TLS (Transport Layer Security) are used to maintain the confidentiality and integrity of the data during transmission. The server comprehensively analyzes the data using various analytical software and an emotion engine. The emotion engine identifies the user's emotional state based on voice tone and facial expression data. For example, if the user is feeling anxious or stressed, the work procedures are adjusted accordingly.

[0573] Once the analysis is complete, the server uses a generative AI model to generate the most appropriate work procedure for the user's current state and the on-site conditions. This AI model has learned from past data and has the ability to infer what kind of work guide is best suited to a particular emotional state. For example, if the user is feeling anxious, it will generate a work guide that includes detailed instructions and additional safety checks.

[0574] The generated work procedures are sent to the terminal, and the instructions are displayed on the user's AR display. In addition, by providing voice guidance, users receive emotionally sensitive work instructions in real time, enabling them to perform their tasks efficiently and safely.

[0575] Users perform tasks according to instructions presented on the device and provide feedback using voice commands and gestures as needed. This feedback includes information about the worker's own emotional state, which is used to optimize future instructions.

[0576] One specific example is a scenario in which safety can be improved at a construction site when workers are performing tasks at height, by displaying a guide that includes a detailed safety check process on a terminal.

[0577] Examples of prompts for the generating AI model include: "Generate the optimal work procedure when the user is feeling anxious or stressed. Specifically, tell me how to provide a work guide that includes detailed instructions."

[0578] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0579] Step 1:

[0580] The device collects data using sensors and other devices. Inputs include the user's facial expressions, voice tone, heart rate, ambient temperature, and lighting levels. This data is acquired from cameras, microphones, heart rate sensors, etc., and converted into a digital format. The output is an integrated digital dataset of this data. Specifically, analog signals obtained from sensors are digitized, and filtering and format conversion are performed as needed.

[0581] Step 2:

[0582] The terminal sends the collected data to the server. The input is the digital dataset generated in step 1. The data is transferred through a secure channel using an encryption protocol such as TLS. The output is the secure data received by the server for analysis. This process involves packetizing the data and converting it to a format suitable for network transmission.

[0583] Step 3:

[0584] The server analyzes the received data. The input is an integrated dataset sent from the terminal. Internally, an emotion engine operates to identify the emotional state based on voice tone and facial expression data. Voice analysis algorithms and image processing technologies are used for the analysis. The output is worker state data that reflects the emotional state. For example, if the emotion identified by the analysis result is determined to be "anxiety," that result is used in the next step.

[0585] Step 4:

[0586] The server generates optimal work procedures using a generative AI model. Inputs include user emotional state data and field condition data. The generative AI model infers the optimal instructions by referencing past data and current inputs. The output is an appropriate work procedure, which includes detailed instructions and additional safety checks. For example, the generated work procedure is formatted as a guide text to be displayed on the user's AR display.

[0587] Step 5:

[0588] The terminal presents the user with work instructions received from the server. The input is the work instructions sent from the server. The terminal displays these as visual instructions on the AR display and provides additional explanations through audio guidance. The output is work instructions that the user can receive visually and audibly. This allows the user to proceed with the work while receiving information in real time.

[0589] Step 6:

[0590] The user performs the task based on the provided instructions while providing feedback. The input is the actual work environment and emotional state experienced by the user. The user sends feedback to the terminal using voice commands and gestures. The output is feedback data sent from the terminal to the server, which includes the worker's emotions and work-related requests. This feedback is used to optimize future work instructions.

[0591] (Application Example 2)

[0592] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0593] In on-site processing facilities, optimizing work procedures is necessary to improve the work efficiency and safety of operators. However, standard work procedures do not take into account the emotional state of operators, so when they feel anxious or fatigued, it can become a burden and hinder efficient work. To solve this problem, a system that can dynamically adjust according to the emotional state of operators is desired.

[0594] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0595] In this invention, the server includes an acquisition means for collecting operator motion data and environmental data at a field processing facility; an analysis means for analyzing the data transmitted from the acquisition means and generating an optimal processing procedure; a presentation means for visually and audibly presenting the processing procedure generated by the analysis means to the operator; and an emotion analysis means for analyzing the operator's emotional state and dynamically adjusting the generated processing procedure according to the operator's mental state. This enables flexible work support that takes into account the operator's emotional state.

[0596] A "site processing facility" is a place where manufacturing and assembly work takes place, and it is an environment where operators process goods using machinery and equipment.

[0597] An "operator" is a person who operates machinery and equipment in a processing facility, performs tasks, and is the entity that provides work data and feedback.

[0598] "Motion data" refers to information about the operator's physical movements, and is digital data collected by sensors.

[0599] "Environmental data" refers to digital data that includes information about the physical conditions of the on-site processing facility, such as temperature, humidity, and illuminance.

[0600] "Emotional state" refers to information indicating the operator's mental and psychological state, which can be inferred from facial expressions, tone of voice, heart rate, etc.

[0601] "Acquisition means" refers to devices and technologies that use sensors and other devices to acquire operator movement data and environmental data.

[0602] "Analysis means" refers to devices or processes that analyze data collected from acquisition means and generate the optimal processing procedure.

[0603] "Presentation means" refers to devices or methods for visually and audibly showing the generated processing procedure to an operator, and includes displays and speakers.

[0604] "Emotional analysis means" refers to devices or methods for analyzing the emotional state of an operator from data and dynamically adjusting the processing procedure.

[0605] "Feedback" refers to the opinions and reactions obtained from operators, which are used to improve the accuracy of future analysis methods.

[0606] This invention is a system that supports operators in on-site processing facilities in performing their work safely and efficiently. It is composed of three main components: a server, a terminal, and a user, each of which plays a specific role to ensure smooth operation.

[0607] First, the terminal collects motion data, environmental data, and emotional state in real time via smart glasses or head-mounted displays worn by the operator and biosensors. This data includes the operator's facial expressions, heart rate, ambient temperature, and humidity. The collected data is transmitted to a server in digital format.

[0608] The server uses software such as Python and TensorFlow to analyze the operator's emotional state. An emotion engine is activated, and the mental state is identified based on the acquired emotional data. Then, based on the analysis results, the optimal processing procedure is generated. Specifically, if the operator is feeling anxious, adjustments are made to increase the level of detailed instructions and safety checks.

[0609] The generated work procedures are sent back to the terminal and displayed on the operator's smart glasses. The presentation system provides both visual and audio guidance, allowing the operator to confirm the procedures in real time. This system enables operators to work efficiently.

[0610] Users provide feedback using voice commands and gestures as they perform tasks. This feedback includes opinions on the user's emotions and is used to optimize instructions for future tasks.

[0611] For example, if an operator temporarily loses focus during assembly, the system detects this and the robot automatically adjusts its work pace. Additionally, smart glasses prompt the operator to check the parts needed for the next step.

[0612] An example of a prompt to the generating AI model might be presented in the form of, "The worker appears stressed; please suggest steps to adjust the work procedure for optimal results." This demonstrates how the system would make adjustments under specific conditions.

[0613] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0614] Step 1:

[0615] The terminal collects motion data, environmental data, and emotional state from the operator's smart glasses or head-mounted display. Inputs include the operator's facial expressions, heart rate, and ambient temperature and humidity, which are acquired as digital data by biosensors. Outputs are the collected biosensors and environmental data. Each data point is transmitted from the sensor to the smart glasses in real time.

[0616] Step 2:

[0617] The terminal sends the collected data to the server. The input consists of the motion data and environmental data collected in step 1, which are sent to the server via the network. The output is digital data stored on the server. This data forms the basis for analysis.

[0618] Step 3:

[0619] The server analyzes the data using software such as Python and TensorFlow to identify the operator's emotional state. The input is the digital data transmitted in step 2, and the emotion engine performs emotion analysis. The output is data indicating the operator's emotional state. Based on these analysis results, the optimal processing procedure is considered.

[0620] Step 4:

[0621] The server generates the optimal processing procedure based on the analysis results. The input is the emotional state data obtained in step 3, and the generated AI model adjusts the work procedure based on this data. The output is the adjusted work instruction data. Flexible and safe procedures are constructed according to the mental state of the operator.

[0622] Step 5:

[0623] The server resends the generated work instructions to the terminal. The input is the work instruction data generated in step 4, which is then sent to the terminal. The output is the visual and audio instruction data on the terminal. This data is displayed on the operator's smart glasses.

[0624] Step 6:

[0625] The user performs the task according to the work procedure presented on the terminal. The input is the work instruction data presented in step 5, and the user performs production activities based on it. The output is the actual work result. Provide feedback using voice commands or gestures as needed.

[0626] Step 7:

[0627] The terminal collects user feedback and sends it to the server. Input is feedback data provided by the operator, obtained through voice and gestures. Output is feedback data stored on the server. This data is used for analysis and instruction generation in subsequent instances.

[0628] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0629] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0630] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0631] [Fourth Embodiment]

[0632] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0633] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0634] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0635] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0636] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0637] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0638] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0639] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0640] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0641] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0642] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0643] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0644] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0645] This invention provides a system designed to enable workers in the field to efficiently perform complex tasks. This system consists of three main elements: a server, a terminal, and a user.

[0646] First, the terminal uses an AR / VR device to collect motion data of the user working on-site and data about the surrounding environment. For example, it collects how the user is using a particular tool, including its movements and operation, and records this data digitally within the terminal. The terminal then transmits this data to a server.

[0647] The server receives and analyzes data sent from the terminal. Using state-of-the-art machine learning algorithms and computer vision technology, it recognizes user actions and generates the optimal procedure for the task. For example, in a specific welding process, it can automatically create a procedure that combines a 3D model with voice instructions to guide the user in making fine adjustments to the timing and angle of their actions.

[0648] The generated instructions are returned to the terminal. The terminal displays these as visual information overlaid on the user's field of view, and simultaneously provides specific guidance to the user using voice prompts. For example, if the welding position is inaccurate, visual guidelines and target marks are displayed on the AR display, allowing the user to correct it immediately.

[0649] Users perform tasks based on instructions received visually and audibly. This system allows even inexperienced individuals to achieve results comparable to those of skilled workers. If a user has questions or requires further instructions during the task, they can provide feedback using voice commands or gestures, which is sent to the server and incorporated into subsequent instructions.

[0650] In this way, the system achieves improved skills and optimized work efficiency in the workplace through close collaboration between servers, terminals, and users.

[0651] The following describes the processing flow.

[0652] Step 1:

[0653] The device uses AR / VR equipment to collect user motion data and surrounding environment data in real time. Specifically, it uses cameras and sensors to acquire data such as the user's hand movements, gaze, and the arrangement of objects around the user.

[0654] Step 2:

[0655] The terminal bundles the collected data into packets and sends them to the server via low-latency communication. This transmission process is designed to ensure data accuracy and timely delivery.

[0656] Step 3:

[0657] The server analyzes the received motion and environmental data. Using machine learning algorithms, it identifies the user's motion patterns and surrounding conditions from the data and generates optimal instructions necessary for the task. For example, it identifies the appropriate tool usage and operating angles.

[0658] Step 4:

[0659] The server reconstructs the generated instructions as data and returns it to the terminal. This data includes visual guidelines and audio instruction files.

[0660] Step 5:

[0661] The terminal displays the received instruction data on the user's AR display and provides supplementary information via audio output. This allows the user to proceed with their work while confirming the work procedure in real time.

[0662] Step 6:

[0663] The user performs the task according to the instructions provided. They perform or modify the instructions as needed, and provide feedback to the device via voice or gestures if they have any questions.

[0664] Step 7:

[0665] The terminal digitizes user feedback and sends it to the server. This data is used to improve instructions and is utilized for analysis in future sessions.

[0666] (Example 1)

[0667] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0668] In on-site work, the skill level and efficiency of workers directly impact the quality of the work. However, not all workers are skilled, and there is a need for support to enable less skilled workers to perform work efficiently and with high quality. Furthermore, since the work environment changes constantly, a system that can provide appropriate instructions in real time is necessary. Conventional methods have the problem of insufficient speed and accuracy in providing feedback to workers, making immediate response on-site difficult.

[0669] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0670] In this invention, the server includes means for collecting operational information and surrounding information in the work environment, means for transmitting the information using communication technology and analyzing it in a central processing unit, and means for generating work procedures using machine learning and visual recognition technology. This provides real-time feedback tailored to the worker's skills and environment, enabling even inexperienced workers to efficiently perform high-quality work.

[0671] "Work environment" refers to the physical or virtual space in which work is performed, and the place where the worker carries out their actions.

[0672] A "worker" refers to an individual who receives instructions through a system and actually performs the work.

[0673] A "visual device" refers to a device that displays digital information on a screen, and can overlay information onto the real-world environment.

[0674] "Motion information" refers to data related to the physical movements of workers and specific actions taken during their work.

[0675] "Ambient information" refers to data that indicates the conditions and circumstances of the work environment, including light intensity, temperature, and humidity.

[0676] "Communication technology" refers to wireless or wired technologies for efficiently and securely sending and receiving data.

[0677] A "central processing unit" refers to the main computer system that receives collected data and performs analysis and processing on it.

[0678] "Machine learning" refers to the technology that enables computers to learn patterns from data and make decisions and predictions.

[0679] "Visual recognition technology" refers to the technology of extracting and understanding specific information or patterns from images and videos.

[0680] A "work procedure" is a set of instructions outlining the steps necessary to perform a specific task efficiently and safely.

[0681] "Visual information" refers to images and video content displayed to the user.

[0682] "Audio information" refers to instructions and guidance provided to users via voice.

[0683] "Response" refers to the reaction that arises from the operator's actions or feedback.

[0684] This invention is a system that provides support for workers to perform their tasks efficiently. The system mainly consists of three components: a server, a terminal, and a user.

[0685] The terminal uses a visual device worn by the worker, specifically an AR / VR device, to collect information about the work environment, including movement and surroundings. This terminal captures the user's hand movements and tool usage in real time, and also measures ambient brightness and temperature. For example, when a user welds an object, the terminal meticulously records their hand movements and tool operation.

[0686] The server receives information sent from the terminal and performs advanced analysis. Specifically, it uses machine learning algorithms implemented in programming languages ​​such as Python and the open-source library OpenCV to analyze user actions in detail. Based on this data, the server generates the optimal work procedure. For example, a high-performance server equipped with an 856 processor will combine historical data with current environmental conditions to determine the most effective procedure for the user.

[0687] The generated instructions are sent to a terminal and presented to the user via a visual device. The terminal overlays visual information such as 3D models and arrows onto the worker's field of view, while simultaneously providing voice instructions. This allows the user to obtain information through multiple senses, improving work efficiency. For example, a user performing gardening tasks might receive voice guidance such as, "Next, cut the left branch at a 45-degree angle," while the terminal visually highlights the branch's position.

[0688] Users perform tasks by following explicit visual and auditory instructions. If necessary, they can send feedback to the terminal using voice commands or hand gestures. This feedback information is analyzed by the server and used to generate the next steps. For example, a possible prompt might be, "Suggest a new welding process step."

[0689] This system will enable improvements in technical skills and optimization of work efficiency at the work site.

[0690] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0691] Step 1:

[0692] The terminal collects motion and ambient information through the user's visual device. Inputs include the user's hand movements, tool usage, and ambient light levels and temperature. This data is converted into a digital format optimized for motion analysis and stored in a data buffer. Specifically, a camera captures the user using a tool, providing this footage as foundational data for real-time analysis.

[0693] Step 2:

[0694] The terminal transmits the collected data to the server using communication technology. The input consists of the operational information and surrounding information collected in step 1. The data is securely transmitted to the server via encryption technology such as TLS. The specific role of the terminal is to batch process the data at regular intervals and efficiently transmit it to the server over the network.

[0695] Step 3:

[0696] The server receives and analyzes data sent from the terminal. The input includes user action information and environmental information. The server processes this data using machine learning algorithms and visual recognition technology. Through this processing, it models the user's work actions and generates the optimal work procedure. Specifically, it uses Python and OpenCV to extract the characteristics of the user's movements.

[0697] Step 4:

[0698] The server sends the generated work procedure to the terminal. The input is the optimal procedure obtained through machine learning. The server uses this to construct work instructions in natural language and a 3D model. The output is visual and audio information presented to the user.

[0699] Step 5:

[0700] The terminal presents instructions received from the server to the user. It receives visual and audio instruction data from the server as input. The output consists of guidelines and audio guidance displayed overlaid on the visual device. Specific actions include displaying arrows on the AR display and playing instructions through the speaker.

[0701] Step 6:

[0702] The user performs the task according to the presented visual and audio instructions. The user utilizes instructions received from the terminal as input. As the user progresses, they send feedback to the terminal via voice commands or gestures if they have any questions. Specific actions include operating tools according to instructions and continuously providing feedback on the status to the terminal.

[0703] Step 7:

[0704] The terminal receives feedback from the user and sends it to the server. Input is the user's voice or gesture response. Output is data sent to the server prompting further analysis and instruction generation. The terminal's operation involves quickly detecting feedback and preparing data for the next cycle.

[0705] (Application Example 1)

[0706] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0707] In on-site work, even inexperienced workers are required to perform tasks accurately and efficiently. However, currently, there is a lack of specific and intuitive instruction, resulting in insufficient quality and safety of work. Furthermore, errors in installation position and angle frequently occur during the assembly and maintenance of structures, sometimes leading to rework and safety problems.

[0708] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0709] In this invention, the server includes acquisition means, analysis means, presentation means, feedback means, and guidance means. This makes it possible to present appropriate work procedures to the worker in real time based on motion data, and in particular to provide visual guidelines regarding the mounting position and angle of structures.

[0710] "Acquisition means" refers to a device or method for collecting worker motion data and environmental data during on-site work.

[0711] "Analysis means" refers to technology for analyzing data obtained from acquisition means and generating the optimal work procedure.

[0712] A "presentation means" is a device for visually and audibly presenting the work procedures generated by the analysis means to the worker.

[0713] A "feedback mechanism" is a function or method for receiving feedback from workers and applying it to an analysis mechanism.

[0714] "Guiding means" refers to a method and apparatus for displaying the mounting position and angle of a structure as a guideline.

[0715] The system that realizes this invention uses a combination of three main elements: a server, a terminal, and a user. The terminal uses an AR / VR device to collect worker motion data and environmental data in the field work environment and transmits this data to the server. The server uses advanced machine learning algorithms to analyze the acquired data and generate the optimal work procedure. Specifically, it analyzes the user's movements and generates guidelines to accurately indicate the installation position and angle of the structure.

[0716] The terminal receives instructions sent back from the server, presents them visually by overlaying them onto the user's field of view, and provides voice guidance. The user performs tasks based on the presented guidelines and voice instructions. Furthermore, the user sends voice input and gesture feedback to the server via the terminal, and this feedback is applied to the analysis system.

[0717] A concrete example of this system is the installation of window frames at a construction site. The user wears an AR headset, which visually displays the precise installation position and angle. Furthermore, if an incorrect procedure is performed during the installation process, immediate voice guidance and corrections are provided. This allows even inexperienced workers to perform the task with the same level of precision as experienced workers.

[0718] Example prompt for a generated AI model: "Design a system that uses an AR headset in construction work to display real-time guidelines for mounting position and angle."

[0719] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0720] Step 1:

[0721] The terminal uses AR / VR devices to collect motion and environmental data in the field work environment. The terminal's sensors capture the user's hand movements and position, recording them as digital data related to specific tasks. This data includes the user's current work state and information about the surrounding environment.

[0722] Step 2:

[0723] The device sends the collected data to the server. The transmitted data is raw data from the sensors, including information about the user's movements and location data. The data is securely transferred to the server via the network without any modifications.

[0724] Step 3:

[0725] The server analyzes the raw data received from the terminal and generates the optimal work procedure. Using a generative AI model and machine learning algorithms, the server characterizes user movements from the data and determines the steps necessary to optimize the task. This process involves pattern recognition of movements and analysis compared to past work data.

[0726] Step 4:

[0727] The server creates guidelines to present to the worker based on the analysis results. These guidelines include visual instructions indicating the installation location and angle of the structure, as well as audio instructions regarding work steps. This ensures that specific procedures are clearly communicated to the user.

[0728] Step 5:

[0729] The device receives guidelines sent from the server and displays them overlaid on the user's field of view. Audio instructions are also played simultaneously to inform the user of the next steps. Installation points and the progress of the work are visually represented on the AR display.

[0730] Step 6:

[0731] Users perform tasks according to the provided guidelines and voice instructions. Users can monitor their actions and modify the work procedure as needed.

[0732] Step 7:

[0733] Users send feedback to the server via their device using voice input or gestures. For example, they can provide feedback if they feel the current guidelines are not practical. The server receives this feedback and applies it to analysis to improve the accuracy of future procedures.

[0734] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0735] This invention provides a system that optimizes work procedures by comprehensively utilizing worker motion data, environmental data, and emotional state in the on-site work environment. This system is composed of three main components: a server, a terminal, and a user.

[0736] First, the device collects user motion data, surrounding environment data, and even the user's emotional state in real time through AR / VR devices and biosensors. Specifically, it acquires data such as the user's facial expressions, voice tone, and heart rate, and stores this data in digital format.

[0737] The device sends all collected data to the server. The server then comprehensively analyzes this data. In particular, an emotion engine operates, analyzing emotional data to identify the user's emotional state. For example, if anxiety or stress levels are high, the system adjusts the work procedures accordingly.

[0738] Based on the analysis, the server generates work procedures best suited to the user's current state and on-site conditions. For example, if the user is feeling anxious, it can generate a work guide that includes detailed instructions and additional safety checks.

[0739] The generated optimal work procedure is then sent back to the terminal. The terminal displays this instruction on the user's AR display and also provides audio guidance. This allows the user to receive emotionally sensitive work instructions in real time, enabling them to perform their tasks efficiently and safely.

[0740] Users follow instructions presented on the device and provide feedback using voice commands and gestures as needed. This feedback includes opinions on the user's emotional state, which is used to optimize future instructions.

[0741] Through this series of processes, this system, which incorporates an emotion engine, not only supports the user's skill improvement but also provides flexible work support that takes into account their emotional state.

[0742] The following describes the processing flow.

[0743] Step 1:

[0744] The device uses cameras and biosensors connected to the AR / VR device to collect user motion data, facial expressions, voice tone, and emotion-related data such as heart rate. For example, it simultaneously records the user's hand movements while using tools and their heart rate fluctuations during the task.

[0745] Step 2:

[0746] The device collects behavioral data, emotional data, and ambient environmental data, bundles them into data packets, and sends them to a server via the internet. This data transmission is configured to minimize latency.

[0747] Step 3:

[0748] The server analyzes the received data and evaluates the user's emotional state, particularly using an emotion engine. For example, it uses facial recognition algorithms to detect signs of stress and fatigue from the user's facial features.

[0749] Step 4:

[0750] Based on the analysis results, the server generates the optimal work procedure tailored to the user's current state. It takes emotional data into consideration and may include instructions to slow down if the user is working too fast, or to encourage a break if fatigue is detected.

[0751] Step 5:

[0752] The server sends the generated work procedure data to the terminal. This includes visual guidance (arrows and highlights) and voice instructions (specific methods of the work and points to pay attention to).

[0753] Step 6:

[0754] The device displays the received work instructions on the user's AR display and plays voice instructions at the appropriate time. This allows the user to understand in detail the specific actions they need to take next.

[0755] Step 7:

[0756] Users perform tasks according to the provided instructions and provide voice feedback on any questions or difficulties they encounter during the process. This feedback may include comments such as "It's too difficult" or "More detailed instructions are needed."

[0757] Step 8:

[0758] The device collects voice feedback data from the user and sends it back to the server. The server uses this feedback to adjust the emotion engine and analysis process to improve the accuracy and suitability of the next work procedure.

[0759] (Example 2)

[0760] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0761] In on-site work environments, it is necessary to appropriately understand the emotional state of workers and present them with the most appropriate work procedures. However, conventional systems do not adequately optimize work procedures while considering emotional states, resulting in insufficient improvements in work efficiency and safety.

[0762] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0763] In this invention, the server includes acquisition means for collecting worker motion data, environmental data, and emotional state data; analysis means for comprehensively analyzing the data transmitted from the acquisition means, identifying the worker's emotional state using an emotion engine, and generating an optimal work procedure; and presentation means for visually and audibly presenting the work procedure generated by the analysis means to the worker. This makes it possible to improve work efficiency and safety by providing appropriate work procedures while taking into account the worker's real-time emotional state.

[0764] A "worker" is an individual engaged in on-site work, whose emotional state and behavioral data are managed by the system.

[0765] "Motion data" refers to information about the physical movements and actions of workers, and is used to improve efficiency and safety in the work environment.

[0766] "Environmental data" includes information about the physical conditions of the work site, and data that indicates factors that affect workers, such as temperature and noise levels.

[0767] "Emotional state data" refers to information that indicates the psychological state of a worker, and includes analysis results from an emotion engine based on facial expressions, voice tone, heart rate, etc.

[0768] "Acquisition means" refers to a device or function in the system for collecting worker motion data, environmental data, and emotional state data.

[0769] "Analysis means" refers to a process and apparatus that uses acquired data to analyze the emotional state of workers using an emotion engine and generate the optimal work procedure.

[0770] "Presentation means" refers to a device or function for conveying visually and audibly generated work procedures to a worker, and includes augmented reality and virtual reality systems.

[0771] A "feedback mechanism" refers to a device or process that accepts responses from workers, such as voices or gestures, and applies them to the analysis mechanism to improve the accuracy of the system.

[0772] An "emotion engine" is an algorithm or technology used to evaluate and analyze a worker's emotional state, and it includes elements that identify emotions based on voice tone and facial expression data.

[0773] A "generative AI model" refers to an artificial intelligence model that learns from a large amount of historical data and generates the optimal work procedure according to the situation.

[0774] This invention provides a system that optimizes work procedures by comprehensively utilizing worker motion data, environmental data, and emotional state in the on-site work environment. This system is composed of three main components: a server, a terminal, and a user.

[0775] First, the device collects user motion data, surrounding environment data, and even the user's emotional state in real time through AR / VR devices and biosensors. Specifically, it captures the user's facial expressions with a camera, acquires voice tone with a microphone, and measures heart rate with a heart rate sensor. This data is stored in a formatted digital format.

[0776] The terminal transmits the collected data to the server via a secure protocol. Encryption technologies such as TLS (Transport Layer Security) are used to maintain the confidentiality and integrity of the data during transmission. The server comprehensively analyzes the data using various analytical software and an emotion engine. The emotion engine identifies the user's emotional state based on voice tone and facial expression data. For example, if the user is feeling anxious or stressed, the work procedures are adjusted accordingly.

[0777] Once the analysis is complete, the server uses a generative AI model to generate the most appropriate work procedure for the user's current state and the on-site conditions. This AI model has learned from past data and has the ability to infer what kind of work guide is best suited to a particular emotional state. For example, if the user is feeling anxious, it will generate a work guide that includes detailed instructions and additional safety checks.

[0778] The generated work procedures are sent to the terminal, and the instructions are displayed on the user's AR display. In addition, by providing voice guidance, users receive emotionally sensitive work instructions in real time, enabling them to perform their tasks efficiently and safely.

[0779] Users perform tasks according to instructions presented on the device and provide feedback using voice commands and gestures as needed. This feedback includes information about the worker's own emotional state, which is used to optimize future instructions.

[0780] One specific example is a scenario in which safety can be improved at a construction site when workers are performing tasks at height, by displaying a guide that includes a detailed safety check process on a terminal.

[0781] Examples of prompts for the generating AI model include: "Generate the optimal work procedure when the user is feeling anxious or stressed. Specifically, tell me how to provide a work guide that includes detailed instructions."

[0782] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0783] Step 1:

[0784] The device collects data using sensors and other devices. Inputs include the user's facial expressions, voice tone, heart rate, ambient temperature, and lighting levels. This data is acquired from cameras, microphones, heart rate sensors, etc., and converted into a digital format. The output is an integrated digital dataset of this data. Specifically, analog signals obtained from sensors are digitized, and filtering and format conversion are performed as needed.

[0785] Step 2:

[0786] The terminal sends the collected data to the server. The input is the digital dataset generated in step 1. The data is transferred through a secure channel using an encryption protocol such as TLS. The output is the secure data received by the server for analysis. This process involves packetizing the data and converting it to a format suitable for network transmission.

[0787] Step 3:

[0788] The server analyzes the received data. The input is an integrated dataset sent from the terminal. Internally, an emotion engine operates to identify the emotional state based on voice tone and facial expression data. Voice analysis algorithms and image processing technologies are used for the analysis. The output is worker state data that reflects the emotional state. For example, if the emotion identified by the analysis result is determined to be "anxiety," that result is used in the next step.

[0789] Step 4:

[0790] The server generates optimal work procedures using a generative AI model. Inputs include user emotional state data and field condition data. The generative AI model infers the optimal instructions by referencing past data and current inputs. The output is an appropriate work procedure, which includes detailed instructions and additional safety checks. For example, the generated work procedure is formatted as a guide text to be displayed on the user's AR display.

[0791] Step 5:

[0792] The terminal presents the user with work instructions received from the server. The input is the work instructions sent from the server. The terminal displays these as visual instructions on the AR display and provides additional explanations through audio guidance. The output is work instructions that the user can receive visually and audibly. This allows the user to proceed with the work while receiving information in real time.

[0793] Step 6:

[0794] The user performs the task based on the provided instructions while providing feedback. The input is the actual work environment and emotional state experienced by the user. The user sends feedback to the terminal using voice commands and gestures. The output is feedback data sent from the terminal to the server, which includes the worker's emotions and work-related requests. This feedback is used to optimize future work instructions.

[0795] (Application Example 2)

[0796] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0797] In on-site processing facilities, optimizing work procedures is necessary to improve the work efficiency and safety of operators. However, standard work procedures do not take into account the emotional state of operators, so when they feel anxious or fatigued, it can become a burden and hinder efficient work. To solve this problem, a system that can dynamically adjust according to the emotional state of operators is desired.

[0798] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0799] In this invention, the server includes an acquisition means for collecting operator motion data and environmental data at a field processing facility; an analysis means for analyzing the data transmitted from the acquisition means and generating an optimal processing procedure; a presentation means for visually and audibly presenting the processing procedure generated by the analysis means to the operator; and an emotion analysis means for analyzing the operator's emotional state and dynamically adjusting the generated processing procedure according to the operator's mental state. This enables flexible work support that takes into account the operator's emotional state.

[0800] A "site processing facility" is a place where manufacturing and assembly work takes place, and it is an environment where operators process goods using machinery and equipment.

[0801] An "operator" is a person who operates machinery and equipment in a processing facility, performs tasks, and is the entity that provides work data and feedback.

[0802] "Motion data" refers to information about the operator's physical movements, and is digital data collected by sensors.

[0803] "Environmental data" refers to digital data that includes information about the physical conditions of the on-site processing facility, such as temperature, humidity, and illuminance.

[0804] "Emotional state" refers to information indicating the operator's mental and psychological state, which can be inferred from facial expressions, tone of voice, heart rate, etc.

[0805] "Acquisition means" refers to devices and technologies that use sensors and other devices to acquire operator movement data and environmental data.

[0806] "Analysis means" refers to devices or processes that analyze data collected from acquisition means and generate the optimal processing procedure.

[0807] "Presentation means" refers to devices or methods for visually and audibly showing the generated processing procedure to an operator, and includes displays and speakers.

[0808] "Emotional analysis means" refers to devices or methods for analyzing the emotional state of an operator from data and dynamically adjusting the processing procedure.

[0809] "Feedback" refers to the opinions and reactions obtained from operators, which are used to improve the accuracy of future analysis methods.

[0810] This invention is a system that supports operators in on-site processing facilities in performing their work safely and efficiently. It is composed of three main components: a server, a terminal, and a user, each of which plays a specific role to ensure smooth operation.

[0811] First, the terminal collects motion data, environmental data, and emotional state in real time via smart glasses or head-mounted displays worn by the operator and biosensors. This data includes the operator's facial expressions, heart rate, ambient temperature, and humidity. The collected data is transmitted to a server in digital format.

[0812] The server uses software such as Python and TensorFlow to analyze the operator's emotional state. An emotion engine is activated, and the mental state is identified based on the acquired emotional data. Then, based on the analysis results, the optimal processing procedure is generated. Specifically, if the operator is feeling anxious, adjustments are made to increase the level of detailed instructions and safety checks.

[0813] The generated work procedures are sent back to the terminal and displayed on the operator's smart glasses. The presentation system provides both visual and audio guidance, allowing the operator to confirm the procedures in real time. This system enables operators to work efficiently.

[0814] Users provide feedback using voice commands and gestures as they perform tasks. This feedback includes opinions on the user's emotions and is used to optimize instructions for future tasks.

[0815] For example, if an operator temporarily loses focus during assembly, the system detects this and the robot automatically adjusts its work pace. Additionally, smart glasses prompt the operator to check the parts needed for the next step.

[0816] An example of a prompt to the generating AI model might be presented in the form of, "The worker appears stressed; please suggest steps to adjust the work procedure for optimal results." This demonstrates how the system would make adjustments under specific conditions.

[0817] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0818] Step 1:

[0819] The terminal collects motion data, environmental data, and emotional state from the operator's smart glasses or head-mounted display. Inputs include the operator's facial expressions, heart rate, and ambient temperature and humidity, which are acquired as digital data by biosensors. Outputs are the collected biosensors and environmental data. Each data point is transmitted from the sensor to the smart glasses in real time.

[0820] Step 2:

[0821] The terminal sends the collected data to the server. The input consists of the motion data and environmental data collected in step 1, which are sent to the server via the network. The output is digital data stored on the server. This data forms the basis for analysis.

[0822] Step 3:

[0823] The server analyzes the data using software such as Python and TensorFlow to identify the operator's emotional state. The input is the digital data transmitted in step 2, and the emotion engine performs emotion analysis. The output is data indicating the operator's emotional state. Based on these analysis results, the optimal processing procedure is considered.

[0824] Step 4:

[0825] The server generates the optimal processing procedure based on the analysis results. The input is the emotional state data obtained in step 3, and the generated AI model adjusts the work procedure based on this data. The output is the adjusted work instruction data. Flexible and safe procedures are constructed according to the mental state of the operator.

[0826] Step 5:

[0827] The server resends the generated work instructions to the terminal. The input is the work instruction data generated in step 4, which is then sent to the terminal. The output is the visual and audio instruction data on the terminal. This data is displayed on the operator's smart glasses.

[0828] Step 6:

[0829] The user performs the task according to the work procedure presented on the terminal. The input is the work instruction data presented in step 5, and the user performs production activities based on it. The output is the actual work result. Provide feedback using voice commands or gestures as needed.

[0830] Step 7:

[0831] The terminal collects user feedback and sends it to the server. Input is feedback data provided by the operator, obtained through voice and gestures. Output is feedback data stored on the server. This data is used for analysis and instruction generation in subsequent instances.

[0832] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0833] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0834] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0835] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0836] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0837] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0838] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0839] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0840] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0841] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0842] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0843] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0844] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0845] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0846] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0847] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0848] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0849] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0850] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0851] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0852] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0853] The following is further disclosed regarding the embodiments described above.

[0854] (Claim 1)

[0855] In the on-site work environment, an acquisition means for collecting worker motion data and environmental data,

[0856] An analysis means that analyzes the data transmitted from the acquisition means and generates the optimal work procedure,

[0857] A presentation means for visually and audibly presenting the work procedure generated by the analysis means to the worker,

[0858] A feedback means that receives the worker's feedback and applies it to the analysis means,

[0859] A system that includes this.

[0860] (Claim 2)

[0861] The system according to claim 1, wherein the presentation means presents work procedures using an augmented reality or virtual reality device.

[0862] (Claim 3)

[0863] The system according to claim 1, wherein the feedback means receives feedback via voice input or gesture, thereby improving the accuracy of the analysis means.

[0864] "Example 1"

[0865] (Claim 1)

[0866] In the work environment, means for collecting motion information and surrounding information using a visual device worn by the worker,

[0867] A means for obtaining the optimal procedure generated by transmitting the information acquired by the aforementioned means using communication technology and analyzing it in a central processing unit,

[0868] A means for generating work procedures that take into account the worker's past information and environmental conditions, utilizing machine learning and visual recognition technologies,

[0869] A means of presenting the generated work procedure to the worker as visual and audio information,

[0870] A means for receiving responses based on the operator's actions and applying them to analysis by a central processing unit,

[0871] A system that includes this.

[0872] (Claim 2)

[0873] The system according to claim 1, wherein the presentation means presents work procedures using visual augmentation technology and auditory guidance.

[0874] (Claim 3)

[0875] The system according to claim 1, wherein the feedback means receives responses in the form of voice commands and motion gestures, thereby improving the effectiveness of the analysis means.

[0876] "Application Example 1"

[0877] (Claim 1)

[0878] In the on-site work environment, an acquisition means for collecting worker motion data and environmental data,

[0879] An analysis means that analyzes the data transmitted from the acquisition means and generates the optimal work procedure,

[0880] A presentation means for visually and audibly presenting the work procedure generated by the analysis means to the worker,

[0881] A feedback means that receives the worker's feedback and applies it to the analysis means,

[0882] A guide means that displays the mounting position and angle of a structure as a guideline,

[0883] A system that includes this.

[0884] (Claim 2)

[0885] The system according to claim 1, wherein the presentation means presents work procedures using an augmented reality or virtual reality device.

[0886] (Claim 3)

[0887] The system according to claim 1, wherein the feedback means receives feedback via voice input or gesture, thereby improving the accuracy of the analysis means.

[0888] "Example 2 of combining an emotion engine"

[0889] (Claim 1)

[0890] A means for collecting worker motion data, environmental data, and emotional state data,

[0891] An analysis means that comprehensively analyzes the data transmitted from the acquisition means, identifies the emotional state of the worker using an emotion engine, and generates the optimal work procedure.

[0892] A presentation means for visually and audibly presenting the work procedure generated by the analysis means to the worker,

[0893] A feedback means that receives feedback including the worker's voice commands or gestures and applies it to the analysis means,

[0894] A system that includes this.

[0895] (Claim 2)

[0896] The system according to claim 1, wherein the presentation means presents work procedures using an augmented reality or virtual reality system.

[0897] (Claim 3)

[0898] The system according to claim 1, wherein the feedback means receives feedback including an evaluation of the worker's emotional state, thereby improving the accuracy of the analysis means and the applicability of the generated AI model.

[0899] "Application example 2 when combining with an emotional engine"

[0900] (Claim 1)

[0901] In an on-site processing facility, an acquisition means for collecting operator motion data and environmental data,

[0902] An analysis means analyzes the data transmitted from the acquisition means and generates the optimal processing procedure,

[0903] A presentation means for visually and audibly presenting the processing procedure generated by the analysis means to the operator,

[0904] A feedback means that receives feedback from the operator and applies it to the analysis means,

[0905] An emotional analysis means that analyzes the emotional state of the operator and dynamically adjusts the generated processing procedure according to the operator's mental state,

[0906] A system that includes this.

[0907] (Claim 2)

[0908] The system according to claim 1, wherein the presentation means presents a processing procedure using an augmented reality or virtual reality device.

[0909] (Claim 3)

[0910] The system according to claim 1, wherein the feedback means receives feedback via voice input or gesture, thereby improving the accuracy of the analysis means. [Explanation of symbols]

[0911] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. In the on-site work environment, an acquisition means for collecting worker motion data and environmental data, An analysis means that analyzes the data transmitted from the acquisition means and generates the optimal work procedure, A presentation means for visually and audibly presenting the work procedure generated by the analysis means to the worker, A feedback means that receives the worker's feedback and applies it to the analysis means, A guide means that displays the mounting position and angle of a structure as a guideline, A system that includes this.

2. The system according to claim 1, wherein the presentation means presents work procedures using an augmented reality or virtual reality device.

3. The system according to claim 1, wherein the feedback means receives feedback via voice input or gesture, and improves the accuracy of the analysis means.

Images