system

Abstract

This system enables accurate real-time monitoring of subordinates' work status, prompt and appropriate instructions, and ultimately improves work efficiency and safety. [Solution] A system comprising: means for collecting physical data using sensor means; means for preprocessing the collected data to extract important features; means for transmitting the preprocessed data to a server; means for the server to analyze the received data and determine a specific situation; means for generating a notification based on the analysis results and transmitting it to a terminal; means for the terminal to display the notification and prompt the user to input instructions; means for transmitting the user's instructions to the server, which then redistributes them to a chip; means for the receiving chip to reproduce or display the content of the instructions; and means for transmitting the recollected data to the server to provide feedback.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In a conventional work monitoring system, there is a problem that it takes labor and time for a manager to grasp the operating status of subordinates. Also, it is difficult to make real-time situation judgments and give instructions in a remote environment, and efficient work management has not been achieved. Furthermore, the collection and analysis of sensor data are insufficient, and there is a lack of means to effectively improve the safety and work efficiency of subordinates.

Means for Solving the Problems

[0005] The present invention solves the above problems by providing a system that includes means for collecting physical data using sensor means, means for preprocessing the collected data to extract important features, means for transmitting the preprocessed data to a server, means for the server to analyze the received data and determine a specific situation, means for generating a notification based on the analysis results and transmitting it to a terminal, means for the terminal to display the notification and prompt the user to input instructions, means for transmitting the user's instructions to the server and for the server to redistribute them to a chip, means for the chip that received the instructions to reproduce or display the contents, and means for transmitting the recollected data to the server to provide feedback.

[0006] A "sensor" refers to a device that measures physical data (such as motion information, location information, temperature information, etc.) and outputs it as digital data.

[0007] "Preprocessing" refers to the process of removing noise from collected raw data and extracting important features.

[0008] A "server" refers to a computer system that receives, stores, and analyzes data via a network, and then transmits the results to other devices (such as terminals or chips).

[0009] "Terminal" refers to a device (such as a personal computer, smartphone, or tablet) used by a user to check notifications and enter instructions.

[0010] A "user" refers to a manager who operates the system, monitors the work status of their subordinates, and issues instructions.

[0011] An "AI model" refers to a program that uses machine learning algorithms and deep learning to analyze data and make judgments about specific situations.

[0012] "Notification" refers to an informational message that the server generates based on analysis results and sends to the terminal.

[0013] A "chip" refers to a small electronic device that incorporates sensors and performs data collection, data transmission and reception, and display and playback of instructions.

[0014] A "real-time communication protocol" refers to a communication protocol (such as MQTT or WebSocket) that enables immediate, bidirectional data communication.

[0015] "Feedback" refers to the process of collecting data on the results of actions taken based on user instructions and sending it back to the server. [Brief explanation of the drawing]

[0016] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when the emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when the emotion engine is combined.

Mode for Carrying Out the Invention

[0017] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), APU (Accelerated Processing Unit), etc.

[0020] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0021] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0022] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0024] [First Embodiment]

[0025] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0026] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0029] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0032] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0036] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0037] This invention relates to a real-time work monitoring and remote instruction system using sensor means. Embodiments of this system are described in detail below.

[0038] System-wide configuration

[0039] This system aims to monitor the work status of subordinates in real time and provide efficient and appropriate instructions. Its main components are sensors, chips, servers, terminals, and users.

[0040] System Overview

[0041] 1. Sensor and chip functions

[0042] The sensors monitor the subordinates' work status in real time. Specifically, they combine IMUs (Inertial Measurement Units), GPS, temperature sensors, and other sensors to acquire operational information, location information, and environmental information.

[0043] The chip receives data from the sensor and performs data preprocessing. Preprocessing includes noise reduction and feature extraction. The preprocessed data is transmitted wirelessly to the server at regular intervals.

[0044] 2. Server Functions

[0045] The server receives data transmitted from the chip and stores it in a database. The stored data is then analyzed in real time by an AI model.

[0046] Based on the analysis results, the server determines a specific situation and generates an appropriate notification. For example, if the situation "Subordinate A is not wearing a safety harness" is detected, that fact will be included in the notification.

[0047] The generated notification is sent to the user's device.

[0048] 3. Device functions

[0049] The device receives a notification and displays it to the user. Based on this notification, the user can input specific instructions. For example, they can use text or voice to give instructions such as "Please readjust your safety harness."

[0050] The user's input is sent back to the server, which then delivers the instructions to the chip.

[0051] 4. User Roles

[0052] The user (administrator) uses a device to monitor notifications and issue instructions as needed. Instructions can be easily given via text or voice input.

[0053] By providing appropriate instructions, users can improve the work efficiency of their subordinates and maintain a safe working environment.

[0054] Specific examples of the system

[0055] Specific example: Use at construction sites

[0056] 1. Collection of sensor data

[0057] The chip is attached to the helmet of a subordinate working at a construction site. The IMU sensor tracks the subordinate's movements, the GPS tracks their location, and the temperature sensor measures the ambient temperature.

[0058] 2. Data preprocessing and transmission

[0059] The chip preprocesses the data obtained from the sensor and extracts specific features (for example, position and movement patterns during work).

[0060] 3. Data Analysis

[0061] The server analyzes the transmitted data in real time and detects if there is a possibility that the subordinate is not wearing their safety harness correctly.

[0062] 4. Creating and sending notifications

[0063] The server generates a notification and sends it to the administrator's terminal.

[0064] 5. Administrator's response

[0065] The user (administrator) checks the notification on their device and enters the instruction, "Please readjust your safety harness."

[0066] 6. Communication and execution of instructions

[0067] Instructions are transmitted to the chip via the server, and the subordinate receives the instructions.

[0068] The subordinate followed instructions and readjusted his safety harness.

[0069] 7. Gathering Feedback

[0070] The chip collects data again and sends it to the server, providing feedback on the results of the instructions' execution.

[0071] This allows managers to monitor their subordinates' on-site work in real time and issue appropriate instructions. This system significantly improves work efficiency and safety.

[0072] The following describes the processing flow.

[0073] Step 1:

[0074] tip

[0075] Sensors (such as IMUs, GPS, and temperature sensors) measure the subordinate's movements, location, and environmental data in real time. This raw data is temporarily stored inside the chip.

[0076] Step 2:

[0077] tip

[0078] The collected raw data is preprocessed. Preprocessing includes noise reduction and feature extraction. For example, features of the subordinate's posture and movements are extracted from IMU data.

[0079] Step 3:

[0080] tip

[0081] Pre-processed data is transmitted to the server wirelessly (Wi-Fi, 4G / 5G, etc.) at regular time intervals (e.g., every second). If transmission is successful, the data is deleted from internal memory. If transmission fails, a retransmission is attempted.

[0082] Step 4:

[0083] server

[0084] The system receives data transmitted from the chip and stores it in a database. Upon receipt, the system performs a data integrity check, and if invalid or missing data is detected, it sends a retransmission instruction to the chip.

[0085] Step 5:

[0086] server

[0087] The system analyzes stored data in real time. It uses machine learning models (e.g., deep learning models) to determine the status of subordinates (e.g., "operating normally," "anomaly detected," etc.) based on the data.

[0088] Step 6:

[0089] server

[0090] The system generates notifications based on the analysis results. For example, if it detects an anomaly such as "Subordinate A is not wearing a safety harness," it will generate a notification including that information.

[0091] Step 7:

[0092] server

[0093] The generated notification is sent to the user's device. The sending method can be push notification or email.

[0094] Step 8:

[0095] terminal

[0096] Receive notifications from the server and display them to the user. Users can check notifications on their devices and view detailed information on the dashboard.

[0097] Step 9:

[0098] User

[0099] Based on the notification, enter the necessary instructions into the terminal. For example, you might enter a message such as "Instruct subordinate A to readjust their safety harness" via voice or text.

[0100] Step 10:

[0101] terminal

[0102] The user's input instructions are sent to the server. Error checking and retransmission are performed during transmission.

[0103] Step 11:

[0104] server

[0105] It receives user instructions and delivers them to the chip. It uses real-time communication protocols (e.g., MQTT or WebSockets).

[0106] Step 12:

[0107] tip

[0108] Receive instructions and notify subordinates. If the instructions are voiced, play them through the speaker; if they are text-based, display them on the screen.

[0109] Step 13:

[0110] subordinate

[0111] Confirm instructions from the user and act accordingly. For example, readjust your safety harness.

[0112] Step 14:

[0113] tip

[0114] The sensor data is collected again and sent to the server. This verifies that the instructions were executed correctly.

[0115] Step 15:

[0116] server

[0117] Analyze the newly collected data to confirm that the instructions were followed correctly. If the problem recurs, send another notification.

[0118] (Example 1)

[0119] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0120] Conventional work monitoring and instruction systems made it difficult to grasp the work status of subordinates in real time and to issue appropriate instructions quickly. Furthermore, the data obtained from sensors contained a lot of noise and irrelevant information, resulting in insufficient pre-processing for accurate situational assessment. In addition, delays in generating notifications based on analysis results and providing feedback on instructions based on those results prevented sufficient improvements in work efficiency and safety.

[0121] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0122] In this invention, the server includes means for collecting physical data using sensor means, means for preprocessing the collected data to extract important features, means for transmitting the preprocessed data to a computer, means for the computer to analyze the received data and determine a specific situation, means for generating a notification based on the analysis results and transmitting it to a terminal, means for the terminal to display the notification and prompt the user to input instructions, means for transmitting the user's instructions to the computer and for the computer to redistribute them to a chip, means for the chip that received the instructions to reproduce or display their contents, means for transmitting the recollected data to the computer to provide feedback, and means for the computer to perform real-time analysis using a domain-specific model based on the instructions input by the user. This makes it possible to accurately grasp the work status of subordinates in real time and quickly issue appropriate instructions. Furthermore, it is possible to improve work efficiency and safety.

[0123] A "sensor" is a device or system used to collect physical data.

[0124] "Preprocessing" refers to the process of removing noise from collected data and extracting important features.

[0125] "Transmission means" refers to a function for transmitting pre-processed data to other devices or systems.

[0126] A "computer" is a device or system that analyzes received data and makes judgments about specific situations.

[0127] "Analysis means" refers to methods or functions for making judgments about specific situations based on data.

[0128] A "notification" is information or a message generated based on the analysis results.

[0129] A "terminal" is a device that displays notifications to the user and allows them to input instructions.

[0130] A "user" is a person or administrator who uses a terminal to input instructions.

[0131] "Feedback" refers to information used to verify the progress of instructions based on recollected data.

[0132] A "domain-specific model" is an AI model that is specialized for a particular industry or domain.

[0133] "Real-time analysis" is an analytical method that processes data immediately and obtains results instantly.

[0134] Modes for carrying out the invention

[0135] This invention relates to a real-time work monitoring and remote instruction system using sensor means. The system aims to monitor the work status of subordinates in real time and provide efficient and appropriate instructions. The main components are sensors, chips, a server, a terminal, and the user. The following describes specific embodiments of this system.

[0136] Sensor and chip functions

[0137] The sensors monitor the work status of subordinates in real time. Specifically, they combine IMUs (Inertial Measurement Units), GPS, temperature sensors, etc., to acquire motion information, location information, and environmental information. These sensors are often attached to the worker's helmet or work clothes. For example, an IMU sensor attached to the helmet measures the subordinate's movements, a GPS determines their location, and a temperature sensor measures the ambient temperature.

[0138] The chip receives data from sensors and performs preprocessing such as noise reduction and feature extraction. For example, it removes unwanted vibration information from collected IMU sensor data and extracts specific operating patterns. The preprocessed data is transmitted to a server via wireless communication at regular intervals.

[0139] Server Functions

[0140] The server receives data transmitted from the chip and stores it in a database. The received data is temporarily stored in memory and then saved in the specified database. For example, databases such as MySQL® or PostgreSQL may be used.

[0141] The server analyzes the stored data in real time. This analysis may utilize generative AI models such as TENSORFLOW® or PyTorch. Through this analysis, it analyzes movement patterns and location information to detect abnormal situations, such as "Subordinate A is not wearing a safety harness."

[0142] Based on the analysis results, the server generates an appropriate notification and sends it to the user's terminal. For example, it might generate a notification stating, "Subordinate A is not wearing a safety harness," and send it to the user's terminal.

[0143] Device functions

[0144] The device receives notifications sent from the server and displays them to the user. Notifications are often displayed as pop-ups on the screen. Based on the notification, the user enters specific instructions. For example, they might enter a text instruction such as "Please readjust your safety harness" into the device. The device also has a voice input function, allowing users to enter instructions by voice.

[0145] The user's input is sent back to the server from the terminal. The server distributes the received instructions to a chip, which then plays or displays the instructions to its subordinates. This allows the subordinates to adjust their work according to the user's instructions.

[0146] Specific example

[0147] Use at construction sites

[0148] 1. Sensor data collection: The chip is attached to the helmet of a subordinate working at the construction site, and the IMU sensor measures operational information, the GPS measures location information, and the temperature sensor measures ambient temperature.

[0149] 2. Data preprocessing and transmission: The chip preprocesses the data obtained from the sensor and extracts specific features (position and movement patterns during operation).

[0150] 3. Data Analysis: The server analyzes the transmitted data in real time to detect the possibility that a subordinate is not wearing their safety harness correctly.

[0151] 4. Creating and sending notifications: The server generates a notification and sends it to the administrator's terminal.

[0152] 5. Administrator's response: The user (administrator) checks the notification on their device and enters the instruction, "Please readjust your safety harness."

[0153] 6. Instruction transmission and execution: Instructions are delivered to the chip via the server, and the subordinate receives them. The subordinate follows the instructions and readjusts their safety harness.

[0154] 7. Feedback Collection: The chip collects data again and sends it to the server to provide feedback on the results of the instructions being executed.

[0155] Example of a prompt

[0156] "At a construction site, there is a system that preprocesses and analyzes data obtained from IMU sensors, GPS, and temperature sensors attached to helmets. Please explain the processing flow of this system in detail."

[0157] This allows us to ask the generated AI model for a detailed explanation of the system.

[0158] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0159] Step 1:

[0160] The user has their subordinates wear sensors. For example, an IMU sensor is attached to a helmet, and a GPS sensor is attached to work clothes. The input is the sensors being worn, and the output is the state of being ready to collect data in real time.

[0161] Step 2:

[0162] Sensors monitor the subordinate's work status in real time. The IMU sensor collects motion data, the GPS collects location data, and the temperature sensor collects ambient temperature data. The input is the subordinate's actions and environmental conditions, and the output is the collection of raw data. Specifically, the IMU sensor measures acceleration and rotation, and the GPS obtains latitude and longitude.

[0163] Step 3:

[0164] The chip receives data from the sensor and performs preprocessing such as noise reduction and feature extraction. It removes unwanted noise from the collected raw data and extracts specific operating patterns. The input is raw data from the sensor, and the output is clean, preprocessed data. Specifically, it performs data filtering and statistical processing.

[0165] Step 4:

[0166] The chip transmits pre-processed data to the server via wireless communication at regular intervals. The input is clean data, and the output is data transmission to the server. Specifically, it transmits data packets using a wireless protocol.

[0167] Step 5:

[0168] The server receives data transmitted from the chip and stores it in the database. The input is the data from the chip, and the output is the stored data. Specifically, it performs a database insertion operation.

[0169] Step 6:

[0170] The server analyzes stored data in real time and performs analysis to determine specific situations. The input is stored data, and the output is the analysis result. Specifically, it performs data analysis using generative AI models such as TensorFlow and PyTorch.

[0171] Step 7:

[0172] The server generates an appropriate notification based on the analysis results and sends it to the user's terminal. The input is the analysis results, and the output is the notification message. Specifically, a notification is generated stating, "Subordinate A is not wearing a safety harness."

[0173] Step 8:

[0174] The device receives notifications from the server and displays them to the user. The input is the notification message, and the output is the notification displayed on the user interface. Specifically, it is displayed as a pop-up notification on the screen.

[0175] Step 9:

[0176] The user enters specific instructions based on a notification displayed on the device. The input is the content of the notification, and the output is the entered instruction. For example, the user might enter text or voice instructions such as "Please readjust your safety harness" into the device.

[0177] Step 10:

[0178] The terminal sends user instructions to the server. The input is the user's instructions, and the output is the data sent to the server. Specifically, text and voice instructions are converted into data packets and sent.

[0179] Step 11:

[0180] The server receives user instructions and redistributes them to the chip. The input is the user's instruction data, and the output is the data to be sent to the chip. Specifically, the instruction content is transmitted using a protocol.

[0181] Step 12:

[0182] The chip receives instructions and plays or displays the content to its subordinates. The input is instruction data from the server, and the output is the notification content for the subordinates. Specifically, it plays the voice message, "Please readjust your safety harness."

[0183] Step 13:

[0184] The chip sends the recollected data to the server and provides feedback. For example, it recollects data to confirm that the safety harness has been reattached. The input is the recollected data, and the output is the feedback data to the server. This allows the administrator to confirm that the instructions have been followed.

[0185] (Application Example 1)

[0186] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0187] Traditional food delivery systems have made it difficult to accurately monitor the work status and efficiency of delivery personnel in real time and to issue appropriate instructions. This has resulted in delivery delays, decreased efficiency, and lower customer satisfaction. Furthermore, delays in providing feedback to delivery personnel and the inability to respond immediately have been a challenge.

[0188] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0189] In this invention, the server includes means for analyzing real-time data, including the movement patterns of delivery personnel; means for evaluating delivery efficiency based on the analysis results and generating instructions to improve efficiency; and means for proposing specific actions to delivery personnel based on user instructions. This makes it possible to accurately grasp the current status of delivery personnel in real time and provide appropriate instructions immediately.

[0190] A "sensor" is a device used to collect physical data in real time.

[0191] "IMU" is an abbreviation for Inertial Measurement Unit, a sensor that detects the movement and acceleration of an object.

[0192] "GPS" is an abbreviation for Global Positioning System, a system used to measure location on Earth.

[0193] An "accelerometer" is a sensor used to measure the acceleration of an object.

[0194] "Preprocessing means" refers to the process of removing noise from collected raw data and extracting important features.

[0195] A "server" is a computer system used to receive, store, and analyze data over a network.

[0196] An "analysis tool" is a mechanism for analyzing collected data and making judgments about a specific situation.

[0197] A "notification system" is a system for sending notifications generated based on analysis results to the user's terminal.

[0198] A "terminal" is a device used by a user to receive notifications and input instructions.

[0199] A "command system" is a system that sends user-inputted instructions to a server, which then redistributes them to the chip.

[0200] A "chip" is a device used to preprocess collected data and send it to a server.

[0201] "Movement patterns" refer to dynamic data such as the delivery person's travel route and speed.

[0202] "Feedback" refers to information about the results and effects of actions provided by sending the recollected data to the server.

[0203] This invention relates to a system that can be applied to food delivery to monitor the work status of delivery personnel in real time and issue appropriate instructions. This system is realized through the cooperation of sensor means, a server, a terminal, and a user.

[0204] System Configuration

[0205] 1. Sensor means:

[0206] Delivery personnel wear smart devices such as smart glasses or smartphones. These devices have built-in IMU sensors, GPS, and accelerometers.

[0207] 2. Data Acquisition and Preprocessing:

[0208] Sensors built into smart devices collect physical data on delivery personnel in real time. This includes location information, movement speed, and motion data.

[0209] The chip preprocesses the data collected from the sensor, performing noise reduction and feature extraction.

[0210] 3. Data transmission and analysis:

[0211] The pre-processed data is transmitted to the server via wireless communication.

[0212] The server stores the received data in a database and analyzes the data in real time using a generative AI model. For example, it analyzes the movement patterns of delivery personnel to detect delivery delays and decreased efficiency.

[0213] 4. Generating and sending notifications:

[0214] The server determines the specific situation based on the analysis results and generates and sends an appropriate notification to the terminal. For example, a notification such as "High acceleration detected. Please check traffic conditions" might be generated.

[0215] 5. User actions:

[0216] The device displays a notification, and the user (administrator) reviews the notification before entering specific instructions. For example, they might enter instructions such as, "There is a rest point nearby; please take a break."

[0217] The input instructions are sent to the server, which then redistributes them to the chip.

[0218] 6. Communication and execution of instructions:

[0219] The chip receives instructions and plays or displays them. The delivery person acts according to the received instructions.

[0220] 7. Gathering feedback:

[0221] The chip collects sensor data again and sends it to the server, providing feedback on the results of the instructions. This allows the server to continuously monitor the delivery person's work status in real time.

[0222] Hardware and software usage examples

[0223] Hardware: Smart devices (smartphones, smart glasses), IMU sensors, GPS, accelerometers.

[0224] software:

[0225] Data preprocessing: Denoising and feature extraction are performed using Python.

[0226] Server: MySQL or PostgreSQL is used for database management, and TensorFlow or PyTorch is used for real-time data analysis.

[0227] Notification system: Web server frameworks such as Flask or Django are used for notifications from the server to the terminal.

[0228] Specific example

[0229] For example, when a delivery person travels towards a designated address, the GPS built into their smartphone collects location information, and the IMU sensor and accelerometer record movement information in real time. This data is preprocessed, features are extracted, and then it is sent to a server. The server analyzes the data using a generative AI model and generates notifications if the delivery is delayed or the delivery person is behaving inappropriately. When a notification appears on the device, the user (administrator) enters specific instructions, such as "Please use the recommended shortcuts," which are then transmitted to the delivery person.

[0230] Example of a prompt

[0231] "We analyze the movement patterns of delivery drivers and evaluate delivery performance in real time."

[0232] This system helps improve delivery efficiency in real time and enhance customer satisfaction.

[0233] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0234] Step 1:

[0235] The delivery person wears a smart device and begins the delivery. Sensors built into the smartphone or smart glasses (IMU sensor, GPS, accelerometer) collect physical data in real time. The input data includes location information, movement information, and speed of movement, which are collected by the sensors.

[0236] Step 2:

[0237] The collected data is preprocessed by a chip within the device. Here, noise reduction is performed, and important features (such as location information and behavioral patterns) are extracted. The input data is the raw, collected data, while the output data is the preprocessed data.

[0238] Step 3:

[0239] The pre-processed data is transmitted to the server via wireless communication. The input data is pre-processed sensor data, which the server receives.

[0240] Step 4:

[0241] The server stores the received data in a database and analyzes the data using a generated AI model. Specifically, it analyzes the movement and action patterns of delivery personnel to determine delivery delays and decreased efficiency. The input data is the received data, and the output data is the analysis results.

[0242] Step 5:

[0243] The server generates a notification based on the analysis results and sends it to the terminal. It generates appropriate notifications for specific situations, such as "High acceleration detected. Please check traffic conditions." The input data is the analysis results, and the output data is the generated notification.

[0244] Step 6:

[0245] The device displays a notification, and the user reviews the notification content and enters specific instructions. For example, the user might enter instructions such as, "There is a rest point nearby; please take a break." The input data is the generated notification, and the output data is the instructions entered by the user.

[0246] Step 7:

[0247] Instructions from the user are sent to the server. The server receives these instructions and redistributes them to the chip. The input data is the user's instructions, and the output data is the redistributed instructions.

[0248] Step 8:

[0249] The chip receives instructions and plays or displays them. The delivery person acts according to the received instructions. The input data is the redistributed instructions, and the output data is the delivery person's actions.

[0250] Step 9:

[0251] Sensor data is collected again and sent to the server. This allows for feedback on the results of the instruction execution. The input data is the newly collected data, and the output data is the feedback information.

[0252] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0253] This invention combines a real-time work monitoring and remote instruction system using sensor means with an emotion engine. The following describes specific embodiments of this system.

[0254] System-wide configuration

[0255] This system aims to monitor subordinates' work status in real time, provide efficient and appropriate instructions, and improve work efficiency by recognizing the user's emotional state. Its main components are sensors, a chip, a server, a terminal, a user interface, and an emotion engine.

[0256] System Overview

[0257] 1. Sensor and chip functions

[0258] The sensors monitor the subordinates' work status in real time. Specifically, they combine IMUs (Inertial Measurement Units), GPS, temperature sensors, and other sensors to acquire operational information, location information, and environmental information.

[0259] The chip receives data from the sensor and performs data preprocessing. Preprocessing includes noise reduction and feature extraction. The preprocessed data is transmitted wirelessly to the server at regular intervals.

[0260] 2. Server Functions

[0261] The server receives data transmitted from the chip and stores it in a database. The stored data is then analyzed in real time by an AI model.

[0262] Based on the analysis results, the server determines a specific situation and generates an appropriate notification. For example, if the situation "Subordinate A is not wearing a safety harness" is detected, that fact will be included in the notification.

[0263] The generated notifications are sent to the user's device. Additionally, the emotion engine analyzes the user's emotional state and generates notification text appropriate to that state.

[0264] 3. Device functions

[0265] The device receives a notification and displays it to the user. Based on this notification, the user can input specific instructions. For example, they can use text or voice to give instructions such as "Please readjust your safety harness."

[0266] The device is equipped with an emotion engine that analyzes the user's voice and facial expression data to determine their emotional state. This emotional state is sent to a server and used to generate notification messages.

[0267] 4. User Roles

[0268] The user (administrator) uses a terminal to monitor notifications and issue instructions as needed. The user's emotional state is analyzed, and the system generates support notifications as needed to help make appropriate decisions.

[0269] Specific examples of the system

[0270] Specific example: Use at construction sites

[0271] 1. Collection of sensor data

[0272] The chip is attached to the helmet of a subordinate working at a construction site. The IMU sensor tracks the subordinate's movements, the GPS tracks their location, and the temperature sensor measures the ambient temperature.

[0273] 2. Data preprocessing and transmission

[0274] The chip preprocesses the data obtained from the sensor and extracts specific features (for example, position and movement patterns during work).

[0275] 3. Data Analysis

[0276] The server analyzes the transmitted data in real time and detects if there is a possibility that the subordinate is not wearing their safety harness correctly.

[0277] 4. Creating a notification

[0278] The server generates a notification and sends it to the administrator's terminal. Simultaneously, the emotion engine analyzes the user's emotional state and adjusts the notification wording as needed.

[0279] 5. Administrator's response

[0280] The user (administrator) checks the notification on their device and enters the instruction, "Please readjust your safety harness." Additional advice and support notifications are displayed depending on the user's emotional state, such as if they are feeling anxious.

[0281] 6. Communication and execution of instructions

[0282] Instructions are transmitted to the chip via the server, and the subordinate receives the instructions. The subordinate follows the instructions and readjusts their safety harness.

[0283] 7. Collection of Feedback

[0284] The chip collects data again and sends it to the server to provide feedback on the execution result of the instruction.

[0285] This enables the administrator to monitor the on-site work of subordinates in real time and issue appropriate instructions. Additionally, by considering the emotional state of the user for assistance, work efficiency and safety can be further improved.

[0286] The following describes the processing flow.

[0287] Step 1:

[0288] Chip

[0289] <> Sensors (such as IMU, GPS, temperature sensors, etc.) measure the actions, positions, and environmental data of subordinates in real time. This raw data is temporarily stored inside the chip.

[0290] Step 2:

[0291] Chip

[0292] The collected raw data is preprocessed, which includes noise removal and feature extraction. For example, feature quantities of the subordinate's posture and actions are extracted from the IMU data.

[0293] Step 3:

[0294] Chip

[0295] The preprocessed data is sent to the server via wireless communication (such as Wi-Fi or 4G / 5G) at regular time intervals (e.g., every 1 second). If the transmission is successful, the data is deleted from the internal memory. If it fails, retransmission is attempted.

[0296] Step 4:

[0297] Server

[0298] Receive the data sent from the chip and store it in the database. When the integrity check of the data is performed during reception, if illegal data or missing data is detected, a retransmission instruction is sent to the chip.

[0299] Step 5:

[0300] Server

[0301] Analyze the stored data in real time. Use a machine learning model (for example, a deep learning model) to determine the status of subordinates (for example, "working normally", "abnormality detected", etc.) based on the data.

[0302] Step 6:

[0303] Server

[0304] Generate a notification based on the analysis result. For example, if an abnormality such as "Subordinate A is not wearing a seatbelt" is detected, generate a notification including that information.

[0305] Step 7:

[0306] Server <N

[0307] Send the generated notification to the user's terminal. As the transmission method, use push notifications or emails.

[0308] Step 8:

[0309] Terminal

[0310] Receive the notification from the server and display it to the user. The user can check the notification on the terminal and view the detailed information on the dashboard.

[0311] Step 9:

[0312] Emotion engine (built into the terminal)

[0313] The system analyzes the user's voice and facial expression data from the camera to determine the user's emotional state (e.g., "tense," "relaxed," etc.). This emotional state is then sent to the server.

[0314] Step 10:

[0315] server

[0316] The system takes the user's emotional state into account and adjusts the notification wording accordingly. For example, if the user is feeling anxious, the message "Please readjust your safety harness" might be changed to "Please take your time, check and fasten your safety harness one more time."

[0317] Step 11:

[0318] terminal

[0319] The system displays notifications to the user based on their emotional state. The user then enters appropriate instructions based on these notifications.

[0320] Step 12:

[0321] User

[0322] Based on the notification, enter the necessary instructions into the terminal. For example, you might enter a message such as "Instruct subordinate A to readjust their safety harness" via voice or text.

[0323] Step 13:

[0324] terminal

[0325] The user's input instructions are sent to the server. Error checking and retransmission are performed during transmission.

[0326] Step 14:

[0327] server

[0328] It receives user instructions and delivers them to the chip. It uses real-time communication protocols (e.g., MQTT or WebSockets).

[0329] Step 15:

[0330] tip

[0331] Receive instructions and notify subordinates. If the instructions are voiced, play them through the speaker; if they are text-based, display them on the screen.

[0332] Step 16:

[0333] subordinate

[0334] Confirm instructions from the user and act accordingly. For example, readjust your safety harness.

[0335] Step 17:

[0336] tip

[0337] The sensor data is collected again and sent to the server. This verifies that the instructions were executed correctly.

[0338] Step 18:

[0339] server

[0340] Analyze the newly collected data to confirm that the instructions were followed correctly. If the problem recurs, send another notification.

[0341] (Example 2)

[0342] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0343] Conventional work monitoring systems have struggled to accurately monitor work status in real time and provide appropriate instructions based on that information. Furthermore, they fail to provide notifications and instructions that take into account the user's emotional state, resulting in a lack of improvement in work efficiency and safety.

[0344] In Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for collecting physical data using sensor means, means for preprocessing the collected data to extract important features, means for transmitting the preprocessed data to the server, means for the server to analyze the received data and determine a specific situation, means for generating a notification based on the analysis results and transmitting it to a terminal, means for the terminal to display the notification and prompt the user to input instructions, means for analyzing the user's emotional state and adjusting the notification wording, means for transmitting the user's instructions to the server and for the server to redistribute them to a chip, means for the chip that received the instructions to reproduce or display the content, and means for transmitting the recollected data to the server to provide feedback. This enables accurate monitoring of the work status in real time and the provision of appropriate instructions according to the user's emotional state.

[0345] "Sensing means" refers to a device used to collect physical data.

[0346] "Data preprocessing" refers to the process of converting data collected by sensor devices into a format suitable for analysis.

[0347] A "chip" is an electronic circuit that receives data from a sensor, performs preprocessing, and transmits the data to a server.

[0348] A "server" is a computing device that receives data transmitted from a chip, analyzes it, generates notifications, and sends them to a terminal.

[0349] A "notification" is a message that the server generates based on the analysis results and sends to the terminal.

[0350] A "terminal" is a device that displays notifications sent from the server to the user and sends instructions from the user to the server.

[0351] A "user" refers to a person who operates a device, checks notifications, and enters instructions as needed.

[0352] "Emotional analysis" is a process that analyzes a user's voice and facial expression data to determine the user's emotional state.

[0353] "Feedback" refers to the process of sending the collected data back to the server and reporting the results of the instructions' execution.

[0354] A "real-time communication protocol" is a communication method used to instantly exchange data between a server and a chip.

[0355] Modes for carrying out the invention

[0356] This invention relates to a system that monitors work status in real time and provides appropriate instructions that take into account the user's emotional state. Specific embodiments of this system are described below.

[0357] System-wide configuration

[0358] The system consists of sensors, chips, servers, terminals, users, and an emotion engine. These elements work together to collect, process, and analyze real-time data, generate notifications, communicate instructions, and provide feedback.

[0359] Sensor and chip functions

[0360] 1. Sensor:

[0361] It is used to monitor the work status of subordinates in real time. Specifically, it includes an IMU (Inertial Measurement Unit), GPS, temperature sensors, etc., to acquire operational information, location information, and environmental information.

[0362] For example, an IMU sensor measures the worker's movements, GPS determines their location, and a temperature sensor obtains the ambient temperature of the site.

[0363] 2. Tips:

[0364] The system receives data from sensors and performs preprocessing such as noise reduction and feature extraction. The preprocessed data is then sent to the server at regular intervals.

[0365] The chip preprocesses the data, extracting features such as movement patterns and changes in position, and then sends them to the server.

[0366] Server Functions

[0367] 1. Server:

[0368] The system receives data transmitted from the chip and stores it in a database. The stored data is then analyzed in real time using an AI model.

[0369] Using an AI model, specific situations are determined from the analysis results. For example, it can detect the situation where "subordinate A is not wearing a safety harness."

[0370] The server generates a notification based on the analysis results and sends it to the user's device. Furthermore, it uses an emotion engine to analyze the user's emotional state and adjust the notification wording accordingly.

[0371] For example, the server might generate a notification stating, "Subordinate A is not wearing a safety harness," and if the user is confused, it might add a message saying, "Please remain calm."

[0372] Device functions

[0373] 1. Terminal:

[0374] The server receives notifications and displays them to the user. The user checks the notifications on their device and enters specific instructions.

[0375] The system is equipped with an emotion engine that analyzes the user's voice and facial expression data to determine their emotional state. This analysis result is sent to a server and used to adjust notification messages.

[0376] For example, a user might input the instruction "Please readjust your safety harness" into their terminal, and this instruction is transmitted to their subordinate via the server.

[0377] User roles

[0378] 1. User (Administrator):

[0379] The device monitors notifications and provides specific instructions as needed. The user's emotional state is analyzed, and supportive notifications tailored to that emotional state are provided to help make appropriate decisions.

[0380] For example, an administrator might issue an instruction to "reattach your safety harness," and then a support notification might appear stating, "Let's stay calm."

[0381] Specific example

[0382] Use at construction sites

[0383] 1. Sensor data collection:

[0384] The helmet is equipped with an IMU, GPS, and temperature sensor that measure the worker's movements, location, and ambient temperature in real time.

[0385] 2. Data preprocessing and transmission:

[0386] The chip preprocesses the data obtained from the sensor, extracts features, and sends them to the server.

[0387] 3. Data Analysis:

[0388] The server analyzes the received data using an AI model and determines that "the worker is not wearing the safety harness correctly."

[0389] 4. Creating and sending notifications:

[0390] The server generates a notification and sends it to the administrator's terminal. The emotion engine analyzes the user's emotional state and adjusts the notification wording as needed.

[0391] 5. Administrator's response:

[0392] The administrator checks the notification on the terminal and enters the instruction, "Please readjust your safety harness." Support notifications tailored to the user's emotional state are also displayed.

[0393] 6. Communication and execution of instructions:

[0394] Instructions are delivered to a chip via a server, and the worker receives the instructions. After that, they readjust their safety harness.

[0395] 7. Gathering feedback:

[0396] The chip collects data again and sends it to the server to provide feedback.

[0397] Examples of prompt statements

[0398] Please describe the process by which users receive real-time notifications and issue instructions for a sensor-based work monitoring system used on construction sites. Also, please describe in detail how support is provided that takes the user's emotional state into consideration.

[0399] This enables accurate monitoring of work status in real time and the provision of appropriate instructions tailored to the user's emotional state.

[0400] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0401] Step 1:

[0402] Data collection using sensors:

[0403] Input: Physical information such as the subordinate's actions, location, and ambient temperature.

[0404] Processing: The IMU sensor acquires operational information, the GPS acquires location information, and the temperature sensor acquires environmental information in real time.

[0405] Output: The raw data obtained.

[0406] Specific operation: Sensors measure the subordinate's movements in milliseconds, GPS determines their location, and temperature sensors collect temperature data.

[0407] Step 2:

[0408] Data preprocessing:

[0409] Input: Raw data acquired from the sensor.

[0410] Processing: The chip performs noise reduction and data correction, and extracts important features.

[0411] Output: Preprocessed data.

[0412] Specific operation: From raw data, outliers and unwanted noise are filtered out, and features such as "patterns of work movements" and "location movement history" are extracted.

[0413] Step 3:

[0414] Sending data:

[0415] Input: Preprocessed data.

[0416] Processing: The chip wirelessly transmits pre-processed data to the server at regular intervals.

[0417] Output: Data transmitted from the chip.

[0418] Specific operation: The chip uses a wireless communication protocol to send data to the server.

[0419] Step 4:

[0420] Data analysis:

[0421] Input: Pre-processed data transmitted from the chip.

[0422] Processing: The server uses an AI model to analyze the data and make a judgment about a specific situation.

[0423] Output: Analysis results (e.g., "Safety harness not worn").

[0424] Specific operation: The server analyzes data in real time and detects things like "a worker is inactive and stopped for a long time" or "an abnormal operation pattern."

[0425] Step 5:

[0426] Notification generation:

[0427] Input: Analysis results.

[0428] Processing: The server generates a notification based on the analysis results and sends it to the user's terminal. The emotion engine analyzes the user's emotional state and adjusts the notification wording.

[0429] Output: Adjusted notification.

[0430] Specific operation: The server generates a notification that "the worker has stopped working," and the emotion engine analyzes the user's stress level and adds the message "Please remain calm."

[0431] Step 6:

[0432] Displaying notifications and entering instructions:

[0433] Input: Adjusted notification.

[0434] Processing: The device displays a notification and prompts the user to enter specific instructions.

[0435] Output: Instructions.

[0436] Specific action: The terminal displays a notification saying "Worker A has stopped moving," and the user enters "Please readjust your safety harness."

[0437] Step 7:

[0438] Sending user instructions:

[0439] Input: Instructions from the user.

[0440] Processing: The terminal sends instructions to the server, and the server redistributes them to the chip.

[0441] Output: Instructions sent to the chip.

[0442] Specific operation: The terminal sends instructions to the server, and the server forwards those instructions to the chip.

[0443] Step 8:

[0444] Execute the instructions:

[0445] Input: The instructions sent to the chip.

[0446] Processing: The chip plays or displays the instructions.

[0447] Output: The actions taken by the worker who received the instructions.

[0448] Specific action: The chip notifies the worker via voice or text message, "Please readjust your safety harness."

[0449] Step 9:

[0450] Gathering feedback:

[0451] Input: Data after the instruction has been executed.

[0452] Processing: The chip collects data again and sends it to the server to provide feedback.

[0453] Output: Feedback data sent to the server.

[0454] Specific operation: The chip collects data from the sensor again, for example, to confirm that "the safety harness is properly fitted," and then sends the data to the server.

[0455] This allows the entire system to monitor work in real time and provide appropriate instructions. Furthermore, by considering the user's emotional state, it enables effective decision-making support.

[0456] (Application Example 2)

[0457] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0458] Traditional work monitoring systems focused on real-time monitoring of work status and remote instructions, but they did not provide notifications or instructions that took into account the emotional state of managers. As a result, manager stress increased, and work efficiency sometimes decreased. Furthermore, traditional systems focused only on physical safety and did not consider psychological aspects, thus failing to improve the overall work environment.

[0459] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for collecting physical data using sensor means, means for preprocessing the collected data to extract important features, means for transmitting the preprocessed data to the server, means for the server to analyze the received data and determine a specific situation, means for generating a notification based on the analysis results and transmitting it to a terminal, means for the terminal to display the notification and prompt the user to input instructions, means for transmitting the user's instructions to the server and for the server to redistribute them to a chip, means for the chip that received the instructions to reproduce or display their contents, means for transmitting the recollected data to the server to provide feedback, means for analyzing the user's emotional state using an emotion analysis engine, and means for adjusting the notification wording based on the analyzed emotional state. This makes it possible to provide appropriate notifications and instructions according to the administrator's emotional state and to improve the work environment while also considering psychological aspects.

[0460] A "sensor" is a device used to acquire physical data from the environment or objects.

[0461] "Physical data" refers to data such as operational information, location information, and environmental information measured by IMUs, GPS, temperature sensors, etc.

[0462] "Preprocessing" is a part of data processing that removes noise from collected data and extracts important features.

[0463] A "server" is a computing device that receives, analyzes, and stores data via a network, and generates appropriate notifications and instructions.

[0464] A "chip" is a hardware component that receives data from sensors, performs preprocessing, and sends the data to a server.

[0465] A "terminal" is a device that receives notifications from a server, displays them to the user, and allows the user to input instructions.

[0466] A "user" is an administrator or person in charge who uses the system and performs monitoring and instructions.

[0467] An "emotion analysis engine" is a software module that analyzes a user's voice and facial expression data to determine their emotional state.

[0468] A "notification" is an informational message generated based on analyzed data and sent to the device.

[0469] "Feedback" refers to the evaluation or response provided by the system based on the results of executing instructions and the collected data.

[0470] A "real-time communication protocol" is a communication standard or procedure used to exchange data quickly and efficiently.

[0471] System-wide configuration

[0472] This system consists of sensors, chips, servers, terminals, users, and an emotion analysis engine, and is primarily intended for monitoring work in factories and generating notifications based on emotional states.

[0473] Sensor means

[0474] Sensors are devices used to acquire motion information, location information, and environmental information from the environment or objects. Specifically, IMUs, GPS, and temperature sensors are used. These sensors are attached to robots or workers to collect data in real time.

[0475] Chip functions

[0476] The chip receives data from the sensor and performs preprocessing such as noise reduction and feature extraction. The preprocessed data is then transmitted wirelessly to the server at regular intervals.

[0477] Server Functions

[0478] The server receives data transmitted from the chip and stores it in a database. The stored data is analyzed in real time. The server analyzes sensor data to determine specific situations. It also generates notifications based on the received data and sends them to the terminal. Furthermore, it uses an emotion analysis engine to analyze the user's (administrator's) emotional state and adjust the notification wording accordingly.

[0479] Device functions

[0480] The terminal receives notifications from the server and displays them to the user. The user inputs instructions based on the notification content, and these instructions are sent back to the server. The server redistributes the received instructions to a chip, which then relays the instructions to the robot or worker. After the instructions are executed, data is collected again and sent back to the server to provide feedback.

[0481] User roles

[0482] The user (administrator) uses the terminal to monitor notifications and issue instructions as needed. The emotion analysis engine analyzes the user's voice and facial expression data to determine their emotional state, and the notification wording is adjusted accordingly to support appropriate responses.

[0483] Functions of the emotion analysis engine

[0484] The emotion analysis engine uses a generative AI model that recognizes the user's emotional state by taking user voice and video data as input. Based on the analysis results, it determines stress levels, relaxation levels, etc., and reflects this in the notification message.

[0485] Specific example

[0486] For example, in a factory setting, if a supervisor wears a head-mounted display (HMD) to monitor the robot's work status, the data collected by the sensors is transmitted to a server via a chip. The server analyzes the data and detects situations where the robot requires maintenance. The detected situation is sent as a notification to the terminal, allowing the user to check the notification and issue instructions. An emotion analysis engine analyzes the user's stress level and generates notification messages such as, "Please perform maintenance work. We also recommend taking a break," depending on the user's state.

[0487] Example of a prompt

[0488] The inputs to the generative AI model when using the emotion analysis engine are as follows:

[0489] "Please analyze this user's voice and facial expressions to determine their current emotional state. If possible, please provide a specific emotional state (e.g., tense, relaxed)."

[0490] This system not only facilitates smooth work monitoring and remote instruction, but also reduces the psychological burden on users, resulting in an efficient and safe work environment.

[0491] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0492] Step 1:

[0493] Sensors are attached to robots and workers. These sensors, using IMUs, GPS, and temperature sensors, collect motion information, location information, and environmental information in real time. The input is sensor data, and the output is raw data before preprocessing.

[0494] Step 2:

[0495] The chip receives data from the sensor and performs noise reduction and feature extraction. Specifically, it uses filtering techniques to reduce noise and extract only the necessary parameters. The input is sensor data, raw data, and the output is pre-processed data.

[0496] Step 3:

[0497] The chip wirelessly transmits pre-processed data to the server. The server receives this data and stores it in a database. The input is the pre-processed data, and the output is the data stored in the database.

[0498] Step 4:

[0499] The server analyzes the received data and uses data mining techniques to determine specific situations. For example, it uses machine learning algorithms to detect situations where a robot requires maintenance. The input is data stored in a database, and the output is the analysis result.

[0500] Step 5:

[0501] The server generates a notification based on the analysis results and sends it to the terminal. The notification includes situation-specific instructions and warnings. The input is the analysis results, and the output is the generated notification.

[0502] Step 6:

[0503] The terminal receives notifications from the server and displays them to the user. The user reviews the notification content and enters the necessary instructions. The input is the notification from the server, and the output is the user's instructions.

[0504] Step 7:

[0505] User instructions are sent from the terminal to the server. The server receives the instructions and redistributes them to the chip. The input is the user's instructions, and the output is the instructions sent to the chip.

[0506] Step 8:

[0507] The chip that receives the instruction will either play or display its contents. This allows the robot or worker to take a specific action. The input is the instruction from the server, and the output is the played or displayed instruction.

[0508] Step 9:

[0509] After the instruction is executed, the sensor collects data again and sends the new data to the server. The server receives this data and updates the entire process as feedback. The input is the newly collected sensor data, the raw data, and the output is the updated database.

[0510] Step 10:

[0511] The emotion analysis engine analyzes the user's voice and video data to determine their emotional state. The server adjusts the notification message based on the analysis results. The input is the user's voice and video data, and the output is the adjusted notification message.

[0512] Through each of the above steps, real-time work monitoring and remote instructions are provided, along with appropriate notifications tailored to the user's emotional state. This results in an efficient and safe work environment.

[0513] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0514] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0515] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0516] [Second Embodiment]

[0517] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0518] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0519] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0520] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0521] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0522] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0523] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0524] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0525] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0526] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0527] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0528] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0529] This invention relates to a real-time work monitoring and remote instruction system using sensor means. Embodiments of this system are described in detail below.

[0530] System-wide configuration

[0531] This system aims to monitor the work status of subordinates in real time and provide efficient and appropriate instructions. Its main components are sensors, chips, servers, terminals, and users.

[0532] System Overview

[0533] 1. Sensor and chip functions

[0534] The sensors monitor the subordinates' work status in real time. Specifically, they combine IMUs (Inertial Measurement Units), GPS, temperature sensors, and other sensors to acquire operational information, location information, and environmental information.

[0535] The chip receives data from the sensor and performs data preprocessing. Preprocessing includes noise reduction and feature extraction. The preprocessed data is transmitted wirelessly to the server at regular intervals.

[0536] 2. Server Functions

[0537] The server receives data transmitted from the chip and stores it in a database. The stored data is then analyzed in real time by an AI model.

[0538] Based on the analysis results, the server determines a specific situation and generates an appropriate notification. For example, if the situation "Subordinate A is not wearing a safety harness" is detected, that fact will be included in the notification.

[0539] The generated notification is sent to the user's device.

[0540] 3. Device functions

[0541] The device receives a notification and displays it to the user. Based on this notification, the user can input specific instructions. For example, they can use text or voice to give instructions such as "Please readjust your safety harness."

[0542] The user's input is sent back to the server, which then delivers the instructions to the chip.

[0543] 4. User Roles

[0544] The user (administrator) uses a device to monitor notifications and issue instructions as needed. Instructions can be easily given via text or voice input.

[0545] By providing appropriate instructions, users can improve the work efficiency of their subordinates and maintain a safe working environment.

[0546] Specific examples of the system

[0547] Specific example: Use at construction sites

[0548] 1. Collection of sensor data

[0549] The chip is attached to the helmet of a subordinate working at a construction site. The IMU sensor tracks the subordinate's movements, the GPS tracks their location, and the temperature sensor measures the ambient temperature.

[0550] 2. Data preprocessing and transmission

[0551] The chip preprocesses the data obtained from the sensor and extracts specific features (for example, position and movement patterns during work).

[0552] 3. Data Analysis

[0553] The server analyzes the transmitted data in real time and detects if there is a possibility that the subordinate is not wearing their safety harness correctly.

[0554] 4. Creating and sending notifications

[0555] The server generates a notification and sends it to the administrator's terminal.

[0556] 5. Administrator's response

[0557] The user (administrator) checks the notification on their device and enters the instruction, "Please readjust your safety harness."

[0558] 6. Communication and execution of instructions

[0559] Instructions are transmitted to the chip via the server, and the subordinate receives the instructions.

[0560] The subordinate followed instructions and readjusted his safety harness.

[0561] 7. Gathering Feedback

[0562] The chip collects data again and sends it to the server, providing feedback on the results of the instructions' execution.

[0563] This allows managers to monitor their subordinates' on-site work in real time and issue appropriate instructions. This system significantly improves work efficiency and safety.

[0564] The following describes the processing flow.

[0565] Step 1:

[0566] tip

[0567] Sensors (such as IMUs, GPS, and temperature sensors) measure the subordinate's movements, location, and environmental data in real time. This raw data is temporarily stored inside the chip.

[0568] Step 2:

[0569] tip

[0570] The collected raw data is preprocessed. Preprocessing includes noise reduction and feature extraction. For example, features of the subordinate's posture and movements are extracted from IMU data.

[0571] Step 3:

[0572] tip

[0573] Pre-processed data is transmitted to the server wirelessly (Wi-Fi, 4G / 5G, etc.) at regular time intervals (e.g., every second). If transmission is successful, the data is deleted from internal memory. If transmission fails, a retransmission is attempted.

[0574] Step 4:

[0575] server

[0576] The system receives data transmitted from the chip and stores it in a database. Upon receipt, the system performs a data integrity check, and if invalid or missing data is detected, it sends a retransmission instruction to the chip.

[0577] Step 5:

[0578] server

[0579] The system analyzes stored data in real time. It uses machine learning models (e.g., deep learning models) to determine the status of subordinates (e.g., "operating normally," "anomaly detected," etc.) based on the data.

[0580] Step 6:

[0581] server

[0582] The system generates notifications based on the analysis results. For example, if it detects an anomaly such as "Subordinate A is not wearing a safety harness," it will generate a notification including that information.

[0583] Step 7:

[0584] server

[0585] The generated notification is sent to the user's device. The sending method can be push notification or email.

[0586] Step 8:

[0587] terminal

[0588] Receive notifications from the server and display them to the user. Users can check the notifications on their devices and view detailed information on the dashboard.

[0589] Step 9:

[0590] User

[0591] Based on the notification, enter the necessary instructions into the terminal. For example, you might enter a message such as "Instruct subordinate A to readjust their safety harness" via voice or text.

[0592] Step 10:

[0593] terminal

[0594] The user's input instructions are sent to the server. Error checking and retransmission are performed during transmission.

[0595] Step 11:

[0596] server

[0597] It receives user instructions and delivers them to the chip. It uses real-time communication protocols (e.g., MQTT or WebSockets).

[0598] Step 12:

[0599] tip

[0600] Receive instructions and notify subordinates. If the instructions are voiced, play them through the speaker; if they are text-based, display them on the screen.

[0601] Step 13:

[0602] subordinate

[0603] Confirm instructions from the user and act accordingly. For example, readjust your safety harness.

[0604] Step 14:

[0605] tip

[0606] The sensor data is collected again and sent to the server. This verifies that the instructions were executed correctly.

[0607] Step 15:

[0608] server

[0609] Analyze the newly collected data to confirm that the instructions were followed correctly. If the problem recurs, send another notification.

[0610] (Example 1)

[0611] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0612] Conventional work monitoring and instruction systems made it difficult to grasp the work status of subordinates in real time and to issue appropriate instructions quickly. Furthermore, the data obtained from sensors contained a lot of noise and irrelevant information, resulting in insufficient pre-processing for accurate situational assessment. In addition, delays in generating notifications based on analysis results and providing feedback on instructions based on those results prevented sufficient improvements in work efficiency and safety.

[0613] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0614] In this invention, the server includes means for collecting physical data using sensor means, means for preprocessing the collected data to extract important features, means for transmitting the preprocessed data to a computer, means for the computer to analyze the received data and determine a specific situation, means for generating a notification based on the analysis results and transmitting it to a terminal, means for the terminal to display the notification and prompt the user to input instructions, means for transmitting the user's instructions to the computer and for the computer to redistribute them to a chip, means for the chip that received the instructions to reproduce or display their contents, means for transmitting the recollected data to the computer to provide feedback, and means for the computer to perform real-time analysis using a domain-specific model based on the instructions input by the user. This makes it possible to accurately grasp the work status of subordinates in real time and quickly issue appropriate instructions. Furthermore, it is possible to improve work efficiency and safety.

[0615] A "sensor" is a device or system used to collect physical data.

[0616] "Preprocessing" refers to the process of removing noise from collected data and extracting important features.

[0617] "Transmission means" refers to a function for transmitting pre-processed data to other devices or systems.

[0618] A "computer" is a device or system that analyzes received data and makes judgments about specific situations.

[0619] "Analysis means" refers to methods or functions for making judgments about specific situations based on data.

[0620] A "notification" is information or a message generated based on the analysis results.

[0621] A "terminal" is a device that displays notifications to the user and allows them to input instructions.

[0622] A "user" is a person or administrator who uses a terminal to input instructions.

[0623] "Feedback" refers to information used to verify the progress of instructions based on recollected data.

[0624] A "domain-specific model" is an AI model that is specialized for a particular industry or domain.

[0625] "Real-time analysis" is an analytical method that processes data immediately and obtains results instantly.

[0626] Modes for carrying out the invention

[0627] This invention relates to a real-time work monitoring and remote instruction system using sensor means. The system aims to monitor the work status of subordinates in real time and provide efficient and appropriate instructions. The main components are sensors, chips, a server, a terminal, and the user. The following describes specific embodiments of this system.

[0628] Sensor and chip functions

[0629] The sensors monitor the work status of subordinates in real time. Specifically, they combine IMUs (Inertial Measurement Units), GPS, temperature sensors, etc., to acquire motion information, location information, and environmental information. These sensors are often attached to the worker's helmet or work clothes. For example, an IMU sensor attached to the helmet measures the subordinate's movements, a GPS determines their location, and a temperature sensor measures the ambient temperature.

[0630] The chip receives data from sensors and performs preprocessing such as noise reduction and feature extraction. For example, it removes unwanted vibration information from collected IMU sensor data and extracts specific operating patterns. The preprocessed data is transmitted to a server via wireless communication at regular intervals.

[0631] Server Functions

[0632] The server receives data transmitted from the chip and stores it in a database. The received data is temporarily stored in memory and then saved to the specified database. For example, databases such as MySQL or PostgreSQL may be used.

[0633] The server analyzes the stored data in real time. This analysis may utilize generative AI models such as TensorFlow or PyTorch. Through this analysis, it analyzes behavioral patterns and location information to detect abnormal situations, such as "Subordinate A is not wearing a safety harness."

[0634] Based on the analysis results, the server generates an appropriate notification and sends it to the user's terminal. For example, it might generate a notification stating, "Subordinate A is not wearing a safety harness," and send it to the user's terminal.

[0635] Device functions

[0636] The device receives notifications sent from the server and displays them to the user. Notifications are often displayed as pop-ups on the screen. Based on the notification, the user enters specific instructions. For example, they might enter a text instruction such as "Please readjust your safety harness" into the device. The device also has a voice input function, allowing users to enter instructions by voice.

[0637] Instructions entered by the user are sent back to the server from the terminal. The server distributes the received instructions to a chip, which then plays or displays the instructions to its subordinates. This allows the subordinates to adjust their work according to the user's instructions.

[0638] Specific example

[0639] Use at construction sites

[0640] 1. Sensor data collection: The chip is attached to the helmet of a subordinate working at the construction site, and the IMU sensor measures operational information, the GPS measures location information, and the temperature sensor measures ambient temperature.

[0641] 2. Data preprocessing and transmission: The chip preprocesses the data obtained from the sensor and extracts specific features (position and movement patterns during operation).

[0642] 3. Data Analysis: The server analyzes the transmitted data in real time to detect the possibility that a subordinate is not wearing their safety harness correctly.

[0643] 4. Creating and sending notifications: The server generates a notification and sends it to the administrator's terminal.

[0644] 5. Administrator's response: The user (administrator) checks the notification on their device and enters the instruction, "Please readjust your safety harness."

[0645] 6. Instruction transmission and execution: Instructions are delivered to the chip via the server, and the subordinate receives them. The subordinate follows the instructions and readjusts their safety harness.

[0646] 7. Feedback Collection: The chip collects data again and sends it to the server to provide feedback on the results of the instructions being executed.

[0647] Example of a prompt

[0648] "At a construction site, there is a system that preprocesses and analyzes data obtained from IMU sensors, GPS, and temperature sensors attached to helmets. Please explain the processing flow of this system in detail."

[0649] This allows us to request a detailed explanation of the system from the generated AI model.

[0650] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0651] Step 1:

[0652] The user has their subordinates wear sensors. For example, an IMU sensor is attached to a helmet, and a GPS sensor is attached to work clothes. The input is the sensors being worn, and the output is the state of being ready to collect data in real time.

[0653] Step 2:

[0654] Sensors monitor the subordinate's work status in real time. The IMU sensor collects motion data, the GPS collects location data, and the temperature sensor collects ambient temperature data. The input is the subordinate's actions and environmental conditions, and the output is the collection of raw data. Specifically, the IMU sensor measures acceleration and rotation, and the GPS obtains latitude and longitude.

[0655] Step 3:

[0656] The chip receives data from the sensor and performs preprocessing such as noise reduction and feature extraction. It removes unwanted noise from the collected raw data and extracts specific operating patterns. The input is raw data from the sensor, and the output is clean, preprocessed data. Specifically, it performs data filtering and statistical processing.

[0657] Step 4:

[0658] The chip transmits pre-processed data to the server via wireless communication at regular intervals. The input is clean data, and the output is data transmission to the server. Specifically, it transmits data packets using a wireless protocol.

[0659] Step 5:

[0660] The server receives data transmitted from the chip and stores it in the database. The input is the data from the chip, and the output is the stored data. Specifically, it performs database insertion operations.

[0661] Step 6:

[0662] The server analyzes stored data in real time and performs analysis to determine specific situations. The input is stored data, and the output is the analysis result. Specifically, it performs data analysis using generative AI models such as TensorFlow and PyTorch.

[0663] Step 7:

[0664] The server generates an appropriate notification based on the analysis results and sends it to the user's terminal. The input is the analysis results, and the output is the notification message. Specifically, a notification is generated stating, "Subordinate A is not wearing a safety harness."

[0665] Step 8:

[0666] The device receives notifications from the server and displays them to the user. The input is the notification message, and the output is the notification displayed on the user interface. Specifically, it is displayed as a pop-up notification on the screen.

[0667] Step 9:

[0668] The user enters specific instructions based on a notification displayed on the device. The input is the content of the notification, and the output is the entered instruction. For example, the user might enter text or voice instructions such as "Please readjust your safety harness" into the device.

[0669] Step 10:

[0670] The terminal sends user instructions to the server. The input is the user's instructions, and the output is the data sent to the server. Specifically, text and voice instructions are converted into data packets and sent.

[0671] Step 11:

[0672] The server receives user instructions and redistributes them to the chip. The input is the user's instruction data, and the output is the data to be sent to the chip. Specifically, the instruction content is transmitted using a protocol.

[0673] Step 12:

[0674] The chip receives instructions and plays or displays the content to its subordinates. The input is instruction data from the server, and the output is the notification content for the subordinates. Specifically, it plays the voice message, "Please readjust your safety harness."

[0675] Step 13:

[0676] The chip sends the recollected data to the server and provides feedback. For example, it recollects data to confirm that the safety harness has been reattached. The input is the recollected data, and the output is the feedback data to the server. This allows the administrator to confirm that the instructions have been followed.

[0677] (Application Example 1)

[0678] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0679] Traditional food delivery systems have made it difficult to accurately monitor the work status and efficiency of delivery personnel in real time and to issue appropriate instructions. This has resulted in delivery delays, decreased efficiency, and lower customer satisfaction. Furthermore, delays in providing feedback to delivery personnel and the inability to respond immediately have been a challenge.

[0680] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0681] In this invention, the server includes means for analyzing real-time data, including the movement patterns of delivery personnel; means for evaluating delivery efficiency based on the analysis results and generating instructions to improve efficiency; and means for proposing specific actions to delivery personnel based on user instructions. This makes it possible to accurately grasp the current status of delivery personnel in real time and provide appropriate instructions immediately.

[0682] A "sensor" is a device used to collect physical data in real time.

[0683] "IMU" is an abbreviation for Inertial Measurement Unit, a sensor that detects the movement and acceleration of an object.

[0684] "GPS" is an abbreviation for Global Positioning System, a system used to measure location on Earth.

[0685] An "accelerometer" is a sensor used to measure the acceleration of an object.

[0686] "Preprocessing means" refers to the process of removing noise from collected raw data and extracting important features.

[0687] A "server" is a computer system used to receive, store, and analyze data over a network.

[0688] An "analysis tool" is a mechanism for analyzing collected data and making judgments about a specific situation.

[0689] A "notification system" is a system for sending notifications generated based on analysis results to the user's terminal.

[0690] A "terminal" is a device used by a user to receive notifications and input instructions.

[0691] A "command system" is a system that sends user-inputted instructions to a server, which then redistributes them to the chip.

[0692] A "chip" is a device used to preprocess collected data and send it to a server.

[0693] "Movement patterns" refer to dynamic data such as the delivery person's travel route and speed.

[0694] "Feedback" refers to information about the results and effects of actions provided by sending the recollected data to the server.

[0695] This invention relates to a system that can be applied to food delivery to monitor the work status of delivery personnel in real time and issue appropriate instructions. This system is realized through the cooperation of sensor means, a server, a terminal, and a user.

[0696] System Configuration

[0697] 1. Sensor means:

[0698] Delivery personnel wear smart devices such as smart glasses or smartphones. These devices have built-in IMU sensors, GPS, and accelerometers.

[0699] 2. Data Acquisition and Preprocessing:

[0700] Sensors built into smart devices collect physical data on delivery personnel in real time. This includes location information, movement speed, and motion data.

[0701] The chip preprocesses the data collected from the sensor, performing noise reduction and feature extraction.

[0702] 3. Data transmission and analysis:

[0703] The pre-processed data is transmitted to the server via wireless communication.

[0704] The server stores the received data in a database and analyzes the data in real time using a generative AI model. For example, it analyzes the movement patterns of delivery personnel to detect delivery delays and decreased efficiency.

[0705] 4. Generating and sending notifications:

[0706] The server determines the specific situation based on the analysis results and generates and sends an appropriate notification to the terminal. For example, a notification such as "High acceleration detected. Please check traffic conditions" is generated.

[0707] 5. User actions:

[0708] The device displays a notification, and the user (administrator) reviews the notification before entering specific instructions. For example, they might enter instructions such as, "There is a rest point nearby; please take a break."

[0709] The input instructions are sent to the server, which then redistributes them to the chip.

[0710] 6. Communication and execution of instructions:

[0711] The chip receives instructions and plays or displays them. The delivery person acts according to the received instructions.

[0712] 7. Gathering feedback:

[0713] The chip collects sensor data again and sends it to the server, providing feedback on the results of the instructions. This allows the server to continuously monitor the delivery person's work status in real time.

[0714] Hardware and software usage examples

[0715] Hardware: Smart devices (smartphones, smart glasses), IMU sensors, GPS, accelerometers.

[0716] software:

[0717] Data preprocessing: Denoising and feature extraction are performed using Python.

[0718] Server: MySQL or PostgreSQL is used for database management, and TensorFlow or PyTorch is used for real-time data analysis.

[0719] Notification system: Web server frameworks such as Flask or Django are used for notifications from the server to the terminal.

[0720] Specific example

[0721] For example, when a delivery person travels towards a designated address, the GPS built into their smartphone collects location information, and the IMU sensor and accelerometer record movement information in real time. This data is preprocessed, features are extracted, and then it is sent to a server. The server analyzes the data using a generative AI model and generates notifications if the delivery is delayed or the delivery person is behaving inappropriately. When a notification appears on the device, the user (administrator) enters specific instructions, such as "Please use the recommended shortcuts," which are then transmitted to the delivery person.

[0722] Example of a prompt

[0723] "We analyze the movement patterns of delivery drivers and evaluate delivery performance in real time."

[0724] This system helps improve delivery efficiency in real time and enhance customer satisfaction.

[0725] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0726] Step 1:

[0727] The delivery person wears a smart device and begins the delivery. Sensors built into the smartphone or smart glasses (IMU sensor, GPS, accelerometer) collect physical data in real time. The input data includes location information, motion information, and movement speed, which are collected by the sensors.

[0728] Step 2:

[0729] The collected data is preprocessed by a chip within the device. Here, noise reduction is performed, and important features (such as location information and behavioral patterns) are extracted. The input data is the raw, collected data, while the output data is the preprocessed data.

[0730] Step 3:

[0731] The pre-processed data is transmitted to the server via wireless communication. The input data is pre-processed sensor data, which the server receives.

[0732] Step 4:

[0733] The server stores the received data in a database and analyzes the data using a generated AI model. Specifically, it analyzes the movement and action patterns of delivery personnel to determine delivery delays and decreased efficiency. The input data is the received data, and the output data is the analysis results.

[0734] Step 5:

[0735] The server generates a notification based on the analysis results and sends it to the terminal. It generates appropriate notifications for specific situations, such as "High acceleration detected. Please check traffic conditions." The input data is the analysis results, and the output data is the generated notification.

[0736] Step 6:

[0737] The device displays a notification, and the user reviews the notification content and enters specific instructions. For example, the user might enter instructions such as, "There is a rest point nearby; please take a break." The input data is the generated notification, and the output data is the instructions entered by the user.

[0738] Step 7:

[0739] Instructions from the user are sent to the server. The server receives these instructions and redistributes them to the chip. The input data is the user's instructions, and the output data is the redistributed instructions.

[0740] Step 8:

[0741] The chip receives instructions and plays or displays them. The delivery person acts according to the received instructions. The input data is the redistributed instructions, and the output data is the delivery person's actions.

[0742] Step 9:

[0743] Sensor data is collected again and sent to the server. This allows for feedback on the results of the instruction execution. The input data is the newly collected data, and the output data is the feedback information.

[0744] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0745] This invention combines a real-time work monitoring and remote instruction system using sensor means with an emotion engine. The following describes specific embodiments of this system.

[0746] System-wide configuration

[0747] This system aims to monitor subordinates' work status in real time, provide efficient and appropriate instructions, and improve work efficiency by recognizing the user's emotional state. Its main components are sensors, a chip, a server, a terminal, a user interface, and an emotion engine.

[0748] System Overview

[0749] 1. Sensor and chip functions

[0750] The sensors monitor the subordinates' work status in real time. Specifically, they combine IMUs (Inertial Measurement Units), GPS, temperature sensors, and other sensors to acquire operational information, location information, and environmental information.

[0751] The chip receives data from the sensor and performs data preprocessing. Preprocessing includes noise reduction and feature extraction. The preprocessed data is transmitted wirelessly to the server at regular intervals.

[0752] 2. Server Functions

[0753] The server receives data transmitted from the chip and stores it in a database. The stored data is then analyzed in real time by an AI model.

[0754] Based on the analysis results, the server determines a specific situation and generates an appropriate notification. For example, if the situation "Subordinate A is not wearing a safety harness" is detected, that fact will be included in the notification.

[0755] The generated notifications are sent to the user's device. Additionally, the emotion engine analyzes the user's emotional state and generates notification text appropriate to that state.

[0756] 3. Device functions

[0757] The device receives a notification and displays it to the user. Based on this notification, the user can input specific instructions. For example, they can use text or voice to give instructions such as "Please readjust your safety harness."

[0758] The device is equipped with an emotion engine that analyzes the user's voice and facial expression data to determine their emotional state. This emotional state is sent to a server and used to generate notification messages.

[0759] 4. User Roles

[0760] The user (administrator) uses a terminal to monitor notifications and issue instructions as needed. The user's emotional state is analyzed, and the system generates support notifications as needed to help make appropriate decisions.

[0761] Specific examples of the system

[0762] Specific example: Use at construction sites

[0763] 1. Collection of sensor data

[0764] The chip is attached to the helmet of a subordinate working at a construction site. The IMU sensor tracks the subordinate's movements, the GPS tracks their location, and the temperature sensor measures the ambient temperature.

[0765] 2. Data preprocessing and transmission

[0766] The chip preprocesses the data obtained from the sensor and extracts specific features (for example, position and movement patterns during work).

[0767] 3. Data Analysis

[0768] The server analyzes the transmitted data in real time and detects if there is a possibility that the subordinate is not wearing their safety harness correctly.

[0769] 4. Creating a notification

[0770] The server generates a notification and sends it to the administrator's terminal. Simultaneously, the emotion engine analyzes the user's emotional state and adjusts the notification wording as needed.

[0771] 5. Administrator's response

[0772] The user (administrator) checks the notification on their device and enters the instruction, "Please readjust your safety harness." Additional advice and support notifications are displayed depending on the user's emotional state, such as if they are feeling anxious.

[0773] 6. Communication and execution of instructions

[0774] Instructions are transmitted to the chip via the server, and the subordinate receives the instructions. The subordinate follows the instructions and readjusts their safety harness.

[0775] 7. Gathering Feedback

[0776] The chip collects data again and sends it to the server, providing feedback on the results of the instructions' execution.

[0777] This allows managers to monitor their subordinates' on-site work in real time and issue appropriate instructions. Furthermore, by considering the user's emotional state when providing support, work efficiency and safety can be further improved.

[0778] The following describes the processing flow.

[0779] Step 1:

[0780] tip

[0781] Sensors (such as IMUs, GPS, and temperature sensors) measure the subordinate's movements, location, and environmental data in real time. This raw data is temporarily stored inside the chip.

[0782] Step 2:

[0783] tip

[0784] The collected raw data is preprocessed. Preprocessing includes noise reduction and feature extraction. For example, features of the subordinate's posture and movements are extracted from IMU data.

[0785] Step 3:

[0786] tip

[0787] Pre-processed data is transmitted to the server wirelessly (Wi-Fi, 4G / 5G, etc.) at regular time intervals (e.g., every second). If transmission is successful, the data is deleted from internal memory. If transmission fails, a retransmission is attempted.

[0788] Step 4:

[0789] server

[0790] The system receives data transmitted from the chip and stores it in a database. Upon reception, the system performs a data integrity check. If invalid or missing data is detected, it sends a retransmission instruction to the chip.

[0791] Step 5:

[0792] server

[0793] The system analyzes stored data in real time. It uses machine learning models (e.g., deep learning models) to determine the status of subordinates (e.g., "operating normally," "anomaly detected," etc.) based on the data.

[0794] Step 6:

[0795] server

[0796] The system generates notifications based on the analysis results. For example, if it detects an anomaly such as "Subordinate A is not wearing a safety harness," it will generate a notification including that information.

[0797] Step 7:

[0798] server

[0799] The generated notification is sent to the user's device. The sending method can be push notification or email.

[0800] Step 8:

[0801] terminal

[0802] Receive notifications from the server and display them to the user. Users can check the notifications on their devices and view detailed information on the dashboard.

[0803] Step 9:

[0804] Emotion engine (built into the device)

[0805] The system analyzes the user's voice and facial expression data from the camera to determine the user's emotional state (e.g., "tense," "relaxed," etc.). This emotional state is then sent to the server.

[0806] Step 10:

[0807] server

[0808] The system takes the user's emotional state into account and adjusts the notification wording accordingly. For example, if the user is feeling anxious, the message "Please readjust your safety harness" might be changed to "Please take your time, check and fasten your safety harness one more time."

[0809] Step 11:

[0810] terminal

[0811] The system displays notifications to the user based on their emotional state. The user then enters appropriate instructions based on these notifications.

[0812] Step 12:

[0813] User

[0814] Based on the notification, enter the necessary instructions into the terminal. For example, you might enter a message such as "Instruct subordinate A to readjust their safety harness" via voice or text.

[0815] Step 13:

[0816] terminal

[0817] The user's input instructions are sent to the server. Error checking and retransmission are performed during transmission.

[0818] Step 14:

[0819] server

[0820] It receives user instructions and delivers them to the chip. It uses real-time communication protocols (e.g., MQTT or WebSockets).

[0821] Step 15:

[0822] tip

[0823] Receive instructions and notify subordinates. If the instructions are voiced, play them through the speaker; if they are text-based, display them on the screen.

[0824] Step 16:

[0825] subordinate

[0826] Confirm instructions from the user and act accordingly. For example, readjust your safety harness.

[0827] Step 17:

[0828] tip

[0829] The sensor data is collected again and sent to the server. This verifies that the instructions were executed correctly.

[0830] Step 18:

[0831] server

[0832] Analyze the newly collected data to confirm that the instructions were followed correctly. If the problem recurs, send another notification.

[0833] (Example 2)

[0834] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0835] Conventional work monitoring systems have struggled to accurately monitor work status in real time and provide appropriate instructions based on that information. Furthermore, they fail to provide notifications and instructions that take into account the user's emotional state, resulting in a lack of improvement in work efficiency and safety.

[0836] In Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for collecting physical data using sensor means, means for preprocessing the collected data to extract important features, means for transmitting the preprocessed data to the server, means for the server to analyze the received data and determine a specific situation, means for generating a notification based on the analysis results and transmitting it to a terminal, means for the terminal to display the notification and prompt the user to input instructions, means for analyzing the user's emotional state and adjusting the notification wording, means for transmitting the user's instructions to the server and for the server to redistribute them to a chip, means for the chip that received the instructions to reproduce or display the content, and means for transmitting the recollected data to the server to provide feedback. This enables accurate monitoring of the work status in real time and the provision of appropriate instructions according to the user's emotional state.

[0837] "Sensing means" refers to a device used to collect physical data.

[0838] "Data preprocessing" refers to the process of converting data collected by sensor devices into a format suitable for analysis.

[0839] A "chip" is an electronic circuit that receives data from a sensor, performs preprocessing, and transmits the data to a server.

[0840] A "server" is a computing device that receives data transmitted from a chip, analyzes it, generates notifications, and sends them to a terminal.

[0841] A "notification" is a message that the server generates based on the analysis results and sends to the terminal.

[0842] A "terminal" is a device that displays notifications sent from the server to the user and sends instructions from the user to the server.

[0843] A "user" refers to a person who operates a device, checks notifications, and enters instructions as needed.

[0844] "Emotional analysis" is a process that analyzes a user's voice and facial expression data to determine the user's emotional state.

[0845] "Feedback" refers to the process of sending the collected data back to the server and reporting the results of the instructions' execution.

[0846] A "real-time communication protocol" is a communication method used to instantly exchange data between a server and a chip.

[0847] Modes for carrying out the invention

[0848] This invention relates to a system that monitors work status in real time and provides appropriate instructions that take into account the user's emotional state. Specific embodiments of this system are described below.

[0849] System-wide configuration

[0850] The system consists of sensors, chips, servers, terminals, users, and an emotion engine. These elements work together to collect, process, and analyze real-time data, generate notifications, communicate instructions, and provide feedback.

[0851] Sensor and chip functions

[0852] 1. Sensor:

[0853] It is used to monitor the work status of subordinates in real time. Specifically, it includes an IMU (Inertial Measurement Unit), GPS, temperature sensors, etc., to acquire operational information, location information, and environmental information.

[0854] For example, an IMU sensor measures the worker's movements, GPS determines their location, and a temperature sensor obtains the ambient temperature of the site.

[0855] 2. Tips:

[0856] The system receives data from sensors and performs preprocessing such as noise reduction and feature extraction. The preprocessed data is then sent to the server at regular intervals.

[0857] The chip preprocesses the data, extracting features such as movement patterns and changes in position, and then sends them to the server.

[0858] Server Functions

[0859] 1. Server:

[0860] The system receives data transmitted from the chip and stores it in a database. The stored data is then analyzed in real time using an AI model.

[0861] Using an AI model, specific situations are determined from the analysis results. For example, it can detect the situation where "subordinate A is not wearing a safety harness."

[0862] The server generates a notification based on the analysis results and sends it to the user's device. Furthermore, it uses an emotion engine to analyze the user's emotional state and adjust the notification wording accordingly.

[0863] For example, the server might generate a notification stating, "Subordinate A is not wearing a safety harness," and if the user is confused, it might add a message saying, "Please remain calm."

[0864] Device functions

[0865] 1. Terminal:

[0866] The server receives notifications and displays them to the user. The user checks the notifications on their device and enters specific instructions.

[0867] The system is equipped with an emotion engine that analyzes the user's voice and facial expression data to determine their emotional state. This analysis result is sent to a server and used to adjust notification messages.

[0868] For example, a user might input the instruction "Please readjust your safety harness" into their terminal, and this instruction is transmitted to their subordinate via the server.

[0869] User roles

[0870] 1. User (Administrator):

[0871] The device monitors notifications and provides specific instructions as needed. The user's emotional state is analyzed, and supportive notifications tailored to that emotional state are provided to help make appropriate decisions.

[0872] For example, an administrator might issue an instruction to "reattach your safety harness," and then a support notification might appear stating, "Let's stay calm."

[0873] Specific example

[0874] Use at construction sites

[0875] 1. Sensor data collection:

[0876] The helmet is equipped with an IMU, GPS, and temperature sensor that measure the worker's movements, location, and ambient temperature in real time.

[0877] 2. Data preprocessing and transmission:

[0878] The chip preprocesses the data obtained from the sensor, extracts features, and sends them to the server.

[0879] 3. Data Analysis:

[0880] The server analyzes the received data using an AI model and determines that "the worker is not wearing the safety harness correctly."

[0881] 4. Creating and sending notifications:

[0882] The server generates a notification and sends it to the administrator's terminal. The emotion engine analyzes the user's emotional state and adjusts the notification wording as needed.

[0883] 5. Administrator's response:

[0884] The administrator checks the notification on the terminal and enters the instruction, "Please readjust your safety harness." Support notifications tailored to the user's emotional state are also displayed.

[0885] 6. Communication and execution of instructions:

[0886] Instructions are delivered to a chip via a server, and the worker receives the instructions. After that, they readjust their safety harness.

[0887] 7. Gathering feedback:

[0888] The chip collects data again and sends it to the server to provide feedback.

[0889] Examples of prompt statements

[0890] Please describe the process by which users receive real-time notifications and issue instructions for a sensor-based work monitoring system used on construction sites. Also, please describe in detail how support is provided that takes the user's emotional state into consideration.

[0891] This enables accurate monitoring of work status in real time and the provision of appropriate instructions tailored to the user's emotional state.

[0892] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0893] Step 1:

[0894] Data collection using sensors:

[0895] Input: Physical information such as the subordinate's actions, location, and ambient temperature.

[0896] Processing: The IMU sensor acquires operational information, the GPS acquires location information, and the temperature sensor acquires environmental information in real time.

[0897] Output: The raw data obtained.

[0898] Specific operation: Sensors measure the subordinate's movements in milliseconds, GPS determines their location, and temperature sensors collect temperature data.

[0899] Step 2:

[0900] Data preprocessing:

[0901] Input: Raw data acquired from the sensor.

[0902] Processing: The chip performs noise reduction and data correction, and extracts important features.

[0903] Output: Preprocessed data.

[0904] Specific operation: From raw data, outliers and unwanted noise are filtered out, and features such as "patterns of work movements" and "location movement history" are extracted.

[0905] Step 3:

[0906] Sending data:

[0907] Input: Preprocessed data.

[0908] Processing: The chip wirelessly transmits pre-processed data to the server at regular intervals.

[0909] Output: Data transmitted from the chip.

[0910] Specific operation: The chip uses a wireless communication protocol to send data to the server.

[0911] Step 4:

[0912] Data analysis:

[0913] Input: Pre-processed data transmitted from the chip.

[0914] Processing: The server uses an AI model to analyze the data and make a judgment about a specific situation.

[0915] Output: Analysis results (e.g., "Safety harness not worn").

[0916] Specific operation: The server analyzes data in real time and detects things like "a worker is inactive and stopped for a long time" or "an abnormal operation pattern."

[0917] Step 5:

[0918] Notification generation:

[0919] Input: Analysis results.

[0920] Processing: The server generates a notification based on the analysis results and sends it to the user's terminal. The emotion engine analyzes the user's emotional state and adjusts the notification wording.

[0921] Output: Adjusted notification.

[0922] Specific operation: The server generates a notification that "the worker has stopped working," and the emotion engine analyzes the user's stress level and adds the message "Please remain calm."

[0923] Step 6:

[0924] Displaying notifications and entering instructions:

[0925] Input: Adjusted notification.

[0926] Processing: The device displays a notification and prompts the user to enter specific instructions.

[0927] Output: Instructions.

[0928] Specific action: The terminal displays a notification saying "Worker A has stopped moving," and the user enters "Please readjust your safety harness."

[0929] Step 7:

[0930] Sending user instructions:

[0931] Input: Instructions from the user.

[0932] Processing: The terminal sends instructions to the server, and the server redistributes them to the chip.

[0933] Output: Instructions sent to the chip.

[0934] Specific operation: The terminal sends instructions to the server, and the server forwards those instructions to the chip.

[0935] Step 8:

[0936] Execute the instructions:

[0937] Input: The instructions sent to the chip.

[0938] Processing: The chip plays or displays the instructions.

[0939] Output: The actions taken by the worker who received the instructions.

[0940] Specific action: The chip notifies the worker via voice or text message, "Please readjust your safety harness."

[0941] Step 9:

[0942] Gathering feedback:

[0943] Input: Data after the instruction has been executed.

[0944] Processing: The chip collects data again and sends it to the server to provide feedback.

[0945] Output: Feedback data sent to the server.

[0946] Specific operation: The chip collects data from the sensor again, for example, to confirm that "the safety harness is properly fitted," and then sends the data to the server.

[0947] This allows the entire system to monitor work in real time and provide appropriate instructions. Furthermore, by considering the user's emotional state, it enables effective decision-making support.

[0948] (Application Example 2)

[0949] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0950] Traditional work monitoring systems focused on real-time monitoring of work status and remote instructions, but they did not provide notifications or instructions that took into account the emotional state of managers. As a result, manager stress increased, and work efficiency sometimes decreased. Furthermore, traditional systems focused only on physical safety and did not consider psychological aspects, thus failing to improve the overall work environment.

[0951] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for collecting physical data using sensor means, means for preprocessing the collected data to extract important features, means for transmitting the preprocessed data to the server, means for the server to analyze the received data and determine a specific situation, means for generating a notification based on the analysis results and transmitting it to a terminal, means for the terminal to display the notification and prompt the user to input instructions, means for transmitting the user's instructions to the server and for the server to redistribute them to a chip, means for the chip that received the instructions to reproduce or display their contents, means for transmitting the recollected data to the server to provide feedback, means for analyzing the user's emotional state using an emotion analysis engine, and means for adjusting the notification wording based on the analyzed emotional state. This makes it possible to provide appropriate notifications and instructions according to the administrator's emotional state and to improve the work environment while also considering psychological aspects.

[0952] A "sensor" is a device used to acquire physical data from the environment or objects.

[0953] "Physical data" refers to data such as operational information, location information, and environmental information measured by IMUs, GPS, temperature sensors, etc.

[0954] "Preprocessing" is a part of data processing that removes noise from collected data and extracts important features.

[0955] A "server" is a computing device that receives, analyzes, and stores data via a network, and generates appropriate notifications and instructions.

[0956] A "chip" is a hardware component that receives data from sensors, performs preprocessing, and sends the data to a server.

[0957] A "terminal" is a device that receives notifications from a server, displays them to the user, and allows the user to input instructions.

[0958] A "user" is an administrator or person in charge who uses the system and performs monitoring and instructions.

[0959] An "emotion analysis engine" is a software module that analyzes a user's voice and facial expression data to determine their emotional state.

[0960] A "notification" is an informational message generated based on analyzed data and sent to the device.

[0961] "Feedback" refers to the evaluation or response provided by the system based on the results of executing instructions and the collected data.

[0962] A "real-time communication protocol" is a set of communication standards and procedures used to exchange data quickly and efficiently.

[0963] System-wide configuration

[0964] This system consists of sensors, chips, servers, terminals, users, and an emotion analysis engine, and is primarily intended for monitoring work in factories and generating notifications based on emotional states.

[0965] Sensor means

[0966] Sensors are devices used to acquire motion information, location information, and environmental information from the environment or objects. Specifically, IMUs, GPS, and temperature sensors are used. These sensors are attached to robots or workers to collect data in real time.

[0967] Chip functions

[0968] The chip receives data from the sensor and performs preprocessing such as noise reduction and feature extraction. The preprocessed data is then transmitted wirelessly to the server at regular intervals.

[0969] Server Functions

[0970] The server receives data transmitted from the chip and stores it in a database. The stored data is analyzed in real time. The server analyzes sensor data to determine specific situations. It also generates notifications based on the received data and sends them to the terminal. Furthermore, it uses an emotion analysis engine to analyze the user's (administrator's) emotional state and adjust the notification wording accordingly.

[0971] Device functions

[0972] The terminal receives notifications from the server and displays them to the user. The user inputs instructions based on the notification content, and these instructions are sent back to the server. The server redistributes the received instructions to a chip, which then relays the instructions to the robot or worker. After the instructions are executed, data is collected again and sent back to the server to provide feedback.

[0973] User roles

[0974] The user (administrator) uses the terminal to monitor notifications and issue instructions as needed. The emotion analysis engine analyzes the user's voice and facial expression data to determine their emotional state, and the notification wording is adjusted accordingly to support appropriate responses.

[0975] Functions of the emotion analysis engine

[0976] The emotion analysis engine uses a generative AI model that recognizes the user's emotional state by taking user voice and video data as input. Based on the analysis results, it determines stress levels, relaxation levels, etc., and reflects this in the notification message.

[0977] Specific example

[0978] For example, in a factory setting, if a supervisor wears a head-mounted display (HMD) to monitor the robot's work status, the data collected by the sensors is transmitted to a server via a chip. The server analyzes the data and detects situations where the robot requires maintenance. The detected situation is sent as a notification to the terminal, allowing the user to check the notification and issue instructions. An emotion analysis engine analyzes the user's stress level and generates notification messages such as, "Please perform maintenance work. We also recommend taking a break," depending on the user's state.

[0979] Example of a prompt

[0980] The inputs to the generative AI model when using the emotion analysis engine are as follows:

[0981] "Please analyze this user's voice and facial expressions to determine their current emotional state. If possible, please provide a specific emotional state (e.g., tense, relaxed)."

[0982] This system not only facilitates smooth work monitoring and remote instruction, but also reduces the psychological burden on users, resulting in an efficient and safe work environment.

[0983] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0984] Step 1:

[0985] Sensors are attached to robots and workers. These sensors, using IMUs, GPS, and temperature sensors, collect motion information, location information, and environmental information in real time. The input is sensor data, and the output is raw data before preprocessing.

[0986] Step 2:

[0987] The chip receives data from the sensor and performs noise reduction and feature extraction. Specifically, it uses filtering techniques to reduce noise and extract only the necessary parameters. The input is sensor data, raw data, and the output is pre-processed data.

[0988] Step 3:

[0989] The chip wirelessly transmits pre-processed data to the server. The server receives this data and stores it in a database. The input is the pre-processed data, and the output is the data stored in the database.

[0990] Step 4:

[0991] The server analyzes the received data and uses data mining techniques to determine specific situations. For example, it uses machine learning algorithms to detect situations where a robot requires maintenance. The input is data stored in a database, and the output is the analysis result.

[0992] Step 5:

[0993] The server generates a notification based on the analysis results and sends it to the terminal. The notification includes situation-specific instructions and warnings. The input is the analysis results, and the output is the generated notification.

[0994] Step 6:

[0995] The terminal receives notifications from the server and displays them to the user. The user reviews the notification content and enters the necessary instructions. The input is the notification from the server, and the output is the user's instructions.

[0996] Step 7:

[0997] User instructions are sent from the terminal to the server. The server receives the instructions and redistributes them to the chip. The input is the user's instructions, and the output is the instructions sent to the chip.

[0998] Step 8:

[0999] The chip that receives the instruction will play or display its contents. This allows the robot or worker to take a specific action. The input is the instruction from the server, and the output is the played or displayed instruction.

[1000] Step 9:

[1001] After the instruction is executed, the sensor collects data again and sends the new data to the server. The server receives this data and updates the entire process as feedback. The input is the newly collected sensor data, the raw data, and the output is the updated database.

[1002] Step 10:

[1003] The emotion analysis engine analyzes the user's voice and video data to determine their emotional state. The server adjusts the notification message based on the analysis results. The input is the user's voice and video data, and the output is the adjusted notification message.

[1004] Through each of the above steps, real-time work monitoring and remote instructions are provided, along with appropriate notifications tailored to the user's emotional state. This results in an efficient and safe work environment.

[1005] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1006] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1007] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[1008] [Third Embodiment]

[1009] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[1010] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1011] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1012] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[1013] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1014] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1015] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1016] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1017] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1018] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1019] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1020] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[1021] This invention relates to a real-time work monitoring and remote instruction system using sensor means. Embodiments of this system are described in detail below.

[1022] System-wide configuration

[1023] This system aims to monitor the work status of subordinates in real time and provide efficient and appropriate instructions. Its main components are sensors, chips, servers, terminals, and users.

[1024] System Overview

[1025] 1. Sensor and chip functions

[1026] The sensors monitor the subordinates' work status in real time. Specifically, they combine IMUs (Inertial Measurement Units), GPS, temperature sensors, and other sensors to acquire operational information, location information, and environmental information.

[1027] The chip receives data from the sensor and performs data preprocessing. Preprocessing includes noise reduction and feature extraction. The preprocessed data is transmitted wirelessly to the server at regular intervals.

[1028] 2. Server Functions

[1029] The server receives data transmitted from the chip and stores it in a database. The stored data is then analyzed in real time by an AI model.

[1030] Based on the analysis results, the server determines a specific situation and generates an appropriate notification. For example, if the situation "Subordinate A is not wearing a safety harness" is detected, that fact will be included in the notification.

[1031] The generated notification is sent to the user's device.

[1032] 3. Device functions

[1033] The device receives a notification and displays it to the user. Based on this notification, the user can input specific instructions. For example, they can use text or voice to give instructions such as "Please readjust your safety harness."

[1034] The user's input is sent back to the server, which then delivers the instructions to the chip.

[1035] 4. User Roles

[1036] The user (administrator) uses a device to monitor notifications and issue instructions as needed. Instructions can be easily given via text or voice input.

[1037] By providing appropriate instructions, users can improve the work efficiency of their subordinates and maintain a safe working environment.

[1038] Specific examples of the system

[1039] Specific example: Use at construction sites

[1040] 1. Collection of sensor data

[1041] The chip is attached to the helmet of a subordinate working at a construction site. The IMU sensor tracks the subordinate's movements, the GPS tracks their location, and the temperature sensor measures the ambient temperature.

[1042] 2. Data preprocessing and transmission

[1043] The chip preprocesses the data obtained from the sensor and extracts specific features (for example, position and movement patterns during work).

[1044] 3. Data Analysis

[1045] The server analyzes the transmitted data in real time and detects if there is a possibility that the subordinate is not wearing their safety harness correctly.

[1046] 4. Creating and sending notifications

[1047] The server generates a notification and sends it to the administrator's terminal.

[1048] 5. Administrator's response

[1049] The user (administrator) checks the notification on their device and enters the instruction, "Please readjust your safety harness."

[1050] 6. Communication and execution of instructions

[1051] Instructions are transmitted to the chip via the server, and the subordinate receives the instructions.

[1052] The subordinate followed instructions and readjusted his safety harness.

[1053] 7. Gathering Feedback

[1054] The chip collects data again and sends it to the server, providing feedback on the results of the instructions' execution.

[1055] This allows managers to monitor their subordinates' on-site work in real time and issue appropriate instructions. This system significantly improves work efficiency and safety.

[1056] The following describes the processing flow.

[1057] Step 1:

[1058] tip

[1059] Sensors (such as IMUs, GPS, and temperature sensors) measure the subordinate's movements, location, and environmental data in real time. This raw data is temporarily stored inside the chip.

[1060] Step 2:

[1061] tip

[1062] The collected raw data is preprocessed. Preprocessing includes noise reduction and feature extraction. For example, features of the subordinate's posture and movements are extracted from IMU data.

[1063] Step 3:

[1064] tip

[1065] Pre-processed data is transmitted to the server wirelessly (Wi-Fi, 4G / 5G, etc.) at regular time intervals (e.g., every second). If transmission is successful, the data is deleted from internal memory. If transmission fails, a retransmission is attempted.

[1066] Step 4:

[1067] server

[1068] The system receives data transmitted from the chip and stores it in a database. Upon receipt, the system performs a data integrity check, and if invalid or missing data is detected, it sends a retransmission instruction to the chip.

[1069] Step 5:

[1070] server

[1071] The system analyzes stored data in real time. It uses machine learning models (e.g., deep learning models) to determine the status of subordinates (e.g., "operating normally," "anomaly detected," etc.) based on the data.

[1072] Step 6:

[1073] server

[1074] The system generates notifications based on the analysis results. For example, if it detects an anomaly such as "Subordinate A is not wearing a safety harness," it will generate a notification including that information.

[1075] Step 7:

[1076] server

[1077] The generated notification is sent to the user's device. The sending method can be push notification or email.

[1078] Step 8:

[1079] terminal

[1080] Receive notifications from the server and display them to the user. Users can check the notifications on their devices and view detailed information on the dashboard.

[1081] Step 9:

[1082] User

[1083] Based on the notification, enter the necessary instructions into the terminal. For example, you might enter a message such as "Instruct subordinate A to readjust their safety harness" via voice or text.

[1084] Step 10:

[1085] terminal

[1086] The user's input instructions are sent to the server. Error checking and retransmission are performed during transmission.

[1087] Step 11:

[1088] server

[1089] It receives user instructions and delivers them to the chip. It uses real-time communication protocols (e.g., MQTT or WebSockets).

[1090] Step 12:

[1091] tip

[1092] Receive instructions and notify subordinates. If the instructions are voiced, play them through the speaker; if they are text-based, display them on the screen.

[1093] Step 13:

[1094] subordinate

[1095] Confirm instructions from the user and act accordingly. For example, readjust your safety harness.

[1096] Step 14:

[1097] tip

[1098] The sensor data is collected again and sent to the server. This verifies that the instructions were executed correctly.

[1099] Step 15:

[1100] server

[1101] Analyze the newly collected data to confirm that the instructions were followed correctly. If the problem recurs, send another notification.

[1102] (Example 1)

[1103] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1104] Conventional work monitoring and instruction systems made it difficult to grasp the work status of subordinates in real time and to issue appropriate instructions quickly. Furthermore, the data obtained from sensors contained a lot of noise and irrelevant information, resulting in insufficient pre-processing for accurate situational assessment. In addition, delays in generating notifications based on analysis results and providing feedback on instructions based on those results prevented sufficient improvements in work efficiency and safety.

[1105] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1106] In this invention, the server includes means for collecting physical data using sensor means, means for preprocessing the collected data to extract important features, means for transmitting the preprocessed data to a computer, means for the computer to analyze the received data and determine a specific situation, means for generating a notification based on the analysis results and transmitting it to a terminal, means for the terminal to display the notification and prompt the user to input instructions, means for transmitting the user's instructions to the computer and for the computer to redistribute them to a chip, means for the chip that received the instructions to reproduce or display their contents, means for transmitting the recollected data to the computer to provide feedback, and means for the computer to perform real-time analysis using a domain-specific model based on the instructions input by the user. This makes it possible to accurately grasp the work status of subordinates in real time and quickly issue appropriate instructions. Furthermore, it is possible to improve work efficiency and safety.

[1107] A "sensor" is a device or system used to collect physical data.

[1108] "Preprocessing" refers to the process of removing noise from collected data and extracting important features.

[1109] "Transmission means" refers to a function for transmitting pre-processed data to other devices or systems.

[1110] A "computer" is a device or system that analyzes received data and makes judgments about specific situations.

[1111] "Analysis means" refers to methods or functions for making judgments about specific situations based on data.

[1112] A "notification" is information or a message generated based on the analysis results.

[1113] A "terminal" is a device that displays notifications to the user and allows them to input instructions.

[1114] A "user" is a person or administrator who uses a terminal to input instructions.

[1115] "Feedback" refers to information used to verify the progress of instructions based on recollected data.

[1116] A "domain-specific model" is an AI model that is specialized for a particular industry or domain.

[1117] "Real-time analysis" is an analytical method that processes data immediately and obtains results instantly.

[1118] Modes for carrying out the invention

[1119] This invention relates to a real-time work monitoring and remote instruction system using sensor means. The system aims to monitor the work status of subordinates in real time and provide efficient and appropriate instructions. The main components are sensors, chips, a server, a terminal, and the user. The following describes specific embodiments of this system.

[1120] Sensor and chip functions

[1121] The sensors monitor the work status of subordinates in real time. Specifically, they combine IMUs (Inertial Measurement Units), GPS, temperature sensors, etc., to acquire motion information, location information, and environmental information. These sensors are often attached to the worker's helmet or work clothes. For example, an IMU sensor attached to the helmet measures the subordinate's movements, a GPS determines their location, and a temperature sensor measures the ambient temperature.

[1122] The chip receives data from sensors and performs preprocessing such as noise reduction and feature extraction. For example, it removes unwanted vibration information from collected IMU sensor data and extracts specific operating patterns. The preprocessed data is transmitted to a server via wireless communication at regular intervals.

[1123] Server Functions

[1124] The server receives data transmitted from the chip and stores it in a database. The received data is temporarily stored in memory and then saved to the specified database. For example, databases such as MySQL or PostgreSQL may be used.

[1125] The server analyzes the stored data in real time. This analysis may utilize generative AI models such as TensorFlow or PyTorch. Through this analysis, it analyzes behavioral patterns and location information to detect abnormal situations, such as "Subordinate A is not wearing a safety harness."

[1126] Based on the analysis results, the server generates an appropriate notification and sends it to the user's terminal. For example, it might generate a notification stating, "Subordinate A is not wearing a safety harness," and send it to the user's terminal.

[1127] Device functions

[1128] The device receives notifications sent from the server and displays them to the user. Notifications are often displayed as pop-ups on the screen. Based on the notification, the user enters specific instructions. For example, they might enter a text instruction such as "Please readjust your safety harness" into the device. The device also has a voice input function, allowing users to enter instructions by voice.

[1129] Instructions entered by the user are sent back to the server from the terminal. The server distributes the received instructions to a chip, which then plays or displays the instructions to its subordinates. This allows the subordinates to adjust their work according to the user's instructions.

[1130] Specific example

[1131] Use at construction sites

[1132] 1. Sensor data collection: The chip is attached to the helmet of a subordinate working at the construction site, and the IMU sensor measures operational information, the GPS measures location information, and the temperature sensor measures ambient temperature.

[1133] 2. Data preprocessing and transmission: The chip preprocesses the data obtained from the sensor and extracts specific features (position and movement patterns during operation).

[1134] 3. Data Analysis: The server analyzes the transmitted data in real time to detect the possibility that a subordinate is not wearing their safety harness correctly.

[1135] 4. Creating and sending notifications: The server generates a notification and sends it to the administrator's terminal.

[1136] 5. Administrator's response: The user (administrator) checks the notification on their device and enters the instruction, "Please readjust your safety harness."

[1137] 6. Instruction transmission and execution: Instructions are delivered to the chip via the server, and the subordinate receives them. The subordinate follows the instructions and readjusts their safety harness.

[1138] 7. Feedback Collection: The chip collects data again and sends it to the server to provide feedback on the results of the instructions being executed.

[1139] Example of a prompt

[1140] "At a construction site, there is a system that preprocesses and analyzes data obtained from IMU sensors, GPS, and temperature sensors attached to helmets. Please explain the processing flow of this system in detail."

[1141] This allows us to request a detailed explanation of the system from the generated AI model.

[1142] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1143] Step 1:

[1144] The user has their subordinates wear sensors. For example, an IMU sensor is attached to a helmet, and a GPS sensor is attached to work clothes. The input is the sensors being worn, and the output is the state of being ready to collect data in real time.

[1145] Step 2:

[1146] Sensors monitor the subordinate's work status in real time. The IMU sensor collects motion data, the GPS collects location data, and the temperature sensor collects ambient temperature data. The input is the subordinate's actions and environmental conditions, and the output is the collection of raw data. Specifically, the IMU sensor measures acceleration and rotation, and the GPS obtains latitude and longitude.

[1147] Step 3:

[1148] The chip receives data from the sensor and performs preprocessing such as noise reduction and feature extraction. It removes unwanted noise from the collected raw data and extracts specific operating patterns. The input is raw data from the sensor, and the output is clean, preprocessed data. Specifically, it performs data filtering and statistical processing.

[1149] Step 4:

[1150] The chip transmits pre-processed data to the server via wireless communication at regular intervals. The input is clean data, and the output is data transmission to the server. Specifically, it transmits data packets using a wireless protocol.

[1151] Step 5:

[1152] The server receives data transmitted from the chip and stores it in the database. The input is the data from the chip, and the output is the stored data. Specifically, it performs database insertion operations.

[1153] Step 6:

[1154] The server analyzes stored data in real time and performs analysis to determine specific situations. The input is stored data, and the output is the analysis result. Specifically, it performs data analysis using generative AI models such as TensorFlow and PyTorch.

[1155] Step 7:

[1156] The server generates an appropriate notification based on the analysis results and sends it to the user's terminal. The input is the analysis results, and the output is the notification message. Specifically, a notification is generated stating, "Subordinate A is not wearing a safety harness."

[1157] Step 8:

[1158] The device receives notifications from the server and displays them to the user. The input is the notification message, and the output is the notification displayed on the user interface. Specifically, it is displayed as a pop-up notification on the screen.

[1159] Step 9:

[1160] The user enters specific instructions based on a notification displayed on the device. The input is the content of the notification, and the output is the entered instruction. For example, the user might enter text or voice instructions such as "Please readjust your safety harness" into the device.

[1161] Step 10:

[1162] The terminal sends user instructions to the server. The input is the user's instructions, and the output is the data sent to the server. Specifically, text and voice instructions are converted into data packets and sent.

[1163] Step 11:

[1164] The server receives user instructions and redistributes them to the chip. The input is the user's instruction data, and the output is the data to be sent to the chip. Specifically, the instruction content is transmitted using a protocol.

[1165] Step 12:

[1166] The chip receives instructions and plays or displays the content to its subordinates. The input is instruction data from the server, and the output is the notification content for the subordinates. Specifically, it plays the voice message, "Please readjust your safety harness."

[1167] Step 13:

[1168] The chip sends the recollected data to the server and provides feedback. For example, it recollects data to confirm that the safety harness has been reattached. The input is the recollected data, and the output is the feedback data to the server. This allows the administrator to confirm that the instructions have been followed.

[1169] (Application Example 1)

[1170] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1171] Traditional food delivery systems have made it difficult to accurately monitor the work status and efficiency of delivery personnel in real time and to issue appropriate instructions. This has resulted in delivery delays, decreased efficiency, and lower customer satisfaction. Furthermore, delays in providing feedback to delivery personnel and the inability to respond immediately have been a challenge.

[1172] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1173] In this invention, the server includes means for analyzing real-time data, including the movement patterns of delivery personnel; means for evaluating delivery efficiency based on the analysis results and generating instructions to improve efficiency; and means for proposing specific actions to delivery personnel based on user instructions. This makes it possible to accurately grasp the current status of delivery personnel in real time and provide appropriate instructions immediately.

[1174] A "sensor" is a device used to collect physical data in real time.

[1175] "IMU" is an abbreviation for Inertial Measurement Unit, a sensor that detects the movement and acceleration of an object.

[1176] "GPS" is an abbreviation for Global Positioning System, a system used to measure location on Earth.

[1177] An "accelerometer" is a sensor used to measure the acceleration of an object.

[1178] "Preprocessing means" refers to the process of removing noise from collected raw data and extracting important features.

[1179] A "server" is a computer system used to receive, store, and analyze data over a network.

[1180] An "analysis tool" is a mechanism for analyzing collected data and making judgments about a specific situation.

[1181] A "notification system" is a system for sending notifications generated based on analysis results to the user's terminal.

[1182] A "terminal" is a device used by a user to receive notifications and input instructions.

[1183] A "command system" is a system that sends user-inputted instructions to a server, which then redistributes them to the chip.

[1184] A "chip" is a device used to preprocess collected data and send it to a server.

[1185] "Movement patterns" refer to dynamic data such as the delivery person's travel route and speed.

[1186] "Feedback" refers to information about the results and effects of actions provided by sending the recollected data to the server.

[1187] This invention relates to a system that can be applied to food delivery to monitor the work status of delivery personnel in real time and issue appropriate instructions. This system is realized through the cooperation of sensor means, a server, a terminal, and a user.

[1188] System Configuration

[1189] 1. Sensor means:

[1190] Delivery personnel wear smart devices such as smart glasses or smartphones. These devices have built-in IMU sensors, GPS, and accelerometers.

[1191] 2. Data Acquisition and Preprocessing:

[1192] Sensors built into smart devices collect physical data on delivery personnel in real time. This includes location information, movement speed, and motion data.

[1193] The chip preprocesses the data collected from the sensor, performing noise reduction and feature extraction.

[1194] 3. Data transmission and analysis:

[1195] The pre-processed data is transmitted to the server via wireless communication.

[1196] The server stores the received data in a database and analyzes the data in real time using a generative AI model. For example, it analyzes the movement patterns of delivery personnel to detect delivery delays and decreased efficiency.

[1197] 4. Generating and sending notifications:

[1198] The server determines the specific situation based on the analysis results and generates and sends an appropriate notification to the terminal. For example, a notification such as "High acceleration detected. Please check traffic conditions" is generated.

[1199] 5. User actions:

[1200] The device displays a notification, and the user (administrator) reviews the notification before entering specific instructions. For example, they might enter instructions such as, "There is a rest point nearby; please take a break."

[1201] The input instructions are sent to the server, which then redistributes them to the chip.

[1202] 6. Communication and execution of instructions:

[1203] The chip receives instructions and plays or displays them. The delivery person acts according to the received instructions.

[1204] 7. Gathering feedback:

[1205] The chip collects sensor data again and sends it to the server, providing feedback on the results of the instructions. This allows the server to continuously monitor the delivery person's work status in real time.

[1206] Hardware and software usage examples

[1207] Hardware: Smart devices (smartphones, smart glasses), IMU sensors, GPS, accelerometers.

[1208] software:

[1209] Data preprocessing: Denoising and feature extraction are performed using Python.

[1210] Server: MySQL or PostgreSQL is used for database management, and TensorFlow or PyTorch is used for real-time data analysis.

[1211] Notification system: Web server frameworks such as Flask or Django are used for notifications from the server to the terminal.

[1212] Specific example

[1213] For example, when a delivery person travels towards a designated address, the GPS built into their smartphone collects location information, and the IMU sensor and accelerometer record movement information in real time. This data is preprocessed, features are extracted, and then it is sent to a server. The server analyzes the data using a generative AI model and generates notifications if the delivery is delayed or the delivery person is behaving inappropriately. When a notification appears on the device, the user (administrator) enters specific instructions, such as "Please use the recommended shortcuts," which are then transmitted to the delivery person.

[1214] Example of a prompt

[1215] "We analyze the movement patterns of delivery drivers and evaluate delivery performance in real time."

[1216] This system helps improve delivery efficiency in real time and enhance customer satisfaction.

[1217] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1218] Step 1:

[1219] The delivery person wears a smart device and begins the delivery. Sensors built into the smartphone or smart glasses (IMU sensor, GPS, accelerometer) collect physical data in real time. The input data includes location information, motion information, and movement speed, which are collected by the sensors.

[1220] Step 2:

[1221] The collected data is preprocessed by a chip within the device. Here, noise reduction is performed, and important features (such as location information and behavioral patterns) are extracted. The input data is the raw, collected data, while the output data is the preprocessed data.

[1222] Step 3:

[1223] The pre-processed data is transmitted to the server via wireless communication. The input data is pre-processed sensor data, which the server receives.

[1224] Step 4:

[1225] The server stores the received data in a database and analyzes the data using a generated AI model. Specifically, it analyzes the movement and action patterns of delivery personnel to determine delivery delays and decreased efficiency. The input data is the received data, and the output data is the analysis results.

[1226] Step 5:

[1227] The server generates a notification based on the analysis results and sends it to the terminal. It generates appropriate notifications for specific situations, such as "High acceleration detected. Please check traffic conditions." The input data is the analysis results, and the output data is the generated notification.

[1228] Step 6:

[1229] The device displays a notification, and the user reviews the notification content and enters specific instructions. For example, the user might enter instructions such as, "There is a rest point nearby; please take a break." The input data is the generated notification, and the output data is the instructions entered by the user.

[1230] Step 7:

[1231] Instructions from the user are sent to the server. The server receives these instructions and redistributes them to the chip. The input data is the user's instructions, and the output data is the redistributed instructions.

[1232] Step 8:

[1233] The chip receives instructions and plays or displays them. The delivery person acts according to the received instructions. The input data is the redistributed instructions, and the output data is the delivery person's actions.

[1234] Step 9:

[1235] Sensor data is collected again and sent to the server. This allows for feedback on the results of the instruction execution. The input data is the newly collected data, and the output data is the feedback information.

[1236] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1237] This invention combines a real-time work monitoring and remote instruction system using sensor means with an emotion engine. The following describes specific embodiments of this system.

[1238] System-wide configuration

[1239] This system aims to monitor subordinates' work status in real time, provide efficient and appropriate instructions, and improve work efficiency by recognizing the user's emotional state. Its main components are sensors, a chip, a server, a terminal, a user interface, and an emotion engine.

[1240] System Overview

[1241] 1. Sensor and chip functions

[1242] The sensors monitor the subordinates' work status in real time. Specifically, they combine IMUs (Inertial Measurement Units), GPS, temperature sensors, and other sensors to acquire operational information, location information, and environmental information.

[1243] The chip receives data from the sensor and performs data preprocessing. Preprocessing includes noise reduction and feature extraction. The preprocessed data is transmitted wirelessly to the server at regular intervals.

[1244] 2. Server Functions

[1245] The server receives data transmitted from the chip and stores it in a database. The stored data is then analyzed in real time by an AI model.

[1246] Based on the analysis results, the server determines a specific situation and generates an appropriate notification. For example, if the situation "Subordinate A is not wearing a safety harness" is detected, that fact will be included in the notification.

[1247] The generated notifications are sent to the user's device. Additionally, the emotion engine analyzes the user's emotional state and generates notification text appropriate to that state.

[1248] 3. Device functions

[1249] The device receives a notification and displays it to the user. Based on this notification, the user can input specific instructions. For example, they can use text or voice to give instructions such as "Please readjust your safety harness."

[1250] The device is equipped with an emotion engine that analyzes the user's voice and facial expression data to determine their emotional state. This emotional state is sent to a server and used to generate notification messages.

[1251] 4. User Roles

[1252] The user (administrator) uses a terminal to monitor notifications and issue instructions as needed. The user's emotional state is analyzed, and the system generates support notifications as needed to help make appropriate decisions.

[1253] Specific examples of the system

[1254] Specific example: Use at construction sites

[1255] 1. Collection of sensor data

[1256] The chip is attached to the helmet of a subordinate working at a construction site. The IMU sensor tracks the subordinate's movements, the GPS tracks their location, and the temperature sensor measures the ambient temperature.

[1257] 2. Data preprocessing and transmission

[1258] The chip preprocesses the data obtained from the sensor and extracts specific features (for example, position and movement patterns during work).

[1259] 3. Data Analysis

[1260] The server analyzes the transmitted data in real time and detects if there is a possibility that the subordinate is not wearing their safety harness correctly.

[1261] 4. Creating a notification

[1262] The server generates a notification and sends it to the administrator's terminal. Simultaneously, the emotion engine analyzes the user's emotional state and adjusts the notification wording as needed.

[1263] 5. Administrator's response

[1264] The user (administrator) checks the notification on their device and enters the instruction, "Please readjust your safety harness." Additional advice and support notifications are displayed depending on the user's emotional state, such as if they are feeling anxious.

[1265] 6. Communication and execution of instructions

[1266] Instructions are transmitted to the chip via the server, and the subordinate receives the instructions. The subordinate follows the instructions and readjusts their safety harness.

[1267] 7. Gathering Feedback

[1268] The chip collects data again and sends it to the server, providing feedback on the results of the instructions' execution.

[1269] This allows managers to monitor their subordinates' on-site work in real time and issue appropriate instructions. Furthermore, by considering the user's emotional state when providing support, work efficiency and safety can be further improved.

[1270] The following describes the processing flow.

[1271] Step 1:

[1272] tip

[1273] Sensors (such as IMUs, GPS, and temperature sensors) measure the subordinate's movements, location, and environmental data in real time. This raw data is temporarily stored inside the chip.

[1274] Step 2:

[1275] tip

[1276] The collected raw data is preprocessed. Preprocessing includes noise reduction and feature extraction. For example, features of the subordinate's posture and movements are extracted from IMU data.

[1277] Step 3:

[1278] tip

[1279] Pre-processed data is transmitted to the server wirelessly (Wi-Fi, 4G / 5G, etc.) at regular time intervals (e.g., every second). If transmission is successful, the data is deleted from internal memory. If transmission fails, a retransmission is attempted.

[1280] Step 4:

[1281] server

[1282] The system receives data transmitted from the chip and stores it in a database. Upon reception, the system performs a data integrity check. If invalid or missing data is detected, it sends a retransmission instruction to the chip.

[1283] Step 5:

[1284] server

[1285] The system analyzes stored data in real time. It uses machine learning models (e.g., deep learning models) to determine the status of subordinates (e.g., "operating normally," "anomaly detected," etc.) based on the data.

[1286] Step 6:

[1287] server

[1288] The system generates notifications based on the analysis results. For example, if it detects an anomaly such as "Subordinate A is not wearing a safety harness," it will generate a notification including that information.

[1289] Step 7:

[1290] server

[1291] The generated notification is sent to the user's device. The sending method can be push notification or email.

[1292] Step 8:

[1293] terminal

[1294] Receive notifications from the server and display them to the user. Users can check the notifications on their devices and view detailed information on the dashboard.

[1295] Step 9:

[1296] Emotion engine (built into the device)

[1297] The system analyzes the user's voice and facial expression data from the camera to determine the user's emotional state (e.g., "tense," "relaxed," etc.). This emotional state is then sent to the server.

[1298] Step 10:

[1299] server

[1300] The system takes the user's emotional state into account and adjusts the notification wording accordingly. For example, if the user is feeling anxious, the message "Please readjust your safety harness" might be changed to "Please take your time, check and fasten your safety harness one more time."

[1301] Step 11:

[1302] terminal

[1303] The system displays notifications to the user based on their emotional state. The user then enters appropriate instructions based on these notifications.

[1304] Step 12:

[1305] User

[1306] Based on the notification, enter the necessary instructions into the terminal. For example, you might enter a message such as "Instruct subordinate A to readjust their safety harness" via voice or text.

[1307] Step 13:

[1308] terminal

[1309] The user's input instructions are sent to the server. Error checking and retransmission are performed during transmission.

[1310] Step 14:

[1311] server

[1312] It receives user instructions and delivers them to the chip. It uses real-time communication protocols (e.g., MQTT or WebSockets).

[1313] Step 15:

[1314] tip

[1315] Receive instructions and notify subordinates. If the instructions are voiced, play them through the speaker; if they are text-based, display them on the screen.

[1316] Step 16:

[1317] subordinate

[1318] Confirm instructions from the user and act accordingly. For example, readjust your safety harness.

[1319] Step 17:

[1320] tip

[1321] The sensor data is collected again and sent to the server. This verifies that the instructions were executed correctly.

[1322] Step 18:

[1323] server

[1324] Analyze the newly collected data to confirm that the instructions were followed correctly. If the problem recurs, send another notification.

[1325] (Example 2)

[1326] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1327] Conventional work monitoring systems have struggled to accurately monitor work status in real time and provide appropriate instructions based on that information. Furthermore, they fail to provide notifications and instructions that take into account the user's emotional state, resulting in a lack of improvement in work efficiency and safety.

[1328] In Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for collecting physical data using sensor means, means for preprocessing the collected data to extract important features, means for transmitting the preprocessed data to the server, means for the server to analyze the received data and determine a specific situation, means for generating a notification based on the analysis results and transmitting it to a terminal, means for the terminal to display the notification and prompt the user to input instructions, means for analyzing the user's emotional state and adjusting the notification wording, means for transmitting the user's instructions to the server and for the server to redistribute them to a chip, means for the chip that received the instructions to reproduce or display the content, and means for transmitting the recollected data to the server to provide feedback. This enables accurate monitoring of the work status in real time and the provision of appropriate instructions according to the user's emotional state.

[1329] "Sensing means" refers to a device used to collect physical data.

[1330] "Data preprocessing" refers to the process of converting data collected by sensor devices into a format suitable for analysis.

[1331] A "chip" is an electronic circuit that receives data from a sensor, performs preprocessing, and transmits the data to a server.

[1332] A "server" is a computing device that receives data transmitted from a chip, analyzes it, generates notifications, and sends them to a terminal.

[1333] A "notification" is a message that the server generates based on the analysis results and sends to the terminal.

[1334] A "terminal" is a device that displays notifications sent from the server to the user and sends instructions from the user to the server.

[1335] A "user" refers to a person who operates a device, checks notifications, and enters instructions as needed.

[1336] "Emotional analysis" is a process that analyzes a user's voice and facial expression data to determine the user's emotional state.

[1337] "Feedback" refers to the process of sending the collected data back to the server and reporting the results of the instructions' execution.

[1338] A "real-time communication protocol" is a communication method used to instantly exchange data between a server and a chip.

[1339] Modes for carrying out the invention

[1340] This invention relates to a system that monitors work status in real time and provides appropriate instructions that take into account the user's emotional state. Specific embodiments of this system are described below.

[1341] System-wide configuration

[1342] The system consists of sensors, chips, servers, terminals, users, and an emotion engine. These elements work together to collect, process, and analyze real-time data, generate notifications, communicate instructions, and provide feedback.

[1343] Sensor and chip functions

[1344] 1. Sensor:

[1345] It is used to monitor the work status of subordinates in real time. Specifically, it includes an IMU (Inertial Measurement Unit), GPS, temperature sensors, etc., to acquire operational information, location information, and environmental information.

[1346] For example, an IMU sensor measures the worker's movements, GPS determines their location, and a temperature sensor obtains the ambient temperature of the site.

[1347] 2. Tips:

[1348] The system receives data from sensors and performs preprocessing such as noise reduction and feature extraction. The preprocessed data is then sent to the server at regular intervals.

[1349] The chip preprocesses the data, extracting features such as movement patterns and changes in position, and then sends them to the server.

[1350] Server Functions

[1351] 1. Server:

[1352] The system receives data transmitted from the chip and stores it in a database. The stored data is then analyzed in real time using an AI model.

[1353] Using an AI model, specific situations are determined from the analysis results. For example, it can detect the situation where "subordinate A is not wearing a safety harness."

[1354] The server generates a notification based on the analysis results and sends it to the user's device. Furthermore, it uses an emotion engine to analyze the user's emotional state and adjust the notification wording accordingly.

[1355] For example, the server might generate a notification stating, "Subordinate A is not wearing a safety harness," and if the user is confused, it might add a message saying, "Please remain calm."

[1356] Device functions

[1357] 1. Terminal:

[1358] The server receives notifications and displays them to the user. The user checks the notifications on their device and enters specific instructions.

[1359] The system is equipped with an emotion engine that analyzes the user's voice and facial expression data to determine their emotional state. This analysis result is sent to a server and used to adjust notification messages.

[1360] For example, a user might input the instruction "Please readjust your safety harness" into their terminal, and this instruction is transmitted to their subordinate via the server.

[1361] User roles

[1362] 1. User (Administrator):

[1363] The device monitors notifications and provides specific instructions as needed. The user's emotional state is analyzed, and supportive notifications tailored to that emotional state are provided to help make appropriate decisions.

[1364] For example, an administrator might issue an instruction to "reattach your safety harness," and then a support notification might appear stating, "Let's stay calm."

[1365] Specific example

[1366] Use at construction sites

[1367] 1. Sensor data collection:

[1368] The helmet is equipped with an IMU, GPS, and temperature sensor that measure the worker's movements, location, and ambient temperature in real time.

[1369] 2. Data preprocessing and transmission:

[1370] The chip preprocesses the data obtained from the sensor, extracts features, and sends them to the server.

[1371] 3. Data Analysis:

[1372] The server analyzes the received data using an AI model and determines that "the worker is not wearing the safety harness correctly."

[1373] 4. Creating and sending notifications:

[1374] The server generates a notification and sends it to the administrator's terminal. The emotion engine analyzes the user's emotional state and adjusts the notification wording as needed.

[1375] 5. Administrator's response:

[1376] The administrator checks the notification on the terminal and enters the instruction, "Please readjust your safety harness." Support notifications tailored to the user's emotional state are also displayed.

[1377] 6. Communication and execution of instructions:

[1378] Instructions are delivered to a chip via a server, and the worker receives the instructions. After that, they readjust their safety harness.

[1379] 7. Gathering feedback:

[1380] The chip collects data again and sends it to the server to provide feedback.

[1381] Examples of prompt statements

[1382] Please describe the process by which users receive real-time notifications and issue instructions for a sensor-based work monitoring system used on construction sites. Also, please describe in detail how support is provided that takes the user's emotional state into consideration.

[1383] This enables accurate monitoring of work status in real time and the provision of appropriate instructions tailored to the user's emotional state.

[1384] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1385] Step 1:

[1386] Data collection using sensors:

[1387] Input: Physical information such as the subordinate's actions, location, and ambient temperature.

[1388] Processing: The IMU sensor acquires operational information, the GPS acquires location information, and the temperature sensor acquires environmental information in real time.

[1389] Output: The raw data obtained.

[1390] Specific operation: Sensors measure the subordinate's movements in milliseconds, GPS determines their location, and temperature sensors collect temperature data.

[1391] Step 2:

[1392] Data preprocessing:

[1393] Input: Raw data acquired from the sensor.

[1394] Processing: The chip performs noise reduction and data correction, and extracts important features.

[1395] Output: Preprocessed data.

[1396] Specific operation: From raw data, outliers and unwanted noise are filtered out, and features such as "patterns of work movements" and "location movement history" are extracted.

[1397] Step 3:

[1398] Sending data:

[1399] Input: Preprocessed data.

[1400] Processing: The chip wirelessly transmits pre-processed data to the server at regular intervals.

[1401] Output: Data transmitted from the chip.

[1402] Specific operation: The chip uses a wireless communication protocol to send data to the server.

[1403] Step 4:

[1404] Data analysis:

[1405] Input: Pre-processed data transmitted from the chip.

[1406] Processing: The server uses an AI model to analyze the data and make a judgment about a specific situation.

[1407] Output: Analysis results (e.g., "Safety harness not worn").

[1408] Specific operation: The server analyzes data in real time and detects things like "a worker is inactive and stopped for a long time" or "an abnormal operation pattern."

[1409] Step 5:

[1410] Notification generation:

[1411] Input: Analysis results.

[1412] Processing: The server generates a notification based on the analysis results and sends it to the user's terminal. The emotion engine analyzes the user's emotional state and adjusts the notification wording.

[1413] Output: Adjusted notification.

[1414] Specific operation: The server generates a notification that "the worker has stopped working," and the emotion engine analyzes the user's stress level and adds the message "Please remain calm."

[1415] Step 6:

[1416] Displaying notifications and entering instructions:

[1417] Input: Adjusted notification.

[1418] Processing: The device displays a notification and prompts the user to enter specific instructions.

[1419] Output: Instructions.

[1420] Specific action: The terminal displays a notification saying "Worker A has stopped moving," and the user enters "Please readjust your safety harness."

[1421] Step 7:

[1422] Sending user instructions:

[1423] Input: Instructions from the user.

[1424] Processing: The terminal sends instructions to the server, and the server redistributes them to the chip.

[1425] Output: Instructions sent to the chip.

[1426] Specific operation: The terminal sends instructions to the server, and the server forwards those instructions to the chip.

[1427] Step 8:

[1428] Execute the instructions:

[1429] Input: The instructions sent to the chip.

[1430] Processing: The chip plays or displays the instructions.

[1431] Output: The actions taken by the worker who received the instructions.

[1432] Specific action: The chip notifies the worker via voice or text message, "Please readjust your safety harness."

[1433] Step 9:

[1434] Gathering feedback:

[1435] Input: Data after the instruction has been executed.

[1436] Processing: The chip collects data again and sends it to the server to provide feedback.

[1437] Output: Feedback data sent to the server.

[1438] Specific operation: The chip collects data from the sensor again, for example, to confirm that "the safety harness is properly fitted," and then sends the data to the server.

[1439] This allows the entire system to monitor work in real time and provide appropriate instructions. Furthermore, by considering the user's emotional state, it enables effective decision-making support.

[1440] (Application Example 2)

[1441] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1442] Traditional work monitoring systems focused on real-time monitoring of work status and remote instructions, but they did not provide notifications or instructions that took into account the emotional state of managers. As a result, manager stress increased, and work efficiency sometimes decreased. Furthermore, traditional systems focused only on physical safety and did not consider psychological aspects, thus failing to improve the overall work environment.

[1443] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for collecting physical data using sensor means, means for preprocessing the collected data to extract important features, means for transmitting the preprocessed data to the server, means for the server to analyze the received data and determine a specific situation, means for generating a notification based on the analysis results and transmitting it to a terminal, means for the terminal to display the notification and prompt the user to input instructions, means for transmitting the user's instructions to the server and for the server to redistribute them to a chip, means for the chip that received the instructions to reproduce or display their contents, means for transmitting the recollected data to the server to provide feedback, means for analyzing the user's emotional state using an emotion analysis engine, and means for adjusting the notification wording based on the analyzed emotional state. This makes it possible to provide appropriate notifications and instructions according to the administrator's emotional state and to improve the work environment while also considering psychological aspects.

[1444] A "sensor" is a device used to acquire physical data from the environment or objects.

[1445] "Physical data" refers to data such as operational information, location information, and environmental information measured by IMUs, GPS, temperature sensors, etc.

[1446] "Preprocessing" is a part of data processing that removes noise from collected data and extracts important features.

[1447] A "server" is a computing device that receives, analyzes, and stores data via a network, and generates appropriate notifications and instructions.

[1448] A "chip" is a hardware component that receives data from sensors, performs preprocessing, and sends the data to a server.

[1449] A "terminal" is a device that receives notifications from a server, displays them to the user, and allows the user to input instructions.

[1450] A "user" is an administrator or person in charge who uses the system and performs monitoring and instructions.

[1451] An "emotion analysis engine" is a software module that analyzes a user's voice and facial expression data to determine their emotional state.

[1452] A "notification" is an informational message generated based on analyzed data and sent to the device.

[1453] "Feedback" refers to the evaluation or response provided by the system based on the results of executing instructions and the collected data.

[1454] A "real-time communication protocol" is a set of communication standards and procedures used to exchange data quickly and efficiently.

[1455] System-wide configuration

[1456] This system consists of sensors, chips, servers, terminals, users, and an emotion analysis engine, and is primarily intended for monitoring work in factories and generating notifications based on emotional states.

[1457] Sensor means

[1458] Sensors are devices used to acquire motion information, location information, and environmental information from the environment or objects. Specifically, IMUs, GPS, and temperature sensors are used. These sensors are attached to robots or workers to collect data in real time.

[1459] Chip functions

[1460] The chip receives data from the sensor and performs preprocessing such as noise reduction and feature extraction. The preprocessed data is then transmitted wirelessly to the server at regular intervals.

[1461] Server Functions

[1462] The server receives data transmitted from the chip and stores it in a database. The stored data is analyzed in real time. The server analyzes sensor data to determine specific situations. It also generates notifications based on the received data and sends them to the terminal. Furthermore, it uses an emotion analysis engine to analyze the user's (administrator's) emotional state and adjust the notification wording accordingly.

[1463] Device functions

[1464] The terminal receives notifications from the server and displays them to the user. The user inputs instructions based on the notification content, and these instructions are sent back to the server. The server redistributes the received instructions to a chip, which then relays the instructions to the robot or worker. After the instructions are executed, data is collected again and sent back to the server to provide feedback.

[1465] User roles

[1466] The user (administrator) uses the terminal to monitor notifications and issue instructions as needed. The emotion analysis engine analyzes the user's voice and facial expression data to determine their emotional state, and the notification wording is adjusted accordingly to support appropriate responses.

[1467] Functions of the emotion analysis engine

[1468] The emotion analysis engine uses a generative AI model that recognizes the user's emotional state by taking user voice and video data as input. Based on the analysis results, it determines stress levels, relaxation levels, etc., and reflects this in the notification message.

[1469] Specific example

[1470] For example, in a factory setting, if a supervisor wears a head-mounted display (HMD) to monitor the robot's work status, the data collected by the sensors is transmitted to a server via a chip. The server analyzes the data and detects situations where the robot requires maintenance. The detected situation is sent as a notification to the terminal, allowing the user to check the notification and issue instructions. An emotion analysis engine analyzes the user's stress level and generates notification messages such as, "Please perform maintenance work. We also recommend taking a break," depending on the user's state.

[1471] Example of a prompt

[1472] The inputs to the generative AI model when using the emotion analysis engine are as follows:

[1473] "Please analyze this user's voice and facial expressions to determine their current emotional state. If possible, please provide a specific emotional state (e.g., tense, relaxed)."

[1474] This system not only facilitates smooth work monitoring and remote instruction, but also reduces the psychological burden on users, resulting in an efficient and safe work environment.

[1475] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1476] Step 1:

[1477] Sensors are attached to robots and workers. These sensors, using IMUs, GPS, and temperature sensors, collect motion information, location information, and environmental information in real time. The input is sensor data, and the output is raw data before preprocessing.

[1478] Step 2:

[1479] The chip receives data from the sensor and performs noise reduction and feature extraction. Specifically, it uses filtering techniques to reduce noise and extract only the necessary parameters. The input is sensor data, raw data, and the output is pre-processed data.

[1480] Step 3:

[1481] The chip wirelessly transmits pre-processed data to the server. The server receives this data and stores it in a database. The input is the pre-processed data, and the output is the data stored in the database.

[1482] Step 4:

[1483] The server analyzes the received data and uses data mining techniques to determine specific situations. For example, it uses machine learning algorithms to detect situations where a robot requires maintenance. The input is data stored in a database, and the output is the analysis result.

[1484] Step 5:

[1485] The server generates a notification based on the analysis results and sends it to the terminal. The notification includes situation-specific instructions and warnings. The input is the analysis results, and the output is the generated notification.

[1486] Step 6:

[1487] The terminal receives notifications from the server and displays them to the user. The user reviews the notification content and enters the necessary instructions. The input is the notification from the server, and the output is the user's instructions.

[1488] Step 7:

[1489] User instructions are sent from the terminal to the server. The server receives the instructions and redistributes them to the chip. The input is the user's instructions, and the output is the instructions sent to the chip.

[1490] Step 8:

[1491] The chip that receives the instruction will play or display its contents. This allows the robot or worker to take a specific action. The input is the instruction from the server, and the output is the played or displayed instruction.

[1492] Step 9:

[1493] After the instruction is executed, the sensor collects data again and sends the new data to the server. The server receives this data and updates the entire process as feedback. The input is the newly collected sensor data, the raw data, and the output is the updated database.

[1494] Step 10:

[1495] The emotion analysis engine analyzes the user's voice and video data to determine their emotional state. The server adjusts the notification message based on the analysis results. The input is the user's voice and video data, and the output is the adjusted notification message.

[1496] Through each of the above steps, real-time work monitoring and remote instructions are provided, along with appropriate notifications tailored to the user's emotional state. This results in an efficient and safe work environment.

[1497] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1498] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1499] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1500] [Fourth Embodiment]

[1501] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1502] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1503] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1504] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1505] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1506] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1507] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1508] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1509] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1510] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1511] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1512] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1513] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1514] This invention relates to a real-time work monitoring and remote instruction system using sensor means. Embodiments of this system are described in detail below.

[1515] System-wide configuration

[1516] This system aims to monitor the work status of subordinates in real time and provide efficient and appropriate instructions. Its main components are sensors, chips, servers, terminals, and users.

[1517] System Overview

[1518] 1. Sensor and chip functions

[1519] The sensors monitor the subordinates' work status in real time. Specifically, they combine IMUs (Inertial Measurement Units), GPS, temperature sensors, and other sensors to acquire operational information, location information, and environmental information.

[1520] The chip receives data from the sensor and performs data preprocessing. Preprocessing includes noise reduction and feature extraction. The preprocessed data is transmitted wirelessly to the server at regular intervals.

[1521] 2. Server Functions

[1522] The server receives data transmitted from the chip and stores it in a database. The stored data is then analyzed in real time by an AI model.

[1523] Based on the analysis results, the server determines a specific situation and generates an appropriate notification. For example, if the situation "Subordinate A is not wearing a safety harness" is detected, that fact will be included in the notification.

[1524] The generated notification is sent to the user's device.

[1525] 3. Device functions

[1526] The device receives a notification and displays it to the user. Based on this notification, the user can input specific instructions. For example, they can use text or voice to give instructions such as "Please readjust your safety harness."

[1527] The user's input is sent back to the server, which then delivers the instructions to the chip.

[1528] 4. User Roles

[1529] The user (administrator) uses a device to monitor notifications and issue instructions as needed. Instructions can be easily given via text or voice input.

[1530] By providing appropriate instructions, users can improve the work efficiency of their subordinates and maintain a safe working environment.

[1531] Specific examples of the system

[1532] Specific example: Use at construction sites

[1533] 1. Collection of sensor data

[1534] The chip is attached to the helmet of a subordinate working at a construction site. The IMU sensor tracks the subordinate's movements, the GPS tracks their location, and the temperature sensor measures the ambient temperature.

[1535] 2. Data preprocessing and transmission

[1536] The chip preprocesses the data obtained from the sensor and extracts specific features (for example, position and movement patterns during work).

[1537] 3. Data Analysis

[1538] The server analyzes the transmitted data in real time and detects if there is a possibility that the subordinate is not wearing their safety harness correctly.

[1539] 4. Creating and sending notifications

[1540] The server generates a notification and sends it to the administrator's terminal.

[1541] 5. Administrator's response

[1542] The user (administrator) checks the notification on their device and enters the instruction, "Please readjust your safety harness."

[1543] 6. Communication and execution of instructions

[1544] Instructions are transmitted to the chip via the server, and the subordinate receives the instructions.

[1545] The subordinate followed instructions and readjusted his safety harness.

[1546] 7. Gathering Feedback

[1547] The chip collects data again and sends it to the server, providing feedback on the results of the instructions' execution.

[1548] This allows managers to monitor their subordinates' on-site work in real time and issue appropriate instructions. This system significantly improves work efficiency and safety.

[1549] The following describes the processing flow.

[1550] Step 1:

[1551] tip

[1552] Sensors (such as IMUs, GPS, and temperature sensors) measure the subordinate's movements, location, and environmental data in real time. This raw data is temporarily stored inside the chip.

[1553] Step 2:

[1554] tip

[1555] The collected raw data is preprocessed. Preprocessing includes noise reduction and feature extraction. For example, features of the subordinate's posture and movements are extracted from IMU data.

[1556] Step 3:

[1557] tip

[1558] Pre-processed data is transmitted to the server wirelessly (Wi-Fi, 4G / 5G, etc.) at regular time intervals (e.g., every second). If transmission is successful, the data is deleted from internal memory. If transmission fails, a retransmission is attempted.

[1559] Step 4:

[1560] server

[1561] The system receives data transmitted from the chip and stores it in a database. Upon receipt, the system performs a data integrity check, and if invalid or missing data is detected, it sends a retransmission instruction to the chip.

[1562] Step 5:

[1563] server

[1564] The system analyzes stored data in real time. It uses machine learning models (e.g., deep learning models) to determine the status of subordinates (e.g., "operating normally," "anomaly detected," etc.) based on the data.

[1565] Step 6:

[1566] server

[1567] The system generates notifications based on the analysis results. For example, if it detects an anomaly such as "Subordinate A is not wearing a safety harness," it will generate a notification including that information.

[1568] Step 7:

[1569] server

[1570] The generated notification is sent to the user's device. The sending method can be push notification or email.

[1571] Step 8:

[1572] terminal

[1573] Receive notifications from the server and display them to the user. Users can check the notifications on their devices and view detailed information on the dashboard.

[1574] Step 9:

[1575] User

[1576] Based on the notification, enter the necessary instructions into the terminal. For example, you might enter a message such as "Instruct subordinate A to readjust their safety harness" via voice or text.

[1577] Step 10:

[1578] terminal

[1579] The user's input instructions are sent to the server. Error checking and retransmission are performed during transmission.

[1580] Step 11:

[1581] server

[1582] It receives user instructions and delivers them to the chip. It uses real-time communication protocols (e.g., MQTT or WebSockets).

[1583] Step 12:

[1584] tip

[1585] Receive instructions and notify subordinates. If the instructions are voiced, play them through the speaker; if they are text-based, display them on the screen.

[1586] Step 13:

[1587] subordinate

[1588] Confirm instructions from the user and act accordingly. For example, readjust your safety harness.

[1589] Step 14:

[1590] tip

[1591] The sensor data is collected again and sent to the server. This verifies that the instructions were executed correctly.

[1592] Step 15:

[1593] server

[1594] Analyze the newly collected data to confirm that the instructions were followed correctly. If the problem recurs, send another notification.

[1595] (Example 1)

[1596] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1597] Conventional work monitoring and instruction systems made it difficult to grasp the work status of subordinates in real time and to issue appropriate instructions quickly. Furthermore, the data obtained from sensors contained a lot of noise and irrelevant information, resulting in insufficient pre-processing for accurate situational assessment. In addition, delays in generating notifications based on analysis results and providing feedback on instructions based on those results prevented sufficient improvements in work efficiency and safety.

[1598] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1599] In this invention, the server includes means for collecting physical data using sensor means, means for preprocessing the collected data to extract important features, means for transmitting the preprocessed data to a computer, means for the computer to analyze the received data and determine a specific situation, means for generating a notification based on the analysis results and transmitting it to a terminal, means for the terminal to display the notification and prompt the user to input instructions, means for transmitting the user's instructions to the computer and for the computer to redistribute them to a chip, means for the chip that received the instructions to reproduce or display their contents, means for transmitting the recollected data to the computer to provide feedback, and means for the computer to perform real-time analysis using a domain-specific model based on the instructions input by the user. This makes it possible to accurately grasp the work status of subordinates in real time and quickly issue appropriate instructions. Furthermore, it is possible to improve work efficiency and safety.

[1600] A "sensor" is a device or system used to collect physical data.

[1601] "Preprocessing" refers to the process of removing noise from collected data and extracting important features.

[1602] "Transmission means" refers to a function for transmitting pre-processed data to other devices or systems.

[1603] A "computer" is a device or system that analyzes received data and makes judgments about specific situations.

[1604] "Analysis means" refers to methods or functions for making judgments about specific situations based on data.

[1605] A "notification" is information or a message generated based on the analysis results.

[1606] A "terminal" is a device that displays notifications to the user and allows them to input instructions.

[1607] A "user" is a person or administrator who uses a terminal to input instructions.

[1608] "Feedback" refers to information used to verify the progress of instructions based on recollected data.

[1609] A "domain-specific model" is an AI model that is specialized for a particular industry or domain.

[1610] "Real-time analysis" is an analytical method that processes data immediately and obtains results instantly.

[1611] Modes for carrying out the invention

[1612] This invention relates to a real-time work monitoring and remote instruction system using sensor means. The system aims to monitor the work status of subordinates in real time and provide efficient and appropriate instructions. The main components are sensors, chips, a server, a terminal, and the user. The following describes specific embodiments of this system.

[1613] Sensor and chip functions

[1614] The sensors monitor the work status of subordinates in real time. Specifically, they combine IMUs (Inertial Measurement Units), GPS, temperature sensors, etc., to acquire motion information, location information, and environmental information. These sensors are often attached to the worker's helmet or work clothes. For example, an IMU sensor attached to the helmet measures the subordinate's movements, a GPS determines their location, and a temperature sensor measures the ambient temperature.

[1615] The chip receives data from sensors and performs preprocessing such as noise reduction and feature extraction. For example, it removes unwanted vibration information from collected IMU sensor data and extracts specific operating patterns. The preprocessed data is transmitted to a server via wireless communication at regular intervals.

[1616] Server Functions

[1617] The server receives data transmitted from the chip and stores it in a database. The received data is temporarily stored in memory and then saved to the specified database. For example, databases such as MySQL or PostgreSQL may be used.

[1618] The server analyzes the stored data in real time. This analysis may utilize generative AI models such as TensorFlow or PyTorch. Through this analysis, it analyzes behavioral patterns and location information to detect abnormal situations, such as "Subordinate A is not wearing a safety harness."

[1619] Based on the analysis results, the server generates an appropriate notification and sends it to the user's terminal. For example, it might generate a notification stating, "Subordinate A is not wearing a safety harness," and send it to the user's terminal.

[1620] Device functions

[1621] The device receives notifications sent from the server and displays them to the user. Notifications are often displayed as pop-ups on the screen. Based on the notification, the user enters specific instructions. For example, they might enter a text instruction such as "Please readjust your safety harness" into the device. The device also has a voice input function, allowing users to enter instructions by voice.

[1622] Instructions entered by the user are sent back to the server from the terminal. The server distributes the received instructions to a chip, which then plays or displays the instructions to its subordinates. This allows the subordinates to adjust their work according to the user's instructions.

[1623] Specific example

[1624] Use at construction sites

[1625] 1. Sensor data collection: The chip is attached to the helmet of a subordinate working at the construction site, and the IMU sensor measures operational information, the GPS measures location information, and the temperature sensor measures ambient temperature.

[1626] 2. Data preprocessing and transmission: The chip preprocesses the data obtained from the sensor and extracts specific features (position and movement patterns during operation).

[1627] 3. Data Analysis: The server analyzes the transmitted data in real time to detect the possibility that a subordinate is not wearing their safety harness correctly.

[1628] 4. Creating and sending notifications: The server generates a notification and sends it to the administrator's terminal.

[1629] 5. Administrator's response: The user (administrator) checks the notification on their device and enters the instruction, "Please readjust your safety harness."

[1630] 6. Instruction transmission and execution: Instructions are delivered to the chip via the server, and the subordinate receives them. The subordinate follows the instructions and readjusts their safety harness.

[1631] 7. Feedback Collection: The chip collects data again and sends it to the server to provide feedback on the results of the instructions being executed.

[1632] Example of a prompt

[1633] "At a construction site, there is a system that preprocesses and analyzes data obtained from IMU sensors, GPS, and temperature sensors attached to helmets. Please explain the processing flow of this system in detail."

[1634] This allows us to request a detailed explanation of the system from the generated AI model.

[1635] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1636] Step 1:

[1637] The user has their subordinates wear sensors. For example, an IMU sensor is attached to a helmet, and a GPS sensor is attached to work clothes. The input is the sensors being worn, and the output is the state of being ready to collect data in real time.

[1638] Step 2:

[1639] Sensors monitor the subordinate's work status in real time. The IMU sensor collects motion data, the GPS collects location data, and the temperature sensor collects ambient temperature data. The input is the subordinate's actions and environmental conditions, and the output is the collection of raw data. Specifically, the IMU sensor measures acceleration and rotation, and the GPS obtains latitude and longitude.

[1640] Step 3:

[1641] The chip receives data from the sensor and performs preprocessing such as noise reduction and feature extraction. It removes unwanted noise from the collected raw data and extracts specific operating patterns. The input is raw data from the sensor, and the output is clean, preprocessed data. Specifically, it performs data filtering and statistical processing.

[1642] Step 4:

[1643] The chip transmits pre-processed data to the server via wireless communication at regular intervals. The input is clean data, and the output is data transmission to the server. Specifically, it transmits data packets using a wireless protocol.

[1644] Step 5:

[1645] The server receives data transmitted from the chip and stores it in the database. The input is the data from the chip, and the output is the stored data. Specifically, it performs database insertion operations.

[1646] Step 6:

[1647] The server analyzes stored data in real time and performs analysis to determine specific situations. The input is stored data, and the output is the analysis result. Specifically, it performs data analysis using generative AI models such as TensorFlow and PyTorch.

[1648] Step 7:

[1649] The server generates an appropriate notification based on the analysis results and sends it to the user's terminal. The input is the analysis results, and the output is the notification message. Specifically, a notification is generated stating, "Subordinate A is not wearing a safety harness."

[1650] Step 8:

[1651] The device receives notifications from the server and displays them to the user. The input is the notification message, and the output is the notification displayed on the user interface. Specifically, it is displayed as a pop-up notification on the screen.

[1652] Step 9:

[1653] The user enters specific instructions based on a notification displayed on the device. The input is the content of the notification, and the output is the entered instruction. For example, the user might enter text or voice instructions such as "Please readjust your safety harness" into the device.

[1654] Step 10:

[1655] The terminal sends user instructions to the server. The input is the user's instructions, and the output is the data sent to the server. Specifically, text and voice instructions are converted into data packets and sent.

[1656] Step 11:

[1657] The server receives user instructions and redistributes them to the chip. The input is the user's instruction data, and the output is the data to be sent to the chip. Specifically, the instruction content is transmitted using a protocol.

[1658] Step 12:

[1659] The chip receives instructions and plays or displays the content to its subordinates. The input is instruction data from the server, and the output is the notification content for the subordinates. Specifically, it plays the voice message, "Please readjust your safety harness."

[1660] Step 13:

[1661] The chip sends the recollected data to the server and provides feedback. For example, it recollects data to confirm that the safety harness has been reattached. The input is the recollected data, and the output is the feedback data to the server. This allows the administrator to confirm that the instructions have been followed.

[1662] (Application Example 1)

[1663] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1664] Traditional food delivery systems have made it difficult to accurately monitor the work status and efficiency of delivery personnel in real time and to issue appropriate instructions. This has resulted in delivery delays, decreased efficiency, and lower customer satisfaction. Furthermore, delays in providing feedback to delivery personnel and the inability to respond immediately have been a challenge.

[1665] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1666] In this invention, the server includes means for analyzing real-time data, including the movement patterns of delivery personnel; means for evaluating delivery efficiency based on the analysis results and generating instructions to improve efficiency; and means for proposing specific actions to delivery personnel based on user instructions. This makes it possible to accurately grasp the current status of delivery personnel in real time and provide appropriate instructions immediately.

[1667] A "sensor" is a device used to collect physical data in real time.

[1668] "IMU" is an abbreviation for Inertial Measurement Unit, a sensor that detects the movement and acceleration of an object.

[1669] "GPS" is an abbreviation for Global Positioning System, a system used to measure location on Earth.

[1670] An "accelerometer" is a sensor used to measure the acceleration of an object.

[1671] "Preprocessing means" refers to the process of removing noise from collected raw data and extracting important features.

[1672] A "server" is a computer system used to receive, store, and analyze data over a network.

[1673] An "analysis tool" is a mechanism for analyzing collected data and making judgments about a specific situation.

[1674] A "notification system" is a system for sending notifications generated based on analysis results to the user's terminal.

[1675] A "terminal" is a device used by a user to receive notifications and input instructions.

[1676] A "command system" is a system that sends user-inputted instructions to a server, which then redistributes them to the chip.

[1677] A "chip" is a device used to preprocess collected data and send it to a server.

[1678] "Movement patterns" refer to dynamic data such as the delivery person's travel route and speed.

[1679] "Feedback" refers to information about the results and effects of actions provided by sending the recollected data to the server.

[1680] This invention relates to a system that can be applied to food delivery to monitor the work status of delivery personnel in real time and issue appropriate instructions. This system is realized through the cooperation of sensor means, a server, a terminal, and a user.

[1681] System Configuration

[1682] 1. Sensor means:

[1683] Delivery personnel wear smart devices such as smart glasses or smartphones. These devices have built-in IMU sensors, GPS, and accelerometers.

[1684] 2. Data Acquisition and Preprocessing:

[1685] Sensors built into smart devices collect physical data on delivery personnel in real time. This includes location information, movement speed, and motion data.

[1686] The chip preprocesses the data collected from the sensor, performing noise reduction and feature extraction.

[1687] 3. Data transmission and analysis:

[1688] The pre-processed data is transmitted to the server via wireless communication.

[1689] The server stores the received data in a database and analyzes the data in real time using a generative AI model. For example, it analyzes the movement patterns of delivery personnel to detect delivery delays and decreased efficiency.

[1690] 4. Generating and sending notifications:

[1691] The server determines the specific situation based on the analysis results and generates and sends an appropriate notification to the terminal. For example, a notification such as "High acceleration detected. Please check traffic conditions" is generated.

[1692] 5. User actions:

[1693] The device displays a notification, and the user (administrator) reviews the notification before entering specific instructions. For example, they might enter instructions such as, "There is a rest point nearby; please take a break."

[1694] The input instructions are sent to the server, which then redistributes them to the chip.

[1695] 6. Communication and execution of instructions:

[1696] The chip receives instructions and plays or displays them. The delivery person acts according to the received instructions.

[1697] 7. Gathering feedback:

[1698] The chip collects sensor data again and sends it to the server, providing feedback on the results of the instructions. This allows the server to continuously monitor the delivery person's work status in real time.

[1699] Hardware and software usage examples

[1700] Hardware: Smart devices (smartphones, smart glasses), IMU sensors, GPS, accelerometers.

[1701] software:

[1702] Data preprocessing: Denoising and feature extraction are performed using Python.

[1703] Server: MySQL or PostgreSQL is used for database management, and TensorFlow or PyTorch is used for real-time data analysis.

[1704] Notification system: Web server frameworks such as Flask or Django are used for notifications from the server to the terminal.

[1705] Specific example

[1706] For example, when a delivery person travels towards a designated address, the GPS built into their smartphone collects location information, and the IMU sensor and accelerometer record movement information in real time. This data is preprocessed, features are extracted, and then it is sent to a server. The server analyzes the data using a generative AI model and generates notifications if the delivery is delayed or the delivery person is behaving inappropriately. When a notification appears on the device, the user (administrator) enters specific instructions, such as "Please use the recommended shortcuts," which are then transmitted to the delivery person.

[1707] Example of a prompt

[1708] "We analyze the movement patterns of delivery drivers and evaluate delivery performance in real time."

[1709] This system helps improve delivery efficiency in real time and enhance customer satisfaction.

[1710] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1711] Step 1:

[1712] The delivery person wears a smart device and begins the delivery. Sensors built into the smartphone or smart glasses (IMU sensor, GPS, accelerometer) collect physical data in real time. The input data includes location information, motion information, and movement speed, which are collected by the sensors.

[1713] Step 2:

[1714] The collected data is preprocessed by a chip within the device. Here, noise reduction is performed, and important features (such as location information and behavioral patterns) are extracted. The input data is the raw, collected data, while the output data is the preprocessed data.

[1715] Step 3:

[1716] The pre-processed data is transmitted to the server via wireless communication. The input data is pre-processed sensor data, which the server receives.

[1717] Step 4:

[1718] The server stores the received data in a database and analyzes the data using a generated AI model. Specifically, it analyzes the movement and action patterns of delivery personnel to determine delivery delays and decreased efficiency. The input data is the received data, and the output data is the analysis results.

[1719] Step 5:

[1720] The server generates a notification based on the analysis results and sends it to the terminal. It generates appropriate notifications for specific situations, such as "High acceleration detected. Please check traffic conditions." The input data is the analysis results, and the output data is the generated notification.

[1721] Step 6:

[1722] The device displays a notification, and the user reviews the notification content and enters specific instructions. For example, the user might enter instructions such as, "There is a rest point nearby; please take a break." The input data is the generated notification, and the output data is the instructions entered by the user.

[1723] Step 7:

[1724] Instructions from the user are sent to the server. The server receives these instructions and redistributes them to the chip. The input data is the user's instructions, and the output data is the redistributed instructions.

[1725] Step 8:

[1726] The chip receives instructions and plays or displays them. The delivery person acts according to the received instructions. The input data is the redistributed instructions, and the output data is the delivery person's actions.

[1727] Step 9:

[1728] Sensor data is collected again and sent to the server. This allows for feedback on the results of the instruction execution. The input data is the newly collected data, and the output data is the feedback information.

[1729] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1730] This invention combines a real-time work monitoring and remote instruction system using sensor means with an emotion engine. The following describes specific embodiments of this system.

[1731] System-wide configuration

[1732] This system aims to monitor subordinates' work status in real time, provide efficient and appropriate instructions, and improve work efficiency by recognizing the user's emotional state. Its main components are sensors, a chip, a server, a terminal, a user interface, and an emotion engine.

[1733] System Overview

[1734] 1. Sensor and chip functions

[1735] The sensors monitor the subordinates' work status in real time. Specifically, they combine IMUs (Inertial Measurement Units), GPS, temperature sensors, and other sensors to acquire operational information, location information, and environmental information.

[1736] The chip receives data from the sensor and performs data preprocessing. Preprocessing includes noise reduction and feature extraction. The preprocessed data is transmitted wirelessly to the server at regular intervals.

[1737] 2. Server Functions

[1738] The server receives data transmitted from the chip and stores it in a database. The stored data is then analyzed in real time by an AI model.

[1739] Based on the analysis results, the server determines a specific situation and generates an appropriate notification. For example, if the situation "Subordinate A is not wearing a safety harness" is detected, that fact will be included in the notification.

[1740] The generated notifications are sent to the user's device. Additionally, the emotion engine analyzes the user's emotional state and generates notification text appropriate to that state.

[1741] 3. Device functions

[1742] The device receives a notification and displays it to the user. Based on this notification, the user can input specific instructions. For example, they can use text or voice to give instructions such as "Please readjust your safety harness."

[1743] The device is equipped with an emotion engine that analyzes the user's voice and facial expression data to determine their emotional state. This emotional state is sent to a server and used to generate notification messages.

[1744] 4. User Roles

[1745] The user (administrator) uses a terminal to monitor notifications and issue instructions as needed. The user's emotional state is analyzed, and the system generates support notifications as needed to help make appropriate decisions.

[1746] Specific examples of the system

[1747] Specific example: Use at construction sites

[1748] 1. Collection of sensor data

[1749] The chip is attached to the helmet of a subordinate working at a construction site. The IMU sensor tracks the subordinate's movements, the GPS tracks their location, and the temperature sensor measures the ambient temperature.

[1750] 2. Data preprocessing and transmission

[1751] The chip preprocesses the data obtained from the sensor and extracts specific features (for example, position and movement patterns during work).

[1752] 3. Data Analysis

[1753] The server analyzes the transmitted data in real time and detects if there is a possibility that the subordinate is not wearing their safety harness correctly.

[1754] 4. Creating a notification

[1755] The server generates a notification and sends it to the administrator's terminal. Simultaneously, the emotion engine analyzes the user's emotional state and adjusts the notification wording as needed.

[1756] 5. Administrator's response

[1757] The user (administrator) checks the notification on their device and enters the instruction, "Please readjust your safety harness." Additional advice and support notifications are displayed depending on the user's emotional state, such as if they are feeling anxious.

[1758] 6. Communication and execution of instructions

[1759] Instructions are transmitted to the chip via the server, and the subordinate receives the instructions. The subordinate follows the instructions and readjusts their safety harness.

[1760] 7. Gathering Feedback

[1761] The chip collects data again and sends it to the server, providing feedback on the results of the instructions' execution.

[1762] This allows managers to monitor their subordinates' on-site work in real time and issue appropriate instructions. Furthermore, by considering the user's emotional state when providing support, work efficiency and safety can be further improved.

[1763] The following describes the processing flow.

[1764] Step 1:

[1765] tip

[1766] Sensors (such as IMUs, GPS, and temperature sensors) measure the subordinate's movements, location, and environmental data in real time. This raw data is temporarily stored inside the chip.

[1767] Step 2:

[1768] tip

[1769] The collected raw data is preprocessed. Preprocessing includes noise reduction and feature extraction. For example, features of the subordinate's posture and movements are extracted from IMU data.

[1770] Step 3:

[1771] tip

[1772] Pre-processed data is transmitted to the server wirelessly (Wi-Fi, 4G / 5G, etc.) at regular time intervals (e.g., every second). If transmission is successful, the data is deleted from internal memory. If transmission fails, a retransmission is attempted.

[1773] Step 4:

[1774] server

[1775] The system receives data transmitted from the chip and stores it in a database. Upon reception, the system performs a data integrity check. If invalid or missing data is detected, it sends a retransmission instruction to the chip.

[1776] Step 5:

[1777] server

[1778] The system analyzes stored data in real time. It uses machine learning models (e.g., deep learning models) to determine the status of subordinates (e.g., "operating normally," "anomaly detected," etc.) based on the data.

[1779] Step 6:

[1780] server

[1781] The system generates notifications based on the analysis results. For example, if it detects an anomaly such as "Subordinate A is not wearing a safety harness," it will generate a notification including that information.

[1782] Step 7:

[1783] server

[1784] The generated notification is sent to the user's device. The sending method can be push notification or email.

[1785] Step 8:

[1786] terminal

[1787] Receive notifications from the server and display them to the user. Users can check the notifications on their devices and view detailed information on the dashboard.

[1788] Step 9:

[1789] Emotion engine (built into the device)

[1790] The system analyzes the user's voice and facial expression data from the camera to determine the user's emotional state (e.g., "tense," "relaxed," etc.). This emotional state is then sent to the server.

[1791] Step 10:

[1792] server

[1793] The system takes the user's emotional state into account and adjusts the notification wording accordingly. For example, if the user is feeling anxious, the message "Please readjust your safety harness" might be changed to "Please take your time, check and fasten your safety harness one more time."

[1794] Step 11:

[1795] terminal

[1796] The system displays notifications to the user based on their emotional state. The user then enters appropriate instructions based on these notifications.

[1797] Step 12:

[1798] User

[1799] Based on the notification, enter the necessary instructions into the terminal. For example, you might enter a message such as "Instruct subordinate A to readjust their safety harness" via voice or text.

[1800] Step 13:

[1801] terminal

[1802] The user's input instructions are sent to the server. Error checking and retransmission are performed during transmission.

[1803] Step 14:

[1804] server

[1805] It receives user instructions and delivers them to the chip. It uses real-time communication protocols (e.g., MQTT or WebSockets).

[1806] Step 15:

[1807] tip

[1808] Receive instructions and notify subordinates. If the instructions are voiced, play them through the speaker; if they are text-based, display them on the screen.

[1809] Step 16:

[1810] subordinate

[1811] Confirm instructions from the user and act accordingly. For example, readjust your safety harness.

[1812] Step 17:

[1813] tip

[1814] The sensor data is collected again and sent to the server. This verifies that the instructions were executed correctly.

[1815] Step 18:

[1816] server

[1817] Analyze the newly collected data to confirm that the instructions were followed correctly. If the problem recurs, send another notification.

[1818] (Example 2)

[1819] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1820] Conventional work monitoring systems have struggled to accurately monitor work status in real time and provide appropriate instructions based on that information. Furthermore, they fail to provide notifications and instructions that take into account the user's emotional state, resulting in a lack of improvement in work efficiency and safety.

[1821] In Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for collecting physical data using sensor means, means for preprocessing the collected data to extract important features, means for transmitting the preprocessed data to the server, means for the server to analyze the received data and determine a specific situation, means for generating a notification based on the analysis results and transmitting it to a terminal, means for the terminal to display the notification and prompt the user to input instructions, means for analyzing the user's emotional state and adjusting the notification wording, means for transmitting the user's instructions to the server and for the server to redistribute them to a chip, means for the chip that received the instructions to reproduce or display the content, and means for transmitting the recollected data to the server to provide feedback. This enables accurate monitoring of the work status in real time and the provision of appropriate instructions according to the user's emotional state.

[1822] "Sensing means" refers to a device used to collect physical data.

[1823] "Data preprocessing" refers to the process of converting data collected by sensor devices into a format suitable for analysis.

[1824] A "chip" is an electronic circuit that receives data from a sensor, performs preprocessing, and transmits the data to a server.

[1825] A "server" is a computing device that receives data transmitted from a chip, analyzes it, generates notifications, and sends them to a terminal.

[1826] A "notification" is a message that the server generates based on the analysis results and sends to the terminal.

[1827] A "terminal" is a device that displays notifications sent from the server to the user and sends instructions from the user to the server.

[1828] A "user" refers to a person who operates a device, checks notifications, and enters instructions as needed.

[1829] "Emotional analysis" is a process that analyzes a user's voice and facial expression data to determine the user's emotional state.

[1830] "Feedback" refers to the process of sending the collected data back to the server and reporting the results of the instructions' execution.

[1831] A "real-time communication protocol" is a communication method used to instantly exchange data between a server and a chip.

[1832] Modes for carrying out the invention

[1833] This invention relates to a system that monitors work status in real time and provides appropriate instructions that take into account the user's emotional state. Specific embodiments of this system are described below.

[1834] System-wide configuration

[1835] The system consists of sensors, chips, servers, terminals, users, and an emotion engine. These elements work together to collect, process, and analyze real-time data, generate notifications, communicate instructions, and provide feedback.

[1836] Sensor and chip functions

[1837] 1. Sensor:

[1838] It is used to monitor the work status of subordinates in real time. Specifically, it includes an IMU (Inertial Measurement Unit), GPS, temperature sensors, etc., to acquire operational information, location information, and environmental information.

[1839] For example, an IMU sensor measures the worker's movements, GPS determines their location, and a temperature sensor obtains the ambient temperature of the site.

[1840] 2. Tips:

[1841] The system receives data from sensors and performs preprocessing such as noise reduction and feature extraction. The preprocessed data is then sent to the server at regular intervals.

[1842] The chip preprocesses the data, extracting features such as movement patterns and changes in position, and then sends them to the server.

[1843] Server Functions

[1844] 1. Server:

[1845] The system receives data transmitted from the chip and stores it in a database. The stored data is then analyzed in real time using an AI model.

[1846] Using an AI model, specific situations are determined from the analysis results. For example, it can detect the situation where "subordinate A is not wearing a safety harness."

[1847] The server generates a notification based on the analysis results and sends it to the user's device. Furthermore, it uses an emotion engine to analyze the user's emotional state and adjust the notification wording accordingly.

[1848] For example, the server might generate a notification stating, "Subordinate A is not wearing a safety harness," and if the user is confused, it might add a message saying, "Please remain calm."

[1849] Device functions

[1850] 1. Terminal:

[1851] The server receives notifications and displays them to the user. The user checks the notifications on their device and enters specific instructions.

[1852] The system is equipped with an emotion engine that analyzes the user's voice and facial expression data to determine their emotional state. This analysis result is sent to a server and used to adjust notification messages.

[1853] For example, a user might input the instruction "Please readjust your safety harness" into their terminal, and this instruction is transmitted to their subordinate via the server.

[1854] User roles

[1855] 1. User (Administrator):

[1856] The device monitors notifications and provides specific instructions as needed. The user's emotional state is analyzed, and supportive notifications tailored to that emotional state are provided to help make appropriate decisions.

[1857] For example, an administrator might issue an instruction to "reattach your safety harness," and then a support notification might appear stating, "Let's stay calm."

[1858] Specific example

[1859] Use at construction sites

[1860] 1. Sensor data collection:

[1861] The helmet is equipped with an IMU, GPS, and temperature sensor that measure the worker's movements, location, and ambient temperature in real time.

[1862] 2. Data preprocessing and transmission:

[1863] The chip preprocesses the data obtained from the sensor, extracts features, and sends them to the server.

[1864] 3. Data Analysis:

[1865] The server analyzes the received data using an AI model and determines that "the worker is not wearing the safety harness correctly."

[1866] 4. Creating and sending notifications:

[1867] The server generates a notification and sends it to the administrator's terminal. The emotion engine analyzes the user's emotional state and adjusts the notification wording as needed.

[1868] 5. Administrator's response:

[1869] The administrator checks the notification on the terminal and enters the instruction, "Please readjust your safety harness." Support notifications tailored to the user's emotional state are also displayed.

[1870] 6. Communication and execution of instructions:

[1871] Instructions are delivered to a chip via a server, and the worker receives the instructions. After that, they readjust their safety harness.

[1872] 7. Gathering feedback:

[1873] The chip collects data again and sends it to the server to provide feedback.

[1874] Examples of prompt statements

[1875] Please describe the process by which users receive real-time notifications and issue instructions for a sensor-based work monitoring system used on construction sites. Also, please describe in detail how support is provided that takes the user's emotional state into consideration.

[1876] This enables accurate monitoring of work status in real time and the provision of appropriate instructions tailored to the user's emotional state.

[1877] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1878] Step 1:

[1879] Data collection using sensors:

[1880] Input: Physical information such as the subordinate's actions, location, and ambient temperature.

[1881] Processing: The IMU sensor acquires operational information, the GPS acquires location information, and the temperature sensor acquires environmental information in real time.

[1882] Output: The raw data obtained.

[1883] Specific operation: Sensors measure the subordinate's movements in milliseconds, GPS determines their location, and temperature sensors collect temperature data.

[1884] Step 2:

[1885] Data preprocessing:

[1886] Input: Raw data acquired from the sensor.

[1887] Processing: The chip performs noise reduction and data correction, and extracts important features.

[1888] Output: Preprocessed data.

[1889] Specific operation: From raw data, outliers and unwanted noise are filtered out, and features such as "patterns of work movements" and "location movement history" are extracted.

[1890] Step 3:

[1891] Sending data:

[1892] Input: Preprocessed data.

[1893] Processing: The chip wirelessly transmits pre-processed data to the server at regular intervals.

[1894] Output: Data transmitted from the chip.

[1895] Specific operation: The chip uses a wireless communication protocol to send data to the server.

[1896] Step 4:

[1897] Data analysis:

[1898] Input: Pre-processed data transmitted from the chip.

[1899] Processing: The server uses an AI model to analyze the data and make a judgment about a specific situation.

[1900] Output: Analysis results (e.g., "Safety harness not worn").

[1901] Specific operation: The server analyzes data in real time and detects things like "a worker is inactive and stopped for a long time" or "an abnormal operation pattern."

[1902] Step 5:

[1903] Notification generation:

[1904] Input: Analysis results.

[1905] Processing: The server generates a notification based on the analysis results and sends it to the user's terminal. The emotion engine analyzes the user's emotional state and adjusts the notification wording.

[1906] Output: Adjusted notification.

[1907] Specific operation: The server generates a notification that "the worker has stopped working," and the emotion engine analyzes the user's stress level and adds the message "Please remain calm."

[1908] Step 6:

[1909] Displaying notifications and entering instructions:

[1910] Input: Adjusted notification.

[1911] Processing: The device displays a notification and prompts the user to enter specific instructions.

[1912] Output: Instructions.

[1913] Specific action: The terminal displays a notification saying "Worker A has stopped moving," and the user enters "Please readjust your safety harness."

[1914] Step 7:

[1915] Sending user instructions:

[1916] Input: Instructions from the user.

[1917] Processing: The terminal sends instructions to the server, and the server redistributes them to the chip.

[1918] Output: Instructions sent to the chip.

[1919] Specific operation: The terminal sends instructions to the server, and the server forwards those instructions to the chip.

[1920] Step 8:

[1921] Execute the instructions:

[1922] Input: The instructions sent to the chip.

[1923] Processing: The chip plays or displays the instructions.

[1924] Output: The actions taken by the worker who received the instructions.

[1925] Specific action: The chip notifies the worker via voice or text message, "Please readjust your safety harness."

[1926] Step 9:

[1927] Gathering feedback:

[1928] Input: Data after the instruction has been executed.

[1929] Processing: The chip collects data again and sends it to the server to provide feedback.

[1930] Output: Feedback data sent to the server.

[1931] Specific operation: The chip collects data from the sensor again, for example, to confirm that "the safety harness is properly fitted," and then sends the data to the server.

[1932] This allows the entire system to monitor work in real time and provide appropriate instructions. Furthermore, by considering the user's emotional state, it enables effective decision-making support.

[1933] (Application Example 2)

[1934] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1935] Traditional work monitoring systems focused on real-time monitoring of work status and remote instructions, but they did not provide notifications or instructions that took into account the emotional state of managers. As a result, manager stress increased, and work efficiency sometimes decreased. Furthermore, traditional systems focused only on physical safety and did not consider psychological aspects, thus failing to improve the overall work environment.

[1936] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for collecting physical data using sensor means, means for preprocessing the collected data to extract important features, means for transmitting the preprocessed data to the server, means for the server to analyze the received data and determine a specific situation, means for generating a notification based on the analysis results and transmitting it to a terminal, means for the terminal to display the notification and prompt the user to input instructions, means for transmitting the user's instructions to the server and for the server to redistribute them to a chip, means for the chip that received the instructions to reproduce or display their contents, means for transmitting the recollected data to the server to provide feedback, means for analyzing the user's emotional state using an emotion analysis engine, and means for adjusting the notification wording based on the analyzed emotional state. This makes it possible to provide appropriate notifications and instructions according to the administrator's emotional state and to improve the work environment while also considering psychological aspects.

[1937] A "sensor" is a device used to acquire physical data from the environment or objects.

[1938] "Physical data" refers to data such as operational information, location information, and environmental information measured by IMUs, GPS, temperature sensors, etc.

[1939] "Preprocessing" is a part of data processing that removes noise from collected data and extracts important features.

[1940] A "server" is a computing device that receives, analyzes, and stores data via a network, and generates appropriate notifications and instructions.

[1941] A "chip" is a hardware component that receives data from sensors, performs preprocessing, and sends the data to a server.

[1942] A "terminal" is a device that receives notifications from a server, displays them to the user, and allows the user to input instructions.

[1943] A "user" is an administrator or person in charge who uses the system and performs monitoring and instructions.

[1944] An "emotion analysis engine" is a software module that analyzes a user's voice and facial expression data to determine their emotional state.

[1945] A "notification" is an informational message generated based on analyzed data and sent to the device.

[1946] "Feedback" refers to the evaluation or response provided by the system based on the results of executing instructions and the collected data.

[1947] A "real-time communication protocol" is a set of communication standards and procedures used to exchange data quickly and efficiently.

[1948] System-wide configuration

[1949] This system consists of sensors, chips, servers, terminals, users, and an emotion analysis engine, and is primarily intended for monitoring work in factories and generating notifications based on emotional states.

[1950] Sensor means

[1951] Sensors are devices used to acquire motion information, location information, and environmental information from the environment or objects. Specifically, IMUs, GPS, and temperature sensors are used. These sensors are attached to robots or workers to collect data in real time.

[1952] Chip functions

[1953] The chip receives data from the sensor and performs preprocessing such as noise reduction and feature extraction. The preprocessed data is then transmitted wirelessly to the server at regular intervals.

[1954] Server Functions

[1955] The server receives data transmitted from the chip and stores it in a database. The stored data is analyzed in real time. The server analyzes sensor data to determine specific situations. It also generates notifications based on the received data and sends them to the terminal. Furthermore, it uses an emotion analysis engine to analyze the user's (administrator's) emotional state and adjust the notification wording accordingly.

[1956] Device functions

[1957] The terminal receives notifications from the server and displays them to the user. The user inputs instructions based on the notification content, and these instructions are sent back to the server. The server redistributes the received instructions to a chip, which then relays the instructions to the robot or worker. After the instructions are executed, data is collected again and sent back to the server to provide feedback.

[1958] User roles

[1959] The user (administrator) uses the terminal to monitor notifications and issue instructions as needed. The emotion analysis engine analyzes the user's voice and facial expression data to determine their emotional state, and the notification wording is adjusted accordingly to support appropriate responses.

[1960] Functions of the emotion analysis engine

[1961] The emotion analysis engine uses a generative AI model that recognizes the user's emotional state by taking user voice and video data as input. Based on the analysis results, it determines stress levels, relaxation levels, etc., and reflects this in the notification message.

[1962] Specific example

[1963] For example, in a factory setting, if a supervisor wears a head-mounted display (HMD) to monitor the robot's work status, the data collected by the sensors is transmitted to a server via a chip. The server analyzes the data and detects situations where the robot requires maintenance. The detected situation is sent as a notification to the terminal, allowing the user to check the notification and issue instructions. An emotion analysis engine analyzes the user's stress level and generates notification messages such as, "Please perform maintenance work. We also recommend taking a break," depending on the user's state.

[1964] Example of a prompt

[1965] The inputs to the generative AI model when using the emotion analysis engine are as follows:

[1966] "Please analyze this user's voice and facial expressions to determine their current emotional state. If possible, please provide a specific emotional state (e.g., tense, relaxed)."

[1967] This system not only facilitates smooth work monitoring and remote instruction, but also reduces the psychological burden on users, resulting in an efficient and safe work environment.

[1968] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1969] Step 1:

[1970] Sensors are attached to robots and workers. These sensors, using IMUs, GPS, and temperature sensors, collect motion information, location information, and environmental information in real time. The input is sensor data, and the output is raw data before preprocessing.

[1971] Step 2:

[1972] The chip receives data from the sensor and performs noise reduction and feature extraction. Specifically, it uses filtering techniques to reduce noise and extract only the necessary parameters. The input is sensor data, raw data, and the output is pre-processed data.

[1973] Step 3:

[1974] The chip wirelessly transmits pre-processed data to the server. The server receives this data and stores it in a database. The input is the pre-processed data, and the output is the data stored in the database.

[1975] Step 4:

[1976] The server analyzes the received data and uses data mining techniques to determine specific situations. For example, it uses machine learning algorithms to detect situations where a robot requires maintenance. The input is data stored in a database, and the output is the analysis result.

[1977] Step 5:

[1978] The server generates a notification based on the analysis results and sends it to the terminal. The notification includes situation-specific instructions and warnings. The input is the analysis results, and the output is the generated notification.

[1979] Step 6:

[1980] The terminal receives notifications from the server and displays them to the user. The user reviews the notification content and enters the necessary instructions. The input is the notification from the server, and the output is the user's instructions.

[1981] Step 7:

[1982] User instructions are sent from the terminal to the server. The server receives the instructions and redistributes them to the chip. The input is the user's instructions, and the output is the instructions sent to the chip.

[1983] Step 8:

[1984] The chip that receives the instruction will play or display its contents. This allows the robot or worker to take a specific action. The input is the instruction from the server, and the output is the played or displayed instruction.

[1985] Step 9:

[1986] After the instruction is executed, the sensor collects data again and sends the new data to the server. The server receives this data and updates the entire process as feedback. The input is the newly collected sensor data, the raw data, and the output is the updated database.

[1987] Step 10:

[1988] The emotion analysis engine analyzes the user's voice and video data to determine their emotional state. The server adjusts the notification message based on the analysis results. The input is the user's voice and video data, and the output is the adjusted notification message.

[1989] Through each of the above steps, real-time work monitoring and remote instructions are provided, along with appropriate notifications tailored to the user's emotional state. This results in an efficient and safe work environment.

[1990] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1991] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1992] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[1993] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1994] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[1995] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[1996] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[1997] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[1998] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[1999] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[2000] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[2001] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[2002] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[2003] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2004] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[2005] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[2006] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[2007] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[2008] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[2009] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[2010] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[2011] The following is further disclosed regarding the embodiments described above.

[2012] (Claim 1)

[2013] A means for collecting physical data using sensor means,

[2014] A means of preprocessing the collected data to extract important features,

[2015] A means for sending pre-processed data to a server,

[2016] A means of analyzing the data received by the server and determining a specific situation,

[2017] A means for generating a notification based on the analysis results and sending it to the terminal,

[2018] A means by which the device displays a notification and prompts the user to input instructions,

[2019] A means of sending user instructions to a server, which then redistributes them to the chip,

[2020] The instructed chip has means to play or display its contents,

[2021] A means of sending the recollected data to a server to provide feedback,

[2022] A system that includes this.

[2023] (Claim 2)

[2024] The system according to claim 1, wherein the sensor means includes an IMU, a GPS, and a temperature sensor.

[2025] (Claim 3)

[2026] The system according to claim 1, wherein the server communicates with the chip using a real-time communication protocol.

[2027] "Example 1"

[2028] (Claim 1)

[2029] A means for collecting physical data using sensor means,

[2030] A means of preprocessing the collected data to extract important features,

[2031] Means for transmitting pre-processed data to a computer,

[2032] A means of analyzing data received by a computer and determining a specific situation,

[2033] A means for generating a notification based on the analysis results and sending it to the terminal,

[2034] A means by which the device displays a notification and prompts the user to input instructions,

[2035] A means for sending user instructions to a computer, and for the computer to redistribute them to a chip,

[2036] The instructed chip has means to play or display its contents,

[2037] A means of sending the recollected data to a computer to provide feedback,

[2038] A means by which a computer performs real-time analysis using a domain-specific model based on instructions entered by the user,

[2039] A system that includes this.

[2040] (Claim 2)

[2041] The system according to claim 1, wherein the sensor means includes an inertial measurement unit, a positioning system, and an environmental measurement sensor.

[2042] (Claim 3)

[2043] The system according to claim 1, wherein the computer communicates with the chip using a real-time communication protocol.

[2044] "Application Example 1"

[2045] (Claim 1)

[2046] A means for collecting physical data using sensor means,

[2047] A means of preprocessing the collected data to extract important features,

[2048] A means for sending pre-processed data to a server,

[2049] A means of analyzing the data received by the server and determining a specific situation,

[2050] A means for generating a notification based on the analysis results and sending it to the terminal,

[2051] A means by which the device displays a notification and prompts the user to input instructions,

[2052] A means of sending user instructions to a server, which then redistributes them to the chip,

[2053] The instructed chip has means to play or display its contents,

[2054] A means of analyzing real-time data including the movement patterns of delivery personnel,

[2055] A means for evaluating delivery efficiency based on analysis results and generating instructions to improve efficiency,

[2056] A means of proposing specific actions to delivery personnel based on user instructions,

[2057] A means of sending the recollected data to a server to provide feedback,

[2058] A system that includes this.

[2059] (Claim 2)

[2060] The system according to claim 1, wherein the sensor means includes an IMU, a GPS, and an accelerometer.

[2061] (Claim 3)

[2062] The system according to claim 1, wherein the server communicates with the chip using a real-time communication protocol.

[2063] "Example 2 of combining an emotion engine"

[2064] (Claim 1)

[2065] A means for collecting physical data using sensor means,

[2066] A means of preprocessing the collected data to extract important features,

[2067] A means for sending pre-processed data to a server,

[2068] A means of analyzing the data received by the server and determining a specific situation,

[2069] A means for generating a notification based on the analysis results and sending it to the terminal,

[2070] A means by which the device displays a notification and prompts the user to input instructions,

[2071] A means of analyzing the user's emotional state and adjusting the notification wording,

[2072] A means of sending user instructions to a server, which then redistributes them to the chip,

[2073] The instructed chip has means to play or display its contents,

[2074] A means of sending the recollected data to a server to provide feedback,

[2075] A system that includes this.

[2076] (Claim 2)

[2077] The system according to claim 1, wherein the sensor means includes an IMU, a GPS, and a temperature sensor.

[2078] (Claim 3)

[2079] The system according to claim 1, wherein the server communicates with the chip using a real-time communication protocol.

[2080] "Application example 2 of combining emotional engines"

[2081] (Claim 1)

[2082] A means for collecting physical data using sensor means,

[2083] A means of preprocessing the collected data to extract important features,

[2084] A means for sending pre-processed data to a server,

[2085] A means of analyzing the data received by the server and determining a specific situation,

[2086] A means for generating a notification based on the analysis results and sending it to the terminal,

[2087] A means by which the device displays a notification and prompts the user to input instructions,

[2088] A means of sending user instructions to a server, which then redistributes them to the chip,

[2089] The instructed chip has means to play or display its contents,

[2090] A means of sending the recollected data to a server to provide feedback,

[2091] A means of analyzing a user's emotional state using an emotion analysis engine,

[2092] A means...

Claims

1. A means for collecting physical data using sensor means, A means of preprocessing the collected data to extract important features, A means for sending pre-processed data to a server, A means of analyzing the data received by the server and determining a specific situation, A means for generating a notification based on the analysis results and sending it to the terminal, A means by which the device displays a notification and prompts the user to input instructions, A means of sending user instructions to a server, which then redistributes them to the chip, The instructed chip has means to play or display its contents, A means of sending the recollected data to a server to provide feedback, A system that includes this.

2. The system according to claim 1, wherein the sensor means includes an IMU, a GPS, and a temperature sensor.

3. The system according to claim 1, wherein the server communicates with the chip using a real-time communication protocol.

Images