system

Abstract

We provide the system. [Solution] A means of acquiring on-site data using multiple input devices placed on-site, Means for transmitting acquired data to a central device, In the central system, there are means for analyzing acquired data and evaluating the risk level at the site, A means for generating alerts based on evaluation results and notifying user devices from the central device, A system that includes means for transmitting instructions provided by user equipment to the field.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, and includes steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of the chatbot's character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance that responds to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] At a construction site, there are problems such as over-concentration of work due to a shortage of on-site agents and non-uniformity in on-site safety management. This problem may cause a delay in the process depending on the work confirmation items because the on-site agent may often be absent from the site when in charge of multiple sites, and as a result, it may lead to the occurrence of accidents due to unreasonable schedules and insufficient confirmations. In addition, since the safety and quality of work at each site depend on the proficiency of the workers, there is a problem that the safety of the site is not consistent.

Means for Solving the Problems

[0005] This invention provides a system for monitoring site conditions in real time and continuously by acquiring site data using multiple input devices placed on-site and transmitting it to a central device. The central device analyzes the collected data and evaluates risk levels to predict hazards at the site. Based on the evaluation results, alerts are generated and notified from the central device to user devices, allowing users to provide instructions to the site in real time as needed. This improves site safety even when a site manager is absent, prevents process delays, and increases work efficiency.

[0006] An "input device" is a device installed to understand the situation on-site and to acquire data and information.

[0007] "Site data" refers to information about the work and environmental conditions carried out at a construction site, and includes various types of data such as video, audio, temperature, and vibration.

[0008] The "central unit" is the core of the system that receives, analyzes, and manages data transmitted from the field.

[0009] "Data analysis" is the process of processing received data to identify trends and anomalies.

[0010] "Risk level" is an indicator that shows the degree of potential danger at a site, evaluated based on analyzed data.

[0011] An "alert" is a signal indicating a potential hazard at a site, and is a notification generated to warn those involved.

[0012] A "user device" is a terminal used by administrators and users to receive notifications from the central device and send instructions.

[0013] "Instructions" refer to specific actions or responses that are commanded from the user's equipment to the on-site workers. [Brief explanation of the drawing]

[0014] [Figure 1] It is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

MODE FOR CARRYING OUT THE INVENTION

[0015] An example of an embodiment of the system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0018] In the following embodiments, a numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0019] In the following embodiments, a numbered storage is one or more non-volatile storage devices that store various programs, various parameters, and the like. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0020] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0022] [First Embodiment]

[0023] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0024] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0027] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0030] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0034] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0035] This invention aims to solve the problem of labor shortages at construction sites and improve safety. The system for this purpose consists of an advanced data collection and analysis system using multiple input devices and a central device installed at the site, as well as user devices.

[0036] Multiple input devices are installed at the site to continuously collect on-site data. These input devices include video cameras, temperature sensors, vibration sensors, and sound detection devices, enabling comprehensive monitoring of all situations. The collected data is initially filtered by terminals and transmitted to a central system according to its importance.

[0037] The server functions as a central device, analyzing transmitted data in real time. The server uses advanced algorithms to evaluate the data and quantify risk levels. This process incorporates machine learning techniques that reference historical data, enabling highly accurate risk assessments through predictions based on accumulated knowledge.

[0038] If a high risk is detected, the server immediately sends an alert to the user's device. This user device is used by on-site managers and supervisors, enabling real-time situation monitoring and immediate response.

[0039] Users can check notifications from the server, input instructions as needed, and transmit them to field workers via their terminals. Instructions are provided specifically via voice and display, and workers are required to act immediately in accordance with them.

[0040] For example, if the temperature rises abnormally at the site, the terminal immediately reports this anomaly to the central system, and the server assesses the level of risk based on past data. If a high risk is assessed, the server sends a system alert to the user's device, and the user instructs the site to take swift action. Through this entire process, it is possible to ensure safety at the site and improve work efficiency.

[0041] The following describes the processing flow.

[0042] Step 1:

[0043] The terminal activates various sensors and cameras installed on-site, acquiring video and sensor data in real time. This includes a variety of data such as temperature, vibration, and sound.

[0044] Step 2:

[0045] The terminal performs an initial filter on the acquired data to detect anomalies and unusual changes. This filtering prevents the transmission of unnecessary data, and only important data is extracted.

[0046] Step 3:

[0047] The terminal sends filtered data to a central server. Here, communication stability is verified, and data integrity is guaranteed.

[0048] Step 4:

[0049] The server processes the received data using an analysis algorithm to evaluate the situation on site. By using historical data and pattern recognition technology in the analysis, it performs a highly accurate risk assessment.

[0050] Step 5:

[0051] The server determines the risk level based on the data analysis results and generates alert messages as needed. In this process, it compares the situation to similar past situations if necessary and considers possible countermeasures.

[0052] Step 6:

[0053] The server sends an alert to the user's device. This alert contains detailed information about the risks at the site, allowing the user to quickly consider countermeasures.

[0054] Step 7:

[0055] After receiving an alert from the server, the user decides on a course of action and sends that action as an instruction command from their device to a terminal at the site. The instructions are specific and require immediate action at the site.

[0056] Step 8:

[0057] The terminal transmits instructions received from the user to on-site workers via voice or display. This allows workers to respond quickly based on the instructions.

[0058] Step 9:

[0059] The server confirms that the entire process has completed successfully and continues recording and analyzing data. This data will be used for future improvements and updates to predictive models.

[0060] (Example 1)

[0061] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0062] In construction sites, manufacturing plants, and other similar environments, there is a need to respond quickly and accurately to changes in the work environment and unforeseen circumstances, thereby improving safety. However, conventional systems have struggled to efficiently collect and analyze individual data, conduct immediate risk assessments based on the results, and provide appropriate instructions to workers. Furthermore, low accuracy in risk prediction and delays in effective responses have been contributing to compromised safety.

[0063] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0064] In this invention, the server includes means for acquiring environmental data from multiple sensor devices placed on-site, means for analyzing the data transferred to a central device using advanced algorithms to perform a safety risk assessment, and means for performing risk prediction by utilizing machine learning technology and referring to past data. This enables rapid data collection and evaluation on-site, highly accurate risk prediction based on past data, and real-time instruction provision to workers.

[0065] A "sensor device" is a device that is placed to detect changes in the environment and collect data.

[0066] A "relay device" is a device that performs initial filtering of data acquired from sensor devices before transmitting it to the central device.

[0067] A "central system" is a computer system that functions to analyze received environmental data and perform safety risk assessments.

[0068] An "advanced algorithm" is a computational method used to analyze complex and diverse data and perform accurate risk assessments.

[0069] "Machine learning technology" is a technique that learns patterns based on past data and uses that to predict future risks.

[0070] "Risk assessment" is the process of quantifying and determining the degree of safety risks based on local situation data.

[0071] A "management device" is a terminal used by on-site managers to receive alerts notified from the central device.

[0072] A "worker" is a person responsible for carrying out actual work according to instructions at the work site.

[0073] A "generative AI model" is an artificial intelligence technology that generates countermeasures based on given data and prompts.

[0074] A "prompt statement" is an input command given to a generative AI model to cause it to generate specific information.

[0075] To implement the invention, this system is realized in the following form.

[0076] Multiple sensor devices are deployed at the site to acquire environmental data such as temperature, vibration, visual information, and sound. This data is transmitted to a central device via a relay device. The relay device first evaluates the importance of the data, performs initial filtering, and sends only the necessary data to the central device.

[0077] The server functions as a central device, analyzing the received data. Here, advanced algorithms are used to analyze the data in detail and assess safety risks. Furthermore, machine learning techniques are utilized to predict future risks by referencing past patterns. This technique, where the server uses generative AI models to predict risks, is a particularly crucial process.

[0078] Based on the risk level assessment, the server sends alerts to the management device in real time. These alerts include emergency response measures tailored to the situation on site, and, in particular, instructions for immediate action when necessary.

[0079] The user uses a management device to receive notifications from the server, consider countermeasures as needed, and communicate specific instructions to the workers. For example, one possible instruction might be, "All workers, immediately move to the designated evacuation area."

[0080] As a concrete example, consider a situation where the temperature rises rapidly at a construction site. In this case, a sensor device detects the anomaly, and the information is transmitted to the central device via a relay device. The server, referring to past data, immediately sends an alert to the management device if it determines that the risk level is high. Based on this alert, the user can quickly issue instructions to the workers.

[0081] An example of a prompt using a generative AI model is a text-based instruction such as, "Assess the risk based on the current temperature data and generate recommended countermeasures." This prompt instructs the AI ​​model to perform appropriate data processing and generate countermeasures.

[0082] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0083] Step 1:

[0084] The terminal acquires environmental data from multiple sensor devices placed on-site. This includes video data from video cameras, temperature data from temperature sensors, vibration data from vibration sensors, and audio data from sound detection devices. The terminal receives this data as input and performs initial filtering based on importance. As output, filtered important data is generated and sent to the relay device.

[0085] Step 2:

[0086] Upon receiving filtered data from a terminal, the server begins data analysis using advanced algorithms. It analyzes the received data in detail and conducts a safety-related risk assessment. Using machine learning techniques, the server references past data and calculates a risk assessment score based on the current situation. This process involves identifying anomaly patterns and performing calculations to determine urgency.

[0087] Step 3:

[0088] The server generates an alert based on the risk assessment score obtained after analysis. This alert is sent to the management device in real time. The alert generated as output includes emergency response measures tailored to the specific situation on site. For example, it may include instructions such as, "The temperature is rising abnormally in Area B, so all workers must evacuate."

[0089] Step 4:

[0090] The user receives alerts from the server via a management device and reviews their contents. Based on the alert information received as input, the user quickly creates specific instructions if action is required. The user then transmits these instructions as output to field workers, prompting them to take action to ensure safety at the site. In some cases, the user may also optimize the instructions by using prompts generated by an AI model, taking into account the countermeasures suggested by the AI.

[0091] An example of a prompt message for a countermeasure proposed by the generative AI model is, "Assess the risk based on the current temperature data and generate recommended countermeasures." Through this process, rapid and accurate risk management is achieved throughout the entire system.

[0092] (Application Example 1)

[0093] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0094] In factories and work environments, labor shortages and ensuring safety are serious problems. Especially in workplaces where automated machinery predominates, human intervention is limited, making the detection of abnormalities and hazards, and rapid response, crucial. However, conventional systems have limitations in real-time capabilities and predictive accuracy, making it difficult to implement appropriate safety measures on-site. This invention aims to solve these problems.

[0095] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0096] In this invention, the server includes means for transmitting collected data to a central processing unit, and the central processing unit includes means for analyzing the collected data and evaluating the degree of risk at the work site, and means for instructing mechanical devices within the work space to take safe actions. This enables real-time risk detection and prediction, and rapid safety response.

[0097] An "information gathering device" refers to devices such as sensors and cameras installed to collect environmental data at a work site.

[0098] A "central processing unit" is a central computer or server used to analyze collected data and assess the level of risk.

[0099] "Analysis" refers to the process of extracting useful information from collected data, and specifically refers to calculations and analyses related to risk assessment.

[0100] "Evaluation" refers to quantifying and judging the degree of danger at a work site based on information obtained through analysis.

[0101] A "warning" is a notification generated based on evaluation results, and is a signal or message conveyed to workers or managers through an operating terminal.

[0102] An "operation terminal" is a device or interface used by users to receive warnings and transmit necessary instructions to the field.

[0103] An "order" is a specific instruction regarding actions and tasks at the work site, and is transmitted from the control terminal to personnel and machinery at the site.

[0104] "Mechanical equipment" refers to robots and other automation devices that perform specific tasks automatically at a work site.

[0105] "Safety actions" refer to the actions that a machine or device should take when a hazard is detected, and are the actions and processes that ensure the safety of people and the environment.

[0106] As an embodiment for carrying out the invention, a safety monitoring system for factories and work sites will be described. This system consists of an information gathering device, a central processing unit, and an operating terminal.

[0107] The information gathering device includes numerous sensors and cameras installed at the work site to collect temperature, vibration, or acoustic data in real time. The data collected by this device is transmitted to a central processing unit via a network.

[0108] The central processing unit (server) analyzes received data using machine learning algorithms such as TENSORFLOW®. Based on past data, it assesses the level of risk from the current data and makes a quantitative judgment. Based on the analysis results, it immediately generates a warning and notifies the operating terminal.

[0109] The control terminal is a device used by managers and workers within the factory, allowing them to check warnings in real time. Upon receiving a warning, the user immediately transmits the necessary commands to the machinery. The machinery then follows the received commands and implements safe actions, thereby ensuring safety on site.

[0110] For example, if a temperature sensor detects an abnormal value and the central processing unit assesses this as a high risk, a warning is sent to the administrator via the control terminal. Subsequently, the machine automatically stops operating and is instructed to move to a safe location.

[0111] Example prompt message:

[0112] The factory temperature sensor has detected an abnormal reading. Assess the level of risk, issue an immediate notification, and instruct the robot to move safely.

[0113] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0114] Step 1:

[0115] The information gathering device continuously collects data at the work site.

[0116] (Input: Sensor data such as temperature, vibration, and sound)

[0117] The information gathering device acquires various data from the field environment in real time and performs initial data processing. Afterward, important data is transmitted to the central processing unit via the network.

[0118] (Output: Filtered dataset)

[0119] Step 2:

[0120] The server analyzes the data it receives.

[0121] (Input: Filtered dataset)

[0122] The server uses TensorFlow algorithms to analyze incoming data, and during this process, it evaluates the level of risk by comparing it to past data. This process utilizes generative AI models to perform risk assessment in real time.

[0123] (Output: Risk assessment value)

[0124] Step 3:

[0125] The server generates a warning based on the risk assessment value and notifies the operating terminal.

[0126] (Input: Risk assessment value)

[0127] If a high risk is detected, the server generates a warning message and promptly sends it to the user's terminal via the network. This message includes specific instructions to enable the user to take immediate action.

[0128] (Output: Warning message)

[0129] Step 4:

[0130] The operating terminal receives a warning message and notifies the user.

[0131] (Input: Warning message)

[0132] When the operating terminal receives a warning, it immediately displays the content on the terminal screen and emits an alert sound if necessary. The user checks the notification and takes appropriate action.

[0133] (Output: Notification to the user)

[0134] Step 5:

[0135] The user instructs the machine to perform safety actions via a control terminal.

[0136] (Input: Warning content and user response instructions)

[0137] The user inputs a command from the control terminal, which is then transmitted to the machinery at the work site. Based on the received instructions, the machinery begins safe operation.

[0138] (Output: Instruction signal to the machine)

[0139] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0140] This invention provides a system equipped with emotion recognition capabilities to improve safety and efficiency at construction sites. This system consists of an input device that monitors site conditions in real time, a central device (server) that analyzes the data, and an emotion engine that recognizes the user's emotions and provides information according to their state.

[0141] First, multiple input devices installed on-site continuously acquire on-site data in real time. This includes acquiring video, audio, temperature, and vibration data from video cameras and various sensors. Terminals analyze this data and transmit the necessary information to a central device.

[0142] The server functions as a central device, analyzing the acquired data. Data analysis utilizes machine learning algorithms that refer to past accident data and quantitatively assess risk. If a high risk is detected, the server generates an alert and notifies the user device.

[0143] The user device is equipped with an emotion engine that analyzes the user's tone of voice, facial expressions, and speed of operation to recognize the user's emotional state. When it receives an alert from the server, the emotion engine assesses the user's stress level and adjusts the notification method accordingly. For example, if the system determines that the user is in a high-stress state, the notification will be adjusted to be more detailed and calming.

[0144] Users receive notifications based on their analyzed emotional state and issue instructions to the field as needed. These instructions are provided in a format easily understood by workers, taking into account feedback from the emotion engine. Information regarding the user's emotional state is fed back to the server and stored in a database. This data is used for future system improvements and instruction optimization.

[0145] For example, if abnormal vibrations are detected on-site, the terminal immediately reports the data to the server, which assesses the risk and generates an alert. The user receives this alert along with an evaluation from the emotion engine, enabling them to give calm instructions to the site. This process maximizes the safety and efficiency of the work.

[0146] The following describes the processing flow.

[0147] Step 1:

[0148] The terminal activates various sensors and cameras installed on-site, acquiring video data and sensor data (temperature, vibration, sound, etc.) in real time. This makes it possible to detect any environmental changes at the site.

[0149] Step 2:

[0150] The terminal performs initial filtering, narrowing down the acquired data to only the essentials. This process extracts unusual changes and data exceeding thresholds, ensuring that only critical information is sent to the central device.

[0151] Step 3:

[0152] The server receives data sent from the terminal and performs data analysis using machine learning algorithms. This analysis quantitatively evaluates the risk level at the site and detects signs of high risk.

[0153] Step 4:

[0154] Based on the risk assessment results, the server generates an alert message appropriate to the user's emotional state. If the risk is higher than historical data, an alert is immediately generated and notified to the user's device.

[0155] Step 5:

[0156] The user device operates an emotion engine that analyzes the user's emotional state based on their actions and voice data. It evaluates the user's stress level and current emotions, and adjusts the notification method and content accordingly.

[0157] Step 6:

[0158] The user considers feedback from the emotion engine, reviews the alerts from the server, and consciously considers instructions for the field. The instructions are specific and adjusted to a format that is easy for workers to understand.

[0159] Step 7:

[0160] The user's instructions are transmitted to a terminal at the site via the user's device. The terminal then relays these instructions to the on-site worker via voice or display. The worker then immediately performs the task or takes action according to these instructions.

[0161] Step 8:

[0162] The server monitors how instructions are executed on-site and feeds the results back into the database. This feedback information is used to improve the system and optimize instructions in the future.

[0163] (Example 2)

[0164] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0165] Conventional systems aimed at improving safety and efficiency at construction sites have limitations in real-time risk assessment and information provision, and are particularly poor at flexibly responding to human factors such as emotional states. Furthermore, it has been difficult to acquire data and transmit appropriate and timely instructions to the site based on the analysis results, so safety and work efficiency have not been fully achieved.

[0166] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0167] In this invention, the server includes means for acquiring various types of data in real time from multiple input devices placed on-site, means for quantitatively evaluating the risk level by comparing it with past accident data using a machine learning algorithm, and means for analyzing the emotional state of the user and adjusting the content of notifications. This enables flexible and effective risk assessment and notification adjustment that takes human factors into account, as well as the rapid transmission of on-site instructions.

[0168] An "input device" is a device installed on-site to acquire various types of data in real time.

[0169] A "central system" is a device that analyzes acquired data and evaluates risk levels by referring to past data.

[0170] A "machine learning algorithm" is a calculation method used to quantitatively evaluate risk by comparing it with past accident data.

[0171] A "user device" is a device that receives alerts, analyzes the user's emotional state, and adjusts notifications accordingly.

[0172] "Emotional state" refers to the psychological state that can be inferred from the user's tone of voice, facial expressions, and speed of operation.

[0173] An "alert" is a warning message generated when a high risk is identified.

[0174] "Instructions" refer to information provided by user equipment and communicated to on-site workers to prompt actions or responses.

[0175] This invention provides a system for real-time data acquisition, analysis, and notification coordination to improve safety and efficiency at construction sites. The system mainly consists of multiple "input devices," a "central device" (server) that analyzes the data, and a "user device" that receives notifications.

[0176] The terminal collects data such as video, audio, temperature, and vibration in real time through input devices installed on-site, such as video cameras and various sensors. This data acquisition is performed by multiple built-in sensor devices and camera systems. The terminal temporarily stores the continuously acquired data and efficiently sends it to the central device based on priority according to requirements.

[0177] The server functions as a central device, using machine learning algorithms such as TensorFlow and PyTorch to analyze the received data. This analysis quantitatively assesses the current risk level by referencing past accident data. If a high risk is detected, an alert is immediately generated and notified to the user's device with a detailed explanation.

[0178] As soon as the user device receives an alert, its built-in emotion engine analyzes the user's tone of voice, facial expressions, and the speed of their actions. This allows the engine to recognize the user's emotional state and adjust the notification accordingly. For example, if the user is determined to be stressed, the alert will be adjusted to be concise and calming.

[0179] As a concrete example, consider a situation where abnormal vibrations are detected on-site. The terminal sends this abnormal data to the server, and if the server analyzes the risk and determines it to be high risk, the user can quickly learn about the situation. Based on this information, the user can issue accurate instructions to the site and avoid potential dangers. By designing such a system, safe and efficient construction site management can be achieved.

[0180] Examples of prompts that can be input to the generating AI model include, "Please explain the overview of data analysis methods in safety management systems for construction sites."

[0181] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0182] Step 1:

[0183] The terminal acquires diverse data such as video, audio, temperature, and vibration in real time through input devices installed on-site. Inputs include on-site sensor information and visual data. This data is temporarily stored within the terminal and organized based on its importance. This organized data is then prepared for analysis in the next processing step.

[0184] Step 2:

[0185] The terminal transmits the acquired data to the central server. The data held by the terminal is used as input. Data transmission is performed using a network-based communication protocol, with high-priority data being transmitted first. The server receives organized field data as output.

[0186] Step 3:

[0187] The server analyzes the received data and performs a risk assessment. Field data sent to the server is used as input. Machine learning algorithms such as TensorFlow and PyTorch are applied to this analysis, quantitatively evaluating the risk by comparing it with past accident data. The output generated in this process is data indicating the risk level at the site.

[0188] Step 4:

[0189] The server generates an alert based on the analysis results and notifies the user's device. Analyzed risk assessment data is used as input. Within the server, the alert generation logic runs, preparing alert information that includes detailed explanations. This alert information is then sent to the user's device as output.

[0190] Step 5:

[0191] The user device receives an alert, and the emotion engine evaluates the user's emotional state. Inputs include the received alert information and emotion-related data obtained from the user. The emotion engine analyzes the user's tone of voice, facial expressions, operation speed, etc., and evaluates stress levels, etc. Based on the results, adjusted notification information is generated as output.

[0192] Step 6:

[0193] The user receives coordinated notifications and then decides on and sends instructions to the field. The input is coordinated notification information. Based on this information, the user calmly and quickly provides specific instructions to the field workers. These instructions are then transmitted to the field as output information from the user.

[0194] (Application Example 2)

[0195] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0196] In on-site work, ensuring both safety and efficiency simultaneously is always required. However, safety and efficiency can be affected by the emotional state of workers, making recognition and countermeasures crucial. To properly solve this problem, a system is needed that can grasp the emotional state of workers in real time and adjust machine operations based on that information.

[0197] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0198] In this invention, the server includes means for acquiring field data from multiple input devices placed on-site, means for recognizing the emotional state of workers and adjusting the notification method in user devices, and means for generating instructions to optimize machine operation based on the adjusted notifications. This makes it possible to maximize on-site safety and efficiency while taking into account the emotional state of workers.

[0199] An "input device" is a device used to acquire on-site data in real time, and includes video cameras and various sensors.

[0200] A "central device" is a device that aggregates and analyzes acquired data, and has the function of assessing risk levels.

[0201] A "user device" is a device that receives alerts and instructions at the work site and provides information to facilitate the smooth progress of the work.

[0202] "Emotional state recognition" is the process of determining a worker's mental state by analyzing their voice tone and facial expressions.

[0203] "Adjusting notification methods" refers to a means of changing the format of alerts and information received according to the emotional state of the worker, in order to encourage appropriate responses.

[0204] "Optimizing machine operation" means adjusting instructions so that the machine operates most efficiently and safely, while also taking into account the emotional state of the operator.

[0205] This system utilizes multiple input devices, a central unit, and user devices. The server continuously acquires field data using video cameras and various sensors via Raspberry Pi and other devices. This includes collecting video, audio, temperature, and vibration data. Data transmitted from the input devices is transferred to the central unit in the cloud. The central unit uses machine learning algorithms, such as Amazon SageMaker, to assess risk levels.

[0206] User devices function as smartphones or tablets, receiving alerts from a central device. Furthermore, using OpenCV and Google Cloud's AI services, the system recognizes the worker's emotional state in real time from their voice and facial expressions, and adjusts notification methods accordingly. For example, if a worker is determined to be in a high-stress state, the notification is adjusted to include more detailed content encouraging calm behavior.

[0207] For example, if the system detects abnormal vibrations, the server immediately analyzes the data and generates an alert if it determines the risk is high. Users receive this alert along with an evaluation by the emotion engine, and appropriate instructions are communicated to the site, improving work safety and efficiency. An example of a prompt to the generating AI model is, "Use the sensor data obtained in real time to evaluate the stress level of the work environment and propose safety measures based on the results."

[0208] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0209] Step 1:

[0210] The terminal uses multiple input devices to acquire real-time video, audio, temperature, and vibration data from the field. This input data includes information indicating changes and anomalies in the field's work environment. The terminal packages this data and transmits it to a central device.

[0211] Step 2:

[0212] The server, acting as a central device, receives data aggregated in the cloud and performs analysis using machine learning algorithms via Amazon SageMaker. The input is the field data acquired earlier. Based on this, the server evaluates the risk level and generates an alert if an anomaly is detected. The output includes the risk assessment results and the generated alerts.

[0213] Step 3:

[0214] The server sends the generated alerts to the user's device. The user's device is a smartphone or tablet, which can receive the alert and immediately understand the situation on-site. In this step, the alert content is directly transmitted to the user's device.

[0215] Step 4:

[0216] The user device uses OpenCV and Google Cloud's AI services to recognize the worker's emotional state from their voice and facial expressions. Input data includes information about the worker's actions and facial expressions. The user device analyzes this information and outputs an evaluation of the emotional state.

[0217] Step 5:

[0218] The user device adjusts the notification method based on the results of the emotional state assessment. For example, if the user is determined to be in a high-stress state, the notification content and format are adjusted to ensure that the information is conveyed in the most effective way for the worker. This output is the adjusted notification content.

[0219] Step 6:

[0220] The user communicates specific instructions to the field based on coordinated notifications. These instructions are optimized to enhance work safety and efficiency. As output, appropriately coordinated instructions are delivered to the field.

[0221] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0222] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0223] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0224] [Second Embodiment]

[0225] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0226] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0227] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0228] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0229] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0230] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0231] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0232] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0233] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0234] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0235] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0236] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0237] The present invention aims to solve the problem of labor shortages at construction sites and improve safety. The system for this purpose consists of an advanced data collection and analysis system using multiple input devices and a central device installed at the site, as well as user devices.

[0238] Multiple input devices are installed at the site to continuously collect on-site data. These input devices include video cameras, temperature sensors, vibration sensors, and sound detection devices, enabling comprehensive monitoring of all situations. The collected data is initially filtered by terminals and transmitted to a central system according to its importance.

[0239] The server functions as a central device, analyzing transmitted data in real time. The server uses advanced algorithms to evaluate the data and quantify risk levels. This process incorporates machine learning techniques that reference historical data, enabling highly accurate risk assessments through predictions based on accumulated knowledge.

[0240] If a high risk is detected, the server immediately sends an alert to the user's device. This user device is used by on-site managers and supervisors, enabling real-time situation monitoring and immediate response.

[0241] Users can check notifications from the server, input instructions as needed, and transmit them to field workers via their terminals. Instructions are provided specifically via voice and display, and workers are required to act immediately in accordance with them.

[0242] For example, if the temperature rises abnormally at the site, the terminal immediately reports this anomaly to the central system, and the server assesses the level of risk based on past data. If a high risk is assessed, the server sends a system alert to the user's device, and the user instructs the site to take swift action. Through this entire process, it is possible to ensure safety at the site and improve work efficiency.

[0243] The following describes the processing flow.

[0244] Step 1:

[0245] The terminal activates various sensors and cameras installed on-site, acquiring video and sensor data in real time. This includes a variety of data such as temperature, vibration, and sound.

[0246] Step 2:

[0247] The terminal performs an initial filter on the acquired data to detect anomalies and unusual changes. This filtering prevents the transmission of unnecessary data, and only important data is extracted.

[0248] Step 3:

[0249] The terminal sends filtered data to a central server. Here, communication stability is verified, and data integrity is guaranteed.

[0250] Step 4:

[0251] The server processes the received data using an analysis algorithm to evaluate the situation on site. By using historical data and pattern recognition technology in the analysis, it performs a highly accurate risk assessment.

[0252] Step 5:

[0253] The server determines the risk level based on the data analysis results and generates alert messages as needed. In this process, it compares the situation to similar past situations if necessary and considers possible countermeasures.

[0254] Step 6:

[0255] The server sends an alert to the user's device. This alert contains detailed information about the risks at the site, allowing the user to quickly consider countermeasures.

[0256] Step 7:

[0257] After receiving an alert from the server, the user decides on a course of action and sends that action as an instruction command from their device to a terminal at the site. The instructions are specific and require immediate action at the site.

[0258] Step 8:

[0259] The terminal transmits instructions received from the user to on-site workers via voice or display. This allows workers to respond quickly based on the instructions.

[0260] Step 9:

[0261] The server confirms that the entire process has completed successfully and continues recording and analyzing data. This data will be used for future improvements and updates to predictive models.

[0262] (Example 1)

[0263] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0264] In construction sites, manufacturing plants, and other similar environments, there is a need to respond quickly and accurately to changes in the work environment and unforeseen circumstances, thereby improving safety. However, conventional systems have struggled to efficiently collect and analyze individual data, conduct immediate risk assessments based on the results, and provide appropriate instructions to workers. Furthermore, low accuracy in risk prediction and delays in effective responses have been contributing to compromised safety.

[0265] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0266] In this invention, the server includes means for acquiring environmental data from multiple sensor devices placed on-site, means for analyzing the data transferred to a central device using advanced algorithms to perform a safety risk assessment, and means for performing risk prediction by utilizing machine learning technology and referring to past data. This enables rapid data collection and evaluation on-site, highly accurate risk prediction based on past data, and real-time instruction provision to workers.

[0267] A "sensor device" is a device that is placed to detect changes in the environment and collect data.

[0268] A "relay device" is a device that performs initial filtering of data acquired from sensor devices before transmitting it to the central device.

[0269] A "central system" is a computer system that functions to analyze received environmental data and perform safety risk assessments.

[0270] An "advanced algorithm" is a computational method used to analyze complex and diverse data and perform accurate risk assessments.

[0271] "Machine learning technology" is a technique that learns patterns based on past data and uses that to predict future risks.

[0272] "Risk assessment" is the process of quantifying and determining the degree of safety risks based on local situation data.

[0273] A "management device" is a terminal used by on-site managers to receive alerts notified from the central device.

[0274] A "worker" is a person responsible for carrying out actual work according to instructions at the work site.

[0275] A "generative AI model" is an artificial intelligence technology that generates countermeasures based on given data and prompts.

[0276] A "prompt statement" is an input command given to a generative AI model to cause it to generate specific information.

[0277] To implement the invention, this system is realized in the following form.

[0278] Multiple sensor devices are deployed at the site to acquire environmental data such as temperature, vibration, visual information, and sound. This data is transmitted to a central device via a relay device. The relay device first evaluates the importance of the data, performs initial filtering, and sends only the necessary data to the central device.

[0279] The server functions as a central device, analyzing the received data. Here, advanced algorithms are used to analyze the data in detail and assess safety risks. Furthermore, machine learning techniques are utilized to predict future risks by referencing past patterns. This technique, where the server uses generative AI models to predict risks, is a particularly crucial process.

[0280] Based on the risk level assessment, the server sends alerts to the management device in real time. These alerts include emergency response measures tailored to the situation on site, and, in particular, instructions for immediate action when necessary.

[0281] The user uses a management device to receive notifications from the server, consider countermeasures as needed, and communicate specific instructions to the workers. For example, one possible instruction might be, "All workers, immediately move to the designated evacuation area."

[0282] As a concrete example, consider a situation where the temperature rises rapidly at a construction site. In this case, a sensor device detects the anomaly, and the information is transmitted to the central device via a relay device. The server, referring to past data, immediately sends an alert to the management device if it determines that the risk level is high. Based on this alert, the user can quickly issue instructions to the workers.

[0283] An example of a prompt using a generative AI model is a text-based instruction such as, "Assess the risk based on the current temperature data and generate recommended countermeasures." This prompt instructs the AI ​​model to perform appropriate data processing and generate countermeasures.

[0284] The flow of a specific process in Example 1 will be described using FIG. 11.

[0285] Step 1:

[0286] The terminal acquires environmental data from a plurality of sensor devices arranged at the site. This includes video data from a video camera, temperature data from a temperature sensor, vibration data from a vibration sensor, and audio data from an audio detection device. The terminal receives these data as input and performs initial filtering based on importance. As an output, filtered important data is generated and sent to the relay device.

[0287] Step 2:

[0288] The server that has received the filtered data transmitted from the terminal starts data analysis using an advanced algorithm. It analyzes the received data in detail and performs a risk assessment related to safety. By using machine learning techniques, the server refers to past data and calculates a risk assessment score based on the current situation as an output. In this process, calculations are performed to identify abnormal patterns and determine urgency.

[0289] Step 3:

[0290] Based on the risk assessment score obtained after analysis, the server generates an alert. This alert is sent to the management device in real time. The alert generated as an output includes emergency response measures according to the specific situation at the site. For example, instructions such as "Due to the abnormal rise in temperature in Area B, all workers should take shelter" are included.

[0291] Step 4:

[0292] The user receives alerts from the server via a management device and reviews their contents. Based on the alert information received as input, the user quickly creates specific instructions if action is required. The user then transmits these instructions as output to field workers, prompting them to take action to ensure safety at the site. In some cases, the user may also optimize the instructions by using prompts generated by an AI model, taking into account the countermeasures suggested by the AI.

[0293] An example of a prompt message for a countermeasure proposed by the generative AI model is, "Assess the risk based on the current temperature data and generate recommended countermeasures." Through this process, rapid and accurate risk management is achieved throughout the entire system.

[0294] (Application Example 1)

[0295] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0296] In factories and work environments, labor shortages and ensuring safety are serious problems. Especially in workplaces where automated machinery predominates, human intervention is limited, making the detection of abnormalities and hazards, and rapid response, crucial. However, conventional systems have limitations in real-time capabilities and predictive accuracy, making it difficult to implement appropriate safety measures on-site. This invention aims to solve these problems.

[0297] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0298] In this invention, the server includes means for transmitting collected data to a central processing unit, and the central processing unit includes means for analyzing the collected data and evaluating the degree of risk at the work site, and means for instructing mechanical devices within the work space to take safe actions. This enables real-time risk detection and prediction, and rapid safety response.

[0299] The "information collection device" is a device such as a sensor or camera installed to collect environmental data at the work site.

[0300] The "central processing unit" is a central computer or server for analyzing the collected data and evaluating the degree of risk.

[0301] "Analysis" means performing a process to find useful information from the collected data, and particularly refers to calculations and analyses related to the evaluation of the degree of risk.

[0302] "Evaluation" means quantifying and judging the degree of risk at the work site based on the information obtained by analysis.

[0303] "Warning" is a notification generated based on the evaluation result, and is a signal or message transmitted to workers and managers through the operation terminal.

[0304] The "operation terminal" is a device or interface used by the user to receive warnings and transmit necessary instructions to the site.

[0305] "Instruction" specifically indicates the actions and work instructions at the work site, and is transmitted from the operation terminal to the personnel and mechanical devices at the site.

[0306] The "mechanical device" is a robot or other automation device for automatically performing specific work at the work site.

[0307] "Safe action" is the action that the mechanical device should take when danger is detected, and is an operation or process for ensuring the safety of people and the environment.

[0308] As a form for implementing the invention, a safety monitoring system in a factory or work site will be described. This system consists of an information collection device, a central processing unit, and an operation terminal.

[0309] The information gathering device includes numerous sensors and cameras installed at the work site to collect temperature, vibration, or acoustic data in real time. The data collected by this device is transmitted to a central processing unit via a network.

[0310] The central processing unit (server) analyzes received data using machine learning algorithms such as TensorFlow. Based on past data, it assesses the level of risk from the current data and makes a quantitative judgment. Based on the analysis results, it immediately generates a warning and notifies the operating terminal.

[0311] The control terminal is a device used by managers and workers within the factory, allowing them to check warnings in real time. Upon receiving a warning, the user immediately transmits the necessary commands to the machinery. The machinery then follows the received commands and implements safe actions, thereby ensuring safety on site.

[0312] For example, if a temperature sensor detects an abnormal value and the central processing unit assesses this as a high risk, a warning is sent to the administrator via the control terminal. Subsequently, the machine automatically stops operating and is instructed to move to a safe location.

[0313] Example prompt message:

[0314] The factory temperature sensor has detected an abnormal reading. Assess the level of risk, issue an immediate notification, and instruct the robot to move safely.

[0315] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0316] Step 1:

[0317] The information gathering device continuously collects data at the work site.

[0318] (Input: Sensor data such as temperature, vibration, and sound)

[0319] The information gathering device acquires various data from the field environment in real time and performs initial data processing. Afterward, important data is transmitted to the central processing unit via the network.

[0320] (Output: Filtered dataset)

[0321] Step 2:

[0322] The server analyzes the data it receives.

[0323] (Input: Filtered dataset)

[0324] The server uses TensorFlow algorithms to analyze incoming data, and during this process, it evaluates the level of risk by comparing it to past data. This process utilizes generative AI models to perform risk assessment in real time.

[0325] (Output: Risk assessment value)

[0326] Step 3:

[0327] The server generates a warning based on the risk assessment value and notifies the operating terminal.

[0328] (Input: Risk assessment value)

[0329] If a high risk is detected, the server generates a warning message and promptly sends it to the user's terminal via the network. This message includes specific instructions to enable the user to take immediate action.

[0330] (Output: Warning message)

[0331] Step 4:

[0332] The operating terminal receives a warning message and notifies the user.

[0333] (Input: Warning message)

[0334] When the operating terminal receives a warning, it immediately displays the content on the terminal screen and emits an alert sound if necessary. The user checks the notification and takes appropriate action.

[0335] (Output: Notification to the user)

[0336] Step 5:

[0337] The user instructs the machine to perform safety actions via a control terminal.

[0338] (Input: Warning content and user response instructions)

[0339] The user inputs a command from the control terminal, which is then transmitted to the machinery at the work site. Based on the received instructions, the machinery begins safe operation.

[0340] (Output: Instruction signal to the machine)

[0341] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0342] This invention provides a system equipped with emotion recognition capabilities to improve safety and efficiency at construction sites. This system consists of an input device that monitors site conditions in real time, a central device (server) that analyzes the data, and an emotion engine that recognizes the user's emotions and provides information according to their state.

[0343] First, multiple input devices installed on-site continuously acquire on-site data in real time. This includes acquiring video, audio, temperature, and vibration data from video cameras and various sensors. Terminals analyze this data and transmit the necessary information to a central device.

[0344] The server functions as a central device, analyzing the acquired data. Data analysis utilizes machine learning algorithms that refer to past accident data and quantitatively assess risk. If a high risk is detected, the server generates an alert and notifies the user device.

[0345] The user device is equipped with an emotion engine that analyzes the user's tone of voice, facial expressions, and speed of operation to recognize the user's emotional state. When it receives an alert from the server, the emotion engine assesses the user's stress level and adjusts the notification method accordingly. For example, if the system determines that the user is in a high-stress state, the notification will be adjusted to be more detailed and calming.

[0346] Users receive notifications based on their analyzed emotional state and issue instructions to the field as needed. These instructions are provided in a format easily understood by workers, taking into account feedback from the emotion engine. Information regarding the user's emotional state is fed back to the server and stored in a database. This data is used for future system improvements and instruction optimization.

[0347] For example, if abnormal vibrations are detected on-site, the terminal immediately reports the data to the server, which assesses the risk and generates an alert. The user receives this alert along with an evaluation from the emotion engine, enabling them to give calm instructions to the site. This process maximizes the safety and efficiency of the work.

[0348] The following describes the processing flow.

[0349] Step 1:

[0350] The terminal activates various sensors and cameras installed on-site, acquiring video data and sensor data (temperature, vibration, sound, etc.) in real time. This makes it possible to detect any environmental changes at the site.

[0351] Step 2:

[0352] The terminal performs initial filtering, narrowing down the acquired data to only the essentials. This process extracts unusual changes and data exceeding thresholds, ensuring that only critical information is sent to the central device.

[0353] Step 3:

[0354] The server receives data sent from the terminal and performs data analysis using machine learning algorithms. This analysis quantitatively evaluates the risk level at the site and detects signs of high risk.

[0355] Step 4:

[0356] Based on the risk assessment results, the server generates an alert message appropriate to the user's emotional state. If the risk is higher than historical data, an alert is immediately generated and notified to the user's device.

[0357] Step 5:

[0358] The user device operates an emotion engine that analyzes the user's emotional state based on their actions and voice data. It evaluates the user's stress level and current emotions, and adjusts the notification method and content accordingly.

[0359] Step 6:

[0360] The user considers feedback from the emotion engine, reviews the alerts from the server, and consciously considers instructions for the field. The instructions are specific and adjusted to a format that is easy for workers to understand.

[0361] Step 7:

[0362] The user's instructions are transmitted to a terminal at the site via the user's device. The terminal then relays these instructions to the on-site worker via voice or display. The worker then immediately performs the task or takes action according to these instructions.

[0363] Step 8:

[0364] The server monitors how instructions are executed on-site and feeds the results back into the database. This feedback information is used to improve the system and optimize instructions in the future.

[0365] (Example 2)

[0366] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0367] Conventional systems aimed at improving safety and efficiency at construction sites have limitations in real-time risk assessment and information provision, and are particularly poor at flexibly responding to human factors such as emotional states. Furthermore, it has been difficult to acquire data and transmit appropriate and timely instructions to the site based on the analysis results, so safety and work efficiency have not been fully achieved.

[0368] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0369] In this invention, the server includes means for acquiring various types of data in real time from multiple input devices placed on-site, means for quantitatively evaluating the risk level by comparing it with past accident data using a machine learning algorithm, and means for analyzing the emotional state of the user and adjusting the content of notifications. This enables flexible and effective risk assessment and notification adjustment that takes human factors into account, as well as the rapid transmission of on-site instructions.

[0370] An "input device" is a device installed on-site to acquire various types of data in real time.

[0371] A "central system" is a device that analyzes acquired data and evaluates risk levels by referring to past data.

[0372] A "machine learning algorithm" is a calculation method used to quantitatively evaluate risk by comparing it with past accident data.

[0373] A "user device" is a device that receives alerts, analyzes the user's emotional state, and adjusts notifications accordingly.

[0374] "Emotional state" refers to the psychological state that can be inferred from the user's tone of voice, facial expressions, and speed of operation.

[0375] An "alert" is a warning message generated when a high risk is identified.

[0376] "Instructions" refer to information provided by user equipment and communicated to on-site workers to prompt actions or responses.

[0377] This invention provides a system for real-time data acquisition, analysis, and notification coordination to improve safety and efficiency at construction sites. The system mainly consists of multiple "input devices," a "central device" (server) that analyzes the data, and a "user device" that receives notifications.

[0378] The terminal collects data such as video, audio, temperature, and vibration in real time through input devices installed on-site, such as video cameras and various sensors. This data acquisition is performed by multiple built-in sensor devices and camera systems. The terminal temporarily stores the continuously acquired data and efficiently sends it to the central device based on priority according to requirements.

[0379] The server functions as a central device, using machine learning algorithms such as TensorFlow and PyTorch to analyze the received data. This analysis quantitatively assesses the current risk level by referencing past accident data. If a high risk is detected, an alert is immediately generated and notified to the user's device with a detailed explanation.

[0380] As soon as the user device receives an alert, its built-in emotion engine analyzes the user's tone of voice, facial expressions, and the speed of their actions. This allows the engine to recognize the user's emotional state and adjust the notification accordingly. For example, if the user is determined to be stressed, the alert will be adjusted to be concise and calming.

[0381] As a concrete example, consider a situation where abnormal vibrations are detected on-site. The terminal sends this abnormal data to the server, and if the server analyzes the risk and determines it to be high risk, the user can quickly learn about the situation. Based on this information, the user can issue accurate instructions to the site and avoid potential dangers. By designing such a system, safe and efficient construction site management can be achieved.

[0382] Examples of prompts that can be input to the generating AI model include, "Please explain the overview of data analysis methods in safety management systems for construction sites."

[0383] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0384] Step 1:

[0385] The terminal acquires diverse data such as video, audio, temperature, and vibration in real time through input devices installed on-site. Inputs include on-site sensor information and visual data. This data is temporarily stored within the terminal and organized based on its importance. This organized data is then prepared for analysis in the next processing step.

[0386] Step 2:

[0387] The terminal transmits the acquired data to the central server. The data held by the terminal is used as input. Data transmission is performed using a network-based communication protocol, with high-priority data being transmitted first. The server receives organized field data as output.

[0388] Step 3:

[0389] The server analyzes the received data and performs a risk assessment. Field data sent to the server is used as input. Machine learning algorithms such as TensorFlow and PyTorch are applied to this analysis, quantitatively evaluating the risk by comparing it with past accident data. The output generated in this process is data indicating the risk level at the site.

[0390] Step 4:

[0391] The server generates an alert based on the analysis results and notifies the user's device. Analyzed risk assessment data is used as input. Within the server, the alert generation logic runs, preparing alert information that includes detailed explanations. This alert information is then sent to the user's device as output.

[0392] Step 5:

[0393] The user device receives an alert, and the emotion engine evaluates the user's emotional state. Inputs include the received alert information and emotion-related data obtained from the user. The emotion engine analyzes the user's tone of voice, facial expressions, operation speed, etc., and evaluates stress levels, etc. Based on the results, adjusted notification information is generated as output.

[0394] Step 6:

[0395] The user receives coordinated notifications and then decides on and sends instructions to the field. The input is coordinated notification information. Based on this information, the user calmly and quickly provides specific instructions to the field workers. These instructions are then transmitted to the field as output information from the user.

[0396] (Application Example 2)

[0397] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0398] In on-site work, ensuring both safety and efficiency simultaneously is always required. However, safety and efficiency can be affected by the emotional state of workers, making recognition and countermeasures crucial. To properly solve this problem, a system is needed that can grasp the emotional state of workers in real time and adjust machine operations based on that information.

[0399] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0400] In this invention, the server includes means for acquiring field data from multiple input devices placed on-site, means for recognizing the emotional state of workers and adjusting the notification method in user devices, and means for generating instructions to optimize machine operation based on the adjusted notifications. This makes it possible to maximize on-site safety and efficiency while taking into account the emotional state of workers.

[0401] An "input device" is a device used to acquire on-site data in real time, and includes video cameras and various sensors.

[0402] A "central device" is a device that aggregates and analyzes acquired data, and has the function of assessing risk levels.

[0403] A "user device" is a device that receives alerts and instructions at the work site and provides information to facilitate the smooth progress of the work.

[0404] "Emotional state recognition" is the process of determining a worker's mental state by analyzing their voice tone and facial expressions.

[0405] "Adjusting notification methods" refers to a means of changing the format of alerts and information received according to the emotional state of the worker, in order to encourage appropriate responses.

[0406] "Optimizing machine operation" means adjusting instructions so that the machine operates most efficiently and safely, while also taking into account the emotional state of the operator.

[0407] This system utilizes multiple input devices, a central unit, and user devices. The server continuously acquires field data using video cameras and various sensors via Raspberry Pi and other devices. This includes collecting video, audio, temperature, and vibration data. Data transmitted from the input devices is transferred to the central unit in the cloud. The central unit uses machine learning algorithms, such as Amazon SageMaker, to assess risk levels.

[0408] User devices function as smartphones or tablets, receiving alerts from a central device. Furthermore, using OpenCV and Google Cloud AI services, the system recognizes the worker's emotional state in real time from their voice and facial expressions, and adjusts notification methods accordingly. For example, if a worker is determined to be in a high-stress state, the notification is adjusted to include more detailed content encouraging calm behavior.

[0409] For example, if the system detects abnormal vibrations, the server immediately analyzes the data and generates an alert if it determines the risk is high. Users receive this alert along with an evaluation by the emotion engine, and appropriate instructions are communicated to the site, improving work safety and efficiency. An example of a prompt to the generating AI model is, "Use the sensor data obtained in real time to evaluate the stress level of the work environment and propose safety measures based on the results."

[0410] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0411] Step 1:

[0412] The terminal uses multiple input devices to acquire real-time video, audio, temperature, and vibration data from the field. This input data includes information indicating changes and anomalies in the field's work environment. The terminal packages this data and transmits it to a central device.

[0413] Step 2:

[0414] The server, acting as a central device, receives data aggregated in the cloud and performs analysis using machine learning algorithms via Amazon SageMaker. The input is the field data acquired earlier. Based on this, the server evaluates the risk level and generates an alert if an anomaly is detected. The output includes the risk assessment results and the generated alerts.

[0415] Step 3:

[0416] The server sends the generated alerts to the user's device. The user's device is a smartphone or tablet, which can receive the alert and immediately understand the situation on-site. In this step, the alert content is directly transmitted to the user's device.

[0417] Step 4:

[0418] The user device uses OpenCV and Google Cloud's AI services to recognize the worker's emotional state from their voice and facial expressions. Input data includes information about the worker's actions and facial expressions. The user device analyzes this information and outputs an evaluation of the emotional state.

[0419] Step 5:

[0420] The user device adjusts the notification method based on the results of the emotional state assessment. For example, if the user is determined to be in a high-stress state, the notification content and format are adjusted to ensure that the information is conveyed in the most effective way for the worker. This output is the adjusted notification content.

[0421] Step 6:

[0422] The user communicates specific instructions to the field based on coordinated notifications. These instructions are optimized to enhance work safety and efficiency. As output, appropriately coordinated instructions are delivered to the field.

[0423] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0424] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0425] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0426] [Third Embodiment]

[0427] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0428] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0429] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0430] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0431] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0432] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0433] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0434] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0435] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0436] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0437] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0438] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0439] This invention aims to solve the problem of labor shortages at construction sites and improve safety. The system for this purpose consists of an advanced data collection and analysis system using multiple input devices and a central device installed at the site, as well as user devices.

[0440] Multiple input devices are installed at the site to continuously collect on-site data. These input devices include video cameras, temperature sensors, vibration sensors, and sound detection devices, enabling comprehensive monitoring of all situations. The collected data is initially filtered by terminals and transmitted to a central system according to its importance.

[0441] The server functions as a central device, analyzing transmitted data in real time. The server uses advanced algorithms to evaluate the data and quantify risk levels. This process incorporates machine learning techniques that reference historical data, enabling highly accurate risk assessments through predictions based on accumulated knowledge.

[0442] If a high risk is detected, the server immediately sends an alert to the user's device. This user device is used by on-site managers and supervisors, enabling real-time situation monitoring and immediate response.

[0443] Users can check notifications from the server, input instructions as needed, and transmit them to field workers via their terminals. Instructions are provided specifically via voice and display, and workers are required to act immediately in accordance with them.

[0444] For example, if the temperature rises abnormally at the site, the terminal immediately reports this anomaly to the central system, and the server assesses the level of risk based on past data. If a high risk is assessed, the server sends a system alert to the user's device, and the user instructs the site to take swift action. Through this entire process, it is possible to ensure safety at the site and improve work efficiency.

[0445] The following describes the processing flow.

[0446] Step 1:

[0447] The terminal activates various sensors and cameras installed on-site, acquiring video and sensor data in real time. This includes a variety of data such as temperature, vibration, and sound.

[0448] Step 2:

[0449] The terminal performs an initial filter on the acquired data to detect anomalies and unusual changes. This filtering prevents the transmission of unnecessary data, and only important data is extracted.

[0450] Step 3:

[0451] The terminal sends filtered data to a central server. Here, communication stability is verified, and data integrity is guaranteed.

[0452] Step 4:

[0453] The server processes the received data using an analysis algorithm to evaluate the situation on site. By using historical data and pattern recognition technology in the analysis, it performs a highly accurate risk assessment.

[0454] Step 5:

[0455] The server determines the risk level based on the data analysis results and generates alert messages as needed. In this process, it compares the situation to similar past situations if necessary and considers possible countermeasures.

[0456] Step 6:

[0457] The server sends an alert to the user's device. This alert contains detailed information about the risks at the site, allowing the user to quickly consider countermeasures.

[0458] Step 7:

[0459] After receiving an alert from the server, the user decides on a course of action and sends that action as an instruction command from their device to a terminal at the site. The instructions are specific and require immediate action at the site.

[0460] Step 8:

[0461] The terminal transmits instructions received from the user to on-site workers via voice or display. This allows workers to respond quickly based on the instructions.

[0462] Step 9:

[0463] The server confirms that the entire process has completed successfully and continues recording and analyzing data. This data will be used for future improvements and updates to predictive models.

[0464] (Example 1)

[0465] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0466] In construction sites, manufacturing plants, and other similar environments, there is a need to respond quickly and accurately to changes in the work environment and unforeseen circumstances, thereby improving safety. However, conventional systems have struggled to efficiently collect and analyze individual data, conduct immediate risk assessments based on the results, and provide appropriate instructions to workers. Furthermore, low accuracy in risk prediction and delays in effective responses have been contributing to compromised safety.

[0467] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0468] In this invention, the server includes means for acquiring environmental data from multiple sensor devices placed on-site, means for analyzing the data transferred to a central device using advanced algorithms to perform a safety risk assessment, and means for performing risk prediction by utilizing machine learning technology and referring to past data. This enables rapid data collection and evaluation on-site, highly accurate risk prediction based on past data, and real-time instruction provision to workers.

[0469] A "sensor device" is a device that is placed to detect changes in the environment and collect data.

[0470] A "relay device" is a device that performs initial filtering of data acquired from sensor devices before transmitting it to the central device.

[0471] A "central system" is a computer system that functions to analyze received environmental data and perform safety risk assessments.

[0472] An "advanced algorithm" is a computational method used to analyze complex and diverse data and perform accurate risk assessments.

[0473] "Machine learning technology" is a technique that learns patterns based on past data and uses that to predict future risks.

[0474] "Risk assessment" is the process of quantifying and determining the degree of safety risks based on local situation data.

[0475] A "management device" is a terminal used by on-site managers to receive alerts notified from the central device.

[0476] A "worker" is a person responsible for carrying out actual work according to instructions at the work site.

[0477] A "generative AI model" is an artificial intelligence technology that generates countermeasures based on given data and prompts.

[0478] A "prompt statement" is an input command given to a generative AI model to cause it to generate specific information.

[0479] To implement the invention, this system is realized in the following form.

[0480] Multiple sensor devices are deployed at the site to acquire environmental data such as temperature, vibration, visual information, and sound. This data is transmitted to a central device via a relay device. The relay device first evaluates the importance of the data, performs initial filtering, and sends only the necessary data to the central device.

[0481] The server functions as a central device, analyzing the received data. Here, advanced algorithms are used to analyze the data in detail and assess safety risks. Furthermore, machine learning techniques are utilized to predict future risks by referencing past patterns. This technique, where the server uses generative AI models to predict risks, is a particularly crucial process.

[0482] Based on the risk level assessment, the server sends alerts to the management device in real time. These alerts include emergency response measures tailored to the situation on site, and, in particular, instructions for immediate action when necessary.

[0483] The user uses a management device to receive notifications from the server, consider countermeasures as needed, and communicate specific instructions to the workers. For example, one possible instruction might be, "All workers, immediately move to the designated evacuation area."

[0484] As a concrete example, consider a situation where the temperature rises rapidly at a construction site. In this case, a sensor device detects the anomaly, and the information is transmitted to the central device via a relay device. The server, referring to past data, immediately sends an alert to the management device if it determines that the risk level is high. Based on this alert, the user can quickly issue instructions to the workers.

[0485] An example of a prompt using a generative AI model is a text-based instruction such as, "Assess the risk based on the current temperature data and generate recommended countermeasures." This prompt instructs the AI ​​model to perform appropriate data processing and generate countermeasures.

[0486] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0487] Step 1:

[0488] The terminal acquires environmental data from multiple sensor devices placed on-site. This includes video data from video cameras, temperature data from temperature sensors, vibration data from vibration sensors, and audio data from sound detection devices. The terminal receives this data as input and performs initial filtering based on importance. As output, filtered important data is generated and sent to the relay device.

[0489] Step 2:

[0490] Upon receiving filtered data from a terminal, the server begins data analysis using advanced algorithms. It analyzes the received data in detail and conducts a safety-related risk assessment. Using machine learning techniques, the server references past data and calculates a risk assessment score based on the current situation. This process involves identifying anomaly patterns and performing calculations to determine urgency.

[0491] Step 3:

[0492] The server generates an alert based on the risk assessment score obtained after analysis. This alert is sent to the management device in real time. The alert generated as output includes emergency response measures tailored to the specific situation on site. For example, it may include instructions such as, "The temperature is rising abnormally in Area B, so all workers must evacuate."

[0493] Step 4:

[0494] The user receives alerts from the server via a management device and reviews their contents. Based on the alert information received as input, the user quickly creates specific instructions if action is required. The user then transmits these instructions as output to field workers, prompting them to take action to ensure safety at the site. In some cases, the user may also optimize the instructions by using prompts generated by an AI model, taking into account the countermeasures suggested by the AI.

[0495] An example of a prompt message for a countermeasure proposed by the generative AI model is, "Assess the risk based on the current temperature data and generate recommended countermeasures." Through this process, rapid and accurate risk management is achieved throughout the entire system.

[0496] (Application Example 1)

[0497] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0498] In factories and work environments, labor shortages and ensuring safety are serious problems. Especially in workplaces where automated machinery predominates, human intervention is limited, making the detection of abnormalities and hazards, and rapid response, crucial. However, conventional systems have limitations in real-time capabilities and predictive accuracy, making it difficult to implement appropriate safety measures on-site. This invention aims to solve these problems.

[0499] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0500] In this invention, the server includes means for transmitting collected data to a central processing unit, and the central processing unit includes means for analyzing the collected data and evaluating the degree of risk at the work site, and means for instructing mechanical devices within the work space to take safe actions. This enables real-time risk detection and prediction, and rapid safety response.

[0501] An "information gathering device" refers to devices such as sensors and cameras installed to collect environmental data at a work site.

[0502] A "central processing unit" is a central computer or server used to analyze collected data and assess the level of risk.

[0503] "Analysis" refers to the process of extracting useful information from collected data, and specifically refers to calculations and analyses related to risk assessment.

[0504] "Evaluation" refers to quantifying and judging the degree of danger at a work site based on information obtained through analysis.

[0505] A "warning" is a notification generated based on evaluation results, and is a signal or message conveyed to workers or managers through an operating terminal.

[0506] An "operation terminal" is a device or interface used by users to receive warnings and transmit necessary instructions to the field.

[0507] An "order" is a specific instruction regarding actions and tasks at the work site, and is transmitted from the control terminal to personnel and machinery at the site.

[0508] "Mechanical equipment" refers to robots and other automation devices that perform specific tasks automatically at a work site.

[0509] "Safety actions" refer to the actions that a machine or device should take when a hazard is detected, and are the actions and processes that ensure the safety of people and the environment.

[0510] As an embodiment for carrying out the invention, a safety monitoring system for factories and work sites will be described. This system consists of an information gathering device, a central processing unit, and an operating terminal.

[0511] The information gathering device includes numerous sensors and cameras installed at the work site to collect temperature, vibration, or acoustic data in real time. The data collected by this device is transmitted to a central processing unit via a network.

[0512] The central processing unit (server) analyzes received data using machine learning algorithms such as TensorFlow. Based on past data, it assesses the level of risk from the current data and makes a quantitative judgment. Based on the analysis results, it immediately generates a warning and notifies the operating terminal.

[0513] The control terminal is a device used by managers and workers within the factory, allowing them to check warnings in real time. Upon receiving a warning, the user immediately transmits the necessary commands to the machinery. The machinery then follows the received commands and implements safe actions, thereby ensuring safety on site.

[0514] For example, if a temperature sensor detects an abnormal value and the central processing unit assesses this as a high risk, a warning is sent to the administrator via the control terminal. Subsequently, the machine automatically stops operating and is instructed to move to a safe location.

[0515] Example prompt message:

[0516] The factory temperature sensor has detected an abnormal reading. Assess the level of risk, issue an immediate notification, and instruct the robot to move safely.

[0517] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0518] Step 1:

[0519] The information gathering device continuously collects data at the work site.

[0520] (Input: Sensor data such as temperature, vibration, and sound)

[0521] The information gathering device acquires various data from the field environment in real time and performs initial data processing. Afterward, important data is transmitted to the central processing unit via the network.

[0522] (Output: Filtered dataset)

[0523] Step 2:

[0524] The server analyzes the data it receives.

[0525] (Input: Filtered dataset)

[0526] The server uses TensorFlow algorithms to analyze incoming data, and during this process, it evaluates the level of risk by comparing it to past data. This process utilizes generative AI models to perform risk assessment in real time.

[0527] (Output: Risk assessment value)

[0528] Step 3:

[0529] The server generates a warning based on the risk assessment value and notifies the operating terminal.

[0530] (Input: Risk assessment value)

[0531] If a high risk is detected, the server generates a warning message and promptly sends it to the user's terminal via the network. This message includes specific instructions to enable the user to take immediate action.

[0532] (Output: Warning message)

[0533] Step 4:

[0534] The operating terminal receives a warning message and notifies the user.

[0535] (Input: Warning message)

[0536] When the operating terminal receives a warning, it immediately displays the content on the terminal screen and emits an alert sound if necessary. The user checks the notification and takes appropriate action.

[0537] (Output: Notification to the user)

[0538] Step 5:

[0539] The user instructs the machine to perform safety actions via a control terminal.

[0540] (Input: Warning content and user response instructions)

[0541] The user inputs a command from the control terminal, which is then transmitted to the machinery at the work site. Based on the received instructions, the machinery begins safe operation.

[0542] (Output: Instruction signal to the machine)

[0543] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0544] This invention provides a system equipped with emotion recognition capabilities to improve safety and efficiency at construction sites. This system consists of an input device that monitors site conditions in real time, a central device (server) that analyzes the data, and an emotion engine that recognizes the user's emotions and provides information according to their state.

[0545] First, multiple input devices installed on-site continuously acquire on-site data in real time. This includes acquiring video, audio, temperature, and vibration data from video cameras and various sensors. Terminals analyze this data and transmit the necessary information to a central device.

[0546] The server functions as a central device, analyzing the acquired data. Data analysis utilizes machine learning algorithms that refer to past accident data and quantitatively assess risk. If a high risk is detected, the server generates an alert and notifies the user device.

[0547] The user device is equipped with an emotion engine that analyzes the user's tone of voice, facial expressions, and speed of operation to recognize the user's emotional state. When it receives an alert from the server, the emotion engine assesses the user's stress level and adjusts the notification method accordingly. For example, if the system determines that the user is in a high-stress state, the notification will be adjusted to be more detailed and calming.

[0548] Users receive notifications based on their analyzed emotional state and issue instructions to the field as needed. These instructions are provided in a format easily understood by workers, taking into account feedback from the emotion engine. Information regarding the user's emotional state is fed back to the server and stored in a database. This data is used for future system improvements and instruction optimization.

[0549] For example, if abnormal vibrations are detected on-site, the terminal immediately reports the data to the server, which assesses the risk and generates an alert. The user receives this alert along with an evaluation from the emotion engine, enabling them to give calm instructions to the site. This process maximizes the safety and efficiency of the work.

[0550] The following describes the processing flow.

[0551] Step 1:

[0552] The terminal activates various sensors and cameras installed on-site, acquiring video data and sensor data (temperature, vibration, sound, etc.) in real time. This makes it possible to detect any environmental changes at the site.

[0553] Step 2:

[0554] The terminal performs initial filtering, narrowing down the acquired data to only the essentials. This process extracts unusual changes and data exceeding thresholds, ensuring that only critical information is sent to the central device.

[0555] Step 3:

[0556] The server receives data sent from the terminal and performs data analysis using machine learning algorithms. This analysis quantitatively evaluates the risk level at the site and detects signs of high risk.

[0557] Step 4:

[0558] Based on the risk assessment results, the server generates an alert message appropriate to the user's emotional state. If the risk is higher than historical data, an alert is immediately generated and notified to the user's device.

[0559] Step 5:

[0560] The user device operates an emotion engine that analyzes the user's emotional state based on their actions and voice data. It evaluates the user's stress level and current emotions, and adjusts the notification method and content accordingly.

[0561] Step 6:

[0562] The user considers feedback from the emotion engine, reviews the alerts from the server, and consciously considers instructions for the field. The instructions are specific and adjusted to a format that is easy for workers to understand.

[0563] Step 7:

[0564] The user's instructions are transmitted to a terminal at the site via the user's device. The terminal then relays these instructions to the on-site worker via voice or display. The worker then immediately performs the task or takes action according to these instructions.

[0565] Step 8:

[0566] The server monitors how instructions are executed on-site and feeds the results back into the database. This feedback information is used to improve the system and optimize instructions in the future.

[0567] (Example 2)

[0568] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0569] Conventional systems aimed at improving safety and efficiency at construction sites have limitations in real-time risk assessment and information provision, and are particularly poor at flexibly responding to human factors such as emotional states. Furthermore, it has been difficult to acquire data and transmit appropriate and timely instructions to the site based on the analysis results, so safety and work efficiency have not been fully achieved.

[0570] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0571] In this invention, the server includes means for acquiring various types of data in real time from multiple input devices placed on-site, means for quantitatively evaluating the risk level by comparing it with past accident data using a machine learning algorithm, and means for analyzing the emotional state of the user and adjusting the content of notifications. This enables flexible and effective risk assessment and notification adjustment that takes human factors into account, as well as the rapid transmission of on-site instructions.

[0572] An "input device" is a device installed on-site to acquire various types of data in real time.

[0573] A "central system" is a device that analyzes acquired data and evaluates risk levels by referring to past data.

[0574] A "machine learning algorithm" is a calculation method used to quantitatively evaluate risk by comparing it with past accident data.

[0575] A "user device" is a device that receives alerts, analyzes the user's emotional state, and adjusts notifications accordingly.

[0576] "Emotional state" refers to the psychological state that can be inferred from the user's tone of voice, facial expressions, and speed of operation.

[0577] An "alert" is a warning message generated when a high risk is identified.

[0578] "Instructions" refer to information provided by user equipment and communicated to on-site workers to prompt actions or responses.

[0579] This invention provides a system for real-time data acquisition, analysis, and notification coordination to improve safety and efficiency at construction sites. The system mainly consists of multiple "input devices," a "central device" (server) that analyzes the data, and a "user device" that receives notifications.

[0580] The terminal collects data such as video, audio, temperature, and vibration in real time through input devices installed on-site, such as video cameras and various sensors. This data acquisition is performed by multiple built-in sensor devices and camera systems. The terminal temporarily stores the continuously acquired data and efficiently sends it to the central device based on priority according to requirements.

[0581] The server functions as a central device, using machine learning algorithms such as TensorFlow and PyTorch to analyze the received data. This analysis quantitatively assesses the current risk level by referencing past accident data. If a high risk is detected, an alert is immediately generated and notified to the user's device with a detailed explanation.

[0582] As soon as the user device receives an alert, its built-in emotion engine analyzes the user's tone of voice, facial expressions, and the speed of their actions. This allows the engine to recognize the user's emotional state and adjust the notification accordingly. For example, if the user is determined to be stressed, the alert will be adjusted to be concise and calming.

[0583] As a concrete example, consider a situation where abnormal vibrations are detected on-site. The terminal sends this abnormal data to the server, and if the server analyzes the risk and determines it to be high risk, the user can quickly learn about the situation. Based on this information, the user can issue accurate instructions to the site and avoid potential dangers. By designing such a system, safe and efficient construction site management can be achieved.

[0584] Examples of prompts that can be input to the generating AI model include, "Please explain the overview of data analysis methods in safety management systems for construction sites."

[0585] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0586] Step 1:

[0587] The terminal acquires diverse data such as video, audio, temperature, and vibration in real time through input devices installed on-site. Inputs include on-site sensor information and visual data. This data is temporarily stored within the terminal and organized based on its importance. This organized data is then prepared for analysis in the next processing step.

[0588] Step 2:

[0589] The terminal transmits the acquired data to the central server. The data held by the terminal is used as input. Data transmission is performed using a network-based communication protocol, with high-priority data being transmitted first. The server receives organized field data as output.

[0590] Step 3:

[0591] The server analyzes the received data and performs a risk assessment. Field data sent to the server is used as input. Machine learning algorithms such as TensorFlow and PyTorch are applied to this analysis, quantitatively evaluating the risk by comparing it with past accident data. The output generated in this process is data indicating the risk level at the site.

[0592] Step 4:

[0593] The server generates an alert based on the analysis results and notifies the user's device. Analyzed risk assessment data is used as input. Within the server, the alert generation logic runs, preparing alert information that includes detailed explanations. This alert information is then sent to the user's device as output.

[0594] Step 5:

[0595] The user device receives an alert, and the emotion engine evaluates the user's emotional state. Inputs include the received alert information and emotion-related data obtained from the user. The emotion engine analyzes the user's tone of voice, facial expressions, operation speed, etc., and evaluates stress levels, etc. Based on the results, adjusted notification information is generated as output.

[0596] Step 6:

[0597] The user receives coordinated notifications and then decides on and sends instructions to the field. The input is coordinated notification information. Based on this information, the user calmly and quickly provides specific instructions to the field workers. These instructions are then transmitted to the field as output information from the user.

[0598] (Application Example 2)

[0599] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0600] In on-site work, ensuring both safety and efficiency simultaneously is always required. However, safety and efficiency can be affected by the emotional state of workers, making recognition and countermeasures crucial. To properly solve this problem, a system is needed that can grasp the emotional state of workers in real time and adjust machine operations based on that information.

[0601] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0602] In this invention, the server includes means for acquiring field data from multiple input devices placed on-site, means for recognizing the emotional state of workers and adjusting the notification method in user devices, and means for generating instructions to optimize machine operation based on the adjusted notifications. This makes it possible to maximize on-site safety and efficiency while taking into account the emotional state of workers.

[0603] An "input device" is a device used to acquire on-site data in real time, and includes video cameras and various sensors.

[0604] A "central device" is a device that aggregates and analyzes acquired data, and has the function of assessing risk levels.

[0605] A "user device" is a device that receives alerts and instructions at the work site and provides information to facilitate the smooth progress of the work.

[0606] "Emotional state recognition" is the process of determining a worker's mental state by analyzing their voice tone and facial expressions.

[0607] "Adjusting notification methods" refers to a means of changing the format of alerts and information received according to the emotional state of the worker, in order to encourage appropriate responses.

[0608] "Optimizing machine operation" means adjusting instructions so that the machine operates most efficiently and safely, while also taking into account the emotional state of the operator.

[0609] This system utilizes multiple input devices, a central unit, and user devices. The server continuously acquires field data using video cameras and various sensors via Raspberry Pi and other devices. This includes collecting video, audio, temperature, and vibration data. Data transmitted from the input devices is transferred to the central unit in the cloud. The central unit uses machine learning algorithms, such as Amazon SageMaker, to assess risk levels.

[0610] User devices function as smartphones or tablets, receiving alerts from a central device. Furthermore, using OpenCV and Google Cloud AI services, the system recognizes the worker's emotional state in real time from their voice and facial expressions, and adjusts notification methods accordingly. For example, if a worker is determined to be in a high-stress state, the notification is adjusted to include more detailed content encouraging calm behavior.

[0611] For example, if the system detects abnormal vibrations, the server immediately analyzes the data and generates an alert if it determines the risk is high. Users receive this alert along with an evaluation by the emotion engine, and appropriate instructions are communicated to the site, improving work safety and efficiency. An example of a prompt to the generating AI model is, "Use the sensor data obtained in real time to evaluate the stress level of the work environment and propose safety measures based on the results."

[0612] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0613] Step 1:

[0614] The terminal uses multiple input devices to acquire real-time video, audio, temperature, and vibration data from the field. This input data includes information indicating changes and anomalies in the field's work environment. The terminal packages this data and transmits it to a central device.

[0615] Step 2:

[0616] The server, acting as a central device, receives data aggregated in the cloud and performs analysis using machine learning algorithms via Amazon SageMaker. The input is the field data acquired earlier. Based on this, the server evaluates the risk level and generates an alert if an anomaly is detected. The output includes the risk assessment results and the generated alerts.

[0617] Step 3:

[0618] The server sends the generated alerts to the user's device. The user's device is a smartphone or tablet, which can receive the alert and immediately understand the situation on-site. In this step, the alert content is directly transmitted to the user's device.

[0619] Step 4:

[0620] The user device uses OpenCV and Google Cloud's AI services to recognize the worker's emotional state from their voice and facial expressions. Input data includes information about the worker's actions and facial expressions. The user device analyzes this information and outputs an evaluation of the emotional state.

[0621] Step 5:

[0622] The user device adjusts the notification method based on the results of the emotional state assessment. For example, if the user is determined to be in a high-stress state, the notification content and format are adjusted to ensure that the information is conveyed in the most effective way for the worker. This output is the adjusted notification content.

[0623] Step 6:

[0624] The user communicates specific instructions to the field based on coordinated notifications. These instructions are optimized to enhance work safety and efficiency. As output, appropriately coordinated instructions are delivered to the field.

[0625] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0626] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0627] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0628] [Fourth Embodiment]

[0629] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0630] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0631] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0632] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0633] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0634] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0635] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0636] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0637] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0638] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0639] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0640] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0641] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0642] This invention aims to solve the problem of labor shortages at construction sites and improve safety. The system for this purpose consists of an advanced data collection and analysis system using multiple input devices and a central device installed at the site, as well as user devices.

[0643] Multiple input devices are installed at the site to continuously collect on-site data. These input devices include video cameras, temperature sensors, vibration sensors, and sound detection devices, enabling comprehensive monitoring of all situations. The collected data is initially filtered by terminals and transmitted to a central system according to its importance.

[0644] The server functions as a central device, analyzing transmitted data in real time. The server uses advanced algorithms to evaluate the data and quantify risk levels. This process incorporates machine learning techniques that reference historical data, enabling highly accurate risk assessments through predictions based on accumulated knowledge.

[0645] If a high risk is detected, the server immediately sends an alert to the user's device. This user device is used by on-site managers and supervisors, enabling real-time situation monitoring and immediate response.

[0646] Users can check notifications from the server, input instructions as needed, and transmit them to field workers via their terminals. Instructions are provided specifically via voice and display, and workers are required to act immediately in accordance with them.

[0647] For example, if the temperature rises abnormally at the site, the terminal immediately reports this anomaly to the central system, and the server assesses the level of risk based on past data. If a high risk is assessed, the server sends a system alert to the user's device, and the user instructs the site to take swift action. Through this entire process, it is possible to ensure safety at the site and improve work efficiency.

[0648] The following describes the processing flow.

[0649] Step 1:

[0650] The terminal activates various sensors and cameras installed on-site, acquiring video and sensor data in real time. This includes a variety of data such as temperature, vibration, and sound.

[0651] Step 2:

[0652] The terminal performs an initial filter on the acquired data to detect anomalies and unusual changes. This filtering prevents the transmission of unnecessary data, and only important data is extracted.

[0653] Step 3:

[0654] The terminal sends filtered data to a central server. Here, communication stability is verified, and data integrity is guaranteed.

[0655] Step 4:

[0656] The server processes the received data using an analysis algorithm to evaluate the situation on site. By using historical data and pattern recognition technology in the analysis, it performs a highly accurate risk assessment.

[0657] Step 5:

[0658] The server determines the risk level based on the data analysis results and generates alert messages as needed. In this process, it compares the situation to similar past situations if necessary and considers possible countermeasures.

[0659] Step 6:

[0660] The server sends an alert to the user's device. This alert contains detailed information about the risks at the site, allowing the user to quickly consider countermeasures.

[0661] Step 7:

[0662] After receiving an alert from the server, the user decides on a course of action and sends that action as an instruction command from their device to a terminal at the site. The instructions are specific and require immediate action at the site.

[0663] Step 8:

[0664] The terminal transmits instructions received from the user to on-site workers via voice or display. This allows workers to respond quickly based on the instructions.

[0665] Step 9:

[0666] The server confirms that the entire process has completed successfully and continues recording and analyzing data. This data will be used for future improvements and updates to predictive models.

[0667] (Example 1)

[0668] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0669] In construction sites, manufacturing plants, and other similar environments, there is a need to respond quickly and accurately to changes in the work environment and unforeseen circumstances, thereby improving safety. However, conventional systems have struggled to efficiently collect and analyze individual data, conduct immediate risk assessments based on the results, and provide appropriate instructions to workers. Furthermore, low accuracy in risk prediction and delays in effective responses have been contributing to compromised safety.

[0670] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0671] In this invention, the server includes means for acquiring environmental data from multiple sensor devices placed on-site, means for analyzing the data transferred to a central device using advanced algorithms to perform a safety risk assessment, and means for performing risk prediction by utilizing machine learning technology and referring to past data. This enables rapid data collection and evaluation on-site, highly accurate risk prediction based on past data, and real-time instruction provision to workers.

[0672] A "sensor device" is a device that is placed to detect changes in the environment and collect data.

[0673] A "relay device" is a device that performs initial filtering of data acquired from sensor devices before transmitting it to the central device.

[0674] A "central system" is a computer system that functions to analyze received environmental data and perform safety risk assessments.

[0675] An "advanced algorithm" is a computational method used to analyze complex and diverse data and perform accurate risk assessments.

[0676] "Machine learning technology" is a technique that learns patterns based on past data and uses that to predict future risks.

[0677] "Risk assessment" is the process of quantifying and determining the degree of safety risks based on local situation data.

[0678] A "management device" is a terminal used by on-site managers to receive alerts notified from the central device.

[0679] A "worker" is a person responsible for carrying out actual work according to instructions at the work site.

[0680] A "generative AI model" is an artificial intelligence technology that generates countermeasures based on given data and prompts.

[0681] A "prompt statement" is an input command given to a generative AI model to cause it to generate specific information.

[0682] To implement the invention, this system is realized in the following form.

[0683] Multiple sensor devices are deployed at the site to acquire environmental data such as temperature, vibration, visual information, and sound. This data is transmitted to a central device via a relay device. The relay device first evaluates the importance of the data, performs initial filtering, and sends only the necessary data to the central device.

[0684] The server functions as a central device, analyzing the received data. Here, advanced algorithms are used to analyze the data in detail and assess safety risks. Furthermore, machine learning techniques are utilized to predict future risks by referencing past patterns. This technique, where the server uses generative AI models to predict risks, is a particularly crucial process.

[0685] Based on the risk level assessment, the server sends alerts to the management device in real time. These alerts include emergency response measures tailored to the situation on site, and, in particular, instructions for immediate action when necessary.

[0686] The user uses a management device to receive notifications from the server, consider countermeasures as needed, and communicate specific instructions to the workers. For example, one possible instruction might be, "All workers, immediately move to the designated evacuation area."

[0687] As a concrete example, consider a situation where the temperature rises rapidly at a construction site. In this case, a sensor device detects the anomaly, and the information is transmitted to the central device via a relay device. The server, referring to past data, immediately sends an alert to the management device if it determines that the risk level is high. Based on this alert, the user can quickly issue instructions to the workers.

[0688] An example of a prompt using a generative AI model is a text-based instruction such as, "Assess the risk based on the current temperature data and generate recommended countermeasures." This prompt instructs the AI ​​model to perform appropriate data processing and generate countermeasures.

[0689] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0690] Step 1:

[0691] The terminal acquires environmental data from multiple sensor devices placed on-site. This includes video data from video cameras, temperature data from temperature sensors, vibration data from vibration sensors, and audio data from sound detection devices. The terminal receives this data as input and performs initial filtering based on importance. As output, filtered important data is generated and sent to the relay device.

[0692] Step 2:

[0693] Upon receiving filtered data from a terminal, the server begins data analysis using advanced algorithms. It analyzes the received data in detail and conducts a safety-related risk assessment. Using machine learning techniques, the server references past data and calculates a risk assessment score based on the current situation. This process involves identifying anomaly patterns and performing calculations to determine urgency.

[0694] Step 3:

[0695] The server generates an alert based on the risk assessment score obtained after analysis. This alert is sent to the management device in real time. The alert generated as output includes emergency response measures tailored to the specific situation on site. For example, it may include instructions such as, "The temperature is rising abnormally in Area B, so all workers must evacuate."

[0696] Step 4:

[0697] The user receives alerts from the server via a management device and reviews their contents. Based on the alert information received as input, the user quickly creates specific instructions if action is required. The user then transmits these instructions as output to field workers, prompting them to take action to ensure safety at the site. In some cases, the user may also optimize the instructions by using prompts generated by an AI model, taking into account the countermeasures suggested by the AI.

[0698] An example of a prompt message for a countermeasure proposed by the generative AI model is, "Assess the risk based on the current temperature data and generate recommended countermeasures." Through this process, rapid and accurate risk management is achieved throughout the entire system.

[0699] (Application Example 1)

[0700] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0701] In factories and work environments, labor shortages and ensuring safety are serious problems. Especially in workplaces where automated machinery predominates, human intervention is limited, making the detection of abnormalities and hazards, and rapid response, crucial. However, conventional systems have limitations in real-time capabilities and predictive accuracy, making it difficult to implement appropriate safety measures on-site. This invention aims to solve these problems.

[0702] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0703] In this invention, the server includes means for transmitting collected data to a central processing unit, and the central processing unit includes means for analyzing the collected data and evaluating the degree of risk at the work site, and means for instructing mechanical devices within the work space to take safe actions. This enables real-time risk detection and prediction, and rapid safety response.

[0704] An "information gathering device" refers to devices such as sensors and cameras installed to collect environmental data at a work site.

[0705] A "central processing unit" is a central computer or server used to analyze collected data and assess the level of risk.

[0706] "Analysis" refers to the process of extracting useful information from collected data, and specifically refers to calculations and analyses related to risk assessment.

[0707] "Evaluation" refers to quantifying and judging the degree of danger at a work site based on information obtained through analysis.

[0708] A "warning" is a notification generated based on evaluation results, and is a signal or message conveyed to workers or managers through an operating terminal.

[0709] An "operation terminal" is a device or interface used by users to receive warnings and transmit necessary instructions to the field.

[0710] An "order" is a specific instruction regarding actions and tasks at the work site, and is transmitted from the control terminal to personnel and machinery at the site.

[0711] "Mechanical equipment" refers to robots and other automation devices that perform specific tasks automatically at a work site.

[0712] "Safety actions" refer to the actions that a machine or device should take when a hazard is detected, and are the actions and processes that ensure the safety of people and the environment.

[0713] As an embodiment for carrying out the invention, a safety monitoring system for factories and work sites will be described. This system consists of an information gathering device, a central processing unit, and an operating terminal.

[0714] The information gathering device includes numerous sensors and cameras installed at the work site to collect temperature, vibration, or acoustic data in real time. The data collected by this device is transmitted to a central processing unit via a network.

[0715] The central processing unit (server) analyzes received data using machine learning algorithms such as TensorFlow. Based on past data, it assesses the level of risk from the current data and makes a quantitative judgment. Based on the analysis results, it immediately generates a warning and notifies the operating terminal.

[0716] The control terminal is a device used by managers and workers within the factory, allowing them to check warnings in real time. Upon receiving a warning, the user immediately transmits the necessary commands to the machinery. The machinery then follows the received commands and implements safe actions, thereby ensuring safety on site.

[0717] For example, if a temperature sensor detects an abnormal value and the central processing unit assesses this as a high risk, a warning is sent to the administrator via the control terminal. Subsequently, the machine automatically stops operating and is instructed to move to a safe location.

[0718] Example prompt message:

[0719] The factory temperature sensor has detected an abnormal reading. Assess the level of risk, issue an immediate notification, and instruct the robot to move safely.

[0720] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0721] Step 1:

[0722] The information gathering device continuously collects data at the work site.

[0723] (Input: Sensor data such as temperature, vibration, and sound)

[0724] The information gathering device acquires various data from the field environment in real time and performs initial data processing. Afterward, important data is transmitted to the central processing unit via the network.

[0725] (Output: Filtered dataset)

[0726] Step 2:

[0727] The server analyzes the data it receives.

[0728] (Input: Filtered dataset)

[0729] The server uses TensorFlow algorithms to analyze incoming data, and during this process, it evaluates the level of risk by comparing it to past data. This process utilizes generative AI models to perform risk assessment in real time.

[0730] (Output: Risk assessment value)

[0731] Step 3:

[0732] The server generates a warning based on the risk assessment value and notifies the operating terminal.

[0733] (Input: Risk assessment value)

[0734] If a high risk is detected, the server generates a warning message and promptly sends it to the user's terminal via the network. This message includes specific instructions to enable the user to take immediate action.

[0735] (Output: Warning message)

[0736] Step 4:

[0737] The operating terminal receives a warning message and notifies the user.

[0738] (Input: Warning message)

[0739] When the operating terminal receives a warning, it immediately displays the content on the terminal screen and emits an alert sound if necessary. The user checks the notification and takes appropriate action.

[0740] (Output: Notification to the user)

[0741] Step 5:

[0742] The user instructs the machine to perform safety actions via a control terminal.

[0743] (Input: Warning content and user response instructions)

[0744] The user inputs a command from the control terminal, which is then transmitted to the machinery at the work site. Based on the received instructions, the machinery begins safe operation.

[0745] (Output: Instruction signal to the machine)

[0746] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0747] This invention provides a system equipped with emotion recognition capabilities to improve safety and efficiency at construction sites. This system consists of an input device that monitors site conditions in real time, a central device (server) that analyzes the data, and an emotion engine that recognizes the user's emotions and provides information according to their state.

[0748] First, multiple input devices installed on-site continuously acquire on-site data in real time. This includes acquiring video, audio, temperature, and vibration data from video cameras and various sensors. Terminals analyze this data and transmit the necessary information to a central device.

[0749] The server functions as a central device, analyzing the acquired data. Data analysis utilizes machine learning algorithms that refer to past accident data and quantitatively assess risk. If a high risk is detected, the server generates an alert and notifies the user device.

[0750] The user device is equipped with an emotion engine that analyzes the user's tone of voice, facial expressions, and speed of operation to recognize the user's emotional state. When it receives an alert from the server, the emotion engine assesses the user's stress level and adjusts the notification method accordingly. For example, if the system determines that the user is in a high-stress state, the notification will be adjusted to be more detailed and calming.

[0751] Users receive notifications based on their analyzed emotional state and issue instructions to the field as needed. These instructions are provided in a format easily understood by workers, taking into account feedback from the emotion engine. Information regarding the user's emotional state is fed back to the server and stored in a database. This data is used for future system improvements and instruction optimization.

[0752] For example, if abnormal vibrations are detected on-site, the terminal immediately reports the data to the server, which assesses the risk and generates an alert. The user receives this alert along with an evaluation from the emotion engine, enabling them to give calm instructions to the site. This process maximizes the safety and efficiency of the work.

[0753] The following describes the processing flow.

[0754] Step 1:

[0755] The terminal activates various sensors and cameras installed on-site, acquiring video data and sensor data (temperature, vibration, sound, etc.) in real time. This makes it possible to detect any environmental changes at the site.

[0756] Step 2:

[0757] The terminal performs initial filtering, narrowing down the acquired data to only the essentials. This process extracts unusual changes and data exceeding thresholds, ensuring that only critical information is sent to the central device.

[0758] Step 3:

[0759] The server receives data sent from the terminal and performs data analysis using machine learning algorithms. This analysis quantitatively evaluates the risk level at the site and detects signs of high risk.

[0760] Step 4:

[0761] Based on the risk assessment results, the server generates an alert message appropriate to the user's emotional state. If the risk is higher than historical data, an alert is immediately generated and notified to the user's device.

[0762] Step 5:

[0763] The user device operates an emotion engine that analyzes the user's emotional state based on their actions and voice data. It evaluates the user's stress level and current emotions, and adjusts the notification method and content accordingly.

[0764] Step 6:

[0765] The user considers feedback from the emotion engine, reviews the alerts from the server, and consciously considers instructions for the field. The instructions are specific and adjusted to a format that is easy for workers to understand.

[0766] Step 7:

[0767] The user's instructions are transmitted to a terminal at the site via the user's device. The terminal then relays these instructions to the on-site worker via voice or display. The worker then immediately performs the task or takes action according to these instructions.

[0768] Step 8:

[0769] The server monitors how instructions are executed on-site and feeds the results back into the database. This feedback information is used to improve the system and optimize instructions in the future.

[0770] (Example 2)

[0771] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0772] Conventional systems aimed at improving safety and efficiency at construction sites have limitations in real-time risk assessment and information provision, and are particularly poor at flexibly responding to human factors such as emotional states. Furthermore, it has been difficult to acquire data and transmit appropriate and timely instructions to the site based on the analysis results, so safety and work efficiency have not been fully achieved.

[0773] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0774] In this invention, the server includes means for acquiring various types of data in real time from multiple input devices placed on-site, means for quantitatively evaluating the risk level by comparing it with past accident data using a machine learning algorithm, and means for analyzing the emotional state of the user and adjusting the content of notifications. This enables flexible and effective risk assessment and notification adjustment that takes human factors into account, as well as the rapid transmission of on-site instructions.

[0775] An "input device" is a device installed on-site to acquire various types of data in real time.

[0776] A "central system" is a device that analyzes acquired data and evaluates risk levels by referring to past data.

[0777] A "machine learning algorithm" is a calculation method used to quantitatively evaluate risk by comparing it with past accident data.

[0778] A "user device" is a device that receives alerts, analyzes the user's emotional state, and adjusts notifications accordingly.

[0779] "Emotional state" refers to the psychological state that can be inferred from the user's tone of voice, facial expressions, and speed of operation.

[0780] An "alert" is a warning message generated when a high risk is identified.

[0781] "Instructions" refer to information provided by user equipment and communicated to on-site workers to prompt actions or responses.

[0782] This invention provides a system for real-time data acquisition, analysis, and notification coordination to improve safety and efficiency at construction sites. The system mainly consists of multiple "input devices," a "central device" (server) that analyzes the data, and a "user device" that receives notifications.

[0783] The terminal collects data such as video, audio, temperature, and vibration in real time through input devices installed on-site, such as video cameras and various sensors. This data acquisition is performed by multiple built-in sensor devices and camera systems. The terminal temporarily stores the continuously acquired data and efficiently sends it to the central device based on priority according to requirements.

[0784] The server functions as a central device, using machine learning algorithms such as TensorFlow and PyTorch to analyze the received data. This analysis quantitatively assesses the current risk level by referencing past accident data. If a high risk is detected, an alert is immediately generated and notified to the user's device with a detailed explanation.

[0785] As soon as the user device receives an alert, its built-in emotion engine analyzes the user's tone of voice, facial expressions, and the speed of their actions. This allows the engine to recognize the user's emotional state and adjust the notification accordingly. For example, if the user is determined to be stressed, the alert will be adjusted to be concise and calming.

[0786] As a concrete example, consider a situation where abnormal vibrations are detected on-site. The terminal sends this abnormal data to the server, and if the server analyzes the risk and determines it to be high risk, the user can quickly learn about the situation. Based on this information, the user can issue accurate instructions to the site and avoid potential dangers. By designing such a system, safe and efficient construction site management can be achieved.

[0787] Examples of prompts that can be input to the generating AI model include, "Please explain the overview of data analysis methods in safety management systems for construction sites."

[0788] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0789] Step 1:

[0790] The terminal acquires diverse data such as video, audio, temperature, and vibration in real time through input devices installed on-site. Inputs include on-site sensor information and visual data. This data is temporarily stored within the terminal and organized based on its importance. This organized data is then prepared for analysis in the next processing step.

[0791] Step 2:

[0792] The terminal transmits the acquired data to the central server. The data held by the terminal is used as input. Data transmission is performed using a network-based communication protocol, with high-priority data being transmitted first. The server receives organized field data as output.

[0793] Step 3:

[0794] The server analyzes the received data and performs a risk assessment. Field data sent to the server is used as input. Machine learning algorithms such as TensorFlow and PyTorch are applied to this analysis, quantitatively evaluating the risk by comparing it with past accident data. The output generated in this process is data indicating the risk level at the site.

[0795] Step 4:

[0796] The server generates an alert based on the analysis results and notifies the user's device. Analyzed risk assessment data is used as input. Within the server, the alert generation logic runs, preparing alert information that includes detailed explanations. This alert information is then sent to the user's device as output.

[0797] Step 5:

[0798] The user device receives an alert, and the emotion engine evaluates the user's emotional state. Inputs include the received alert information and emotion-related data obtained from the user. The emotion engine analyzes the user's tone of voice, facial expressions, operation speed, etc., and evaluates stress levels, etc. Based on the results, adjusted notification information is generated as output.

[0799] Step 6:

[0800] The user receives coordinated notifications and then decides on and sends instructions to the field. The input is coordinated notification information. Based on this information, the user calmly and quickly provides specific instructions to the field workers. These instructions are then transmitted to the field as output information from the user.

[0801] (Application Example 2)

[0802] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0803] In on-site work, ensuring both safety and efficiency simultaneously is always required. However, safety and efficiency can be affected by the emotional state of workers, making recognition and countermeasures crucial. To properly solve this problem, a system is needed that can grasp the emotional state of workers in real time and adjust machine operations based on that information.

[0804] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0805] In this invention, the server includes means for acquiring field data from multiple input devices placed on-site, means for recognizing the emotional state of workers and adjusting the notification method in user devices, and means for generating instructions to optimize machine operation based on the adjusted notifications. This makes it possible to maximize on-site safety and efficiency while taking into account the emotional state of workers.

[0806] An "input device" is a device used to acquire on-site data in real time, and includes video cameras and various sensors.

[0807] A "central device" is a device that aggregates and analyzes acquired data, and has the function of assessing risk levels.

[0808] A "user device" is a device that receives alerts and instructions at the work site and provides information to facilitate the smooth progress of the work.

[0809] "Emotional state recognition" is the process of determining a worker's mental state by analyzing their voice tone and facial expressions.

[0810] "Adjusting notification methods" refers to a means of changing the format of alerts and information received according to the emotional state of the worker, in order to encourage appropriate responses.

[0811] "Optimizing machine operation" means adjusting instructions so that the machine operates most efficiently and safely, while also taking into account the emotional state of the operator.

[0812] This system utilizes multiple input devices, a central unit, and user devices. The server continuously acquires field data using video cameras and various sensors via Raspberry Pi and other devices. This includes collecting video, audio, temperature, and vibration data. Data transmitted from the input devices is transferred to the central unit in the cloud. The central unit uses machine learning algorithms, such as Amazon SageMaker, to assess risk levels.

[0813] User devices function as smartphones or tablets, receiving alerts from a central device. Furthermore, using OpenCV and Google Cloud AI services, the system recognizes the worker's emotional state in real time from their voice and facial expressions, and adjusts notification methods accordingly. For example, if a worker is determined to be in a high-stress state, the notification is adjusted to include more detailed content encouraging calm behavior.

[0814] For example, if the system detects abnormal vibrations, the server immediately analyzes the data and generates an alert if it determines the risk is high. Users receive this alert along with an evaluation by the emotion engine, and appropriate instructions are communicated to the site, improving work safety and efficiency. An example of a prompt to the generating AI model is, "Use the sensor data obtained in real time to evaluate the stress level of the work environment and propose safety measures based on the results."

[0815] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0816] Step 1:

[0817] The terminal uses multiple input devices to acquire real-time video, audio, temperature, and vibration data from the field. This input data includes information indicating changes and anomalies in the field's work environment. The terminal packages this data and transmits it to a central device.

[0818] Step 2:

[0819] The server, acting as a central device, receives data aggregated in the cloud and performs analysis using machine learning algorithms via Amazon SageMaker. The input is the field data acquired earlier. Based on this, the server evaluates the risk level and generates an alert if an anomaly is detected. The output includes the risk evaluation result and the generated alert.

[0820] Step 3:

[0821] The server sends the generated alerts to the user's device. The user's device is a smartphone or tablet, which can receive the alert and immediately understand the situation on-site. In this step, the alert content is directly transmitted to the user's device.

[0822] Step 4:

[0823] The user device uses OpenCV and Google Cloud's AI services to recognize the worker's emotional state from their voice and facial expressions. Input data includes information about the worker's actions and facial expressions. The user device analyzes this information and outputs an evaluation of the emotional state.

[0824] Step 5:

[0825] The user device adjusts the notification method based on the results of the emotional state assessment. For example, if the user is determined to be in a high-stress state, the notification content and format are adjusted to ensure that the information is conveyed in the most effective way for the worker. This output is the adjusted notification content.

[0826] Step 6:

[0827] The user communicates specific instructions to the field based on coordinated notifications. These instructions are optimized to enhance work safety and efficiency. As output, appropriately coordinated instructions are delivered to the field.

[0828] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0829] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0830] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0831] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0832] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0833] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0834] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0835] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0836] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0837] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0838] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0839] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0840] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0841] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0842] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0843] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0844] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0845] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0846] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0847] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0848] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0849] The following is further disclosed regarding the embodiments described above.

[0850] (Claim 1)

[0851] A means of acquiring on-site data using multiple input devices placed on-site,

[0852] Means for transmitting acquired data to a central device,

[0853] In the central system, there are means for analyzing acquired data and evaluating the risk level at the site,

[0854] A means for generating alerts based on evaluation results and notifying user devices from the central device,

[0855] A system that includes means for transmitting instructions provided by user equipment to the field.

[0856] (Claim 2)

[0857] The system according to claim 1, which performs risk prediction based on past data when generating an alert.

[0858] (Claim 3)

[0859] The system according to claim 1, which provides real-time instructions to on-site workers based on the results of data analysis.

[0860] "Example 1"

[0861] (Claim 1)

[0862] A means of acquiring environmental data using multiple sensor devices placed on-site,

[0863] A means for transmitting the acquired data to a relay device and performing initial filtering,

[0864] A means for transferring filtered data from a relay device to a central device,

[0865] In the central device, there is a means for analyzing the transferred data using advanced algorithms and conducting a safety risk assessment,

[0866] A method for performing risk prediction by referencing past data using machine learning technology,

[0867] A means for automatically generating alerts based on evaluation results and notifying the management device from the central device in real time,

[0868] A system that includes means for transmitting instructions provided on a management device to on-site workers.

[0869] (Claim 2)

[0870] The system according to claim 1, which uses a generated AI model during the data analysis process to perform highly accurate risk prediction.

[0871] (Claim 3)

[0872] The system according to claim 1, which provides identified instructions to workers in real time and proposes countermeasures in accordance with risk assessment.

[0873] "Application Example 1"

[0874] (Claim 1)

[0875] A means of collecting environmental data using multiple information gathering devices placed at the work site,

[0876] A means for transmitting the collected data to a central processing unit,

[0877] In the central processing unit, there is a means for analyzing the collected data and evaluating the degree of risk at the work site,

[0878] A means for generating a warning based on the evaluation and notifying the operating terminal from the central processing unit,

[0879] A means of transmitting commands provided from the control terminal to the work site,

[0880] A system that includes means for instructing mechanical devices within a workspace to take safe actions.

[0881] (Claim 2)

[0882] The system according to claim 1, which performs risk prediction based on past data when generating a warning.

[0883] (Claim 3)

[0884] The system according to claim 1, which provides immediate instructions to personnel at the work site based on the results of data analysis.

[0885] "Example 2 of combining an emotion engine"

[0886] (Claim 1)

[0887] A means of acquiring various types of data in real time using multiple input devices placed on-site,

[0888] A means for temporarily holding acquired data in a central device and transmitting it based on priority,

[0889] In the central system, a means is provided to analyze data using machine learning algorithms and quantitatively evaluate the risk level by comparing it with past accident data,

[0890] A means of immediately generating an alert based on the evaluation results and notifying the user device with a detailed explanation,

[0891] A means of analyzing the emotional state of users and adjusting notification content,

[0892] A system that includes means for transmitting instructions provided by user equipment to the field in an easily understandable format.

[0893] (Claim 2)

[0894] The system according to claim 1, which performs quantitative risk prediction based on past data when generating an alert.

[0895] (Claim 3)

[0896] The system according to claim 1, which provides real-time instructions to field workers to encourage them to remain calm and act quickly, based on data analysis and the emotional state of the user.

[0897] "Application example 2 when combining with an emotional engine"

[0898] (Claim 1)

[0899] A means of acquiring on-site data using multiple input devices placed on-site,

[0900] Means for transmitting acquired data to a central device,

[0901] In the central system, there are means for analyzing acquired data and evaluating the risk level at the site,

[0902] A means for generating alerts based on evaluation results and notifying user devices from the central device,

[0903] A means of transmitting instructions provided by the user device to the field,

[0904] A means for recognizing the emotional state of the worker and adjusting the notification method in the user device,

[0905] A system including means for generating instructions to optimize the operation of a machine based on adjusted notifications.

[0906] (Claim 2)

[0907] The system according to claim 1, which performs risk prediction based on past data when generating an alert.

[0908] (Claim 3)

[0909] The system according to claim 1, which provides real-time instructions to on-site workers based on the results of data analysis. [Explanation of Symbols]

[0910] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of acquiring on-site data using multiple input devices placed on-site, Means for transmitting acquired data to a central device, In the central system, there are means for analyzing acquired data and evaluating the risk level at the site, A means for generating alerts based on evaluation results and notifying user devices from the central device, A system that includes means for transmitting instructions provided by user equipment to the field.

2. The system according to claim 1, which performs risk prediction based on past data when generating an alert.

3. The system according to claim 1, which provides real-time instructions to on-site workers based on the results of data analysis.

Images