system

Abstract

We provide the system. [Solution] Means for acquiring data from unmanned aerial vehicles, ground information gathering devices, and worker equipment, A means of integrating acquired data to perform risk assessment, A means for notifying workers of a warning based on the results of the risk assessment, A means for collecting and integrating biometric and location information of workers, A means for performing real-time data analysis and communicating the analysis results to the worker, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, the method including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In a work environment with risks such as a construction site, workers are constantly exposed to the risk of accidents. In particular, it is difficult to grasp the surrounding situation in real time and ensure safety, so it is not easy to prevent accidents from occurring. This situation may result in the safety of workers not being ensured and may cause significant damage to individuals and organizations. Therefore, there is a need for new technical means to dramatically improve safety at the work site.

Means for Solving the Problems

[0005] This invention provides a system that acquires data from drones, ground-based imaging devices, and worker devices, and integrates this data to assess risks. Specifically, it has the function of analyzing the acquired data in real time and providing immediate warnings to workers. This makes it possible to predict dangerous situations in the work environment and respond quickly. Furthermore, the worker device is equipped with an output device for displaying warnings, so that workers can immediately receive safety instructions. In addition, the system uses video data from drones and ground-based imaging devices to accurately determine the degree of danger in the work environment and comprehensively ensure worker safety.

[0006] A "drone" is an unmanned aerial vehicle that flies through the air to take photographs and collect information.

[0007] A "ground-based imaging device" is a device installed on the ground that records images and videos of the surrounding environment.

[0008] A "worker's device" is a device worn and used by workers at a construction site to transmit and receive data and display warnings.

[0009] "Means of acquiring data" refers to technical means of collecting necessary information using various sensors and communication devices.

[0010] "Risk assessment" is the process of analyzing the degree of hazards in the work environment based on acquired data.

[0011] "Means of providing warnings" are methods for ensuring safety by communicating warnings and instructions to workers in real time. [Brief explanation of the drawing]

[0012] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3]It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Embodiments for Carrying Out the Invention

[0013] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described according to the accompanying drawings.

[0014] First, the language used in the following description will be explained.

[0015] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0016] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0017] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0018] In the following embodiments, the numbered communication I / F (Interface) is an interface that includes a communication processor and an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark), etc.

[0019] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0020] [First Embodiment]

[0021] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0022] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0023] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0024] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0025] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0026] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0027] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0028] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0029] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0030] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0031] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0032] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0033] This invention is a system for improving safety at construction sites, and it enables real-time risk assessment and warning provision through data acquisition and analysis using drones, ground imaging devices, and worker equipment. The embodiments thereof are described below.

[0034] The server receives video data transmitted from drones and ground-based cameras. This enables wide-area environmental monitoring and aims to identify hazards that cannot be captured from a specific viewpoint. In addition, biometric and location information is transmitted from worker devices, and the server analyzes this data in an integrated manner.

[0035] The server uses machine learning algorithms to compare this data with historical accident data in order to analyze it in real time. This process is regularly updated, allowing it to adapt to changing field conditions.

[0036] For example, if a worker is performing work at height, the server checks the condition of the scaffolding and equipment using video from a drone. It monitors the surrounding movements and layout using data from ground cameras and evaluates heart rate, abnormal posture, and other factors using sensor data from the worker's equipment. Based on this, the server makes a comprehensive assessment of the site situation and immediately generates a warning if the risk increases.

[0037] The terminal has the function of notifying workers of warnings generated by the server. These notifications are sent in various forms, such as voice instructions, screen displays, and vibrations, and are designed to allow workers to take appropriate action immediately.

[0038] The user, as the worker, is required to receive warnings and instructions from the terminal and promptly implement safe work procedures. This makes it possible to prevent dangerous situations and significantly improve the overall safety of the work environment.

[0039] The following describes the processing flow.

[0040] Step 1:

[0041] The server receives aerial video data from drones and video data from ground-based cameras. This allows it to acquire visual information about the overall work environment and specific areas.

[0042] Step 2:

[0043] The server receives location and biometric information from the worker's device. This makes it possible to understand the worker's current status and location.

[0044] Step 3:

[0045] The server preprocesses all received data. Video data is broken down frame by frame, and sensor data is filtered to remove noise.

[0046] Step 4:

[0047] The server inputs pre-processed data into a machine learning algorithm to perform a real-time risk assessment. This process predicts the level of risk in the current work environment by referring to past accident data.

[0048] Step 5:

[0049] Based on the risk assessment results, the server generates warnings about potential hazards that workers may face. Specifically, these include precautions to take and emergency response measures in the event of an imminent hazard.

[0050] Step 6:

[0051] The terminal receives warnings sent from the server and notifies the worker visually and audibly. This includes displaying warning messages on the screen, providing audio alerts, and vibrating warnings.

[0052] Step 7:

[0053] The user (worker) will check the warnings from the terminal and take the necessary actions to continue working safely. For example, they may evacuate from dangerous areas or recheck their protective equipment.

[0054] (Example 1)

[0055] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0056] Ensuring worker safety in real time at construction sites is extremely important. However, current systems suffer from delays in data collection and difficulty in rapidly assessing risks in response to environmental changes. Furthermore, they lack the accuracy and timeliness of warnings, making it difficult for workers to make immediate decisions to avoid danger.

[0057] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0058] In this invention, the server includes means for receiving multiple types of information from a data acquisition device, means for integrating the acquired information and pre-processing it using data processing technology, and means for evaluating the degree of risk based on the pre-processed information. This enables real-time, highly accurate risk assessment and rapid warning generation.

[0059] A "data acquisition device" refers to hardware and software used to collect multiple types of information and transmit it within a system.

[0060] "Information integration" is the process of converting different types of data into a single, analyzable format to streamline processing.

[0061] "Data processing technology" refers to the algorithms and computational methods used to analyze received data, which are used for data preprocessing and analysis.

[0062] "Preprocessing" refers to a series of operations to prepare data for analysis, such as removing noise and filling in missing data.

[0063] "Assessing the level of risk" is an analytical process conducted to determine the risks in the current work environment based on collected data, with the aim of ensuring safety.

[0064] "Warning generation" is a process carried out to notify workers when a hazard is detected, and is a means of prompting appropriate action.

[0065] This system is designed to improve safety at construction sites. First, the server receives video data from drones and ground cameras. This allows the server to monitor the site environment over a wide area and identify hazards in real time. In addition, biometric information (e.g., heart rate and body temperature) and location information are collected from worker devices. This data is integrated and analyzed on the server.

[0066] The server utilizes advanced data processing technology to pre-process all received data. Specifically, it employs image processing algorithms to recognize workers and obstacles from video data. Furthermore, for biometric information, statistical methods are used to filter out abnormal values. This enables early detection of anomalies.

[0067] Next, the server uses the pre-processed data to assess the level of risk. It employs machine learning algorithms to analyze the similarities between past accident data and the current situation. This process enables highly accurate assessment of on-site risks and real-time warning generation.

[0068] The terminal's role is to notify workers of warnings sent from the server. These notifications are provided in the form of voice instructions, screen displays, vibrations, etc., allowing workers to immediately detect danger and take action.

[0069] The user, the worker, is required to receive instructions from the terminal and take prompt and appropriate action. This helps prevent dangerous situations and ensures a safe working environment.

[0070] As a specific example, in a scenario involving work at height, the server monitors the worker's safety equipment using drone footage and observes surrounding movements using data from ground-based cameras. It then immediately issues a warning if sensor data from the worker's equipment indicates an anomaly.

[0071] An example of a prompt for a generating AI model is, "Explain how to generate and communicate a warning if a worker performing work at height may not be wearing a helmet." This prompt allows the AI ​​model to provide specific risk avoidance steps.

[0072] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0073] Step 1:

[0074] The server receives video data from drones and ground cameras, as well as biometric and location information from worker equipment. It takes real-time data streams from these devices as input and generates standardized datasets as output. This standardizes different data formats to facilitate subsequent processing.

[0075] Step 2:

[0076] The server integrates the received data and preprocesses it using data processing techniques. Specifically, video data is processed using image processing algorithms to clearly identify workers and equipment. Biometric information is filtered using statistical methods to detect anomalies. The input is an integrated dataset, and the output is clean, analyzable data. This allows only important information to be extracted, enabling efficient analysis.

[0077] Step 3:

[0078] The server assesses the level of risk based on pre-processed data. Using machine learning algorithms, it compares past accident data with the current situation to determine the risk level. Clean data and past accident data are used as input, and the risk assessment results are obtained as output. This quantifies on-site safety and sets criteria for determining the next course of action.

[0079] Step 4:

[0080] The server generates warnings based on the risk assessment results. If the risk level is high, it immediately creates a warning message and sends it to the terminal. The risk assessment results are used as input, and a specific warning message is generated as output. This ensures that workers can respond to risks quickly.

[0081] Step 5:

[0082] The terminal notifies the worker of warnings sent from the server. Warnings are provided in various forms, such as voice messages, screen displays, and vibrations. The input is a warning message, and the output is information presented in a format that the worker can recognize. This creates an environment where workers can make appropriate decisions immediately.

[0083] Step 6:

[0084] The user, the worker, receives a warning from the terminal and takes immediate, safe action. The input is the provided warning, and the output is an appropriate and safe response. This ensures safety on site and prevents hazards before they occur.

[0085] (Application Example 1)

[0086] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0087] To ensure a safe working environment at construction sites, it is necessary to grasp workers' biometric and location information, as well as site conditions in real time, and to quickly assess risks. However, conventional methods make it difficult to accurately and quickly predict sudden changes in the work environment or the occurrence of unexpected hazards, resulting in delays in providing appropriate warnings and instructions to workers.

[0088] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0089] In this invention, the server includes means for acquiring data from unmanned aerial vehicles, ground information gathering devices, and worker equipment; means for integrating the acquired data to perform a risk assessment; and means for notifying workers of warnings based on the results of the risk assessment. This makes it possible to integrate workers' biometric and location information at construction sites, assess risks in real time, and issue warnings quickly.

[0090] An "unmanned aerial vehicle" is an aircraft that flies remotely or autonomously and has the function of collecting information from the air.

[0091] A "ground information gathering device" is a device installed on the ground that collects visual information, audio information, and other data.

[0092] A "worker device" is a device that can be worn by a worker and has the function of acquiring the worker's biometric information and location information, and receiving warnings.

[0093] "Methods for integrating data and conducting risk assessments" refer to the process of combining data obtained from multiple sources and evaluating risk by comparing it with past cases and anticipated hazards.

[0094] "Means of notifying warnings" refers to methods such as sound, visual displays, or vibrations used to inform workers when a hazard is recognized.

[0095] This invention is a system that integrates data from unmanned aerial vehicles, ground information gathering devices, and worker equipment to enhance safety at construction sites. The following describes a specific configuration for realizing this system.

[0096] The server receives data transmitted from unmanned aerial vehicles (UAVs) and ground information gathering devices, allowing it to understand the working environment from different perspectives. This makes it possible to monitor in detail working environments at high altitudes and over wide areas that are not visible from the ground. Examples of hardware used include general-purpose UAVs and surveillance cameras.

[0097] Next, the server centrally manages the data and uses machine learning algorithms to perform risk assessments based on past accident data. The software used here includes TENSORFLOW® as a machine learning platform for data analysis and Apache® Kafka for real-time data streaming. If the analysis predicts a hazard, a warning is immediately sent to the worker's device.

[0098] Terminals (smartphones and smart glasses) receive warnings from the server and notify workers visually, audibly, and through vibration. Worker terminals collect biometric and location information from users and transmit it to the server in real time to support more detailed risk assessments.

[0099] For example, if strong winds are detected while a user is working at a high altitude, the system analyzes video data from the unmanned aerial vehicle and the user's biometric information to immediately warn the user of the effects of the strong winds and instruct them to temporarily suspend their work.

[0100] Using a generative AI model, it is possible to generate specific warning messages that respond to changes in the situation. The following example prompts can be used as a reference.

[0101] Example of a prompt:

[0102] "I want to develop an app that detects risks during construction work. It will analyze video from unmanned aerial vehicles and biometric data of workers in real time and have a function to issue immediate warnings. Please suggest specific warning message phrases."

[0103] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0104] Step 1:

[0105] Data is acquired from unmanned aerial vehicles (UAVs) and ground information gathering equipment. The server receives video data streamed in real time from UAVs and audio and image data obtained from ground information gathering equipment. The input consists of various data from UAVs and ground equipment, and the output is an integrated set of this data.

[0106] Step 2:

[0107] The server integrates the collected data and constructs a dataset for analysis. It performs data processing, converting multiple data sets with different types and formats into a unified format. The input is the integrated set obtained in step 1, and the output is the dataset formatted for analysis.

[0108] Step 3:

[0109] The server uses machine learning algorithms to analyze the integrated dataset and perform risk assessment. During this process, it compares the data with past accident data to evaluate future risks. The input is the dataset formatted in step 2, and the output is the risk assessment result.

[0110] Step 4:

[0111] Based on the risk assessment results, the server uses a generative AI model to generate specific warning messages. It utilizes prompts to select appropriate phrases for the situation. The input is the risk assessment results obtained in step 3, and the output is the generated warning message.

[0112] Step 5:

[0113] The terminal receives warning messages sent from the server and notifies the user. This notification can be made via sound, screen display, or vibration. The input is the warning message from the server, and the output is the notification to the user.

[0114] Step 6:

[0115] The user receives a warning from the terminal and takes appropriate action. During this process, the terminal verifies the user's biometric and location information and sends it to the server. Inputs consist of the warning message and on-site information from the user, while output is feedback data sent to the server.

[0116] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0117] The system according to the present invention integrates a drone, ground imaging equipment, worker equipment, and an emotion engine to improve safety in dangerous work environments such as construction sites. The server receives data from multiple information sources and uses this data to simultaneously assess the risks of the work environment and the emotional state of the workers.

[0118] The server integrates video data obtained from drones and ground cameras to check a wide range of environmental conditions. This allows for the detection of risks that cannot be captured by visual checks by workers. In addition, it analyzes location and biometric information acquired from worker devices to evaluate individual risk factors.

[0119] A distinctive feature of this system is that the server uses an emotion engine to analyze the worker's facial expressions and voice data to determine their current emotional state. For example, if a worker is extremely tense or stressed, the system can qualitatively adjust the content of warnings and add reassuring instructions.

[0120] The terminal notifies workers of warnings and emotion-based instructions generated by the server. These include specific safety instructions and motivational messages. If the terminal determines that a worker is experiencing stress, it will issue a corresponding alert.

[0121] The user, as the worker, checks notifications and warnings from the terminal and takes appropriate safety measures. By adhering to safe work procedures according to the notification content and acting on advice based on their emotional state, the worker can reduce the risk of accidents.

[0122] Thus, in order to implement the invention, it is necessary to optimally configure data communication between devices and to implement flexible risk management in response to changes in the field.

[0123] The following describes the processing flow.

[0124] Step 1:

[0125] The server receives aerial footage transmitted from the drone and video data from ground-based cameras. This allows for the collection of data that provides an overall understanding of the work environment.

[0126] Step 2:

[0127] The server receives location and biometric information from the worker's device. This includes data such as the worker's current location, body temperature, and heart rate.

[0128] Step 3:

[0129] The server uses an emotion engine to analyze the worker's facial expression and voice data to evaluate their emotional state. For example, it can detect stress and anxiety from facial expressions.

[0130] Step 4:

[0131] The server integrates all collected data and performs a real-time risk assessment. Here, machine learning algorithms are used to compare current data with past accident patterns and determine the current level of risk.

[0132] Step 5:

[0133] The server customizes the warnings provided to workers based on the results of risk assessment and emotional assessment. If high stress is detected, a relaxation message for safety confirmation is added in addition to the normal warning.

[0134] Step 6:

[0135] The terminal receives customized warning messages generated by the server and notifies the worker visually and audibly. For example, the message may be displayed on the terminal's screen, and an audible warning may be emitted from the speaker.

[0136] Step 7:

[0137] The user (worker) checks notifications from the terminal and takes necessary actions to maintain safe work practices. Based on warnings and advice, they adjust work procedures and avoid hazards.

[0138] (Example 2)

[0139] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0140] In traditional work environments, workers rely on their intuition and experience to ensure safety, but this method is prone to oversights. Furthermore, the lack of objective means to assess workers' mental stress and fatigue raises concerns about an increased risk of accidents. Therefore, there is a growing need for a system that can objectively and in real-time assess the risks of the work environment, understand workers' emotional states, and provide appropriate warnings and instructions.

[0141] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0142] In this invention, the server includes means for acquiring data from machines, means for integrating the acquired data to assess risks, and means for evaluating the emotional state of workers using sentiment analysis. This makes it possible to objectively assess risks in the work environment and provide appropriate warnings and instructions in real time according to the emotional state of workers.

[0143] "Machinery" refers to multi-purpose automated devices for acquiring data, including drones and ground-based imaging equipment.

[0144] "Data" refers to all information acquired by machines, including video data, location information, and biometric information.

[0145] "Means of assessing risk" refers to the process of integrating and analyzing acquired data to determine risks related to the work environment and worker safety.

[0146] "Emotional analysis" refers to a technology that analyzes a worker's facial expressions and voice data to evaluate the worker's emotional state at that moment.

[0147] "Notifications" refer to warnings, instructions, and general information provided to workers based on their evaluation results, and are delivered through the worker's device.

[0148] The invention will now be described in terms of its embodiments. This system operates by coordinating a server, terminals, and workers to improve safety in construction sites and hazardous work environments. The server first acquires video data from machines such as drones and ground cameras. This data is used to understand the entire work environment and assess potential risks.

[0149] The server then analyzes location and biometric information (e.g., heart rate and body temperature) acquired from worker equipment to assess the individual worker's risk status. This enables real-time monitoring of safety in the work environment. Generative AI models are used in this process to perform advanced data analysis.

[0150] Furthermore, the server uses emotion analysis technology to understand the worker's emotional state from their facial expressions and voice data. For example, if a worker inputs a voice message such as "I'm a little nervous today" into the terminal, the server analyzes the degree of that nervousness and prepares an appropriate message to reassure them.

[0151] The warnings and instructions generated in this way are communicated to workers via the terminal. The notifications include specific safety instructions and messages to boost work motivation. If the terminal detects that a worker is experiencing stress, it instantly provides appropriate warnings and encouraging messages.

[0152] The user, as the worker, can take safety measures in real time based on notifications from the terminal. For example, based on the user's voice input, a prompt such as "Assess the current work risk and generate appropriate safety instructions" is entered into the system. This prompt allows the system to quickly assess the risks the user faces and provide appropriate countermeasures.

[0153] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0154] Step 1:

[0155] The server acquires video data from drones and ground cameras. This input data includes a wide range of conditions at the work site. The server analyzes this video and uses a generated AI model to identify potential hazards such as terrain and obstacles. The output is a risk assessment of the site, including the identification of high-risk areas.

[0156] Step 2:

[0157] The server acquires location and biometric information from the worker's equipment. Input data includes the worker's current location, heart rate, body temperature, and other physical data. Based on this data, the server evaluates the individual worker's risk status. The data processing process includes detecting abnormal biometric parameters and analyzing risk factors. The output is the risk assessment result for each worker.

[0158] Step 3:

[0159] The server collects facial and voice data from workers to perform emotion analysis. Input data includes voice and video input from the worker's terminal. The server analyzes this data using a generative AI model to determine the worker's emotional state. Specific data calculations include creating stress level clusters from voice data and recognizing patterns in facial expressions. The output is an evaluation of the worker's emotional state.

[0160] Step 4:

[0161] Based on the analysis results from the previous steps, the server generates warnings and instructions for the worker. Here, it considers the combined risk assessment and emotional state to create appropriate safety instructions and encouraging messages. The input data is the result of the risk assessment and emotional state, and the output is the specific message content to be presented to the worker.

[0162] Step 5:

[0163] The terminal notifies workers of warnings and instructions received from the server. Input is message data from the server, and output is delivered to workers as audio alarms and visual displays. The terminal provides notifications to enable workers to respond immediately and supports them in carrying out safety instructions.

[0164] Step 6:

[0165] The user (worker) receives notifications from the terminal and takes safety measures based on them. The user confirms safe operating procedures and appropriately performs the instructed actions. The input is the content of the notification from the terminal, and the output is the user's specific safety actions. This allows workers to perform their duties while reducing the risk of accidents.

[0166] (Application Example 2)

[0167] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0168] Improving safety in on-site work and maintaining the mental health of workers are crucial challenges. In particular, appropriate responses are required that address the environmental hazards workers face and their individual emotional states. However, conventional systems have limitations in assessing the hazards of the work environment and judging workers' emotional states, making it difficult to provide appropriate safety instructions. This poses a risk of reduced work efficiency and safety.

[0169] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0170] In this invention, the server includes means for acquiring information from a drone, a ground imaging device, and equipment for workers; means for integrating the acquired information to perform a risk assessment; and means for evaluating the emotional state of workers using an emotion analysis device. This makes it possible to more accurately assess the hazards of the work environment and provide appropriate warnings and instructions tailored to the individual emotional state of workers.

[0171] A "drone" is an unmanned aerial system that flies through the air to take photographs and collect data.

[0172] A "ground-based camera" is a device used to capture images and photographs from the ground, playing a role in visually understanding the situation at the site.

[0173] "Worker equipment" refers to portable devices used by workers that have the function of collecting biometric and location information.

[0174] "Risk assessment" is the process of analyzing the degree of danger in the work environment based on collected information and considering countermeasures.

[0175] An "emotion analysis device" is a system that analyzes the worker's facial expressions, voice, and other biological responses to determine their emotional state.

[0176] "Warnings and instructions" refer to cautionary messages and work-related instructions given to workers based on risk assessments and emotional state analyses.

[0177] The system for implementing this invention primarily consists of a drone, a ground-based camera, and equipment for workers. The server is responsible for acquiring information from these devices. Specifically, it receives wide-area environmental images obtained from the drone and ground-based camera, as well as biometric and location information obtained from the worker's equipment. This information is integrated to comprehensively evaluate the level of risk at the site.

[0178] Furthermore, the server uses an emotion analysis device to analyze the emotional state of workers from their facial expressions and voice. This analysis utilizes an AI engine and applies software such as Affectiva. This technology makes it possible to understand the stress and tension levels of workers in real time.

[0179] The terminal notifies workers of warnings and instructions based on the results of risk assessment and sentiment analysis generated by the server. Messages are delivered visually or audibly via portable devices such as smart glasses. This allows workers to take safe actions at the appropriate time.

[0180] For example, if an increase in a worker's heart rate is detected while working at height, a message saying "Please take a break" will be displayed on the smart glasses. Furthermore, if the worker approaches a dangerous area, a message saying "Caution: Working at height. Please check your harness" will be displayed.

[0181] An example of a prompt message is as follows:

[0182] We are planning a smart glasses application designed to improve safety during work at heights. It will integrate video data from drones and ground cameras with worker biodata and use an algorithm to assess the worker's emotional state to provide appropriate safety procedures to the worker in real time.

[0183] This system aims to improve safety on site and reduce the psychological burden on workers.

[0184] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0185] Step 1:

[0186] The server receives video data as input from drones and ground cameras. This allows it to acquire video information to understand the overall picture of the work site. The server analyzes this video data frame by frame to assess the level of risk at the site. By applying algorithms to analyze specific areas within the video and identifying potential hazards, it performs an initial risk assessment.

[0187] Step 2:

[0188] The server acquires biometric and location information from the worker's equipment as input. This includes heart rate and stress level. The server analyzes this data and evaluates the worker's biological state in real time. An AI module is used for the analysis, and if an abnormal value is detected, a risk flag is set.

[0189] Step 3:

[0190] The server receives facial expression and voice data as input to evaluate the worker's emotional state using an emotion analysis device. Emotion analysis software (e.g., facial recognition technology and voice analysis engine) is used to determine the worker's level of tension and stress. Based on these results, the server determines the necessary actions.

[0191] Step 4:

[0192] The server integrates previously obtained risk assessment data and sentiment analysis results to generate warnings and instructions for each worker. This output includes specific safety instructions based on the worker's physical condition and the hazard level of the work environment. The generated instructions are designed to support safe work procedures.

[0193] Step 5:

[0194] The terminal receives instructions from the server and notifies the worker. Using visual devices such as smart glasses, necessary warnings and instructions are displayed on the screen. The worker reviews these notifications and modifies their actions according to the indicated safety procedures. This feedback loop improves safety and work efficiency.

[0195] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0196] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0197] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0198] [Second Embodiment]

[0199] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0200] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0201] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0202] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0203] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0204] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0205] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0206] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0207] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0208] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0209] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0210] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0211] This invention is a system for improving safety at construction sites, and it enables real-time risk assessment and warning provision through data acquisition and analysis using drones, ground imaging devices, and worker equipment. The embodiments thereof are described below.

[0212] The server receives video data transmitted from drones and ground-based cameras. This enables wide-area environmental monitoring and aims to identify hazards that cannot be captured from a specific viewpoint. In addition, biometric and location information is transmitted from worker devices, and the server analyzes this data in an integrated manner.

[0213] The server uses machine learning algorithms to compare this data with historical accident data in order to analyze it in real time. This process is regularly updated, allowing it to adapt to changing field conditions.

[0214] For example, if a worker is performing work at height, the server checks the condition of the scaffolding and equipment using video from a drone. It monitors the surrounding movements and layout using data from ground cameras and evaluates heart rate, abnormal posture, and other factors using sensor data from the worker's equipment. Based on this, the server makes a comprehensive assessment of the site situation and immediately generates a warning if the risk increases.

[0215] The terminal has the function of notifying workers of warnings generated by the server. These notifications are sent in various forms, such as voice instructions, screen displays, and vibrations, and are designed to allow workers to take appropriate action immediately.

[0216] The user, as the worker, is required to receive warnings and instructions from the terminal and promptly implement safe work procedures. This makes it possible to prevent dangerous situations and significantly improve the overall safety of the work environment.

[0217] The following describes the processing flow.

[0218] Step 1:

[0219] The server receives aerial video data from drones and video data from ground-based cameras. This allows it to acquire visual information about the overall work environment and specific areas.

[0220] Step 2:

[0221] The server receives location and biometric information from the worker's device. This makes it possible to understand the worker's current status and location.

[0222] Step 3:

[0223] The server preprocesses all received data. Video data is broken down frame by frame, and sensor data is filtered to remove noise.

[0224] Step 4:

[0225] The server inputs pre-processed data into a machine learning algorithm to perform a real-time risk assessment. This process predicts the level of risk in the current work environment by referring to past accident data.

[0226] Step 5:

[0227] Based on the risk assessment results, the server generates warnings about potential hazards that workers may face. Specifically, these include precautions to take and emergency response measures in the event of an imminent hazard.

[0228] Step 6:

[0229] The terminal receives warnings sent from the server and notifies the worker visually and audibly. This includes displaying warning messages on the screen, providing audio alerts, and vibrating warnings.

[0230] Step 7:

[0231] The user (worker) will check the warnings from the terminal and take the necessary actions to continue working safely. For example, they may evacuate from dangerous areas or recheck their protective equipment.

[0232] (Example 1)

[0233] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0234] Ensuring worker safety in real time at construction sites is extremely important. However, current systems suffer from delays in data collection and difficulty in rapidly assessing risks in response to environmental changes. Furthermore, they lack the accuracy and timeliness of warnings, making it difficult for workers to make immediate decisions to avoid danger.

[0235] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0236] In this invention, the server includes means for receiving multiple types of information from a data acquisition device, means for integrating the acquired information and pre-processing it using data processing technology, and means for evaluating the degree of risk based on the pre-processed information. This enables real-time, highly accurate risk assessment and rapid warning generation.

[0237] A "data acquisition device" refers to hardware and software used to collect multiple types of information and transmit it within a system.

[0238] "Information integration" is the process of converting different types of data into a single, analyzable format to streamline processing.

[0239] "Data processing technology" refers to the algorithms and computational methods used to analyze received data, which are used for data preprocessing and analysis.

[0240] "Preprocessing" refers to a series of operations to prepare data for analysis, such as removing noise and filling in missing data.

[0241] "Assessing the level of risk" is an analytical process conducted to determine the risks in the current work environment based on collected data, with the aim of ensuring safety.

[0242] "Warning generation" is a process carried out to notify workers when a hazard is detected, and is a means of prompting appropriate action.

[0243] This system is designed to improve safety at construction sites. First, the server receives video data from drones and ground cameras. This allows the server to monitor the site environment over a wide area and identify hazards in real time. In addition, biometric information (e.g., heart rate and body temperature) and location information are collected from worker devices. This data is integrated and analyzed on the server.

[0244] The server utilizes advanced data processing technology to pre-process all received data. Specifically, it employs image processing algorithms to recognize workers and obstacles from video data. Furthermore, for biometric information, statistical methods are used to filter out abnormal values. This enables early detection of anomalies.

[0245] Next, the server uses the pre-processed data to assess the level of risk. It employs machine learning algorithms to analyze the similarities between past accident data and the current situation. This process enables highly accurate assessment of on-site risks and real-time warning generation.

[0246] The terminal's role is to notify workers of warnings sent from the server. These notifications are provided in the form of voice instructions, screen displays, vibrations, etc., allowing workers to immediately detect danger and take action.

[0247] The user, the worker, is required to receive instructions from the terminal and take prompt and appropriate action. This helps prevent dangerous situations and ensures a safe working environment.

[0248] As a specific example, in a scenario involving work at height, the server monitors the worker's safety equipment using drone footage and observes surrounding movements using data from ground-based cameras. It then immediately issues a warning if sensor data from the worker's equipment indicates an anomaly.

[0249] An example of a prompt for a generating AI model is, "Explain how to generate and communicate a warning if a worker performing work at height may not be wearing a helmet." This prompt allows the AI ​​model to provide specific risk avoidance steps.

[0250] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0251] Step 1:

[0252] The server receives video data from drones and ground cameras, as well as biometric and location information from worker equipment. It takes real-time data streams from these devices as input and generates standardized datasets as output. This standardizes different data formats to facilitate subsequent processing.

[0253] Step 2:

[0254] The server integrates the received data and preprocesses it using data processing techniques. Specifically, video data is processed using image processing algorithms to clearly identify workers and equipment. Biometric information is filtered using statistical methods to detect anomalies. The input is an integrated dataset, and the output is clean, analyzable data. This allows only important information to be extracted, enabling efficient analysis.

[0255] Step 3:

[0256] The server assesses the level of risk based on pre-processed data. Using machine learning algorithms, it compares past accident data with the current situation to determine the risk level. Clean data and past accident data are used as input, and the risk assessment results are obtained as output. This quantifies on-site safety and sets criteria for determining the next course of action.

[0257] Step 4:

[0258] The server generates warnings based on the risk assessment results. If the risk level is high, it immediately creates a warning message and sends it to the terminal. The risk assessment results are used as input, and a specific warning message is generated as output. This ensures that workers can respond to risks quickly.

[0259] Step 5:

[0260] The terminal notifies the worker of warnings sent from the server. Warnings are provided in various forms, such as voice messages, screen displays, and vibrations. The input is a warning message, and the output is information presented in a format that the worker can recognize. This creates an environment where workers can make appropriate decisions immediately.

[0261] Step 6:

[0262] The user, the worker, receives a warning from the terminal and takes immediate, safe action. The input is the provided warning, and the output is an appropriate and safe response. This ensures safety on site and prevents hazards before they occur.

[0263] (Application Example 1)

[0264] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0265] To ensure a safe working environment at construction sites, it is necessary to grasp workers' biometric and location information, as well as site conditions in real time, and to quickly assess risks. However, conventional methods make it difficult to accurately and quickly predict sudden changes in the work environment or the occurrence of unexpected hazards, resulting in delays in providing appropriate warnings and instructions to workers.

[0266] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0267] In this invention, the server includes means for acquiring data from unmanned aerial vehicles, ground information gathering devices, and worker equipment; means for integrating the acquired data to perform a risk assessment; and means for notifying workers of warnings based on the results of the risk assessment. This makes it possible to integrate workers' biometric and location information at construction sites, assess risks in real time, and issue warnings quickly.

[0268] An "unmanned aerial vehicle" is an aircraft that flies remotely or autonomously and has the function of collecting information from the air.

[0269] A "ground information gathering device" is a device installed on the ground that collects visual information, audio information, and other data.

[0270] A "worker device" is a device that can be worn by a worker and has the function of acquiring the worker's biometric information and location information, and receiving warnings.

[0271] "Methods for integrating data and conducting risk assessments" refer to the process of combining data obtained from multiple sources and evaluating risk by comparing it with past cases and anticipated hazards.

[0272] "Means of notifying warnings" refers to methods such as sound, visual displays, or vibrations used to inform workers when a hazard is recognized.

[0273] This invention is a system that integrates data from unmanned aerial vehicles, ground information gathering devices, and worker equipment to enhance safety at construction sites. The following describes a specific configuration for realizing this system.

[0274] The server receives data transmitted from unmanned aerial vehicles (UAVs) and ground information gathering devices, allowing it to understand the working environment from different perspectives. This makes it possible to monitor in detail working environments at high altitudes and over wide areas that are not visible from the ground. Examples of hardware used include general-purpose UAVs and surveillance cameras.

[0275] Next, the server centrally manages the data and uses machine learning algorithms to perform risk assessments based on past accident data. The software used here includes TensorFlow as a machine learning platform for data analysis and Apache Kafka for real-time data streaming. If the analysis predicts a hazard, a warning is immediately sent to the worker's device.

[0276] Terminals (smartphones and smart glasses) receive warnings from the server and notify workers visually, audibly, and through vibration. Worker terminals collect biometric and location information from users and transmit it to the server in real time to support more detailed risk assessments.

[0277] For example, if strong winds are detected while a user is working at a high altitude, the system analyzes video data from the unmanned aerial vehicle and the user's biometric information to immediately warn the user of the effects of the strong winds and instruct them to temporarily suspend their work.

[0278] Using a generative AI model, it is possible to generate specific warning messages that respond to changes in the situation. The following example prompts can be used as a reference.

[0279] Example of a prompt:

[0280] "I want to develop an app that detects risks during construction work. It will analyze video from unmanned aerial vehicles and biometric data of workers in real time and have a function to issue immediate warnings. Please suggest specific warning message phrases."

[0281] The flow of the specific process in Application Example 1 will be described using FIG. 12.

[0282] Step 1:

[0283] Data is acquired from the unmanned aircraft and the ground information collection device. The server receives video data streamed in real time from the unmanned aircraft and audio and image data obtained from the ground information collection device. The input is various data from the unmanned aircraft and the ground device, and the output is an integrated set of these data.

[0284] Step 2:

[0285] The server integrates the collected data and constructs a data set for analysis. Data processing is performed to convert a plurality of data with different types and formats into a unified format. The input is the integrated set obtained in Step 1, and the output is a data set formatted for analysis.

[0286] Step 3:

[0287] The server analyzes the integrated data set using a machine learning algorithm and performs a risk assessment. At this time, it is also compared with past accident data, and data calculations are performed to evaluate future risks. The input is the data set formatted in Step 2, and the output is the risk assessment result.

[0288] Step 4:

[0289] Based on the result of the risk assessment, the server uses a generative AI model to generate a specific warning message. Utilize the prompt text to select phrases according to the situation. The input is the risk assessment result obtained in Step 3, and the output is the generated warning message.

[0290] Step 5:

[0291] The terminal receives warning messages sent from the server and notifies the user. This notification can be made via sound, screen display, or vibration. The input is the warning message from the server, and the output is the notification to the user.

[0292] Step 6:

[0293] The user receives a warning from the terminal and takes appropriate action. During this process, the terminal verifies the user's biometric and location information and sends it to the server. Inputs consist of the warning message and on-site information from the user, while output is feedback data sent to the server.

[0294] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0295] The system according to the present invention integrates a drone, ground imaging equipment, worker equipment, and an emotion engine to improve safety in dangerous work environments such as construction sites. The server receives data from multiple information sources and uses this data to simultaneously assess the risks of the work environment and the emotional state of the workers.

[0296] The server integrates video data obtained from drones and ground cameras to check a wide range of environmental conditions. This allows for the detection of risks that cannot be captured by visual checks by workers. In addition, it analyzes location and biometric information acquired from worker devices to evaluate individual risk factors.

[0297] A distinctive feature of this system is that the server uses an emotion engine to analyze the worker's facial expressions and voice data to determine their current emotional state. For example, if a worker is extremely tense or stressed, the system can qualitatively adjust the content of warnings and add reassuring instructions.

[0298] The terminal notifies the worker of the warnings and instructions generated by the server based on emotions. This includes specific safety instructions and encouraging messages to boost motivation. If the worker is judged to be feeling stressed, the terminal will issue corresponding warnings.

[0299] The worker, who is the user, checks the notifications and warnings from the terminal and takes appropriate safety measures. By following safe working procedures according to the notification content and implementing advice based on the emotional state, the worker can reduce the risk of accidents.

[0300] Thus, to implement the invention, it is necessary to optimally configure data communication between devices and perform flexible risk management according to changes on-site.

[0301] <0OO0950>The following explains the process flow.

[0302] Step 1:

[0303] The server receives the aerial video transmitted from the drone and the video data from the ground imaging device. This gathers data for understanding the overall situation of the working environment.

[0304] Step 2:

[0305] The server receives the location information and biometric information from the worker's device. This includes data such as the worker's current location, body temperature, and heart rate.

[0306] Step 3:

[0307] The server uses an emotion engine to analyze the worker's facial expression data and voice data and evaluate the worker's emotional state. For example, it is possible to detect stress and uneasiness from facial expressions. [[ID=—]]

[0308] Step 4:

[0309] The server integrates all collected data and performs a real-time risk assessment. Here, machine learning algorithms are used to compare current data with past accident patterns and determine the current level of risk.

[0310] Step 5:

[0311] The server customizes the warnings provided to workers based on the results of risk assessment and emotional assessment. If high stress is detected, a relaxation message for safety confirmation is added in addition to the normal warning.

[0312] Step 6:

[0313] The terminal receives customized warning messages generated by the server and notifies the worker visually and audibly. For example, the message may be displayed on the terminal's screen, and an audible warning may be emitted from the speaker.

[0314] Step 7:

[0315] The user (worker) checks notifications from the terminal and takes necessary actions to maintain safe work practices. Based on warnings and advice, they adjust work procedures and avoid hazards.

[0316] (Example 2)

[0317] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0318] In traditional work environments, workers rely on their intuition and experience to ensure safety, but this method is prone to oversights. Furthermore, the lack of objective means to assess workers' mental stress and fatigue raises concerns about an increased risk of accidents. Therefore, there is a growing need for a system that can objectively and in real-time assess the risks of the work environment, understand workers' emotional states, and provide appropriate warnings and instructions.

[0319] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0320] In this invention, the server includes means for acquiring data from machines, means for integrating the acquired data to assess risks, and means for evaluating the emotional state of workers using sentiment analysis. This makes it possible to objectively assess risks in the work environment and provide appropriate warnings and instructions in real time according to the emotional state of workers.

[0321] "Machinery" refers to multi-purpose automated devices for acquiring data, including drones and ground-based imaging equipment.

[0322] "Data" refers to all information acquired by machines, including video data, location information, and biometric information.

[0323] "Means of assessing risk" refers to the process of integrating and analyzing acquired data to determine risks related to the work environment and worker safety.

[0324] "Emotional analysis" refers to a technology that analyzes a worker's facial expressions and voice data to evaluate the worker's emotional state at that moment.

[0325] "Notifications" refer to warnings, instructions, and general information provided to workers based on their evaluation results, and are delivered through the worker's device.

[0326] The invention will now be described in terms of its embodiments. This system operates by coordinating a server, terminals, and workers to improve safety in construction sites and hazardous work environments. The server first acquires video data from machines such as drones and ground cameras. This data is used to understand the entire work environment and assess potential risks.

[0327] The server then analyzes location and biometric information (e.g., heart rate and body temperature) acquired from worker equipment to assess the individual worker's risk status. This enables real-time monitoring of safety in the work environment. Generative AI models are used in this process to perform advanced data analysis.

[0328] Furthermore, the server uses emotion analysis technology to understand the worker's emotional state from their facial expressions and voice data. For example, if a worker inputs a voice message such as "I'm a little nervous today" into the terminal, the server analyzes the degree of that nervousness and prepares an appropriate message to reassure them.

[0329] The warnings and instructions generated in this way are communicated to workers via the terminal. The notifications include specific safety instructions and messages to boost work motivation. If the terminal detects that a worker is experiencing stress, it instantly provides appropriate warnings and encouraging messages.

[0330] The user, as the worker, can take safety measures in real time based on notifications from the terminal. For example, based on the user's voice input, a prompt such as "Assess the current work risk and generate appropriate safety instructions" is entered into the system. This prompt allows the system to quickly assess the risks the user faces and provide appropriate countermeasures.

[0331] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0332] Step 1:

[0333] The server acquires video data from drones and ground cameras. This input data includes a wide range of conditions at the work site. The server analyzes this video and uses a generated AI model to identify potential hazards such as terrain and obstacles. The output is a risk assessment of the site, including the identification of high-risk areas.

[0334] Step 2:

[0335] The server acquires location and biometric information from the worker's equipment. Input data includes the worker's current location, heart rate, body temperature, and other physical data. Based on this data, the server evaluates the individual worker's risk status. The data processing process includes detecting abnormal biometric parameters and analyzing risk factors. The output is the risk assessment result for each worker.

[0336] Step 3:

[0337] The server collects facial and voice data from workers to perform emotion analysis. Input data includes voice and video input from the worker's terminal. The server analyzes this data using a generative AI model to determine the worker's emotional state. Specific data calculations include creating stress level clusters from voice data and recognizing patterns in facial expressions. The output is an evaluation of the worker's emotional state.

[0338] Step 4:

[0339] Based on the analysis results from the previous steps, the server generates warnings and instructions for the worker. Here, it considers the combined risk assessment and emotional state to create appropriate safety instructions and encouraging messages. The input data is the result of the risk assessment and emotional state, and the output is the specific message content to be presented to the worker.

[0340] Step 5:

[0341] The terminal notifies workers of warnings and instructions received from the server. Input is message data from the server, and output is delivered to workers as audio alarms and visual displays. The terminal provides notifications to enable workers to respond immediately and supports them in carrying out safety instructions.

[0342] Step 6:

[0343] The user (worker) receives notifications from the terminal and takes safety measures based on them. The user confirms safe operating procedures and appropriately performs the instructed actions. The input is the content of the notification from the terminal, and the output is the user's specific safety actions. This allows workers to perform their duties while reducing the risk of accidents.

[0344] (Application Example 2)

[0345] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0346] Improving safety in on-site work and maintaining the mental health of workers are crucial challenges. In particular, appropriate responses are required that address the environmental hazards workers face and their individual emotional states. However, conventional systems have limitations in assessing the hazards of the work environment and judging workers' emotional states, making it difficult to provide appropriate safety instructions. This poses a risk of reduced work efficiency and safety.

[0347] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0348] In this invention, the server includes means for acquiring information from a drone, a ground imaging device, and equipment for workers; means for integrating the acquired information to perform a risk assessment; and means for evaluating the emotional state of workers using an emotion analysis device. This makes it possible to more accurately assess the hazards of the work environment and provide appropriate warnings and instructions tailored to the individual emotional state of workers.

[0349] A "drone" is an unmanned aerial system that flies through the air to take photographs and collect data.

[0350] A "ground-based camera" is a device used to capture images and photographs from the ground, playing a role in visually understanding the situation at the site.

[0351] "Worker equipment" refers to portable devices used by workers that have the function of collecting biometric and location information.

[0352] "Risk assessment" is the process of analyzing the degree of danger in the work environment based on collected information and considering countermeasures.

[0353] An "emotion analysis device" is a system that analyzes the worker's facial expressions, voice, and other biological responses to determine their emotional state.

[0354] "Warnings and instructions" refer to cautionary messages and work-related instructions given to workers based on risk assessments and emotional state analyses.

[0355] The system for implementing this invention primarily consists of a drone, a ground-based camera, and equipment for workers. The server is responsible for acquiring information from these devices. Specifically, it receives wide-area environmental images obtained from the drone and ground-based camera, as well as biometric and location information obtained from the worker's equipment. This information is integrated to comprehensively evaluate the level of risk at the site.

[0356] Furthermore, the server uses an emotion analysis device to analyze the emotional state of workers from their facial expressions and voice. This analysis utilizes an AI engine and applies software such as Affectiva. This technology makes it possible to understand the stress and tension levels of workers in real time.

[0357] The terminal notifies workers of warnings and instructions based on the results of risk assessment and sentiment analysis generated by the server. Messages are delivered visually or audibly via portable devices such as smart glasses. This allows workers to take safe actions at the appropriate time.

[0358] For example, if an increase in a worker's heart rate is detected while working at height, a message saying "Please take a break" will be displayed on the smart glasses. Furthermore, if the worker approaches a dangerous area, a message saying "Caution: Working at height. Please check your harness" will be displayed.

[0359] An example of a prompt message is as follows:

[0360] We are planning a smart glasses application designed to improve safety during work at heights. It will integrate video data from drones and ground cameras with worker biodata and use an algorithm to assess the worker's emotional state to provide appropriate safety procedures to the worker in real time.

[0361] This system aims to improve safety on site and reduce the psychological burden on workers.

[0362] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0363] Step 1:

[0364] The server receives video data as input from drones and ground cameras. This allows it to acquire video information to understand the overall picture of the work site. The server analyzes this video data frame by frame to assess the level of risk at the site. By applying algorithms to analyze specific areas within the video and identifying potential hazards, it performs an initial risk assessment.

[0365] Step 2:

[0366] The server acquires biometric and location information from the worker's equipment as input. This includes heart rate and stress level. The server analyzes this data and evaluates the worker's biological state in real time. An AI module is used for the analysis, and if an abnormal value is detected, a risk flag is set.

[0367] Step 3:

[0368] The server receives facial expression and voice data as input to evaluate the worker's emotional state using an emotion analysis device. Emotion analysis software (e.g., facial recognition technology and voice analysis engine) is used to determine the worker's level of tension and stress. Based on these results, the server determines the necessary actions.

[0369] Step 4:

[0370] The server integrates previously obtained risk assessment data and sentiment analysis results to generate warnings and instructions for each worker. This output includes specific safety instructions based on the worker's physical condition and the hazard level of the work environment. The generated instructions are designed to support safe work procedures.

[0371] Step 5:

[0372] The terminal receives instructions from the server and notifies the worker. Using visual devices such as smart glasses, necessary warnings and instructions are displayed on the screen. The worker reviews these notifications and modifies their actions according to the indicated safety procedures. This feedback loop improves safety and work efficiency.

[0373] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0374] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0375] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0376] [Third Embodiment]

[0377] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0378] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0379] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0380] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0381] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0382] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0383] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0384] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0385] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0386] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0387] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0388] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0389] This invention is a system for improving safety at construction sites, and it enables real-time risk assessment and warning provision through data acquisition and analysis using drones, ground imaging devices, and worker equipment. The embodiments thereof are described below.

[0390] The server receives video data transmitted from drones and ground-based cameras. This enables wide-area environmental monitoring and aims to identify hazards that cannot be captured from a specific viewpoint. In addition, biometric and location information is transmitted from worker devices, and the server analyzes this data in an integrated manner.

[0391] The server uses machine learning algorithms to compare this data with historical accident data in order to analyze it in real time. This process is regularly updated, allowing it to adapt to changing field conditions.

[0392] For example, if a worker is performing work at height, the server checks the condition of the scaffolding and equipment using video from a drone. It monitors the surrounding movements and layout using data from ground cameras and evaluates heart rate, abnormal posture, and other factors using sensor data from the worker's equipment. Based on this, the server makes a comprehensive assessment of the site situation and immediately generates a warning if the risk increases.

[0393] The terminal has the function of notifying workers of warnings generated by the server. These notifications are sent in various forms, such as voice instructions, screen displays, and vibrations, and are designed to allow workers to take appropriate action immediately.

[0394] The user, as the worker, is required to receive warnings and instructions from the terminal and promptly implement safe work procedures. This makes it possible to prevent dangerous situations and significantly improve the overall safety of the work environment.

[0395] The following describes the processing flow.

[0396] Step 1:

[0397] The server receives aerial video data from drones and video data from ground-based cameras. This allows it to acquire visual information about the overall work environment and specific areas.

[0398] Step 2:

[0399] The server receives location and biometric information from the worker's device. This makes it possible to understand the worker's current status and location.

[0400] Step 3:

[0401] The server preprocesses all received data. Video data is broken down frame by frame, and sensor data is filtered to remove noise.

[0402] Step 4:

[0403] The server inputs pre-processed data into a machine learning algorithm to perform a real-time risk assessment. This process predicts the level of risk in the current work environment by referring to past accident data.

[0404] Step 5:

[0405] Based on the risk assessment results, the server generates warnings about potential hazards that workers may face. Specifically, these include precautions to take and emergency response measures in the event of an imminent hazard.

[0406] Step 6:

[0407] The terminal receives warnings sent from the server and notifies the worker visually and audibly. This includes displaying warning messages on the screen, providing audio alerts, and vibrating warnings.

[0408] Step 7:

[0409] The user (worker) will check the warnings from the terminal and take the necessary actions to continue working safely. For example, they may evacuate from dangerous areas or recheck their protective equipment.

[0410] (Example 1)

[0411] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0412] Ensuring worker safety in real time at construction sites is extremely important. However, current systems suffer from delays in data collection and difficulty in rapidly assessing risks in response to environmental changes. Furthermore, they lack the accuracy and timeliness of warnings, making it difficult for workers to make immediate decisions to avoid danger.

[0413] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0414] In this invention, the server includes means for receiving multiple types of information from a data acquisition device, means for integrating the acquired information and pre-processing it using data processing technology, and means for evaluating the degree of risk based on the pre-processed information. This enables real-time, highly accurate risk assessment and rapid warning generation.

[0415] A "data acquisition device" refers to hardware and software used to collect multiple types of information and transmit it within a system.

[0416] "Information integration" is the process of converting different types of data into a single, analyzable format to streamline processing.

[0417] "Data processing technology" refers to the algorithms and computational methods used to analyze received data, which are used for data preprocessing and analysis.

[0418] "Preprocessing" refers to a series of operations to prepare data for analysis, such as removing noise and filling in missing data.

[0419] "Assessing the level of risk" is an analytical process conducted to determine the risks in the current work environment based on collected data, with the aim of ensuring safety.

[0420] "Warning generation" is a process carried out to notify workers when a hazard is detected, and is a means of prompting appropriate action.

[0421] This system is designed to improve safety at construction sites. First, the server receives video data from drones and ground cameras. This allows the server to monitor the site environment over a wide area and identify hazards in real time. In addition, biometric information (e.g., heart rate and body temperature) and location information are collected from worker devices. This data is integrated and analyzed on the server.

[0422] The server utilizes advanced data processing technology to pre-process all received data. Specifically, it employs image processing algorithms to recognize workers and obstacles from video data. Furthermore, for biometric information, statistical methods are used to filter out abnormal values. This enables early detection of anomalies.

[0423] Next, the server uses the pre-processed data to assess the level of risk. It employs machine learning algorithms to analyze the similarities between past accident data and the current situation. This process enables highly accurate assessment of on-site risks and real-time warning generation.

[0424] The terminal's role is to notify workers of warnings sent from the server. These notifications are provided in the form of voice instructions, screen displays, vibrations, etc., allowing workers to immediately detect danger and take action.

[0425] The user, the worker, is required to receive instructions from the terminal and take prompt and appropriate action. This helps prevent dangerous situations and ensures a safe working environment.

[0426] As a specific example, in a scenario involving work at height, the server monitors the worker's safety equipment using drone footage and observes surrounding movements using data from ground-based cameras. It then immediately issues a warning if sensor data from the worker's equipment indicates an anomaly.

[0427] An example of a prompt for a generating AI model is, "Explain how to generate and communicate a warning if a worker performing work at height may not be wearing a helmet." This prompt allows the AI ​​model to provide specific risk avoidance steps.

[0428] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0429] Step 1:

[0430] The server receives video data from drones and ground cameras, as well as biometric and location information from worker equipment. It takes real-time data streams from these devices as input and generates standardized datasets as output. This standardizes different data formats to facilitate subsequent processing.

[0431] Step 2:

[0432] The server integrates the received data and preprocesses it using data processing techniques. Specifically, video data is processed using image processing algorithms to clearly identify workers and equipment. Biometric information is filtered using statistical methods to detect anomalies. The input is an integrated dataset, and the output is clean, analyzable data. This allows only important information to be extracted, enabling efficient analysis.

[0433] Step 3:

[0434] The server assesses the level of risk based on pre-processed data. Using machine learning algorithms, it compares past accident data with the current situation to determine the risk level. Clean data and past accident data are used as input, and the risk assessment results are obtained as output. This quantifies on-site safety and sets criteria for determining the next course of action.

[0435] Step 4:

[0436] The server generates warnings based on the risk assessment results. If the risk level is high, it immediately creates a warning message and sends it to the terminal. The risk assessment results are used as input, and a specific warning message is generated as output. This ensures that workers can respond to risks quickly.

[0437] Step 5:

[0438] The terminal notifies the worker of warnings sent from the server. Warnings are provided in various forms, such as voice messages, screen displays, and vibrations. The input is a warning message, and the output is information presented in a format that the worker can recognize. This creates an environment where workers can make appropriate decisions immediately.

[0439] Step 6:

[0440] The user, the worker, receives a warning from the terminal and takes immediate, safe action. The input is the provided warning, and the output is an appropriate and safe response. This ensures safety on site and prevents hazards before they occur.

[0441] (Application Example 1)

[0442] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0443] To ensure a safe working environment at construction sites, it is necessary to grasp workers' biometric and location information, as well as site conditions in real time, and to quickly assess risks. However, conventional methods make it difficult to accurately and quickly predict sudden changes in the work environment or the occurrence of unexpected hazards, resulting in delays in providing appropriate warnings and instructions to workers.

[0444] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0445] In this invention, the server includes means for acquiring data from unmanned aerial vehicles, ground information gathering devices, and worker equipment; means for integrating the acquired data to perform a risk assessment; and means for notifying workers of warnings based on the results of the risk assessment. This makes it possible to integrate workers' biometric and location information at construction sites, assess risks in real time, and issue warnings quickly.

[0446] An "unmanned aerial vehicle" is an aircraft that flies remotely or autonomously and has the function of collecting information from the air.

[0447] A "ground information gathering device" is a device installed on the ground that collects visual information, audio information, and other data.

[0448] A "worker device" is a device that can be worn by a worker and has the function of acquiring the worker's biometric information and location information, and receiving warnings.

[0449] "Methods for integrating data and conducting risk assessments" refer to the process of combining data obtained from multiple sources and evaluating risk by comparing it with past cases and anticipated hazards.

[0450] "Means of notifying warnings" refers to methods such as sound, visual displays, or vibrations used to inform workers when a hazard is recognized.

[0451] This invention is a system that integrates data from unmanned aerial vehicles, ground information gathering devices, and worker equipment to enhance safety at construction sites. The following describes a specific configuration for realizing this system.

[0452] The server receives data transmitted from unmanned aerial vehicles (UAVs) and ground information gathering devices, allowing it to understand the working environment from different perspectives. This makes it possible to monitor in detail working environments at high altitudes and over wide areas that are not visible from the ground. Examples of hardware used include general-purpose UAVs and surveillance cameras.

[0453] Next, the server centrally manages the data and uses machine learning algorithms to perform risk assessments based on past accident data. The software used here includes TensorFlow as a machine learning platform for data analysis and Apache Kafka for real-time data streaming. If the analysis predicts a hazard, a warning is immediately sent to the worker's device.

[0454] Terminals (smartphones and smart glasses) receive warnings from the server and notify workers visually, audibly, and through vibration. Worker terminals collect biometric and location information from users and transmit it to the server in real time to support more detailed risk assessments.

[0455] For example, if strong winds are detected while a user is working at a high altitude, the system analyzes video data from the unmanned aerial vehicle and the user's biometric information to immediately warn the user of the effects of the strong winds and instruct them to temporarily suspend their work.

[0456] Using a generative AI model, it is possible to generate specific warning messages that respond to changes in the situation. The following example prompts can be used as a reference.

[0457] Example of a prompt:

[0458] "I want to develop an app that detects risks during construction work. It will analyze video from unmanned aerial vehicles and biometric data of workers in real time and have a function to issue immediate warnings. Please suggest specific warning message phrases."

[0459] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0460] Step 1:

[0461] Data is acquired from unmanned aerial vehicles (UAVs) and ground information gathering equipment. The server receives video data streamed in real time from UAVs and audio and image data obtained from ground information gathering equipment. The input consists of various data from UAVs and ground equipment, and the output is an integrated set of this data.

[0462] Step 2:

[0463] The server integrates the collected data and constructs a dataset for analysis. It performs data processing, converting multiple data sets with different types and formats into a unified format. The input is the integrated set obtained in step 1, and the output is the dataset formatted for analysis.

[0464] Step 3:

[0465] The server uses machine learning algorithms to analyze the integrated dataset and perform risk assessment. During this process, it compares the data with past accident data to evaluate future risks. The input is the dataset formatted in step 2, and the output is the risk assessment result.

[0466] Step 4:

[0467] Based on the risk assessment results, the server uses a generative AI model to generate specific warning messages. It utilizes prompts to select appropriate phrases for the situation. The input is the risk assessment results obtained in step 3, and the output is the generated warning message.

[0468] Step 5:

[0469] The terminal receives warning messages sent from the server and notifies the user. This notification can be made via sound, screen display, or vibration. The input is the warning message from the server, and the output is the notification to the user.

[0470] Step 6:

[0471] The user receives a warning from the terminal and takes appropriate action. During this process, the terminal verifies the user's biometric and location information and sends it to the server. Inputs consist of the warning message and on-site information from the user, while output is feedback data sent to the server.

[0472] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0473] The system according to the present invention integrates a drone, ground imaging equipment, worker equipment, and an emotion engine to improve safety in dangerous work environments such as construction sites. The server receives data from multiple information sources and uses this data to simultaneously assess the risks of the work environment and the emotional state of the workers.

[0474] The server integrates video data obtained from drones and ground cameras to check a wide range of environmental conditions. This allows for the detection of risks that cannot be captured by visual checks by workers. In addition, it analyzes location and biometric information acquired from worker devices to evaluate individual risk factors.

[0475] A distinctive feature of this system is that the server uses an emotion engine to analyze the worker's facial expressions and voice data to determine their current emotional state. For example, if a worker is extremely tense or stressed, the system can qualitatively adjust the content of warnings and add reassuring instructions.

[0476] The terminal notifies workers of warnings and emotion-based instructions generated by the server. These include specific safety instructions and motivational messages. If the terminal determines that a worker is experiencing stress, it will issue a corresponding alert.

[0477] The user, as the worker, checks notifications and warnings from the terminal and takes appropriate safety measures. By adhering to safe work procedures according to the notification content and acting on advice based on their emotional state, the worker can reduce the risk of accidents.

[0478] Thus, in order to implement the invention, it is necessary to optimally configure data communication between devices and to implement flexible risk management in response to changes in the field.

[0479] The following describes the processing flow.

[0480] Step 1:

[0481] The server receives aerial footage transmitted from the drone and video data from ground-based cameras. This allows for the collection of data that provides an overall understanding of the work environment.

[0482] Step 2:

[0483] The server receives location and biometric information from the worker's device. This includes data such as the worker's current location, body temperature, and heart rate.

[0484] Step 3:

[0485] The server uses an emotion engine to analyze the worker's facial expression and voice data to evaluate their emotional state. For example, it can detect stress and anxiety from facial expressions.

[0486] Step 4:

[0487] The server integrates all collected data and performs a real-time risk assessment. Here, machine learning algorithms are used to compare current data with past accident patterns and determine the current level of risk.

[0488] Step 5:

[0489] The server customizes the warnings provided to workers based on the results of risk assessment and emotional assessment. If high stress is detected, a relaxation message for safety confirmation is added in addition to the normal warning.

[0490] Step 6:

[0491] The terminal receives customized warning messages generated by the server and notifies the worker visually and audibly. For example, the message may be displayed on the terminal's screen, and an audible warning may be emitted from the speaker.

[0492] Step 7:

[0493] The user (worker) checks notifications from the terminal and takes necessary actions to maintain safe work practices. Based on warnings and advice, they adjust work procedures and avoid hazards.

[0494] (Example 2)

[0495] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0496] In traditional work environments, workers rely on their intuition and experience to ensure safety, but this method is prone to oversights. Furthermore, the lack of objective means to assess workers' mental stress and fatigue raises concerns about an increased risk of accidents. Therefore, there is a growing need for a system that can objectively and in real-time assess the risks of the work environment, understand workers' emotional states, and provide appropriate warnings and instructions.

[0497] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0498] In this invention, the server includes means for acquiring data from machines, means for integrating the acquired data to assess risks, and means for evaluating the emotional state of workers using sentiment analysis. This makes it possible to objectively assess risks in the work environment and provide appropriate warnings and instructions in real time according to the emotional state of workers.

[0499] "Machinery" refers to multi-purpose automated devices for acquiring data, including drones and ground-based imaging equipment.

[0500] "Data" refers to all information acquired by machines, including video data, location information, and biometric information.

[0501] "Means of assessing risk" refers to the process of integrating and analyzing acquired data to determine risks related to the work environment and worker safety.

[0502] "Emotional analysis" refers to a technology that analyzes a worker's facial expressions and voice data to evaluate the worker's emotional state at that moment.

[0503] "Notifications" refer to warnings, instructions, and general information provided to workers based on their evaluation results, and are delivered through the worker's device.

[0504] The invention will now be described in terms of its embodiments. This system operates by coordinating a server, terminals, and workers to improve safety in construction sites and hazardous work environments. The server first acquires video data from machines such as drones and ground cameras. This data is used to understand the entire work environment and assess potential risks.

[0505] The server then analyzes location and biometric information (e.g., heart rate and body temperature) acquired from worker equipment to assess the individual worker's risk status. This enables real-time monitoring of safety in the work environment. Generative AI models are used in this process to perform advanced data analysis.

[0506] Furthermore, the server uses emotion analysis technology to understand the worker's emotional state from their facial expressions and voice data. For example, if a worker inputs a voice message such as "I'm a little nervous today" into the terminal, the server analyzes the degree of that nervousness and prepares an appropriate message to reassure them.

[0507] The warnings and instructions generated in this way are communicated to workers via the terminal. The notifications include specific safety instructions and messages to boost work motivation. If the terminal detects that a worker is experiencing stress, it instantly provides appropriate warnings and encouraging messages.

[0508] The user, as the worker, can take safety measures in real time based on notifications from the terminal. For example, based on the user's voice input, a prompt such as "Assess the current work risk and generate appropriate safety instructions" is entered into the system. This prompt allows the system to quickly assess the risks the user faces and provide appropriate countermeasures.

[0509] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0510] Step 1:

[0511] The server acquires video data from drones and ground cameras. This input data includes a wide range of conditions at the work site. The server analyzes this video and uses a generated AI model to identify potential hazards such as terrain and obstacles. The output is a risk assessment of the site, including the identification of high-risk areas.

[0512] Step 2:

[0513] The server acquires location and biometric information from the worker's equipment. Input data includes the worker's current location, heart rate, body temperature, and other physical data. Based on this data, the server evaluates the individual worker's risk status. The data processing process includes detecting abnormal biometric parameters and analyzing risk factors. The output is the risk assessment result for each worker.

[0514] Step 3:

[0515] The server collects facial and voice data from workers to perform emotion analysis. Input data includes voice and video input from the worker's terminal. The server analyzes this data using a generative AI model to determine the worker's emotional state. Specific data calculations include creating stress level clusters from voice data and recognizing patterns in facial expressions. The output is an evaluation of the worker's emotional state.

[0516] Step 4:

[0517] Based on the analysis results from the previous steps, the server generates warnings and instructions for the worker. Here, it considers the combined risk assessment and emotional state to create appropriate safety instructions and encouraging messages. The input data is the result of the risk assessment and emotional state, and the output is the specific message content to be presented to the worker.

[0518] Step 5:

[0519] The terminal notifies workers of warnings and instructions received from the server. Input is message data from the server, and output is delivered to workers as audio alarms and visual displays. The terminal provides notifications to enable workers to respond immediately and supports them in carrying out safety instructions.

[0520] Step 6:

[0521] The user (worker) receives notifications from the terminal and takes safety measures based on them. The user confirms safe operating procedures and appropriately performs the instructed actions. The input is the content of the notification from the terminal, and the output is the user's specific safety actions. This allows workers to perform their duties while reducing the risk of accidents.

[0522] (Application Example 2)

[0523] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0524] Improving safety in on-site work and maintaining the mental health of workers are crucial challenges. In particular, appropriate responses are required that address the environmental hazards workers face and their individual emotional states. However, conventional systems have limitations in assessing the hazards of the work environment and judging workers' emotional states, making it difficult to provide appropriate safety instructions. This poses a risk of reduced work efficiency and safety.

[0525] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0526] In this invention, the server includes means for acquiring information from a drone, a ground imaging device, and equipment for workers; means for integrating the acquired information to perform a risk assessment; and means for evaluating the emotional state of workers using an emotion analysis device. This makes it possible to more accurately assess the hazards of the work environment and provide appropriate warnings and instructions tailored to the individual emotional state of workers.

[0527] A "drone" is an unmanned aerial system that flies through the air to take photographs and collect data.

[0528] A "ground-based camera" is a device used to capture images and photographs from the ground, playing a role in visually understanding the situation at the site.

[0529] "Worker equipment" refers to portable devices used by workers that have the function of collecting biometric and location information.

[0530] "Risk assessment" is the process of analyzing the degree of danger in the work environment based on collected information and considering countermeasures.

[0531] An "emotion analysis device" is a system that analyzes the worker's facial expressions, voice, and other biological responses to determine their emotional state.

[0532] "Warnings and instructions" refer to cautionary messages and work-related instructions given to workers based on risk assessments and emotional state analyses.

[0533] The system for implementing this invention primarily consists of a drone, a ground-based camera, and equipment for workers. The server is responsible for acquiring information from these devices. Specifically, it receives wide-area environmental images obtained from the drone and ground-based camera, as well as biometric and location information obtained from the worker's equipment. This information is integrated to comprehensively evaluate the level of risk at the site.

[0534] Furthermore, the server uses an emotion analysis device to analyze the emotional state of workers from their facial expressions and voice. This analysis utilizes an AI engine and applies software such as Affectiva. This technology makes it possible to understand the stress and tension levels of workers in real time.

[0535] The terminal notifies workers of warnings and instructions based on the results of risk assessment and sentiment analysis generated by the server. Messages are delivered visually or audibly via portable devices such as smart glasses. This allows workers to take safe actions at the appropriate time.

[0536] For example, if an increase in a worker's heart rate is detected while working at height, a message saying "Please take a break" will be displayed on the smart glasses. Furthermore, if the worker approaches a dangerous area, a message saying "Caution: Working at height. Please check your harness" will be displayed.

[0537] An example of a prompt message is as follows:

[0538] We are planning a smart glasses application designed to improve safety during work at heights. It will integrate video data from drones and ground cameras with worker biodata and use an algorithm to assess the worker's emotional state to provide appropriate safety procedures to the worker in real time.

[0539] This system aims to improve safety on site and reduce the psychological burden on workers.

[0540] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0541] Step 1:

[0542] The server receives video data as input from drones and ground cameras. This allows it to acquire video information to understand the overall picture of the work site. The server analyzes this video data frame by frame to assess the level of risk at the site. By applying algorithms to analyze specific areas within the video and identifying potential hazards, it performs an initial risk assessment.

[0543] Step 2:

[0544] The server acquires biometric and location information from the worker's equipment as input. This includes heart rate and stress level. The server analyzes this data and evaluates the worker's biological state in real time. An AI module is used for the analysis, and if an abnormal value is detected, a risk flag is set.

[0545] Step 3:

[0546] The server receives facial expression and voice data as input to evaluate the worker's emotional state using an emotion analysis device. Emotion analysis software (e.g., facial recognition technology and voice analysis engine) is used to determine the worker's level of tension and stress. Based on these results, the server determines the necessary actions.

[0547] Step 4:

[0548] The server integrates previously obtained risk assessment data and sentiment analysis results to generate warnings and instructions for each worker. This output includes specific safety instructions based on the worker's physical condition and the hazard level of the work environment. The generated instructions are designed to support safe work procedures.

[0549] Step 5:

[0550] The terminal receives instructions from the server and notifies the worker. Using visual devices such as smart glasses, necessary warnings and instructions are displayed on the screen. The worker reviews these notifications and modifies their actions according to the indicated safety procedures. This feedback loop improves safety and work efficiency.

[0551] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0552] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0553] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0554] [Fourth Embodiment]

[0555] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0556] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0557] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0558] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0559] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0560] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0561] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0562] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0563] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0564] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0565] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0566] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0567] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0568] This invention is a system for improving safety at construction sites, and it enables real-time risk assessment and warning provision through data acquisition and analysis using drones, ground imaging devices, and worker equipment. The embodiments thereof are described below.

[0569] The server receives video data transmitted from drones and ground-based cameras. This enables wide-area environmental monitoring and aims to identify hazards that cannot be captured from a specific viewpoint. In addition, biometric and location information is transmitted from worker devices, and the server analyzes this data in an integrated manner.

[0570] The server uses machine learning algorithms to compare this data with historical accident data in order to analyze it in real time. This process is regularly updated, allowing it to adapt to changing field conditions.

[0571] For example, if a worker is performing work at height, the server checks the condition of the scaffolding and equipment using video from a drone. It monitors the surrounding movements and layout using data from ground cameras and evaluates heart rate, abnormal posture, and other factors using sensor data from the worker's equipment. Based on this, the server makes a comprehensive assessment of the site situation and immediately generates a warning if the risk increases.

[0572] The terminal has the function of notifying workers of warnings generated by the server. These notifications are sent in various forms, such as voice instructions, screen displays, and vibrations, and are designed to allow workers to take appropriate action immediately.

[0573] The user, as the worker, is required to receive warnings and instructions from the terminal and promptly implement safe work procedures. This makes it possible to prevent dangerous situations and significantly improve the overall safety of the work environment.

[0574] The following describes the processing flow.

[0575] Step 1:

[0576] The server receives aerial video data from drones and video data from ground-based cameras. This allows it to acquire visual information about the overall work environment and specific areas.

[0577] Step 2:

[0578] The server receives location and biometric information from the worker's device. This makes it possible to understand the worker's current status and location.

[0579] Step 3:

[0580] The server preprocesses all received data. Video data is broken down frame by frame, and sensor data is filtered to remove noise.

[0581] Step 4:

[0582] The server inputs pre-processed data into a machine learning algorithm to perform a real-time risk assessment. This process predicts the level of risk in the current work environment by referring to past accident data.

[0583] Step 5:

[0584] Based on the risk assessment results, the server generates warnings about potential hazards that workers may face. Specifically, these include precautions to take and emergency response measures in the event of an imminent hazard.

[0585] Step 6:

[0586] The terminal receives warnings sent from the server and notifies the worker visually and audibly. This includes displaying warning messages on the screen, providing audio alerts, and vibrating warnings.

[0587] Step 7:

[0588] The user (worker) will check the warnings from the terminal and take the necessary actions to continue working safely. For example, they may evacuate from dangerous areas or recheck their protective equipment.

[0589] (Example 1)

[0590] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0591] Ensuring worker safety in real time at construction sites is extremely important. However, current systems suffer from delays in data collection and difficulty in rapidly assessing risks in response to environmental changes. Furthermore, they lack the accuracy and timeliness of warnings, making it difficult for workers to make immediate decisions to avoid danger.

[0592] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0593] In this invention, the server includes means for receiving multiple types of information from a data acquisition device, means for integrating the acquired information and pre-processing it using data processing technology, and means for evaluating the degree of risk based on the pre-processed information. This enables real-time, highly accurate risk assessment and rapid warning generation.

[0594] A "data acquisition device" refers to hardware and software used to collect multiple types of information and transmit it within a system.

[0595] "Information integration" is the process of converting different types of data into a single, analyzable format to streamline processing.

[0596] "Data processing technology" refers to the algorithms and computational methods used to analyze received data, which are used for data preprocessing and analysis.

[0597] "Preprocessing" refers to a series of operations to prepare data for analysis, such as removing noise and filling in missing data.

[0598] "Assessing the level of risk" is an analytical process conducted to determine the risks in the current work environment based on collected data, with the aim of ensuring safety.

[0599] "Warning generation" is a process carried out to notify workers when a hazard is detected, and is a means of prompting appropriate action.

[0600] This system is designed to improve safety at construction sites. First, the server receives video data from drones and ground cameras. This allows the server to monitor the site environment over a wide area and identify hazards in real time. In addition, biometric information (e.g., heart rate and body temperature) and location information are collected from worker devices. This data is integrated and analyzed on the server.

[0601] The server utilizes advanced data processing technology to pre-process all received data. Specifically, it employs image processing algorithms to recognize workers and obstacles from video data. Furthermore, for biometric information, statistical methods are used to filter out abnormal values. This enables early detection of anomalies.

[0602] Next, the server uses the pre-processed data to assess the level of risk. It employs machine learning algorithms to analyze the similarities between past accident data and the current situation. This process enables highly accurate assessment of on-site risks and real-time warning generation.

[0603] The terminal's role is to notify workers of warnings sent from the server. These notifications are provided in the form of voice instructions, screen displays, vibrations, etc., allowing workers to immediately detect danger and take action.

[0604] The user, the worker, is required to receive instructions from the terminal and take prompt and appropriate action. This helps prevent dangerous situations and ensures a safe working environment.

[0605] As a specific example, in a scenario involving work at height, the server monitors the worker's safety equipment using drone footage and observes surrounding movements using data from ground-based cameras. It then immediately issues a warning if sensor data from the worker's equipment indicates an anomaly.

[0606] An example of a prompt for a generating AI model is, "Explain how to generate and communicate a warning if a worker performing work at height may not be wearing a helmet." This prompt allows the AI ​​model to provide specific risk avoidance steps.

[0607] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0608] Step 1:

[0609] The server receives video data from drones and ground cameras, as well as biometric and location information from worker equipment. It takes real-time data streams from these devices as input and generates standardized datasets as output. This standardizes different data formats to facilitate subsequent processing.

[0610] Step 2:

[0611] The server integrates the received data and preprocesses it using data processing techniques. Specifically, video data is processed using image processing algorithms to clearly identify workers and equipment. Biometric information is filtered using statistical methods to detect anomalies. The input is an integrated dataset, and the output is clean, analyzable data. This allows only important information to be extracted, enabling efficient analysis.

[0612] Step 3:

[0613] The server assesses the level of risk based on pre-processed data. Using machine learning algorithms, it compares past accident data with the current situation to determine the risk level. Clean data and past accident data are used as input, and the risk assessment results are obtained as output. This quantifies on-site safety and sets criteria for determining the next course of action.

[0614] Step 4:

[0615] The server generates warnings based on the risk assessment results. If the risk level is high, it immediately creates a warning message and sends it to the terminal. The risk assessment results are used as input, and a specific warning message is generated as output. This ensures that workers can respond to risks quickly.

[0616] Step 5:

[0617] The terminal notifies the worker of warnings sent from the server. Warnings are provided in various forms, such as voice messages, screen displays, and vibrations. The input is a warning message, and the output is information presented in a format that the worker can recognize. This creates an environment where workers can make appropriate decisions immediately.

[0618] Step 6:

[0619] The user, the worker, receives a warning from the terminal and takes immediate, safe action. The input is the provided warning, and the output is an appropriate and safe response. This ensures safety on site and prevents hazards before they occur.

[0620] (Application Example 1)

[0621] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0622] To ensure a safe working environment at construction sites, it is necessary to grasp workers' biometric and location information, as well as site conditions in real time, and to quickly assess risks. However, conventional methods make it difficult to accurately and quickly predict sudden changes in the work environment or the occurrence of unexpected hazards, resulting in delays in providing appropriate warnings and instructions to workers.

[0623] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0624] In this invention, the server includes means for acquiring data from unmanned aerial vehicles, ground information gathering devices, and worker equipment; means for integrating the acquired data to perform a risk assessment; and means for notifying workers of warnings based on the results of the risk assessment. This makes it possible to integrate workers' biometric and location information at construction sites, assess risks in real time, and issue warnings quickly.

[0625] An "unmanned aerial vehicle" is an aircraft that flies remotely or autonomously and has the function of collecting information from the air.

[0626] A "ground information gathering device" is a device installed on the ground that collects visual information, audio information, and other data.

[0627] A "worker device" is a device that can be worn by a worker and has the function of acquiring the worker's biometric information and location information, and receiving warnings.

[0628] "Methods for integrating data and conducting risk assessments" refer to the process of combining data obtained from multiple sources and evaluating risk by comparing it with past cases and anticipated hazards.

[0629] "Means of notifying warnings" refers to methods such as sound, visual displays, or vibrations used to inform workers when a hazard is recognized.

[0630] This invention is a system that integrates data from unmanned aerial vehicles, ground information gathering devices, and worker equipment to enhance safety at construction sites. The following describes a specific configuration for realizing this system.

[0631] The server receives data transmitted from unmanned aerial vehicles (UAVs) and ground information gathering devices, allowing it to understand the working environment from different perspectives. This makes it possible to monitor in detail working environments at high altitudes and over wide areas that are not visible from the ground. Examples of hardware used include general-purpose UAVs and surveillance cameras.

[0632] Next, the server centrally manages the data and uses machine learning algorithms to perform risk assessments based on past accident data. The software used here includes TensorFlow as a machine learning platform for data analysis and Apache Kafka for real-time data streaming. If the analysis predicts a hazard, a warning is immediately sent to the worker's device.

[0633] Terminals (smartphones and smart glasses) receive warnings from the server and notify workers visually, audibly, and through vibration. Worker terminals collect biometric and location information from users and transmit it to the server in real time to support more detailed risk assessments.

[0634] For example, if strong winds are detected while a user is working at a high altitude, the system analyzes video data from the unmanned aerial vehicle and the user's biometric information to immediately warn the user of the effects of the strong winds and instruct them to temporarily suspend their work.

[0635] Using a generative AI model, it is possible to generate specific warning messages that respond to changes in the situation. The following example prompts can be used as a reference.

[0636] Example of a prompt:

[0637] "I want to develop an app that detects risks during construction work. It will analyze video from unmanned aerial vehicles and biometric data of workers in real time and have a function to issue immediate warnings. Please suggest specific warning message phrases."

[0638] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0639] Step 1:

[0640] Data is acquired from unmanned aerial vehicles (UAVs) and ground information gathering equipment. The server receives video data streamed in real time from UAVs and audio and image data obtained from ground information gathering equipment. The input consists of various data from UAVs and ground equipment, and the output is an integrated set of this data.

[0641] Step 2:

[0642] The server integrates the collected data and constructs a dataset for analysis. It performs data processing, converting multiple data sets with different types and formats into a unified format. The input is the integrated set obtained in step 1, and the output is the dataset formatted for analysis.

[0643] Step 3:

[0644] The server uses machine learning algorithms to analyze the integrated dataset and perform risk assessment. During this process, it compares the data with past accident data to evaluate future risks. The input is the dataset formatted in step 2, and the output is the risk assessment result.

[0645] Step 4:

[0646] Based on the risk assessment results, the server uses a generative AI model to generate specific warning messages. It utilizes prompts to select appropriate phrases for the situation. The input is the risk assessment results obtained in step 3, and the output is the generated warning message.

[0647] Step 5:

[0648] The terminal receives warning messages sent from the server and notifies the user. This notification can be made via sound, screen display, or vibration. The input is the warning message from the server, and the output is the notification to the user.

[0649] Step 6:

[0650] The user receives a warning from the terminal and takes appropriate action. During this process, the terminal verifies the user's biometric and location information and sends it to the server. Inputs consist of the warning message and on-site information from the user, while output is feedback data sent to the server.

[0651] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0652] The system according to the present invention integrates a drone, ground imaging equipment, worker equipment, and an emotion engine to improve safety in dangerous work environments such as construction sites. The server receives data from multiple information sources and uses this data to simultaneously assess the risks of the work environment and the emotional state of the workers.

[0653] The server integrates video data obtained from drones and ground cameras to check a wide range of environmental conditions. This allows for the detection of risks that cannot be captured by visual checks by workers. In addition, it analyzes location and biometric information acquired from worker devices to evaluate individual risk factors.

[0654] A distinctive feature of this system is that the server uses an emotion engine to analyze the worker's facial expressions and voice data to determine their current emotional state. For example, if a worker is extremely tense or stressed, the system can qualitatively adjust the content of warnings and add reassuring instructions.

[0655] The terminal notifies workers of warnings and emotion-based instructions generated by the server. These include specific safety instructions and motivational messages. If the terminal determines that a worker is experiencing stress, it will issue a corresponding alert.

[0656] The user, as the worker, checks notifications and warnings from the terminal and takes appropriate safety measures. By adhering to safe work procedures according to the notification content and acting on advice based on their emotional state, the worker can reduce the risk of accidents.

[0657] Thus, in order to implement the invention, it is necessary to optimally configure data communication between devices and to implement flexible risk management in response to changes in the field.

[0658] The following describes the processing flow.

[0659] Step 1:

[0660] The server receives aerial footage transmitted from the drone and video data from ground-based cameras. This allows for the collection of data that provides an overall understanding of the work environment.

[0661] Step 2:

[0662] The server receives location and biometric information from the worker's device. This includes data such as the worker's current location, body temperature, and heart rate.

[0663] Step 3:

[0664] The server uses an emotion engine to analyze the worker's facial expression and voice data to evaluate their emotional state. For example, it can detect stress and anxiety from facial expressions.

[0665] Step 4:

[0666] The server integrates all collected data and performs a real-time risk assessment. Here, machine learning algorithms are used to compare current data with past accident patterns and determine the current level of risk.

[0667] Step 5:

[0668] The server customizes the warnings provided to workers based on the results of risk assessment and emotional assessment. If high stress is detected, a relaxation message for safety confirmation is added in addition to the normal warning.

[0669] Step 6:

[0670] The terminal receives customized warning messages generated by the server and notifies the worker visually and audibly. For example, the message may be displayed on the terminal's screen, and an audible warning may be emitted from the speaker.

[0671] Step 7:

[0672] The user (worker) checks notifications from the terminal and takes necessary actions to maintain safe work practices. Based on warnings and advice, they adjust work procedures and avoid hazards.

[0673] (Example 2)

[0674] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0675] In traditional work environments, workers rely on their intuition and experience to ensure safety, but this method is prone to oversights. Furthermore, the lack of objective means to assess workers' mental stress and fatigue raises concerns about an increased risk of accidents. Therefore, there is a growing need for a system that can objectively and in real-time assess the risks of the work environment, understand workers' emotional states, and provide appropriate warnings and instructions.

[0676] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0677] In this invention, the server includes means for acquiring data from machines, means for integrating the acquired data to assess risks, and means for evaluating the emotional state of workers using sentiment analysis. This makes it possible to objectively assess risks in the work environment and provide appropriate warnings and instructions in real time according to the emotional state of workers.

[0678] "Machinery" refers to multi-purpose automated devices for acquiring data, including drones and ground-based imaging equipment.

[0679] "Data" refers to all information acquired by machines, including video data, location information, and biometric information.

[0680] "Means of assessing risk" refers to the process of integrating and analyzing acquired data to determine risks related to the work environment and worker safety.

[0681] "Emotional analysis" refers to a technology that analyzes a worker's facial expressions and voice data to evaluate the worker's emotional state at that moment.

[0682] "Notifications" refer to warnings, instructions, and general information provided to workers based on their evaluation results, and are delivered through the worker's device.

[0683] The invention will now be described in terms of its embodiments. This system operates by coordinating a server, terminals, and workers to improve safety in construction sites and hazardous work environments. The server first acquires video data from machines such as drones and ground cameras. This data is used to understand the entire work environment and assess potential risks.

[0684] The server then analyzes location and biometric information (e.g., heart rate and body temperature) acquired from worker equipment to assess the individual worker's risk status. This enables real-time monitoring of safety in the work environment. Generative AI models are used in this process to perform advanced data analysis.

[0685] Furthermore, the server uses emotion analysis technology to understand the worker's emotional state from their facial expressions and voice data. For example, if a worker inputs a voice message such as "I'm a little nervous today" into the terminal, the server analyzes the degree of that nervousness and prepares an appropriate message to reassure them.

[0686] The warnings and instructions generated in this way are communicated to workers via the terminal. The notifications include specific safety instructions and messages to boost work motivation. If the terminal detects that a worker is experiencing stress, it instantly provides appropriate warnings and encouraging messages.

[0687] The user, as the worker, can take safety measures in real time based on notifications from the terminal. For example, based on the user's voice input, a prompt such as "Assess the current work risk and generate appropriate safety instructions" is entered into the system. This prompt allows the system to quickly assess the risks the user faces and provide appropriate countermeasures.

[0688] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0689] Step 1:

[0690] The server acquires video data from drones and ground cameras. This input data includes a wide range of conditions at the work site. The server analyzes this video and uses a generated AI model to identify potential hazards such as terrain and obstacles. The output is a risk assessment of the site, including the identification of high-risk areas.

[0691] Step 2:

[0692] The server acquires location and biometric information from the worker's equipment. Input data includes the worker's current location, heart rate, body temperature, and other physical data. Based on this data, the server evaluates the individual worker's risk status. The data processing process includes detecting abnormal biometric parameters and analyzing risk factors. The output is the risk assessment result for each worker.

[0693] Step 3:

[0694] The server collects facial and voice data from workers to perform emotion analysis. Input data includes voice and video input from the worker's terminal. The server analyzes this data using a generative AI model to determine the worker's emotional state. Specific data calculations include creating stress level clusters from voice data and recognizing patterns in facial expressions. The output is an evaluation of the worker's emotional state.

[0695] Step 4:

[0696] Based on the analysis results from the previous steps, the server generates warnings and instructions for the worker. Here, it considers the combined risk assessment and emotional state to create appropriate safety instructions and encouraging messages. The input data is the result of the risk assessment and emotional state, and the output is the specific message content to be presented to the worker.

[0697] Step 5:

[0698] The terminal notifies workers of warnings and instructions received from the server. Input is message data from the server, and output is delivered to workers as audio alarms and visual displays. The terminal provides notifications to enable workers to respond immediately and supports them in carrying out safety instructions.

[0699] Step 6:

[0700] The user (worker) receives notifications from the terminal and takes safety measures based on them. The user confirms safe operating procedures and appropriately performs the instructed actions. The input is the content of the notification from the terminal, and the output is the user's specific safety actions. This allows workers to perform their duties while reducing the risk of accidents.

[0701] (Application Example 2)

[0702] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0703] Improving safety in on-site work and maintaining the mental health of workers are crucial challenges. In particular, appropriate responses are required that address the environmental hazards workers face and their individual emotional states. However, conventional systems have limitations in assessing the hazards of the work environment and judging workers' emotional states, making it difficult to provide appropriate safety instructions. This poses a risk of reduced work efficiency and safety.

[0704] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0705] In this invention, the server includes means for acquiring information from a drone, a ground imaging device, and equipment for workers; means for integrating the acquired information to perform a risk assessment; and means for evaluating the emotional state of workers using an emotion analysis device. This makes it possible to more accurately assess the hazards of the work environment and provide appropriate warnings and instructions tailored to the individual emotional state of workers.

[0706] A "drone" is an unmanned aerial system that flies through the air to take photographs and collect data.

[0707] A "ground-based camera" is a device used to capture images and photographs from the ground, playing a role in visually understanding the situation at the site.

[0708] "Worker equipment" refers to portable devices used by workers that have the function of collecting biometric and location information.

[0709] "Risk assessment" is the process of analyzing the degree of danger in the work environment based on collected information and considering countermeasures.

[0710] An "emotion analysis device" is a system that analyzes the worker's facial expressions, voice, and other biological responses to determine their emotional state.

[0711] "Warnings and instructions" refer to cautionary messages and work-related instructions given to workers based on risk assessments and emotional state analyses.

[0712] The system for implementing this invention primarily consists of a drone, a ground-based camera, and equipment for workers. The server is responsible for acquiring information from these devices. Specifically, it receives wide-area environmental images obtained from the drone and ground-based camera, as well as biometric and location information obtained from the worker's equipment. This information is integrated to comprehensively evaluate the level of risk at the site.

[0713] Furthermore, the server uses an emotion analysis device to analyze the emotional state of workers from their facial expressions and voice. This analysis utilizes an AI engine and applies software such as Affectiva. This technology makes it possible to understand the stress and tension levels of workers in real time.

[0714] The terminal notifies workers of warnings and instructions based on the results of risk assessment and sentiment analysis generated by the server. Messages are delivered visually or audibly via portable devices such as smart glasses. This allows workers to take safe actions at the appropriate time.

[0715] For example, if an increase in a worker's heart rate is detected while working at height, a message saying "Please take a break" will be displayed on the smart glasses. Furthermore, if the worker approaches a dangerous area, a message saying "Caution: Working at height. Please check your harness" will be displayed.

[0716] An example of a prompt message is as follows:

[0717] We are planning a smart glasses application designed to improve safety during work at heights. It will integrate video data from drones and ground cameras with worker biodata and use an algorithm to assess the worker's emotional state to provide appropriate safety procedures to the worker in real time.

[0718] This system aims to improve safety on site and reduce the psychological burden on workers.

[0719] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0720] Step 1:

[0721] The server receives video data as input from drones and ground cameras. This allows it to acquire video information to understand the overall picture of the work site. The server analyzes this video data frame by frame to assess the level of risk at the site. By applying algorithms to analyze specific areas within the video and identifying potential hazards, it performs an initial risk assessment.

[0722] Step 2:

[0723] The server acquires biometric and location information from the worker's equipment as input. This includes heart rate and stress level. The server analyzes this data and evaluates the worker's biological state in real time. An AI module is used for the analysis, and if an abnormal value is detected, a risk flag is set.

[0724] Step 3:

[0725] The server receives facial expression and voice data as input to evaluate the worker's emotional state using an emotion analysis device. Emotion analysis software (e.g., facial recognition technology and voice analysis engine) is used to determine the worker's level of tension and stress. Based on these results, the server determines the necessary actions.

[0726] Step 4:

[0727] The server integrates previously obtained risk assessment data and sentiment analysis results to generate warnings and instructions for each worker. This output includes specific safety instructions based on the worker's physical condition and the hazard level of the work environment. The generated instructions are designed to support safe work procedures.

[0728] Step 5:

[0729] The terminal receives instructions from the server and notifies the worker. Using visual devices such as smart glasses, necessary warnings and instructions are displayed on the screen. The worker reviews these notifications and modifies their actions according to the indicated safety procedures. This feedback loop improves safety and work efficiency.

[0730] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0731] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0732] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0733] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0734] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0735] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0736] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0737] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0738] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0739] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0740] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0741] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0742] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0743] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0744] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0745] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0746] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0747] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0748] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0749] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0750] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0751] The following is further disclosed regarding the embodiments described above.

[0752] (Claim 1)

[0753] Means for acquiring data from drones, ground imaging equipment, and worker equipment,

[0754] A means of integrating acquired data to perform risk assessment,

[0755] A means of providing warnings to workers based on the results of the risk assessment,

[0756] A system that includes this.

[0757] (Claim 2)

[0758] The system according to claim 1, wherein the operator's device includes an output device for displaying the warning.

[0759] (Claim 3)

[0760] The system according to claim 1, further comprising means for analyzing video data from a drone and a ground-based camera to determine the degree of danger in the work environment.

[0761] "Example 1"

[0762] (Claim 1)

[0763] Means for receiving multiple types of information from a data acquisition device,

[0764] A means of integrating the acquired information and preprocessing it using data processing technology,

[0765] A means of evaluating the degree of risk based on pre-processed information,

[0766] A means for generating and outputting warnings based on evaluation results,

[0767] A system that includes this.

[0768] (Claim 2)

[0769] The system according to claim 1, comprising a display device for displaying or providing warnings in various formats.

[0770] (Claim 3)

[0771] The system according to claim 1, which uses an analytical technique to compare past accident information with current data.

[0772] "Application Example 1"

[0773] (Claim 1)

[0774] Means for acquiring data from unmanned aerial vehicles, ground information gathering devices, and worker equipment,

[0775] A means of integrating acquired data to perform risk assessment,

[0776] A means for notifying workers of a warning based on the results of the risk assessment,

[0777] A means for collecting and integrating biometric and location information of workers,

[0778] A means for performing real-time data analysis and communicating the analysis results to the worker,

[0779] A system that includes this.

[0780] (Claim 2)

[0781] The system according to claim 1, wherein the operator's device includes an output function for displaying the warning.

[0782] (Claim 3)

[0783] The system according to claim 1, further comprising means for analyzing visual information from unmanned aerial vehicles and ground information gathering devices to evaluate the degree of hazard in the work environment.

[0784] "Example 2 of combining an emotion engine"

[0785] (Claim 1)

[0786] Means of acquiring data from machines,

[0787] A means of integrating acquired data to assess risk,

[0788] A means of evaluating the emotional state of workers using emotion analysis,

[0789] A means of providing notifications to workers based on evaluation results,

[0790] ...

[0791] A system that includes this.

[0792] (Claim 2)

[0793] The system according to claim 1, wherein the worker device includes an information output device for displaying the notification.

[0794] (Claim 3)

[0795] The system according to claim 1, further comprising means for analyzing visual data from a machine to determine risk factors in the work environment.

[0796] "Application example 2 when combining with an emotional engine"

[0797] (Claim 1)

[0798] Means for acquiring information from drones, ground imaging equipment, and worker equipment,

[0799] A means of integrating acquired information to perform risk assessment,

[0800] A means for evaluating the emotional state of a worker using an emotion analysis device,

[0801] Means for providing warnings and instructions to workers based on the results of the risk assessment and emotional state,

[0802] ...

[0803] A system that includes this.

[0804] (Claim 2)

[0805] The system according to claim 1, wherein the operator's device includes an output function for displaying the warning and instructions.

[0806] (Claim 3)

[0807] The system according to claim 1, further comprising means for analyzing video information from a drone and a ground-based camera to determine the degree of danger in the work environment and generating a notification corresponding to the emotional state based on this. [Explanation of Symbols]

[0808] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. Means for acquiring data from unmanned aerial vehicles, ground information gathering devices, and worker equipment, A means of integrating acquired data to perform risk assessment, A means for notifying workers of a warning based on the results of the risk assessment, A means for collecting and integrating biometric and location information of workers, A means for performing real-time data analysis and communicating the analysis results to the worker, A system that includes this.

2. The system according to claim 1, wherein the operator's device includes an output function for displaying the warning.

3. The system according to claim 1, further comprising means for analyzing visual information from unmanned aerial vehicles and ground information gathering devices to evaluate the degree of hazard in the work environment.

Images