Multimodal AI system and method for automating risk assessment and accident type prediction that reflects the changing work environment in industrial setting

Abstract

The present invention relates to a multimodal AI system and method for automating risk assessment and accident type prediction that reflects the changing work environment of an industrial site, comprising: an object recognition unit that detects objects and identifies the area and type of objects by performing instance segmentation on each frame of an input image; a scene graph generation unit that constructs a graph of the structure and situational context of a scene by using the detected objects as nodes and expressing spatial relationships or interaction relationships between objects as edges; a template selection unit that selects a description template corresponding to the scene by comparing information extracted from the scene graph with a predefined hazard type dictionary (Hazard Map); a sentence generation unit that generates a description of a dangerous situation in natural language form by substituting actual object names and situational information into the selected description template; a logging unit that stores intermediate outputs of caption generation for each frame; and a time series analysis unit that analyzes the increase or decrease in risk and the possibility of future risk occurrence in a time series using information extracted from consecutive frames of the image and sensing data from a sensor.

Description

Technology Field

[0001] The present invention relates to a multimodal AI system and method that predicts accident types in advance based on video data at work sites in industrial sectors where fatal accidents are frequent, such as construction sites, and automates the processing of frequent risk assessments that reflect changing risk factors at the work site caused by dynamic accident causative factors such as workers and operating machinery and equipment, and predicts the increase in the level of accident risk resulting therefrom.

[0002] More specifically, the invention relates to a multimodal AI system that automatically processes situational awareness and risk factors by classifying object recognition and the relationships and contexts between objects at a work site through object recognition based on instance segmentation within work site images, detection of changes in relationships between objects based on spatiotemporal scene graphs, and language models specialized for industrial sites, particularly in the field of construction safety, such as DFT (Domain fine tuning) and TFT (Task fine tuning) of ultra-lightweight sLM models like Ko-BART, thereby automatically processing frequent risk assessments that were impossible to handle to the desired extent for actual work sites when relying on existing safety managers.

[0003] Furthermore, the proposed technology relates to a technology that supports the prediction and forecasting of accident types for each individual workplace by analyzing in real time the working conditions and risk factors of individual work sites, which change unexpectedly and frequently due to dynamic accident causes such as workers, operating machinery, and equipment, all of which have different working conditions. Background Technology

[0005] [Research and Development Project Supporting This Invention][Project No.] D2510013[Ministry Name] Gyeonggi-do [Project Management (Specialized) Agency Name] Gyeonggi Economic & Science Promotion Agency [Research Project Name] Gyeonggi R&D First Step Corporate Research Institute Support Project [Research Project Name] Development of Construction Site Structural Collapse Prediction Service System Based on Multimodal AI Convergence Analysis [Project Performing Agency Name] Totalsys Co., Ltd. [Research Period] 2025.08.01 ~ 2026.7.31 Cornia et al. achieved high performance on the COCO dataset in 2020 by encoding object relationships within images at multiple levels using a Meshed-Memory Transformer and incorporating a memory mechanism, and a Transformer-based approach is also standing out in video captioning.

[0006] Lin et al. proposed the SwinBERT model in 2022, which generates subtitles by having the video transformer directly take original frame patches as input instead of pre-extracting video frames as CNN features. This model learns inter-frame relationships using a spatiotemporal self-attention module instead of a 2D / 3D feature extractor and achieved significant performance improvements over previous methods in five benchmarks, including MSVD and MSR-VTT.

[0007] As such, transformer-based captioning techniques are improving the quality of image description generation by leveraging the benefits of parallelization and long-term dependency learning.

[0008] However, this technology has limitations in that it requires a massive model and vast training data for high performance, and there were difficulties in real-time application in actual industrial settings due to inference speed and the requirement for large-scale GPU resources.

[0009] Recently, pre-trained massive multimodal models have been emerging that possess both image-text understanding and generation capabilities. A representative example is BLIP (Bootstrapping Language-Image Pre-training), a vision-language model trained on a massive corpus of web image-text pairs, which has demonstrated excellent performance in various tasks such as image caption generation and question answering. BLIP learns efficiently by generating and filtering synthetic captions from noisy web data using a captioner.

[0010] DeepMind's Flamingo model is a multimodal model specialized for few-shot learning that combines massive language and vision models, trained on web data containing both images and text. Flamingo adapts to new tasks with only a few examples provided in the prompt, without the need for additional fine-tuning, and demonstrates performance that outperforms existing specialized models on various image and video description tasks using only few shots of training.

[0011] Furthermore, Google's Gemini, announced in 2023, is a model that integrates GPT-based LLMs with multimodal cognition. It is available in various scales such as Ultra, Pro, and Nano, and demonstrates excellent capabilities for images, audio, video, and text. Such large-scale multimodal models offer good versatility.

[0012] However, while these models offer good generality, they have practical limitations in that they are slow to infer and incur massive deployment costs, as they were developed on a scale of billions of parameters. Additionally, since they are not optimized for detailed captioning in specialized fields such as construction safety, they are unsuitable for field-specific captioning that reflects domain knowledge and vocabulary.

[0013] Meanwhile, scene graph-based captioning technology utilizing a structural understanding of video has also been announced. A scene graph is a graph in which objects appearing in a scene are represented as nodes and relationships as edges; by inputting this into a captioning model, it is possible to generate richer captions by utilizing information on the relationships between objects.

[0014] Methods proposed by Yao et al. in 2019 improved the accuracy of captions by integrating image scene graph information into RNN-based captioning.

[0015] Nguyen et al. published a scene graph-specific captioning model called SG2Cap in 2021. SG2Caps is a lightweight model that generates captions using only the object / relationship labels of detected scene graphs; by constructing an HOI graph with added Human-Object Interaction (HOI) information, it achieved superior results compared to existing models even without image features.

[0016] However, there was a problem where captioning performance was actually degraded due to errors and inconsistencies in the predicted scene graph.

[0017] Meanwhile, there is a patent application (Application No. 10-2022-0109147) regarding a multimodal-based image captioning system that generates image captions using a pre-trained model. This prior art utilizes the advantages of a general-purpose model trained on large-scale web data, but has limitations in detailed descriptions of specific specialized fields.

[0018] In addition, among patents related to brain disease diagnosis devices using deep learning, there is a registered patent (Registration No. 10-2555139) that performs image analysis by utilizing a vision transformer as a core element. While this demonstrates the powerful feature extraction capabilities of the transformer, this patent focuses on diagnosis / classification, so problems regarding the model's inefficiency and inference speed still remain from the perspective of image description generation (captioning).

[0019] In addition, there is a published patent (Publication No. 10-2019-0086902) regarding a method for material analysis using a multimodal input-based neural network and an apparatus for the same. This patent focuses on performing 'classification' by fusing sensor data other than visual data, and thus lacks the ability to 'generate' the content of an image into sentences.

[0020] Most of these prior patents have limitations, such as using general-purpose multimodal / transformer models, focusing on classification / diagnosis rather than captioning, or lacking efficiency in the inference process. The problem to be solved

[0022] This invention was devised to solve the aforementioned problems. Although existing multimodal VLM (Vision Language Model) AI technologies such as ChatGPT have been released, they have limitations, including the illusion of application in specific fields due to their generality, processing times caused by large-scale parameters, and the difficulty in presenting explainable evidence due to the black-box processing of AI models. This invention provides a method that minimizes the illusion of application in specific fields, minimizes processing time, and enables evidence-based explanation, thereby overcoming the limitations of existing multimodal AI.

[0023] Furthermore, conventional methods for assessing risk involve safety management experts utilizing regulations and guidelines to extract risk factors by construction type and detailed work items, identify corresponding accident types, and then evaluate the severity of the risks. However, there were limitations in frequently assessing risks at desired work sites due to the limited number of safety experts. This invention aims to provide a method for automatically processing risk assessments using multimodal AI simply by inputting the type of construction work and images of the work site.

[0024] Furthermore, although risk assessment results are shared through continuous or daily assessments—specifically via the Work Safety Meeting (TBM) held before the start of work each day—it is difficult to assess risks that reflect the worker environment and working conditions at the site, which change frequently in minute-by-minute intervals. Consequently, changes in risk factors caused by such shifting sites and conditions can lead to unassessed risk factors, making it difficult to prevent accidents. However, this invention aims to provide an automated risk assessment technique that reflects the changing work site in real time.

[0025] Furthermore, while existing vLM models perform video captioning, they have limitations in predicting risks and forecasting accidents in near real-time because they fail to specifically analyze risk factors and the characteristics of safety measures implementation regarding a changing site through time-series video analysis, and fail to reflect the increase / decrease in risk and risk mitigation resulting from the implementation of safety measures. However, the present invention aims to provide a technology that predicts and forecasts risks and accident types regarding a changing work site in near real-time by reflecting risk factors and safety measures of the changing site.

[0026] Furthermore, although various risk prediction technologies utilizing actual construction site accident databases have been introduced to predict accidents at construction sites, it is difficult to reflect the working conditions and risk factors of individual workplaces through natural language-based input. Moreover, methods for predicting accident risks in advance at intervals of minutes or less, one hour later, one day earlier, or one week have limitations. For example, even the Construction Safety Daily Dashboard operated by the Korea Occupational Safety and Health Agency predicts and presents general risk levels based on factors such as accidents that occurred on the current date within the last 10 years or weather conditions; however, there were limitations in predicting and forecasting accident types and risks specific to each individual construction site in a customized manner for construction sites nationwide, where the types of work, work contents, and progress rates vary. This invention aims to provide a method for forecasting accident risk rates on a daily and weekly basis, similar to weather forecasting, by reflecting the accident database from past construction sites and the current working conditions and risk factors of individual workplaces in natural language. means of solving the problem

[0028] A multimodal AI system for automating risk assessment and accident type prediction reflecting a changing work site in an industrial site according to an embodiment of the present invention for solving the aforementioned problems comprises: an object recognition unit that detects objects and identifies the area and type of objects by performing instance segmentation on each frame of an input image; a scene graph generation unit that constructs a graph of the structure and situational context of a scene by using the detected objects as nodes and expressing spatial relationships or interaction relationships between objects as edges; a template selection unit that selects a description template corresponding to the scene by comparing information extracted from the scene graph with a predefined hazard type dictionary (Hazard Map); a sentence generation unit that generates a description of a dangerous situation in natural language form by substituting actual object names and situational information into the selected description template; a logging unit that stores intermediate outputs of caption generation for each frame; and a time series analysis unit that analyzes the increase or decrease in risk and the possibility of future risk occurrence in a time series using information extracted from consecutive frames of the image and sensing data from a sensor.

[0029] According to another embodiment of the present invention, the object recognition unit processes a video stream received from a field camera by dividing it into a time window of a predetermined length, recognizes people, equipment, structures, PPE, and safety equipment through instance segmentation in each frame, and assigns a consistent ID to the same object through object tracking between consecutive frames.

[0030] According to another embodiment of the present invention, the scene graph generation unit integrates instance segmentation and object tracking results within a single time window to form a scene-temporal scene graph, wherein the nodes of the graph are defined as "object × time," and the edges of the graph include spatial relationship edges including center distance, azimuth angle, height difference, relative speed, etc. between objects at the same point in time, and temporal relationship edges connecting the same objects at consecutive points in time, thereby simultaneously expressing the structure of the scene and information on changes in relationships.

[0031] According to another embodiment of the present invention, the spatiotemporal scene graph is combined with an ST-GCN-based posture / behavior recognition system that learns by modeling a human joint keypoint time series as a graph, and a video behavior recognition system that recognizes work actions such as cutting, transporting, or unloading using X3D or a lightweight SlowFast model, thereby detecting high-risk patterns combining multiple grounds, such as "unfastened harness during work at height + signs of loss of balance + lack of handrail," at an early stage.

[0032] According to another embodiment of the present invention, the risk type dictionary constructs a template and a risk map by reflecting not only risk factor cases and safety measure standards by construction type but also SIF (Serious Injury & Fatality) expert knowledge, and the template matching and sentence rendering are performed by rule-based operations and process multiple frames per second through a parallel optimized deep learning inference engine.

[0033] According to another embodiment of the present invention, the sentence generation unit converts object names into Korean according to a Korean term table and fine-tunes the Ko-BART language model with DFT (Domain fine-tuning) and TFT (Task fine-tuning) for industrial safety and construction safety domains to generate natural descriptions of dangerous situations.

[0034] According to another embodiment of the present invention, the logging unit records the object list detected per frame, the generated scene graph structure, the selected template ID, and the final generated danger situation description in JSON format, and displays the generated danger situation description overlaid on the main video frame.

[0035] According to another embodiment of the present invention, the time series analysis unit calculates summary indicators and risk scores from spatiotemporal scene graphs generated in time window units, accumulates them as time series data to monitor risk levels and rates of change in near real-time, performs an alarm when a threshold is exceeded, or performs short-term and medium-term risk prediction using multivariate sequences of the last N windows as input.

[0036] According to another embodiment of the present invention, the time series analysis unit calculates window summary indicators, such as risk scores and relationship change amounts, based on spatiotemporal scene graphs generated for each time window, accumulates and stores the indicators as risk situation time series data, calculates the accumulated risk and the rate of increase or decrease thereof from the time series data in near real-time, and generates an immediate alarm if a preset threshold value is exceeded, or predicts a short-term or medium-term risk level using a multivariate sequence for the last N consecutive windows as input. Effects of the invention

[0038] The present invention has the following effects.

[0039] First, the present invention is characterized by video captioning technology specialized for risk assessment and risk forecasting for industrial site risk management, which integrates object-behavior-environment information within an image scene to accurately describe hazardous situations.

[0040] Technically, by introducing a conditional template matching technique instead of deep learning-based free text generation, it is possible to improve the consistency and accuracy of descriptions and achieve system lightweighting. Additionally, by constructing a scene graph from the results of instance segmentation and utilizing it to integrate domain knowledge (a dictionary of disaster types in specific industries (Hazard Map)), it is possible to generate captions that understand situational context beyond simple object enumeration, and by logging the entire generation process to enable verification when necessary. Furthermore, the final system can operate in real-time or near-real-time, thereby achieving a level of performance suitable for direct application to on-site CCTV monitoring.

[0041] Therefore, it is possible to achieve lightweighting of multimodal AI models, particularly resource-intensive LLM models, while maintaining accuracy performance, thereby improving processing speed by more than three times compared to commercial ChatGPT. In addition, since it can be run using only small GPU resources, it has the effect of enabling service support at the edge and on-premises for organizations where cloud services are not available.

[0042] Second, because the present invention uses self-constructed training data and self-selected modules, it can realize explainable AI by presenting evidence-based inference results and minimize hallucinatory inference results, thus having a useful effect for AI application in fields requiring high reliability and evidence, such as safety.

[0043] Third, although there are safety management-related laws, guidelines, manuals, and supervision systems, including the Serious Accidents Prevention Act, decisions such as safety blind spots and the unreasonable pursuit of construction to shorten the construction period are still possible; however, since this invention can automatically support risk assessment—particularly for changing work sites—and near-real-time risk assessment and verification of the implementation of safety measures, it can realize time-saving, cost-saving, and human error-minimizing effects.

[0044] Fourth, the present invention supports risk forecasting regarding predicted accident types and probability of occurrence by workplace type and worker, such as after minutes, after one hour, and after one day, by reflecting the working conditions of individual workplaces rather than predicting the general risk level of a construction site as high, medium, or low risk. Therefore, it has the effect of preventing risks in advance and reducing social costs through this. Brief explanation of the drawing

[0046] FIG. 1 is a configuration diagram of a multimodal AI system for automating risk assessment and accident type prediction reflecting the changing work environment of an industrial site according to the present invention. FIG. 2 is a hardware configuration diagram in which the multimodal AI system according to the present invention is installed. FIG. 3 and FIG. 4 are diagrams illustrating examples of automatically generated Hazard Maps and Templates according to the present invention, respectively. FIG. 5 is a diagram illustrating a multimodal AI method for automating risk assessment and accident type prediction reflecting the changing work environment of an industrial site according to an embodiment of the present invention. FIG. 6 and FIG. 7 are diagrams illustrating the results of captioning two image frames according to the present invention. FIG. 8 and FIG. 9 are diagrams illustrating the results of automated risk assessment processing of an actual work environment in an industrial site using the multimodal AI model of the present invention. FIG. 10 is an operation flowchart of a module for risk assessment and risk prediction regarding a work environment that changes frequently in real time according to the present invention. FIG. 11 is an operation flowchart of a multimodal AI model for predicting accident types and occurrence probabilities for individual work environments according to the present invention. Specific details for implementing the invention

[0047] The present invention is capable of various modifications and may have various embodiments, and specific embodiments are illustrated in the drawings and described in detail in the description of the invention. However, this is not intended to limit the present invention to specific embodiments, and it should be understood that it includes all modifications, equivalents, and substitutions that fall within the spirit and scope of the invention.

[0048] However, in describing the embodiments, if it is determined that a detailed description of related known functions or configurations could unnecessarily obscure the essence of the invention, such detailed description is omitted. Furthermore, the sizes of each component in the drawings may be exaggerated for illustrative purposes and do not represent the actual sizes applied.

[0049] Furthermore, throughout the specification, when a component is referred to as being "connected" or "joined" with another component, it should be understood that the component may be directly connected or joined to the other component, but unless specifically stated otherwise, it may also be connected or joined through an intermediate component. Additionally, throughout the specification, when a part is described as "including" a component, unless specifically stated otherwise, this means that it may include additional components rather than excluding other components.

[0050] The present invention provides a function that utilizes a real-time video analysis model specialized for industrial sites, such as construction sites, to extract situational analysis, risk factors, and safety measure elements for a construction site, and to predict and explain risks through time-series analysis, as well as a technique for forecasting accident types and occurrence probabilities for individual work sites.

[0051] More specifically, natural language captioning and time series prediction analysis are performed on the conditions and risk factors of the work site from images collected from video sensors (Thermal, RGB cameras, body cams, smartphones, drone cams, etc.) installed at industrial sites.

[0052] In addition, it provides a multimodal AI prediction model that predicts and forecasts accident types and occurrence probabilities by fusing a database of past construction site accident cases with risk factors at individual work sites through natural language input.

[0053] Figure 1 is a configuration diagram of a multimodal AI system for automating risk assessment and accident type prediction that reflects the changing work environment of industrial sites.

[0054] A multimodal AI system for automating risk assessment and accident type prediction is composed of an object recognition unit, a scene graph generation unit, a template selection unit, a sentence generation unit, a logging unit, and a time series analysis unit.

[0055] First, the object recognition unit performs instance segmentation for each frame of the input image to identify the areas and types of objects such as people, equipment, structures, PPE (safety hook fasteners, etc.), and safety control means (safety railings, etc.). For example, two workers, one mobile crane, and one scaffolding can be detected in one frame.

[0056] Next, the scene graph generation unit generates a graph in which the detected objects are nodes and the spatial relationships between objects (e.g., "on," "next to," distance between the worker and the object, etc.) or interactions (e.g., "the worker operates the crane") are represented as edges. This allows for the representation of the scene's structure and situational context, rather than a simple list of objects.

[0057] That is, the object recognition unit performs instance segmentation for each frame of the input image to detect objects and identify the region and type of the objects, and the scene graph generation unit uses the detected objects as nodes and represents the spatial relationships or interaction relationships between objects as edges to construct the structure and situational context of the scene as a graph.

[0058] Next, the template selection unit compares the information extracted from the scene graph with the prior knowledge, the hazard map, to determine the description template to be applied to the scene. The hazard map defines keywords and sentence patterns for various disaster situations.

[0059] That is, the template selection unit compares the information extracted from the scene graph with a predefined hazard type dictionary (Hazard Map) to select a description template corresponding to the scene.

[0060] For example, if a graph interpretation such as "a worker is in a high place without wearing a safety helmet" is given, this corresponds to a fall risk, so the "Fall Risk" template is selected. The template is a sentence framework written in Jinja2 syntax, allowing object names or risk types to be inserted in variable places, such as "since a {{worker}} is working without a safety helmet at the site, {{risk type}} may occur." In the captioning sentence generation stage, the actual object names and situational information are filled into the selected template to render the final Korean sentence. At this stage, object names are replaced with Korean expressions according to the previously defined Korean terminology table (e.g., "worker" → "근로자", "scaffold" → "비계", etc.), and finally, the actual object names and situational information are substituted into the selected description template to generate a natural language description of the risk situation. In this process, a natural Korean sentence such as "since a worker is working without a safety helmet at the site, a fall risk may occur" is generated through the Ko-BART model tuned for the work site of the applicable industry.

[0061] Next, the logging unit stores all intermediate outputs of caption generation for the corresponding frame. Specifically, it records the list of detected objects, the scene graph structure, the selected template ID, and the final generated situation and risk type captions in JSON format, and displays the generated sentences as an overlay on the main screen. Through this log data, operators can review the basis of the AI's judgment retrospectively, and if false positives or undetected cases are found, they can trace back to where the problem occurred.

[0062] For example, when an incorrect caption is generated, checking the logs can identify the cause as a mismatched template or a missing object.

[0063] FIG. 2 is a hardware configuration diagram equipped with a multimodal AI system according to the present invention.

[0064] Referring to FIG. 2, the multimodal AI system according to the present invention detects objects in an input image input through a camera, identifies the area and type of the objects, constructs the structure of the scene and the situational context into a graph, selects a description template corresponding to the scene, generates a description of the risk situation in natural language form, analyzes the increase or decrease in risk and the possibility of future risk occurrence in a time series, provides the analyzed information to a mobile device, and can provide it to an integrated control system through an integrated server (cloud).

[0065] Figures 3 and 4 show examples of a Hazard Map and a Template, respectively, that are automatically generated in the present invention. Both documents are designed to be automatically generated.

[0066] Finally, the time series analysis unit analyzes the increase or decrease in risk and the likelihood of future risk occurrence in a time series using information extracted from consecutive frames of the video and sensor sensing data.

[0067] In particular, the time series analysis unit of the present invention for predicting changing risk levels performs captioning to understand the situation in units of fixed windows (continuous frames) input from CCTV video, structures changes in relationships within the window using spatiotemporal scene graphs, and enriches the context by combining worker behavior recognition to accumulate and analyze changing risk situations caused by dynamic accident factors in a time series, thereby predicting increases or decreases in risk levels and simultaneously performing alarms and forecasts when necessary.

[0068] More specifically, video streams received from field cameras are captioned in fixed-length time windows (1-2 seconds recommended). In each frame, people, equipment, structures, PPE, and safety equipment are identified through instance segmentation, and consistent IDs are assigned to identical objects through object tracking in consecutive frames. Once instance segmentation and tracking are completed for a single window, the results are integrated to construct a spatiotemporal scene graph.

[0069] The nodes of the graph are "object × time" pairs, and there are two types of edges. One is a spatial relationship edge connecting pairs of objects at the same time, representing values ​​directly related to safety judgments, such as center distance, azimuth, height difference, and relative approach speed.

[0070] The other is a time edge connecting identical objects across consecutive timeframes, indicating transitions in location, behavior, or PPE state, as well as occlusion status.

[0071] This allows for the simultaneous representation of not only the structure of the scene (what is where) but also the continuity and change of relationships (how they have evolved), enabling the understanding of danger signals (e.g., gradual approach, persistent failure to wear protective gear during work at height) that are missed by single-frame-based static interpretation.

[0072] While physical risk factors such as distance and height can be extracted using only spatiotemporal scene graphs, worker action recognition must be combined to more faithfully understand the actual work context.

[0073] To this end, the present invention utilizes two series of lightweight models in parallel. First, it sensitively detects posture-based warning signals, such as loss of balance, sudden torso rotation, and dangerous leaning, through the ST-GCN series, which learns by viewing the time series of human joint keypoints as a graph.

[0074] Second, a lightweight clip-based video model (X3D or SlowFast lightweighting) that encompasses foreground movement is used to recognize work actions such as cutting, unloading, and carrying. The outputs of the two series are added to the scene graph with additional labels on nodes and edges (e.g., "person-tool: operate", "balance_loss=0.72"), and as a result, a combined representation of state (space / PPE) × action (pose / motion) becomes possible.

[0075] This enables the early detection of high-risk patterns combining multiple pieces of evidence, such as "unfastened harness during work at height + signs of loss of balance + absence of handrail."

[0076] The spatiotemporal scene graph generated for each window is immediately calculated as a window summary indicator.

[0077] For example, minimum distance, duration of proximity, maximum height difference, maximum value of relative approach speed (in the decreasing direction), accumulated time without PPE, presence of handrails / nets, number of times crossing danger boundaries, etc.

[0078] These indicators are normalized into window-unit risk scores. The score calculation is designed using lightweight formulas such as weighted sum and exponential decay to ensure explainability, and the evidence frame, time interval, and corresponding rule ID are logged together.

[0079] The window-unit risk scores and indicators calculated in this way are accumulated as time-series data of risk situations, and based on this time-series data, the accumulated risk and its growth rate are monitored in near-real-time to generate an immediate alert when a threshold is exceeded, or short-term / medium-term risk forecasting is performed by taking the multivariate sequence of the last N windows as input.

[0080] In particular, the time series analysis unit improves prediction accuracy by analyzing the increase or decrease in risk through fused time series analysis of multimodal data capable of detecting accident precursors—such as collapse-related sound and vibration sensing data, worker safety hook fastening data, and tilt sensing data—as well as analyzing not only risk factors for each frame of the video but also the presence of safety measures.

[0081] For example, it detects risks in a changing work site and uses risk factors and whether safety measures are implemented for each frame of the video in near real-time to reduce the risk level if safety measures are implemented and increase the risk if they are not, and it also detects real-time risks by fusion analysis of risk-causing objects and forecasts risks such as after minutes or after one hour.

[0083] FIG. 5 is a diagram illustrating a multimodal AI method for automating risk assessment and accident type prediction that reflects the changing work environment of an industrial site according to an embodiment of the present invention.

[0084] In Figure 5, the preprocessing process inputs include Excel documents of accident cases, Excel documents of risk assessments, PDF files of laws, guidelines, regulations, and accident case data related to construction sites, and outputs include SFT training data, risk assessment rule data, accident case data, and a construction site-specific corpus. Here, SFT (Supervised Fine-Tuning) is a step of fine-tuning a model to a specific task using labeled data through supervised learning.

[0085] In this invention, fine-tuning is performed in two stages to specialize a Korean language model for the field of construction safety. The first stage of KoBART training involves DAPT / TAPT training, where the input is a construction site-specific corpus and the output is a KoBART model specialized for construction sites. Here, DAPT (Domain-Adaptive Pre-Training) adds pre-training with a specific domain corpus, and TAPT (Task-Adaptive Pre-Training) adds pre-training with a specific task dataset (unlabeled text) to improve downstream task performance.

[0086] Likewise, the second KoBART learning step is the stage of performing SFT learning, the input is SFT learning data (supervised learning), and the output is a KoBART model specialized in construction site risk assessment and situation captioning.

[0087] Jinja is a Python-based template engine and a tool that enables the dynamic generation of text documents. It is a system that automatically creates documents by inserting data (variables) into a fixed document structure (template).

[0088] In the video captioning inference model of the present invention, when a video to be analyzed is input, it is converted into an image, and since various image processing techniques described below are processed in parallel, the situational awareness performance regarding the scene can be improved.

[0089] It processes Instance Segmentation, Bytetrack, and Pose Estimation in parallel to detect objects at the most important pixel level, with the input being an image and the outputs being Instance Segmentation information, object tracking information, and object behavior information. Here, Instance Segmentation has the advantage of accurately distinguishing the outlines of objects (more precise than Bounding Boxes) because, unlike object detection which indicates what is where using Bounding Boxes, Instance Segmentation separates objects at the pixel level to distinguish them even within the same class.

[0090] Basically, object detection only provides location per frame, but ByteTrack improves object tracking success rate (Recall) because it tracks the same object over time, assigns an ID, and considers even low-confidence boxes.

[0091] Pose Estimation (Option) is a technique that extracts keypoints, that is, predicts the position of a person's joints in the form of coordinates (x, y). It is a technique that generally uses 17 to 25 joint points (COCO dataset: 17 keypoints) to generate a skeleton and visualizes the skeletal structure by connecting the extracted keypoints with lines, making it a useful technique for predicting human actions, behaviors, and poses.

[0092] Figures 6 and 7 show the results of captioning two image frames. In this case, Figure 6 extracts a scene through instance segmentation in which formwork and support structures are arranged and two workers are working in the central area while wearing PPE. The video captioning system of the present invention uses this to generate a scene graph and simultaneously determines the "risk of falling in the high-altitude work section" and the "risk of entrapment / falling near formwork / supports" by combining the location of the horse scaffolding, the workers' working height, and the density of surrounding temporary materials, and reflects the safety gloves / safety shoes detected as PPE clues in the caption. The safety helmet was described as "need to check wearing status" because there were obscured / low-reliability sections.

[0093] Captioning was carried out similarly in Figure 7 as well. In both experimental results, the match between the danger template and the scene cue was clear, and it was confirmed that the Korean mapping of nouns and technical vocabulary was also consistent.

[0094] The aforementioned evaluation experiment was conducted in a MacBook Pro (M1, 8-core CPU / 8-core GPU / 16-core Neural Engine) environment using Python 3.10 and the open-source KoBART, and the template was processed using jinja2.

[0095] The data used in the experiment consists of actual video footage of short formwork-related operations. Instance segmentation was performed prior to the experiment, and the results were used to conduct the captioning experiment. Consistency is described using two auxiliary indicators.

[0096] First, we verified how faithfully the captions reflect the core objects actually detected and segmented from the perspective of Object-Caption Alignment. The implemented captioning system used rapidfuzz for synonym normalization to calculate the noun set agreement, and the results confirmed high performance consistency with stable explicit mapping within sentences.

[0097] Second, we checked whether the selected risk template was consistent with the rules of the Hazard Map and confirmed that the data extracted from all frames corresponded coherently with the risk clues (treasury, shoring, formwork, etc.) and the template.

[0098] Latency was measured only for the selector (condition augmentation and require_facts) and Jinja2 renderer sections, and showed response times of several milliseconds (ms) even in M1 CPU-only execution. This is considered sufficiently negligible compared to instance segmentation latency, and the latency of the development system depends on the performance of instance segmentation, which is sufficient for applying real-time or near-real-time risk detection and risk prediction to video.

[0099] The present invention performs the function of, upon receiving an image, analyzing the types, number, and relationships between objects within the image to determine the situation of the work site represented by the image, particularly the nature of the work, extracting risk factors of the work site, and inferring risk factors and safety measures through the risk factors and work content.

[0100] In order to improve the performance of risk assessment, the present invention incorporates expert knowledge, such as SIF (serious injury & fatality) and measures from the Korea Occupational Safety and Health Agency’s draft risk assessment table guidelines, into the template and risk map dictionary in addition to the results of risk factors and measures through the video captioning of the present invention. Although there are some differences depending on the complexity of the scene, template matching and sentence rendering are mostly rule-based operations with very little latency, and processing of more than a few frames per second is possible by using a parallel-optimized deep learning inference engine.

[0101] Figures 8 and 9 are examples of the results of automated risk assessment processing at an actual industrial site using the multimodal AI model of the present invention.

[0102] Meanwhile, Fig. 10 is a flowchart of the operation of a module that assesses risks and predicts risks for a work site that changes frequently in real time.

[0103] In FIG. 10, the present invention is characterized by extracting risk factors through images, risk factors through sensing data such as inclination or strain, risk mitigation factors such as the installation of safety railings and the fastening of safety hooks, and risk factors determined by the worker themselves to evaluate the final risk, and predicting and forecasting near-real-time, short-term, and medium-to-long-term risks by performing time-series analysis on the risk factors analyzed and evaluated over time.

[0104] Risk assessment for such changing work sites involves inputting real-time video from CCTVs or smartphones, and the multimodal AI module converts the video into images at a rate of only a few frames per second for analysis. Additionally, it performs fused analysis with collapse-related sound and vibration sensing data, data on whether workers have fastened their safety harnesses, and tilt sensing data. Based on the analyzed risk factors, a continuous risk assessment is conducted; if the risk level is deemed dangerous enough to warrant a work stoppage, an alarm is immediately transmitted via flashing lights at the site, vibration alerts to workers, and phone calls or vibration alerts to remote safety managers and site supervisors.

[0105] Of course, in the case of non-serious risk factors, event processing is not performed unless the risk factor increases or the type of disaster changes; however, if the number of risk factors increases or the grade of the disaster type increases, a threshold is set to transmit a risk alert. Additionally, a window is set in increments of seconds or one minute, and the extracted risk factors are analyzed in a time series to perform a function of predicting near-real-time, short-term accident risks.

[0106] Meanwhile, FIG. 11 is a flowchart of the operation of a multimodal AI model that predicts accident types and probability of occurrence for individual work sites according to the present invention.

[0107] The structure of the individual construction site-specific risk forecasting neural network model of the present invention is a multi-input neural network structure for simultaneously predicting human accidents and material accidents. The model processes text data (KoBERT embeddings), continuous data (standardized numerical variables), and categorical data (embedded categorical variables) in separate input layers, and then combines them in an integrated neural network structure to perform a final prediction. The main components of the model are as shown in Table 1 below.

[0108] [Table 1]

[0109]

[0110] In Figure 11, KoBERT embeddings were utilized to vectorize detailed accident information, such as work content described in natural language in construction site accident case data. The 768-dimensional embedding vector extracted from KoBERT was passed to a fully connected (Dense) layer with 512 neurons to compress information, and Dropout (0.3) was applied to prevent overfitting. Additionally, non-linearity was introduced by using ReLU as the activation function, and standardized numerical data was input into a fully connected (Dense) layer with 128 neurons. Batch Normalization was applied to enhance the stability of the learning process. Subsequently, the ReLU activation function was used to improve the model's expressiveness, and categorical variables were individually converted into unique vectors through the embedding layer. These were then flattened and used as inputs to the neural network. KoBerT is useful for understanding Korean with a small number of parameters, approximately 110 million, whereas KoBART is advantageous for generating 400 million Korean words.

[0111] The outputs of each input layer are combined in a Concatenation Layer to generate an integrated feature representation, then pass through a dense layer with 256 neurons to apply Dropout (0.3) and ReLU activation functions, and additionally refine the features through a layer with 128 neurons and Dropout (0.2).

[0112] Finally, the model's prediction results were derived using two Softmax output layers that predict human accidents and material accidents, respectively. In the output stage, human accidents and material accidents were configured as separate Softmax layers to predict them simultaneously, and Focal Loss was applied to reduce learning bias caused by imbalanced data.

[0113] Hyperparameter search was performed using KerasTuner to optimize the performance of the forecasting model of the present invention. The key hyperparameters and the optimal values ​​for the search range are shown in Table 2 below. Through the hyperparameter optimization process performed in the present invention, it was possible to maximize the accuracy of the model while simultaneously mitigating the problem of overfitting. ReLU (Rectified Linear Unit) was applied as the activation function of the hidden layer, and the Softmax function was applied in the output layer to calculate the probability value for each class in multi-class classification problems. Through this, the model is able to derive a predicted value by normalizing the probability that the input data belongs to a specific class.

[0114] [Table 2]

[0115]

[0116] In actual accident data sets from construction sites, unstructured text data such as accident names, accident circumstances, and specific causes serve as key elements in explaining the causes of accidents. In this invention, unstructured text data is embedded using a pre-trained KoBERT model. Text columns are combined into a single sentence, and then the KoBERT model is applied to convert each accident case into a 768-dimensional vector. This vector is utilized as input to a deep learning model to improve accident prediction performance. By applying KoBERT embedding, contextual meaning could be reflected more accurately than with existing TF-IDF and Word2Vec-based methods, and this approach contributes to the improvement of the accident prediction model's performance.

[0117] In the actual accident data from construction sites, categorical interval data was converted into representative values ​​(e.g., "10~19%" → mapped to 15), and a standardization method was applied using StandardScaler to standardize the data into a normal distribution with a mean of 0 and a variance of 1, thereby reducing the influence of scale differences in individual variables during model training and maintaining data balance.

[0118] Furthermore, in the case of human accidents, which are the target variable in actual construction accident case data, there is a severe imbalance between classes. To mitigate this, the present invention applies the Synthetic Minority Over-sampling Technique (SMOTE). In the original data, a problem arises where specific types of human accidents occur with very low frequency; however, by applying SMOTE to synthesize minority class data, the learning effect is improved, thereby enhancing the predictive performance for rare accident types.

[0119] In the detailed description of the present invention as described above, specific embodiments have been described. However, various modifications are possible within the scope of the present invention. The technical concept of the present invention should not be limited to the aforementioned embodiments, but should be defined by the claims as well as equivalents thereof.

Claims

Claim 1 An object recognition unit that performs instance segmentation for each frame of an input image to detect objects and identify the region and type of the objects; a scene graph generation unit that constructs a scene-temporal graph representing the structure and situational context of the scene by using detected objects as nodes and expressing spatial or interactional relationships between objects as edges; a template selection unit that generates and stores a description template corresponding to the scene based on Jinja2 by comparing data extracted from the scene-temporal graphs with a predefined hazard map, and selects a description template corresponding to the scene by comparing information extracted from the scene-temporal graph of the input image with the predefined hazard map; a sentence generation unit that generates a description of a dangerous situation in natural language by substituting actual object names and situational information into the selected description template; and a logging unit that stores intermediate outputs of caption generation for each frame. and includes a time-series analysis unit that analyzes the increase or decrease in risk and the possibility of future risk occurrence in a time-series manner based on the scene-temporal, the hazard map graph, and the description template, using information extracted from consecutive frames of the video and sensing data from the sensor; wherein the object recognition unit performs an instance segmentation technique, a ByteTrack method, and a pose estimation method in parallel for each frame of the input video to output instance segmentation information, object-specific tracking information, and object behavior information; wherein the instance segmentation information is configured such that, when displayed as a bounding box, objects are separated by pixel to distinguish them by object, and the scene graph generation unitA scene-temporal scene graph is constructed by integrating instance segmentation and object tracking results within a single time window, wherein the nodes of the graph are defined as "Object x Time," and the edges of the graph include spatial relationship edges containing center distance, azimuth, height difference, relative velocity, etc., between objects at the same point in time, and temporal relationship edges connecting identical objects at consecutive point in time, thereby simultaneously expressing information on the structure of the scene and changes in relationships; the scene-temporal scene graph is combined with an ST-GCN-based posture / behavior recognition system that learns by modeling human joint keypoint time series as a graph, and a video behavior recognition system that recognizes cutting, transport, or unloading operations using X3D or a lightweight SlowFast model, to detect high-risk patterns early by combining multiple grounds such as failure to wear a harness during work at height, signs of loss of balance, and handrail failures; the hazard map is constructed with templates and risk maps by reflecting not only risk factor cases and safety measure standards by work type but also SIF (Serious Injury & Fatality) expert knowledge, and the description generated and stored based on Jinja2 Regarding the template, template matching, which selects a similar description template based on the similarity of the hazard situation of the description template to the input image, and sentence rendering, which modifies the text into terms and sentences suitable for the domain, are performed as rule-based operations; multiple frames per second are processed through a parallel-optimized deep learning inference engine; the template selection unit determines a description template to be applied to the corresponding scene by comparing information extracted from the spatiotemporal scene graph with the hazard type dictionary (Hazard Map); and the sentence generation unit converts object names into Korean according to a Korean term table and generates natural hazard situation descriptions by fine-tuning the Ko-BART language model for industrial safety and construction safety domains using DFT (Domain fine-tuning) and TFT (Task fine-tuning).A system characterized by training the Ko-BART language model using Domain-Adaptive Pre-training (DAPT) which pre-trains with a specific domain corpus and Task-Adaptive Pre-training (TAPT) which pre-trains with a specific unlabeled task dataset, and the logging unit which records the list of objects detected per frame, the generated scene graph structure, the selected template ID, and the final generated risk situation description in JSON format, and overlays and displays the generated risk situation description on key video frames to provide the basis for predicting the risk situation, thereby enabling explanation. Claim 2 A system according to claim 1, wherein the object recognition unit processes a video stream received from a field camera by dividing it into a time window of a predetermined length, recognizes people, equipment, structures, PPE (Personal Protective Equipment), and safety equipment through instance segmentation in each frame, and assigns a consistent ID to the same object through object tracking between consecutive frames. Claim 3 delete Claim 4 delete Claim 5 delete Claim 6 delete Claim 7 delete Claim 8 A system according to claim 1, wherein the time series analysis unit calculates a spatiotemporal scene graph generated in units of time windows into a summary indicator and a risk score, accumulates this as time series data to monitor the risk level and rate of change in near real-time, performs an alarm when a threshold is exceeded, or performs short-term and medium-term risk prediction using a multivariate sequence of the last N windows as input. Claim 9 A system according to claim 1, wherein the time series analysis unit calculates window summary indicators, such as risk scores and relationship change amounts, based on spatiotemporal scene graphs generated for each time window, accumulates and stores said indicators as risk situation time series data, calculates the accumulated risk and the rate of increase or decrease thereof from said time series data in near real-time, generates an immediate alarm if a preset threshold value is exceeded, or predicts a short-term or medium-term risk level by using a multivariate sequence for the last N consecutive windows as input.

Images