A multimodal artificial intelligence-based integrated construction site operation and risk prediction system

Abstract

The present invention relates to a multimodal artificial intelligence-based construction site integrated operation and risk prediction system implemented by a computing device comprising one or more processors and one or more memories for storing instructions executable by said processors, comprising: a multimodal information synchronization unit that, upon receiving first multimodal information from a plurality of linked information generation units, maps detailed information included in said first multimodal information to a reference time stamp and a reference coordinate frame, and stores second multimodal information generated by synchronizing to the same time and spatial coordinates in a database; and a decision information generation unit that, when the function of said multimodal information synchronization unit is completed, reflects said second multimodal information in previously input construction context information to generate site situation information capable of identifying the current situation of the construction site, and analyzes said site situation information through a previously stored artificial intelligence algorithm to generate decision information customized to the current situation of the construction site. The present invention is characterized by including an operational decision provision unit that, once the generation of the above decision information is completed, compares and analyzes the risk assessment information generated by a worker at a construction site with the above decision information, identifies discrepancies through the analysis results, provides verification result information regarding the discrepancies to a manager, inputs response information in which the manager responds to the verification result information into the previously stored artificial intelligence algorithm to generate final decision information in which the response information is reflected as feedback, and provides this information to the manager, thereby enabling the manager to operate the construction site in an integrated manner through the final decision information. In addition to this, various embodiments identified through this document are possible.

Description

Technology Field

[0001] The present invention relates to a multimodal artificial intelligence-based integrated operation and risk prediction system for construction sites. Specifically, it relates to a technology that integrately manages safety, process, quality, and operation of a construction site by synchronizing multimodal information, including weather information, BIM information, sensor information, image information, spatial shape information, and biometric information generated at the construction site, based on construction location and time; analyzing site situation information generated by reflecting construction context information in the multimodal information through a pre-stored artificial intelligence algorithm to predict complex risk situations at the construction site; and generating decision information regarding the predicted complex risk situations. Background Technology

[0002] Recently, construction sites have been classified as high-risk working environments where the risk of various safety accidents is ever-present due to the use of large equipment, work at heights, the operation of heavy machinery, and the progression of complex processes. In particular, since the working environment at construction sites is characterized by frequent changes depending on weather conditions, project schedules, worker status, and equipment operation, real-time management and response are required.

[0003] Accordingly, intelligent CCTVs utilizing AI-based video analysis technology, as well as technologies for detecting safety helmet usage, intrusion into hazardous areas, and fall and collision detection, have been developed and are being utilized for on-site safety management. These technologies are primarily operated by detecting specific risk events and providing warnings based on camera video data.

[0004] However, conventional construction site safety management technologies rely on a single information source or modality to assess hazardous situations, which limits their ability to comprehensively consider various factors such as weather, structural condition, worker vital signs, and process data. Furthermore, as they remain limited to detecting risks based on predefined event patterns, they face difficulties in predicting complex hazardous situations or new types of accidents in advance.

[0005] Accordingly, the industry is developing various technologies to solve the aforementioned problems.

[0006] As an example, Korean registered patent 10-2505765 (IoT-based wearable safety visualization system for construction site worker safety) discloses a technology for monitoring worker safety through an IoT-based wearable device.

[0007] However, the aforementioned prior art only discloses technology that ensures the safety of workers through IoT-based wearable devices, but does not disclose technology that integrately manages safety, process, quality, and operation of a construction site by synchronizing multimodal information including weather information, BIM information, sensor information, image information, spatial shape information, and biometric information generated at the construction site based on construction location and time, analyzing site situation information generated by reflecting construction context information in the multimodal information through a stored artificial intelligence algorithm to predict complex risk situations at the construction site, and generating decision information regarding the predicted complex risk situations. Therefore, there is a need for technology that can solve this problem. The problem to be solved

[0008] Accordingly, the present invention is derived to solve the problems of the aforementioned existing technologies. Its purpose is to improve the overall safety, operational efficiency, process stability, and cost management levels of a construction site by synchronizing multimodal information—including weather information, BIM information, sensor information, image information, spatial shape information, and bio-information generated at the construction site—based on construction location and time, analyzing site situation information generated by reflecting construction context information in the multimodal information through a pre-stored artificial intelligence algorithm to predict complex risk situations at the construction site, and generating decision information regarding the predicted complex risk situations, thereby integrally managing safety, process, quality, and operation of the construction site to predict complex risk situations in advance and support optimal operational decision-making. means of solving the problem

[0009] A multimodal artificial intelligence-based construction site integrated operation and risk prediction system implemented by a computing device comprising one or more processors and one or more memories for storing instructions executable by said processors according to an embodiment of the present invention, wherein, upon receiving first multimodal information from a plurality of linked information generation units, a multimodal information synchronization unit maps detailed information included in said first multimodal information to a reference time stamp and a reference coordinate frame, and stores second multimodal information generated by synchronizing to the same time and spatial coordinates in a database; and a decision information generation unit, upon completion of the function of said multimodal information synchronization unit, generates site situation information capable of identifying the current situation of a construction site by reflecting said second multimodal information in previously input construction context information, and generates decision information customized to the current situation of a construction site by analyzing said site situation information through a previously stored artificial intelligence algorithm. The present invention is characterized by including an operational decision provision unit that, when the generation of the above decision information is completed, compares and analyzes the risk assessment information generated by a worker at a construction site with the above decision information, identifies discrepancies through the analysis results, provides verification result information regarding the discrepancies to a manager, inputs response information in which the manager responds to the verification result information into the previously stored artificial intelligence algorithm to generate final decision information in which the response information is reflected as feedback, and provides this information to the manager, thereby enabling the manager to operate the construction site in an integrated manner through the above final decision information.

[0010] It is preferable that the above-mentioned linked information generation units include: a weather information generation unit that generates weather information including wind speed, precipitation, temperature, humidity, and atmospheric pressure of a region including a construction site; a BIM (Building Information Modeling) information generation unit that generates BIM information including spatial coordinates, floor zones, material placement, and process plans of a construction structure being constructed at the construction site; an IoT information generation unit that generates sensing information including vibration, deformation, load, gas concentration, and noise through a plurality of IoT sensors installed at the construction site; an image information generation unit that generates image information including real-time captured video captured through a camera module installed at the construction site; a spatial shape generation unit that generates spatial shape information including 3D point cloud data acquired through a LiDAR sensor mounted on a drone flying over the construction site; a biometric information generation unit that generates biometric information including a worker's heart rate, body temperature, location, and motion patterns measured through a wearable device worn by a worker at the construction site; and a process management information generation unit that generates process management information including work schedules, pouring plans, personnel input, and equipment placement at the construction site.

[0011] When the reception of the first multimodal information is completed, the multimodal information synchronization unit maps the detailed information included in the first multimodal information to correspond to a common reference time based on the reference time stamp and a common coordinate system based on the reference coordinate frame in order to align them according to a common reference, and completes noise removal and normalization processing, thereby completing the generation of second multimodal information corresponding to the common reference and storing it in the database.

[0012] It is preferable that the above decision information generation unit comprises: a situation information generation unit that, when the generation of the second multimodal information is completed, reflects detailed information based on the second multimodal information into a background scenario based on the previously entered construction context information to generate site situation information capable of identifying the current situation of the construction site; and a situation analysis unit that, when the generation of the site situation information is completed, analyzes the relationship between detailed contexts based on the site situation information through the previously stored artificial intelligence algorithm to identify a complex risk situation derived from the analysis results, compares the similarity between the identified complex risk situation and a plurality of accident history information, and generates decision information to improve the complex risk situation based on a response plan corresponding to the accident history information with the highest similarity.

[0013] If the aforementioned situation analysis unit identifies the aforementioned complex risk situation as a pre-set critical risk situation, it is possible to immediately dispatch and control a drone linked to the construction site before generating the aforementioned decision information, thereby receiving real-time video of the construction site via the drone and transmitting it to the manager while outputting a warning message to the workers within the construction site.

[0014] It is preferable that the above decision information generation unit identifies a request intent value based on request information input by a manager, sets the mode of the previously stored artificial intelligence algorithm to one of an inference mode, a response generation mode, and a computational resource allocation mode based on the identified request intent value, and controls the previously stored artificial intelligence algorithm to be executed in an on-device or on-premise manner according to the set mode.

[0015] The aforementioned stored artificial intelligence algorithm is an algorithm that includes a model for periodically learning pattern values ​​derived by analyzing correlations between field situation information, which is accumulated learning information; complex risk situations based on detailed contexts derived from the field situation information; final decision information based on response measures corresponding to the complex risk situations; and result feedback information based on the final decision information. It is preferably distributed to a device deployed at a construction site via wireless communication, optimized to correspond to the computational specifications of said device, and receives second multimodal information provided to said device and analyzes it in an on-device manner, thereby performing functions independently even in an environment where network connectivity is limited.

[0016] It is preferable that the above operational decision providing unit includes: a verification result providing unit that, when the generation of the above decision information is completed, compares and analyzes the contents of the risk assessment information and the above decision information by item, identifies items corresponding to inconsistencies through the analysis results as inconsistencies, and provides verification result information including contents based on the inconsistencies to the manager; and a final decision providing unit that, when the function of the above verification result providing unit is completed, receives response information responding to the above verification result information from the manager, inputs the above response information as a weight into the above stored artificial intelligence algorithm, thereby causing the above stored artificial intelligence algorithm to generate final decision information in which the above response information is reflected as feedback, and provides it to the manager.

[0017] The above multimodal artificial intelligence-based construction site integrated operation and risk prediction system further includes an information layer management unit; wherein the information layer management unit classifies and stores site operation information, learning analysis information, and personal sensitive information obtained at the construction site based on pre-set classification conditions, and manages them in a layered manner by classifying them into hot classification information for real-time processing based on the pre-set classification conditions, warm classification information for analysis and learning, and cold classification information for storage and disposal, and in the case of the hot classification information, it is preferable to store it after de-identification processing or, if de-identification processing is not possible, destroy it after a pre-set period.

[0018] The above-described multimodal artificial intelligence-based construction site integrated operation and risk prediction system further comprises a learning pipeline construction unit; wherein the learning pipeline construction unit preferably comprises: a raw log collection unit that collects site operation history information in the form of raw log data, the second multimodal information, the site situation information, information on complex risk situations, the decision information, the verification result information, the response information, the final decision information, and result feedback information regarding the final decision information; and a learning information generation unit that, upon completion of the collection of the site operation history information, performs a semi-automatic verification process to tag a tag value based on a pre-set tagging condition for the detailed information included in the site operation history information and then allows an administrator to verify the site operation history information, and upon completion of the verification of the site operation history information, converts and processes the site operation history information into an instruction tuning data format that the pre-stored artificial intelligence algorithm can learn from and stores it in a learning database. Effects of the invention

[0019] Through the pet health management system using digital twin and artificial intelligence learning of the present invention, complex risk situations can be predicted in advance and optimal operational decisions can be automatically derived, thereby significantly improving safety and process stability at construction sites. Brief explanation of the drawing

[0020] FIG. 1 is a block diagram illustrating a multimodal artificial intelligence-based construction site integrated operation and risk prediction system according to one embodiment of the present invention. FIG. 2 is a block diagram illustrating a decision information generation unit of a multimodal artificial intelligence-based construction site integrated operation and risk prediction system according to one embodiment of the present invention. FIG. 3 is another block diagram illustrating a decision information generation unit of a multimodal artificial intelligence-based construction site integrated operation and risk prediction system according to one embodiment of the present invention. FIG. 4 is a block diagram illustrating an operational decision-making providing unit of a multimodal artificial intelligence-based construction site integrated operation and risk prediction system according to one embodiment of the present invention. FIG. 5 is a block diagram illustrating the learning pipeline construction section of a multimodal artificial intelligence-based construction site integrated operation and risk prediction system according to one embodiment of the present invention. FIG. 6 is a drawing for explaining an example of the internal configuration of a computing device according to an embodiment of the present invention. Specific details for implementing the invention

[0021] Hereinafter, various embodiments and / or aspects are disclosed with reference to the drawings. For illustrative purposes, numerous specific details are disclosed in the following description to aid in a general understanding of one or more aspects. However, it will also be recognized by those skilled in the art that these aspects may be practiced without such specific details. The following description and the accompanying drawings describe specific exemplary aspects of one or more aspects in detail. However, these aspects are exemplary, and some of the various methods in the principles of the various aspects may be used, and the description is intended to include all such aspects and their equivalents.

[0022] As used herein, terms such as "examples," "examples," "aspects," "examples," etc., may not be interpreted as implying that any aspect or design described is better or more advantageous than other aspects or designs.

[0023] Additionally, the terms “comprising” and / or “comprising” should be understood to mean that the relevant feature and / or component is present, but not to exclude the presence or addition of one or more other features, components and / or groups thereof.

[0024] Additionally, terms including ordinal numbers, such as first, second, etc., may be used to describe various components, but said components are not limited by said terms. Such terms are used solely for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be named the second component, and similarly, the second component may be named the first component. The term "and / or" includes a combination of a plurality of related described items or any of a plurality of related described items.

[0025] Furthermore, in the embodiments of the present invention, all terms used herein, including technical or scientific terms, unless otherwise defined, have the same meaning as generally understood by those skilled in the art to which the present invention pertains. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in the embodiments of the present invention.

[0026] FIG. 1 is a block diagram illustrating a multimodal artificial intelligence-based construction site integrated operation and risk prediction system according to one embodiment of the present invention.

[0027] Referring to FIG. 1, a multimodal artificial intelligence-based construction site integrated operation and risk prediction system (100) (hereinafter referred to as a site management system) implemented as a computing device comprising one or more processors and one or more memories for storing instructions that can be executed by said processors may include a multimodal information synchronization unit (101), a decision information generation unit (103), and an operation decision provision unit (105).

[0028] According to one embodiment, when the multimodal information synchronization unit (101) receives the first multimodal information (101a) from a plurality of linked information generation units (107), it maps the detailed information included in the first multimodal information (101a) to a reference time stamp and a reference coordinate frame, and stores the second multimodal information generated by synchronizing to the same time and space coordinates in a database.

[0029] According to one embodiment, the plurality of interconnected information generation units (107) may include a weather information generation unit, a BIM information generation unit, an IoT information generation unit, an image information generation unit, a spatial shape generation unit, a bio-information generation unit, and a process management information generation unit.

[0030] According to one embodiment, the weather information generating unit can generate weather information including wind speed, precipitation, temperature, humidity, and atmospheric pressure in an area including a construction site.

[0031] In relation to the above, the weather information is data collected in real time from weather observation equipment installed at the construction site or from an external weather agency API, and can be used to assess the safety of processes sensitive to weather conditions, such as concrete pouring, work at height, and crane operations.

[0032] According to one embodiment, the BIM information generation unit can generate Building Information Modeling (BIM) information including spatial coordinates, floor zones, material placement, and process plans of a construction structure being built at a construction site.

[0033] In relation to the above, the BIM information is three-dimensional model data of a construction structure, including spatial coordinates of each floor area, locations of columns and beams, locations of material placement, and schedule information for each process, and can be used as reference information that provides spatial context for sensor data and image data.

[0034] According to one embodiment, the IoT information generation unit can generate sensing information including vibration, deformation, load, gas concentration, and noise through a plurality of IoT sensors installed at a construction site.

[0035] In relation to the above, the plurality of IoT sensors are sensors distributed among structures, equipment, work zones, etc. at a construction site, and may include acceleration sensors that detect vibration and deformation of structures, load cells that measure loads, gas sensors that measure gas concentration in a confined space, sound meters that measure noise levels, etc.

[0036] According to one embodiment, the image information generating unit can generate image information including real-time captured video captured through a camera module installed at a construction site.

[0037] In relation to the above, the camera module is a CCTV camera fixedly installed in major work areas, entrances, and hazardous areas of a construction site, and can record in real time whether a worker is wearing a safety helmet, whether they have entered a hazardous area, and the operating status of equipment.

[0038] According to one embodiment, the spatial shape generation unit can generate spatial shape information including three-dimensional point cloud data acquired through a LiDAR sensor mounted on a drone flying over a construction site.

[0039] In relation to the above, the three-dimensional point cloud data is a set of three-dimensional coordinates in space generated by a LiDAR sensor mounted on a drone emitting a laser and measuring the time of reflection, and may be data capable of identifying the terrain of a construction site, the shape of a structure, the status of material loading, the location of equipment placement, etc. with millimeter precision.

[0040] According to one embodiment, the biometric information generating unit can generate biometric information including the worker's heart rate, body temperature, location, and movement patterns measured through a wearable device worn by the worker at a construction site.

[0041] In relation to the above, the wearable device may be a sensor device attached to a worker's safety helmet, safety vest, wristband, etc., and may be a device that measures biosignals such as the worker's heart rate and body temperature, GPS-based location information, and acceleration sensor-based motion patterns in real time and transmits them via wireless communication.

[0042] According to one embodiment, the process management information generating unit can generate process management information including a work schedule, pouring plan, personnel input, and equipment placement at a construction site.

[0043] In relation to the above, the process management information is information collected from a construction site schedule management system and may include the types of work to be performed on specific dates and times, the number of personnel and equipment to be deployed, and the planned schedule of major processes such as concrete pouring.

[0044] Accordingly, the weather information, BIM information, image information, spatial shape information, bio-information, and process management information may be included as detailed information of the first multimodal information (101a).

[0045] According to one embodiment, when the multimodal information synchronization unit (101) completes receiving the first multimodal information (101a), it can map the detailed information included in the first multimodal information to correspond to a common reference time based on the reference time stamp and a common coordinate system based on the reference coordinate frame in order to align the details according to a common reference.

[0046] In relation to the above, the first multimodal information (101a) may be composed of heterogeneous data having different time references and coordinate systems, so direct integrated analysis may be impossible.

[0047] According to one embodiment, the reference time stamp is a unified time standard used in the site management system (100), and the reference coordinate frame may be a unified three-dimensional coordinate system for representing all spatial data of the construction site.

[0048] For example, when the multimodal information synchronization unit (101) measures a wind speed of 12 m / s in weather information at 9:30:15 AM, measures a vibration value of 0.8 Hz in area A on the 3rd floor in an IoT sensor, obtains a 3D point cloud of the area in a drone lidar, and measures a heart rate of 95 bpm in a worker wearable device, even if these data each use different time formats (UTC, local timestamp, GPS time, etc.) and coordinate systems (GPS coordinates, local coordinates, relative coordinates, etc.), they can be spatiotemporally synchronized to a position corresponding to a unified reference timestamp and a reference coordinate frame (X:25.3 m, Y:48.7 m, Z:9.2 m) based on a BIM coordinate system, and generate a single integrated data record.

[0049] According to one embodiment, when the mapping is completed, the multimodal information synchronization unit (101) can complete the noise removal and normalization processing to complete the generation of second multimodal information corresponding to the common standard and store it in the database.

[0050] In relation to the above, the noise removal is a process of removing outliers and measurement errors included in the sensor measurement values, and the normalization process may be a process of converting data with different units and scales into the same range.

[0051] According to one embodiment, when the decision information generation unit (103) completes the function of the multimodal information synchronization unit (101), it can generate site situation information that can identify the current situation of the construction site by reflecting the second multimodal information in the previously entered construction context information, and generate decision information (103b) customized to the current situation of the construction site by analyzing the site situation information through a previously stored artificial intelligence algorithm (103a).

[0052] In relation to the above, the previously entered construction context information may be a scenario that sets the ongoing tasks, environment, and risk conditions of a construction site, and may be background set information including process context, spatial context, time context, worker context, environmental context, and operational context as detailed items.

[0053] According to one embodiment, the process context may include information indicating the type and stage of the work currently in progress, such as, for example, "3rd floor slab concrete pouring work in progress, 2 hours elapsed since pouring started, 150 m³ completed out of a total planned pouring volume of 225 m³".

[0054] According to one embodiment, the spatial context is information indicating the physical location and surrounding environment where work is being performed, and may include information such as, for example, "3rd floor Area A, 10.5m above ground, open workspace, 2nd floor work being performed simultaneously nearby."

[0055] According to one embodiment, the time context is information indicating the temporal conditions of the time of the operation, and may include information such as, for example, "11:30 AM, 30 minutes before lunch, 6 hours remaining until sunset, rain is forecast from 2:00 PM."

[0056] According to one embodiment, the worker context is information indicating the status and placement of personnel deployed to the site, and may include information such as, for example, "15 workers deployed, average work time of 3 hours elapsed, including 3 high-altitude workers, 2 newly deployed personnel."

[0057] According to one embodiment, the environmental context is information indicating the physical conditions of the work environment, and may include information such as, for example, "outside temperature 5 degrees, strong wind (wind speed 15 m / s), good road surface condition, normal visibility."

[0058] According to one embodiment, the operational context is information indicating the operational status and constraints of the site, and may include information such as, for example, "equipment utilization rate 90%, sufficient material inventory, nearby area where complaints may occur, no noise restriction time zone."

[0059] According to one embodiment, the decision information generation unit (103) can generate site situation information by reflecting the previously input construction context information in the second multimodal information.

[0060] In relation to the above, the above-mentioned site situation information may be comprehensive situation information of a construction site generated by combining construction context information with sensor data and image data of the second multimodal information.

[0061] For example, if the decision information generation unit (103) identifies "wind speed 15 m / s, 3rd floor slab vibration 2.5 Hz, worker A location (X:15.2 m, Y:12.8 m, Z:10.5 m), heart rate 95 bpm, temperature 5 degrees" from the second multimodal information, and identifies "pouring work in progress in 3rd floor A zone, 2 hours elapsed since pouring started, 150 m³ completed out of 225 m³ planned pouring volume, high-altitude work at 10.5 m above ground, 11:30 AM, rain expected from 2:00 PM, 15 workers deployed, average work time 3 hours elapsed" from the previously entered construction context information, it combines these to identify "pouring work in progress in 3rd floor A zone (progress rate 67%), current wind speed 15 m / s indicating strong wind conditions, structural vibration 2.5 Hz detected, 15 workers performing high-altitude work at 10.5 m above ground, You can generate site situation information including the content “Expected increase in fatigue after 3 hours of average work time, and work needs to be completed within 2.5 hours as rain is expected from 2 PM.”

[0062] According to one embodiment, when the decision information generation unit (103) completes the generation of the site situation information, it can analyze the site situation information through a previously stored artificial intelligence algorithm (103a) to generate decision information (103b) customized to the current situation of the construction site.

[0063] In relation to the above, the artificial intelligence algorithm (103a) stored above may be a deep learning model learned from past accumulated construction site data, which analyzes site situation information and analyzes the relationship between content by detailed context based on site situation information to identify complex risk situations and generates decision information (103b) that comprehensively considers safety, process, quality, and cost.

[0064] According to one embodiment, the decision information (103b) may be composed of a plurality of detailed items including a risk item, a work control item, a target area item, a schedule impact item, a cost impact item, and a safety measure item.

[0065] According to one embodiment, the risk item includes an overall risk grade, accident type, and possible time of occurrence; the work control item includes work suspension and resumption, pouring postponement, and process change; the target area item includes a target area ID, floor / zone, and equipment ID; the schedule impact item includes an expected delay time and alternative schedule; the cost impact item includes additional costs and loss minimization plans; and the safety measures item may include reinforced protective equipment, access control, and enhanced surveillance.

[0066] For example, the decision information generation unit (103) analyzes on-site situation information through the previously stored artificial intelligence algorithm (103a) and includes a "risk level item (overall risk grade: high, accident type: fall of a high-altitude worker due to strong winds, possible time of occurrence: within 30 minutes), work control item (work stoppage: immediately, work resumption: after confirming wind speed 10 m / s or less), target area item (target area ID: 3F-A, floor / zone: 3rd floor A area, equipment ID: pump truck-01, vibratory compactor-03), schedule impact item (estimated delay time: 2 hours, alternative schedule: work needs to be extended until 7 PM if resuming at 4 PM), cost impact item (additional costs: labor costs for extended work 1.5 million won, equipment standby costs 800,000 won, minimum loss plan: minimize delay by concentrating work tomorrow morning), safety measures item (reinforcement of protective equipment: double fastening of safety harnesses for all workers, access control: prohibition of entry to 3rd floor A area, enhanced surveillance: deployment of 2 additional safety managers)" Decision information (103b) can be generated.

[0067] According to one embodiment, when the operation decision providing unit (105) completes the generation of the decision information (103b), it compares and analyzes the risk assessment information generated by the worker at the construction site with the decision information (103b), identifies discrepancies through the analysis results, and provides verification result information regarding the discrepancies to the manager. The manager then inputs the response information (105a) in which it responds to the verification result information into the previously stored artificial intelligence algorithm (103a), thereby generating and providing the final decision information in which the response information (105a) is reflected as feedback, so that the manager can operate the construction site in an integrated manner through the final decision information.

[0068] According to one embodiment, when the generation of the decision information (103b) is completed, the operational decision providing unit (105) can compare and analyze the decision information (103b) with the risk assessment information generated by the worker at the construction site.

[0069] In relation to the above, the risk assessment information is a risk assessment report prepared by a worker or safety manager at a construction site before starting work, and may include expected risk factors, risk grades, safety measures, etc.

[0070] According to one embodiment, the operational decision providing unit (105) can compare and analyze the risk assessment information and the decision information by item to identify inconsistencies through the analysis results.

[0071] For example, the above-mentioned operational decision-making unit (105) can identify the “risk level” item, the “wind risk” item, and the “safety measure” item as inconsistent items when the risk assessment information written by the worker is listed as “risk level: normal, wind risk: low, safety measure: basic safety harness wear”, but the above-mentioned decision-making information (103b) is analyzed as “overall risk level: high, accident type: fall of a high-altitude worker due to strong wind, safety measure: double fastening of safety harness required”.

[0072] According to one embodiment, when the identification of the discrepancy item is completed, the operation decision providing unit (105) may provide verification result information regarding the discrepancy item to the manager.

[0073] In relation to the above, the verification result information is information that quantitatively indicates the difference between the risk assessment information and the decision information (103b) and includes detailed information for each discrepancy item, and can be used as data for an administrator to judge the appropriateness of the risk assessment and decide whether to take additional measures.

[0074] For example, the above operational decision providing unit (105) can generate and provide to the manager the verification result information, "Worker assessment risk grade: Normal vs. AI analysis risk grade: High (discrepancy), Worker assessment strong wind risk: Low vs. AI analysis strong wind risk: High (current wind speed 15 m / s, exceeding the strong wind advisory standard of 14 m / s) (discrepancy), Worker suggested safety measure: Wearing basic safety harness vs. AI recommended safety measure: Double fastening of safety harness + control of access to Zone A on the 3rd floor (discrepancy), Analysis of cause of discrepancy: It is determined that the worker did not check the on-site weather information in real time and underestimated the risk of strong winds during high-altitude work."

[0075] According to one embodiment, when the operational decision providing unit (105) receives response information (105a) in which a manager responds to the verification result information, the response information (105a) is input into the previously stored artificial intelligence algorithm (103a) to generate final decision information in which the response information (105a) is reflected as feedback and provide it to the manager.

[0076] In relation to the above, the response information (105a) is information that includes the approval of some of the contents of the decision information (103b) or the correction of discrepancies after the manager reviews the verification result information, and may include, for example, "Approve AI recommendation," "Immediately execute work stoppage," "Partial adjustment of safety measures (double fastening of safety belt is mandatory, access control is optional)," etc.

[0077] For example, when the above-mentioned operational decision-making unit (105) receives response information (105b) from the manager, such as “Approval of AI analysis results, instruction to immediately stop work in Zone A on the 3rd floor, mandatory double fastening of safety belts, resumption of work after confirming wind speed of 10 m / s or less, and access control applied only when wind speed is 12 m / s or more,” it can reflect this in the above-mentioned stored artificial intelligence algorithm (103a).

[0078] Accordingly, the previously stored artificial intelligence algorithm (103a) can generate final decision information including the following by reanalyzing the field situation information by reflecting the response information (105b) as a weight: "Work control: Immediately stop work in Area A on the 3rd floor, resume after confirming wind speed is 10 m / s or less, Safety measures: Mandate double fastening of safety harnesses for all workers, Access control: Prohibit entry to Area A on the 3rd floor when wind speed is 12 m / s or more, Schedule adjustment: Scheduled to resume at 4 p.m., work extended until 7 p.m., Additional costs: Approval of labor costs for extended work of 1.5 million won."

[0079] According to one embodiment, when the operation decision providing unit (105) completes the generation of the final decision information, it provides the final decision information to the manager, thereby enabling the manager to operate the construction site in an integrated manner through the final decision information.

[0080] According to one embodiment, the field management system (100) may further include an information layer management unit.

[0081] According to one embodiment, the information hierarchy management unit classifies and stores information obtained at the construction site, such as site operation information, learning analysis information, and personal sensitive information, based on pre-set classification conditions. The information is classified into hot classification information for real-time processing based on the pre-set classification conditions, warm classification information for analysis and learning, and cold classification information for storage and disposal, and is managed in a hierarchy. In the case of the hot classification information, it may be stored after de-identification processing, or if de-identification processing is not possible, it may be destroyed after a pre-set period.

[0082] According to one embodiment, the information layer management unit can identify information obtained at the construction site as site operation information, learning analysis information, and personal sensitive information.

[0083] According to one embodiment, the field operation information may include first multimodal information and second multimodal information, and the learning analysis information may include accident history information, field situation information and decision information, in addition to learning information for training the previously stored artificial intelligence algorithm (103a).

[0084] According to one embodiment, the personal sensitive information may include biometric information among the information included in the multimodal information, such as a worker's face, location history, work records, biometric information, and motion patterns.

[0085] According to one embodiment, the information hierarchy management unit can classify and manage information into hot classification information for real-time processing based on the preset classification conditions, warm classification information for analysis and learning, and cold classification information for storage and disposal.

[0086] In relation to the above, the information layer management unit can classify and manage weather information, sensor information, image information, worker location information, first multimodal information, and second multimodal information collected in real time at the construction site into hot classification information subject to real-time processing for immediate determination of dangerous situations and site control.

[0087] According to one embodiment, the information hierarchy management unit can classify and store field situation information, complex risk situation information, accident history information, decision information, and a previously stored artificial intelligence algorithm (103a) learning data set generated as a result of analysis of the hot classification information into hot classification information for use in long-term analysis and learning.

[0088] According to one embodiment, the information hierarchy management unit may classify and manage field operation history information for which the retention period has expired, record information regarding completed projects, original image data for long-term storage purposes, and personal sensitive information that cannot be de-identified into cold classification information for storage or destruction in accordance with regulations.

[0089] According to one embodiment, the information hierarchy management unit may store the hot classification information after de-identification processing, or destroy it after a preset period if de-identification processing is not possible.

[0090] In relation to the above, the de-identification process is a process of converting data so that an individual cannot be identified, and can be performed by blurring or deleting the face area in the case of a worker's face image, replacing the personal ID with an anonymous code in the case of location history, and removing the personal identifier in the case of biometric information.

[0091] FIG. 2 is a block diagram illustrating a decision information generation unit of a multimodal artificial intelligence-based construction site integrated operation and risk prediction system according to one embodiment of the present invention.

[0092] Referring to FIG. 2, a multimodal artificial intelligence-based construction site integrated operation and risk prediction system (e.g., the multimodal artificial intelligence-based construction site integrated operation and risk prediction system (100) of FIG. 1) (hereinafter referred to as a site management system) implemented as a computing device comprising one or more processors and one or more memories for storing instructions that can be executed by said processors may include a decision information generation unit (200) (e.g., the decision information generation unit (103) of FIG. 1).

[0093] According to one embodiment, when the decision information generation unit (200) completes the function of the multimodal information synchronization unit (e.g., the multimodal information synchronization unit (101) of FIG. 1), it can generate site situation information that can identify the current situation of the construction site by reflecting the second multimodal information (201a) in the previously entered construction context information (201b), and generate decision information customized to the current situation of the construction site by analyzing the site situation information through a previously stored artificial intelligence algorithm (205).

[0094] According to one embodiment, the decision information generation unit (200) may include a situation information generation unit (201) and a situation analysis unit (203) as detailed configurations for performing the above-described function.

[0095] According to one embodiment, when the generation of the second multimodal information (201a) is completed, the situation information generation unit (201) can generate site situation information that can identify the current situation of the construction site by reflecting detailed information based on the second multimodal information (201a) into a background scenario based on the previously entered construction context information (20ba).

[0096] According to one embodiment, when the generation of the second multimodal information (201a) is completed, the situation information generation unit (201) can identify the detailed information included in the second multimodal information (201a).

[0097] In relation to the above, the detailed information included in the second multimodal information (201a) is integrated data in which weather information, BIM information, IoT sensing information, image information, spatial shape information, biometric information, and process management information are synchronized in space and time, and each piece of information may be mapped to a unified reference time stamp and reference coordinate frame.

[0098] For example, the situation information generating unit (201) can identify from the second multimodal information (201a) "Weather information: wind speed 15 m / s, precipitation 0 mm / h, temperature 5 degrees, humidity 60%, IoT sensing information: vibration in area A on the 3rd floor 2.5 Hz, load 450 kN, gas concentration normal, video information: CCTV video of the work area on the 3rd floor, spatial shape information: point cloud data of the 3rd floor slab, biometric information: location and heart rate data of 15 workers, process management information: pouring work in area A on the 3rd floor in progress, pouring start time 9:30 AM".

[0099] According to one embodiment, when the identification of the detailed information is completed, the situation information generation unit (201) can generate site situation information by reflecting the second multimodal information (201a) in the previously entered construction context information (201b).

[0100] In relation to the above, the construction context information (201b) may be a background scenario that sets the progress, environment, and risk conditions of a construction site, and may be background set information in which a process context, spatial context, time context, worker context, environment context, and operational context are set.

[0101] In relation to the above, the above site situation information is information describing the comprehensive situation of the construction site by combining detailed information based on the second multimodal information (201a) with detailed context based on the above construction context information (201b), and can be used as input data for a previously stored artificial intelligence algorithm (205) to analyze the relationship between detailed contexts to determine complex risk situations and generate decisions.

[0102] For example, the above-mentioned situation information generation unit (201) combines "wind speed 15 m / s, 3rd floor slab vibration 2.5 Hz, load 450 kN, location data of 15 workers, average heart rate 95 bpm, temperature 5 degrees, pouring work in progress" of the second multimodal information (201a) and "pouring progress rate 67%, 2 hours elapsed since pouring started, high-altitude work 10.5 m above ground, 11:30 AM, rain expected from 2:00 PM, average worker work time 3 hours elapsed" of the construction context information (201b) to obtain "slab concrete pouring work in progress in Zone A on the 3rd floor (progress rate 67%, 2 hours elapsed since pouring started, remaining pouring volume 75 m³), ​​current wind speed 15 m / s, strong wind condition (exceeding the strong wind advisory standard of 14 m / s), structural vibration 2.5 Hz detected, pouring load 450 kN applied" You can generate site situation information including the following: "15 workers are working at a height of 10.5m above the ground (average heart rate of 95bpm is the upper limit of the normal range), fatigue is expected to increase after 3 hours of average work time, work needs to be completed within 2.5 hours as rain is expected from 2:00 PM, and concrete curing conditions need to be considered with a temperature of 5 degrees."

[0103] According to one embodiment, when the generation of the field situation information is completed, the situation analysis unit (203) can analyze the relationship between detailed contexts based on the field situation information through the previously stored artificial intelligence algorithm (205), identify a complex risk situation derived from the analysis results, and compare the similarity between the identified complex risk situation and a plurality of accident history information to generate decision information for improving the complex risk situation based on a response plan corresponding to the accident history information with the highest similarity.

[0104] According to one embodiment, when the generation of the field situation information is completed, the situation analysis unit (203) can analyze the relationship between detailed contexts based on the field situation information through the previously stored artificial intelligence algorithm.

[0105] In relation to the above, the relationship between the detailed contexts is an interaction and influence relationship between multiple elements included in the field situation information, for example, it may mean a complex relationship between "strong wind (wind speed 15 m / s)," "working at height (10.5 m above the ground)," and "worker fatigue (after 3 hours of work)."

[0106] According to one embodiment, the artificial intelligence algorithm (205) stored above may be a multimodal inference model that determines a complex risk situation by comprehensively analyzing the interaction between detailed contexts, rather than independently evaluating each detailed context included in the field situation information.

[0107] For example, the above situation analysis unit (203) can identify multiple detailed contexts from the site situation information through the above-stored artificial intelligence algorithm (205), such as “strong wind (wind speed 15 m / s) + work at height (10.5 m above ground) + increased worker fatigue (work time 3 hours) + structural vibration (2.5 Hz) + pouring load (450 kN) + time pressure (rain scheduled for 2 PM, completion required within 2.5 hours),” and each detailed context individually is at a medium risk level, but when they occur simultaneously, it can be analyzed as a “complex risk situation in which the risk of a worker falling during work at height in a strong wind environment is amplified by increased fatigue and time pressure, and structural vibration caused by the pouring load amplifies work instability.”

[0108] According to one embodiment, the previously stored artificial intelligence algorithm (205) is an algorithm that includes a model that periodically learns pattern values ​​derived by analyzing correlations between field situation information, which is past accumulated learning information; complex risk situations based on detailed contexts based on field situation information; final decision information based on response measures corresponding to complex risk situations; and result feedback information based on final decision information. The algorithm is distributed to a device deployed at a construction site via wireless communication and is optimized to correspond to the computational specifications of the device. By receiving second multimodal information provided to the device and analyzing it in an on-device manner, it can perform functions independently even in an environment where network connectivity is limited.

[0109] According to one embodiment, the artificial intelligence algorithm (205) stored above may include a self-learning structure that periodically learns previously accumulated learning information.

[0110] In relation to the above, the previously accumulated learning information may be a series of data sets including field situation information, complex risk situations based on detailed contexts based on field situation information, final decision information based on response measures corresponding to complex risk situations, and result feedback information based on the final decision information.

[0111] According to one embodiment, the artificial intelligence algorithm (205) stored above can collect field situation information, complex risk situations identified in response thereto, generated decision information, final decision information reflecting manager's feedback, and result feedback information after taking actual action based on the final decision information as training data.

[0112] For example, the artificial intelligence algorithm (205) stored above can collect a series of data as learning information, such as “Site situation information: wind speed 15 m / s, high-altitude work 10.5 m above ground, 15 workers, pouring progress 67% -> Complex risk situation: risk of high-altitude worker falling in strong wind environment (risk level: high) -> Decision information: work immediately stopped, resumed after confirming wind speed 10 m / s or less -> Final decision information: work stopped reflecting manager feedback, double fastening of safety harnesses mandatory, resumed after confirming wind speed 10 m / s or less -> Result feedback information: no actual accident occurred after work stopped, work resumed after 2 hours by descending to wind speed 9 m / s, additional cost of 2.3 million won incurred but no casualties”.

[0113] According to one embodiment, the artificial intelligence algorithm (205) stored above can derive a pattern value by analyzing the correlation between the learning information.

[0114] In relation to the above, the pattern value may be a value that quantifies the probability of a specific complex risk occurring in a specific field situation, the effectiveness of a specific response measure, and the cost-effectiveness of safety improvement resulting from a specific measure.

[0115] For example, the artificial intelligence algorithm (205) stored above can analyze 50 past learning data to derive a pattern value such as "when wind speed is 14 m / s or more + work is 10 m or more above the ground + work time is 3 hours or more, the probability of a worker falling is 85%, the accident prevention effect is 95% when work is immediately stopped, the average delay time is 2.5 hours, and the average additional cost is 2.5 million won."

[0116] According to one embodiment, the artificial intelligence algorithm (205) stored above can periodically perform learning to update the weights and parameters of the model based on the pattern value.

[0117] According to one embodiment, the previously stored artificial intelligence algorithm (205) can be distributed to a device deployed at a construction site via wireless communication.

[0118] In relation to the above, the device deployed at the construction site is an edge computing device at the construction site and may be an on-device system capable of independently performing artificial intelligence functions even in underground spaces, tunnels, mountainous areas, etc., where the internet connection is unstable.

[0119] According to one embodiment, the artificial intelligence algorithm (205) stored above can be optimized to correspond to the computational specifications of the device.

[0120] In relation to the above, the optimization is a process of lightweighting a large-scale AI model trained on a cloud server so that it can be executed on the limited computational resources (CPU, GPU, NPU, memory) of an edge device, and techniques such as model quantization, pruning, and knowledge distillation may be applied.

[0121] For example, the above-mentioned stored artificial intelligence algorithm (205) can reduce the model size by 1 / 4 by quantizing a model trained with FP32 (32-bit floating-point) precision on a cloud server into INT8 (8-bit integer), reduce the amount of computation by 50% by applying pruning to remove weights of low importance, and improve the inference speed by 3 times by applying a computation library optimized for the Mali GPU and 6TOP NPU of the edge device.

[0122] According to one embodiment, the previously stored artificial intelligence algorithm (205) can be distributed to a device deployed at a construction site via wireless communication once optimization is complete.

[0123] In relation to the above, the wireless communication may be an Over-The-Air (OTA) distribution method that transmits AI model files from a cloud server to an edge device via a wireless network such as LTE, 5G, or Wi-Fi.

[0124] For example, the above-mentioned stored artificial intelligence algorithm (205) generates an AI model that is periodically updated (e.g., once a week) on a cloud server and automatically distributes it to edge devices at a construction site (e.g., an edge server installed at the site office, a smart safety terminal deployed in the work area) via a 5G wireless network so that the edge devices can be updated to the latest AI model.

[0125] According to one embodiment, when the deployment of the previously stored artificial intelligence algorithm (205) is completed, the device deployed at the construction site can receive the second multimodal information provided to the device and analyze it on-device.

[0126] In relation to the above, the on-device method is a method of running an AI model inside an edge device without communication with a cloud server, enabling real-time analysis without network latency and ensuring data privacy.

[0127] Accordingly, the device deployed at the construction site performs its functions independently even in an environment where network connectivity is limited, thereby providing real-time risk analysis and decision-making functions even in underground spaces, tunnels, mountainous areas, etc., where internet connectivity is unstable.

[0128] According to one embodiment, when the analysis of the relationship between the detailed contexts is completed, the situation analysis unit (203) can identify the complex risk situation derived from the analysis results.

[0129] In relation to the above, the complex risk situation is a risk situation that arises when risk factors interact according to the relationship between multiple detailed contexts rather than a single risk factor, and it may be a risk situation that is not detected when each detailed context is evaluated individually but is identified when considered in combination.

[0130] For example, the above-mentioned situation analysis unit (203) can determine a complex risk situation through a stored artificial intelligence algorithm (205) as follows: "Complex risk situation: work at height (10.5m above ground) under a strong wind environment (wind speed 15m / s, exceeding the strong wind warning standard), increased worker fatigue (average work time 3 hours), and time pressure (2.5 hours until rain is scheduled, pressure to complete work) by analyzing the relationship between detailed contexts, such as a situation where work instability is amplified due to structural vibration (2.5Hz) caused by the pouring load (450kN), overall risk: high, major risk: worker falling due to strong wind, additional risk: possibility of safety procedures being omitted due to time pressure."

[0131] According to one embodiment, when the identification of the complex risk situation is completed, the situation analysis unit (203) can compare the similarity between the identified complex risk situation and a plurality of accident history information.

[0132] In relation to the above, the multiple accident history information includes information such as the cause of accidents, site conditions, accident type, accident results, and applied response measures of accidents that occurred at past construction sites, is stored in a database, and can be referenced to derive the optimal response measures for similar situations.

[0133] For example, the above situation analysis unit (203) can compare the similarity between the complex risk situation and multiple accident history information through a hybrid structure model based on RAG and MCP of the above-stored artificial intelligence algorithm (205).

[0134] In relation to the above, the above RAG (Retrieval-Augmented Generation) is a technology that searches for accident history information stored in a database and extracts cases with high similarity, and the above MCP (Model Context Protocol) may be a technology that reinterprets the context information of the extracted cases to suit the current situation and generates an optimal response plan.

[0135] For example, the situation analysis unit (203) converts the currently identified complex risk situation "strong wind (wind speed 15 m / s) + work at height (10.5 m above ground) + increased worker fatigue + time pressure" into a vector, and then calculates the cosine similarity with each of the plurality of accident history information, and identifies the one with the highest similarity "Accident History A (November 2024, Site B, worker fell while working at height 12 m above ground in a strong wind 17 m / s environment, cause: safety harness not fastened + strong wind, similarity 0.87)", "Accident History B (September 2024, Site C, equipment fell while working at height 9 m above ground in a wind speed 14 m / s environment, cause: strong wind + poor equipment fixation, similarity 0.82)", "Accident History C (July 2024, Site D, worker attempted to fall while working at height 11 m above ground in a wind speed 13 m / s environment, cause: strong wind + Accident history information with the content “Work stoppage not performed, similarity 0.79” can be extracted.

[0136] According to one embodiment, when the similarity comparison is completed, the situation analysis unit (203) can identify a response plan corresponding to the accident history information with the highest similarity.

[0137] For example, the above situation analysis unit (203) can identify "immediate work stoppage, double fastening of safety belts for all workers, resumption of work after confirming wind speed of 10 m / s or less, additional placement of safety managers" as a response plan corresponding to the first accident history information with the highest similarity.

[0138] According to one embodiment, when the situation analysis unit (203) completes the identification of the response plan, it can generate decision information to improve the complex risk situation based on the response plan.

[0139] In relation to the above, the decision information may consist of a plurality of detailed items including a risk item, a work control item, a target area item, a schedule impact item, a cost impact item, and a safety measure item.

[0140] For example, the above-mentioned situation analysis unit (203) adjusts the response plan for the accident history information with the highest similarity to the current situation, and includes: "Risk level item (Overall risk grade: High, Accident type: Fall of a worker at height due to strong winds, Possible time of occurrence: Within 30 minutes), Work control item (Work stoppage: Immediately, Work resumption: After confirming wind speed 10 m / s or less), Target area item (Target area ID: 3F-A, Floor / Zone: 3rd floor A zone, Equipment ID: Pump truck-01, Vibratory compactor-03), Schedule impact item (Estimated delay time: 2 hours, Alternative schedule: Work needs to be extended until 7 PM if resumed at 4 PM), Cost impact item (Additional cost: Labor costs for extended work 1.5 million won + Equipment standby costs 800,000 won = Total 2.3 million won, Minimization of loss plan: Minimize delay by concentrating work tomorrow morning), Safety measures item (Reinforcement of protective equipment: Mandatory double fastening of safety harnesses for all workers + Verification of safety helmet chin strap fastening, Access control: Prohibition of entry to 3rd floor A zone, Monitoring You can complete the creation of decision information including "Reinforcement: Deployment of 2 additional safety managers + Real-time monitoring of wind speed upon resumption of work".

[0141] According to one embodiment, if the situation analysis unit (203) identifies the identified complex risk situation as a pre-set critical risk situation, it can immediately dispatch and control a drone linked to the construction site before generating the decision information, thereby receiving real-time video of the construction site through the drone and transmitting it to the manager, and output a warning message to the workers at the construction site.

[0142] In relation to the above, the aforementioned pre-established serious risk situations are risk situations with a high probability of causing immediate casualties, such as "risk of workers falling in strong wind environments," "risk of structural collapse," "risk of gas leak explosion," and "risk of fire."

[0143] According to one embodiment, the situation analysis unit (203) can execute a drone dispatch command immediately before generating the decision information if the complex risk situation corresponds to a serious risk situation as identified as "Overall risk level: High, Accident type: Fall of a high-altitude worker due to strong winds, Possible time of occurrence: Within 30 minutes."

[0144] For example, the above situation analysis unit (203) can send a control command to the linked drone, such as “immediately dispatch to area A on the 3rd floor, maintain an altitude of 15m, film real-time video over the work area, transmit video to the manager, and broadcast a warning to the worker, ‘Caution: strong winds, stop work immediately, evacuate to a safe area’,” so that the drone automatically flies over area A on the 3rd floor and transmits real-time video to the manager, and repeatedly broadcasts a warning message to the worker, “Caution: strong winds, stop work immediately, evacuate to a safe area,” through a speaker mounted on the drone.

[0145] FIG. 3 is another block diagram illustrating a decision information generation unit of a multimodal artificial intelligence-based construction site integrated operation and risk prediction system according to one embodiment of the present invention.

[0146] Referring to FIG. 3, a multimodal artificial intelligence-based construction site integrated operation and risk prediction system (e.g., the multimodal artificial intelligence-based construction site integrated operation and risk prediction system (100) of FIG. 1) (hereinafter referred to as a site management system) implemented as a computing device comprising one or more processors and one or more memories for storing instructions that can be executed by said processors may include a decision information generation unit (301) (e.g., the decision information generation unit (103) of FIG. 1).

[0147] According to one embodiment, when the decision information generation unit (301) completes the function of the multimodal information synchronization unit (e.g., the multimodal information synchronization unit (101) of FIG. 1), it can generate site situation information that can identify the current situation of the construction site by reflecting the second multimodal information in the previously entered construction context information, and generate decision information customized to the current situation of the construction site by analyzing the site situation information through a previously stored artificial intelligence algorithm.

[0148] According to one embodiment, the decision information generating unit (301) may perform other functions in addition to the functions described above.

[0149] According to one embodiment, the decision information generation unit (301) identifies a request intention value based on request information (301a) input from a manager, and sets the mode of the previously stored artificial intelligence algorithm (303) based on the identified request intention value to one of an inference mode, a response generation mode, and a computational resource allocation mode, and controls the previously stored artificial intelligence algorithm (303) to be executed on-device or on-premise according to the set mode.

[0150] According to one embodiment, the decision information generation unit (301) can identify a request intention value based on request information (301a) input from the manager.

[0151] In relation to the above, the request information (301a) is a natural language query or command that a manager inputs into the construction site integrated operation system, and may include, for example, "Can we proceed with the pouring work in Area A on the 3rd floor?", "Is working at height safe in the current strong wind conditions?", "Please analyze whether the work schedule for this afternoon needs to be adjusted", "Please prepare a detailed risk analysis report for the entire 3rd floor area".

[0152] In addition, the above request intent value is a value indicating what type of response the manager's request requires, and can be classified, for example, as "immediate response required," "detailed analysis required," "high-precision inference required," "lightweight analysis required," etc.

[0153] According to one embodiment, the decision information generation unit (301) can identify the request intention value by analyzing the request information (301a) through a natural language processing model among the previously stored artificial intelligence algorithms.

[0154] For example, the decision information generation unit (301) can analyze the manager's request information "Can I proceed with the pouring work in Area A on the 3rd floor?" to identify a request intent value such as "Request type: Immediate judgment request, Response time requirement: Real-time (within 3 seconds), Precision requirement: Medium (Go / No-Go judgment), Output format: Simple Yes / No + 1-2 sentences of reasoning," and analyze the manager's request information "Please write a detailed risk analysis report for the entire area on the 3rd floor" to identify a request intent value such as "Request type: In-depth analysis request, Response time requirement: Non-real-time (30 seconds to 1 minute allowed), Precision requirement: High (Multi-level inference required), Output format: Structured report (Including risk analysis, countermeasures, and cost analysis)."

[0155] According to one embodiment, when the decision information generation unit (301) completes the identification of the request intention value, it can set the mode of the previously stored artificial intelligence algorithm (303) to one of different inference modes, response generation modes, and computational resource allocation modes based on the identified request intention value.

[0156] In relation to the above, the inference mode is a mode that controls the inference speed and precision of the artificial intelligence model, the response generation mode is a mode that controls the format and detail of the output information, and the computational resource allocation mode may be a mode that controls the usage of computational resources used for the execution of the artificial intelligence model.

[0157] According to one embodiment, the inference mode may include a high-speed inference mode, a standard inference mode, and a precision inference mode.

[0158] According to one embodiment, the high-speed inference mode is a mode that uses a lightweight artificial intelligence model with the highest priority on inference speed, and can provide a response time within 1 to 3 seconds and an intermediate level of precision.

[0159] According to one embodiment, the standard inference mode is a mode that balances inference speed and precision, and can provide a response time of 5 to 10 seconds and a high level of precision.

[0160] According to one embodiment, the precision inference mode is a mode that uses a large-scale artificial intelligence model and multi-layer inference with precision as the top priority, and can provide a response time of 30 seconds to 1 minute and a very high level of precision.

[0161] According to one embodiment, the response generation mode may include a concise response mode, a standard response mode, and a detailed response mode.

[0162] According to one embodiment, the concise response mode is a mode that provides only the key conclusion, and can provide an output format: Yes / No + 1-2 sentences of reasoning, and an output length: 50-100 words.

[0163] According to one embodiment, the standard response mode is a mode that provides a conclusion and key grounds, and can provide an output format: conclusion + grounds + recommendations, and an output length of 200 to 300 words.

[0164] According to one embodiment, the detailed response mode is a mode provided in the form of a structured report, and may provide output format: comprehensive analysis + detailed basis + countermeasures + cost analysis + schedule impact, output length: 500 to 1000 words.

[0165] According to one embodiment, the computational resource allocation mode may include a low resource mode, a standard resource mode, and a high resource mode.

[0166] According to one embodiment, the low resource mode is a mode that uses minimal computational resources, using only the CPU, not using the GPU / NPU, and using memory usage of 1GB or less.

[0167] According to one embodiment, the standard resource mode is a mode that uses balanced computational resources, and can use CPU + GPU usage, memory usage: 2~4GB.

[0168] According to one embodiment, the high resource mode is a mode that uses maximum computational resources, and can use CPU + GPU + NPU in parallel, and memory usage: 8GB or more.

[0169] According to one embodiment, the decision information generation unit (301) can set the mode of the previously stored artificial intelligence algorithm (303) based on the identified request intention value.

[0170] For example, if the decision information generation unit (301) identifies the request intent value for the manager's request information "Can I proceed with the pouring work in Area A on the 3rd floor?" as "Immediate judgment request, response time requirement: real-time, precision requirement: medium, output format: simple," it can set the inference mode to "high-speed inference mode," set the response generation mode to "concise response mode," and set the computational resource allocation mode to "low-resource mode."

[0171] According to one embodiment, when the mode setting is completed, the decision information generating unit (301) can control the previously stored artificial intelligence algorithm to be executed in an on-device or on-premise manner according to the set mode.

[0172] In relation to the above, the on-device method is a method of executing an AI model on an edge device deployed at the site, which enables real-time response without network latency but is limited to executing only lightweight models due to limited computational resources, and the on-premise method is a method of executing an AI model on a server built within the construction site, which enables the execution of large-scale models but may result in network latency.

[0173] According to one embodiment, the decision information generation unit (301) can control the execution of the previously stored artificial intelligence algorithm (303) on-device when the set mode is "high-speed inference mode + concise response mode + low-resource mode".

[0174] For example, when the decision information generation unit (301) is set to “high-speed inference mode + concise response mode + low-resource mode” in response to the manager’s request “Can I proceed with the pouring work in Area A on the 3rd floor?”, it can run a lightweight AI model deployed to an edge device (equipped with Mali GPU and 6TOP NPU) installed at the field office on-device to analyze field situation information and generate decision information including a concise response “Can proceed. Current wind speed is 8 m / s, within the safe working range. However, rain is expected at 2:00 PM, so it is recommended to complete the work by 1:30 PM” within 2 seconds.

[0175] According to one embodiment, the decision information generation unit (301) can control the previously stored artificial intelligence algorithm (303) to be executed on-premises when the set mode is "precision inference mode + detailed response mode + high resource mode".

[0176] According to one embodiment, the decision information generation unit (300) can control the dynamic selection and execution of an on-device method or an on-premise method depending on the field situation when the set mode is "standard inference mode + standard response mode + standard resource mode".

[0177] In relation to the above, the dynamic selection may be a process of determining the optimal execution method by comprehensively considering the current network connection status, the utilization of computing resources of the edge device, the load status of the on-premises server, etc.

[0178] According to one embodiment, the decision information generation unit (300) can provide the generated response (e.g., decision information) to the manager when the execution of the previously stored artificial intelligence algorithm (303) is completed.

[0179] FIG. 4 is a block diagram illustrating an operational decision-making providing unit of a multimodal artificial intelligence-based construction site integrated operation and risk prediction system according to one embodiment of the present invention.

[0180] Referring to FIG. 4, a multimodal artificial intelligence-based construction site integrated operation and risk prediction system (e.g., the multimodal artificial intelligence-based construction site integrated operation and risk prediction system (100) of FIG. 1) (hereinafter referred to as a site management system) implemented as a computing device comprising one or more processors and one or more memories for storing instructions that can be executed by said processors may include an operational decision providing unit (400) (e.g., the operational decision providing unit (105) of FIG. 1).

[0181] According to one embodiment, when the operation decision providing unit (400) completes the generation of decision information (401a), it compares and analyzes the risk assessment information generated by the worker at the construction site with the decision information (401a), identifies discrepancies through the analysis results, provides verification result information regarding the discrepancies to the manager, and inputs the response information in which the manager responds to the verification result information into the previously stored artificial intelligence algorithm (405) to generate final decision information in which the response information is reflected as feedback, thereby enabling the manager to operate the construction site in an integrated manner through the final decision information.

[0182] According to one embodiment, the operation decision providing unit (400) may include a verification result providing unit (401) and a final decision providing unit (403) as detailed configurations for performing the above-described function.

[0183] According to one embodiment, when the generation of the decision information (401a) is completed, the verification result providing unit (401) can compare and analyze the contents of the risk assessment information and the decision information by item, identify the items corresponding to the inconsistent content through the analysis results as inconsistent items, and provide verification result information including the contents based on the inconsistent items to the manager.

[0184] According to one embodiment, the verification result providing unit (401) can identify risk assessment information generated by a worker at a construction site when the generation of the decision information (401a) is completed.

[0185] In relation to the above, the risk assessment information may be a risk assessment report prepared by a worker or safety manager at a construction site before starting work, and may be a document including expected risk factors, risk grades, safety measures, etc.

[0186] According to one embodiment, when the identification of the risk assessment information is completed, the verification result providing unit (401) can compare and analyze the contents of the risk assessment information and the decision information (401a) by item.

[0187] In relation to the above, the above-mentioned comparative analysis by identical item may be a process of extracting items commonly included in the risk assessment information and the decision information to determine whether the content of each item matches or does not match.

[0188] According to one embodiment, the verification result providing unit (401) can compare the risk grade item of the risk assessment information with the comprehensive risk grade item of the decision information (401a), compare the possible accident type item of the risk assessment information with the accident type item of the decision information (401a), and compare the safety measure item of the risk assessment information with the safety measure item of the decision information (401a).

[0189] According to one embodiment, when the comparison analysis is completed, the verification result providing unit (401) can identify an item corresponding to the inconsistent content through the analysis result as an inconsistent item.

[0190] In relation to the above, the above discrepancy item is an item where the content of the worker's risk assessment and the content of the decision information (401a) are different, and may occur when the worker underestimates or overestimates the risk or omits safety measures.

[0191] According to one embodiment, when the identification of the discrepancy item is completed, the verification result providing unit (401) can generate verification result information including content based on the discrepancy item.

[0192] In relation to the above, the verification result information is information that quantitatively indicates the difference between the risk assessment information and the decision information (401a) and includes detailed information for each discrepancy item, and may be information used as data for a manager to judge the appropriateness of the risk assessment and decide whether to take additional measures.

[0193] According to one embodiment, the verification result information may include a list of discrepancies, details of discrepancies by item, analysis of the causes of discrepancies, quantitative evaluation of risk differences, and recommended measures.

[0194] According to one embodiment, the verification result providing unit (401) can provide the verification result information to the manager when the generation of the verification result information (403a) is completed.

[0195] According to one embodiment, when the final decision providing unit (403) receives response information (403a) responding to the verification result information from the manager while the function of the verification result providing unit (401) is completed, the response information (403a) is input as a weight into the previously stored artificial intelligence algorithm (405), thereby causing the previously stored artificial intelligence algorithm (405) to generate final decision information in which the response information (403a) is reflected as feedback and provide it to the manager.

[0196] According to one embodiment, the final decision providing unit (403) may receive response information (403a) from the manager in response to the verification result information.

[0197] In relation to the above, the response information (403a) is information that includes the approval of some parts of the decision information or the correction of discrepancies after the manager reviews the verification result information, and may include, for example, "Approve AI recommendation," "Immediately execute work stoppage," "Partial adjustment of safety measures," "Modify work stoppage criteria," etc.

[0198] According to one embodiment, when the final decision providing unit (403) completes receiving the response information (403a), it can input the response information (403a) as a weight to the previously stored artificial intelligence algorithm (405).

[0199] In relation to the above, the weight is a parameter that adjusts the stored artificial intelligence algorithm to give higher importance to specific elements when generating a decision, and by inputting the manager's response information (403a) as the weight, the stored artificial intelligence algorithm can generate a final decision by reflecting the manager's judgment.

[0200] According to one embodiment, the final decision providing unit (403) can analyze each item included in the response information (403a) and classify it into approval items, modification items, and rejection items.

[0201] For example, the final decision providing unit (403) can analyze the manager's response information (403a) and classify it as follows: "Approval item: Approval of all AI analysis results, immediate suspension of work in Zone A on the 3rd floor, mandatory double fastening of safety belts, resumption of work after confirming wind speed of 10 m / s or less, Modification item: Adjustment of access control standards to wind speed of 12 m / s or more (AI recommendation is immediate access control), adjustment of additional safety manager placement to 1 person (AI recommendation is 2 people), Rejection item: None".

[0202] According to one embodiment, when the classification is completed, the final decision providing unit (403) may input the approved item to a weight of 1.0, the modified item to a new value reflecting the manager's modifications, and the rejected item to a weight of 0 into the previously stored artificial intelligence algorithm (405).

[0203] For example, the final decision providing unit (403) can input "Immediate suspension of work: weight 1.0 (approved), mandatory double fastening of safety belt: weight 1.0 (approved), resumption after checking wind speed 10 m / s or less: weight 1.0 (approved), access control: weight 0.8 (modified, conditionally applied - when wind speed is 12 m / s or more), additional placement of safety manager: weight 0.5 (modified, reduced from 2 to 1)" into the artificial intelligence algorithm (405) that was previously stored.

[0204] According to one embodiment, when the final decision providing unit (403) completes the weight input of the response information (403a), the previously stored artificial intelligence algorithm (405) may generate final decision information in which the response information (403a) is reflected as feedback.

[0205] In relation to the above, the previously stored artificial intelligence algorithm (405) can generate final decision information reflecting the manager's judgment by reanalyzing the field situation information by reflecting the response information (403a) as a weight.

[0206] For example, the final decision providing unit (403) uses the previously stored artificial intelligence algorithm (405) to provide "Final Decision Information: [Work Control] Immediate suspension of work in Area A on the 3rd floor (Approved), resumption of work after confirming wind speed 10 m / s or less (Approved), [Safety Measures] Mandatory double fastening of safety harnesses for all workers (Approved), verification of safety helmet chin strap fastening (Approved), Access Control: Prohibition of entry to Area A on the 3rd floor when wind speed is 12 m / s or more (Reflecting manager's modification), Deployment of 1 additional safety manager (Reflecting manager's modification, existing 2 -> 1), [Target Area] Target Area ID: 3F-A, Floor / Zone: Area A on the 3rd floor, Equipment ID: Pump Truck-01, Vibratory Compactor-03, [Schedule Adjustment] Estimated delay time: 2 hours, Scheduled resumption of work: 4 PM (after confirming wind speed 10 m / s or less), Work extension: Until 7 PM (Night work required), [Cost Impact] Additional cost: Labor costs for extended work You can generate final decision information including "1,500,000 KRW + Equipment standby cost 800,000 KRW = Total 2,300,000 KRW (Approved), [Manager Approval Item] This decision has been finalized after review and approval by the manager, Approval Date: February 10, 2025, 11:35 AM, Approver: Site Manager Hong Gil-dong".

[0207] According to one embodiment, the final decision providing unit (403) can provide the final decision information to the manager when the generation of the final decision information is completed.

[0208] FIG. 5 is a block diagram illustrating the learning pipeline construction section of a multimodal artificial intelligence-based construction site integrated operation and risk prediction system according to one embodiment of the present invention.

[0209] Referring to FIG. 5, a multimodal artificial intelligence-based construction site integrated operation and risk prediction system (e.g., the multimodal artificial intelligence-based construction site integrated operation and risk prediction system of FIG. 1 (100)) (hereinafter referred to as a site management system) implemented as a computing device comprising one or more processors and one or more memories for storing instructions that can be executed by said processors may further include a learning pipeline construction unit (500).

[0210] According to one embodiment, the learning pipeline construction unit (500) may include a raw log collection unit (501) and a learning information generation unit (503).

[0211] According to one embodiment, the raw log collection unit (501) can collect field operation history information (501a) in the form of raw log data, which includes the second multimodal information, the field situation information, information on complex risk situations, the decision information, the verification result information, the response information, the final decision information, and result feedback information regarding the final decision information.

[0212] According to one embodiment, the raw log collection unit (501) can collect all data generated during the operation of the field management system in real time.

[0213] In relation to the above, the above field operation history information (501a) is a series of data generated during the entire operation process of the field management system, and may be a comprehensive data set that includes all processes from when a stored artificial intelligence algorithm generates a decision, when a manager reviews and approves it, and when actual measures are taken and results are verified.

[0214] According to one embodiment, the raw log collection unit (501) can collect the field operation history information (501a) in the form of raw log data.

[0215] In relation to the above, the raw log data (501a) may be data stored in a structured format such as JSON, XML, CSV, or unstructured text format, and may be data that is not yet directly usable by the artificial intelligence algorithm for learning.

[0216] According to one embodiment, the learning information generation unit (503) performs a semi-automatic verification process in which, upon completion of the collection of the field operation history information, tag values ​​based on tagging conditions (503a) pre-set in the detailed information included in the field operation history information, and then allows an administrator to verify the field operation history information. When the verification of the field operation history information is completed, the field operation history information can be converted and processed into an instruction tuning data format that the pre-stored artificial intelligence algorithm can learn, and stored in a learning database.

[0217] According to one embodiment, when the collection of the field operation history information is completed, the learning information generation unit (503) can tag a tag value according to a tagging condition (503a) pre-set in the detailed information included in the field operation history information.

[0218] In relation to the above, the above-mentioned tagging condition (503a) is a rule for automatically assigning tags to each piece of information included in raw log data, and may be a condition for classifying tags according to criteria such as data type, risk level, decision type, and whether an accident occurred.

[0219] According to one embodiment, the pre-set tagging condition (503a) may include a data type tag, a risk tag, a decision tag, a result tag, and a learning quality tag.

[0220] For example, the above-mentioned learning information generation unit (503) may assign a tag “Data type: Site situation information, Risk level: High, Learning quality: Excellent” to the site situation information regarding the site operation history information related to the pouring work in Area A on the 3rd floor that occurred at 11:30 a.m. on February 10, 2025, assign a tag “Data type: Risk identification information, Risk level: High, Risk type: Falling from a height work, Learning quality: Excellent” to the complex risk situation information, assign a tag “Data type: Decision information, Decision type: Work stoppage, Learning quality: Excellent” to the decision information, and assign a tag “Data type: Result feedback information, Result: Accident prevention success, Cost: Budget exceeded, Learning quality: Excellent” to the result feedback information.

[0221] According to one embodiment, the learning information generation unit (503) can perform a semi-automatic verification process that allows an administrator to verify the field operation history information when the tagging is completed.

[0222] In relation to the above, the semi-automatic verification process is a process in which an administrator reviews automatically tagged data to verify the accuracy of the tags and, if necessary, modifies or assigns additional tags, and may be a quality control process to prevent an AI model from learning with incorrect data.

[0223] According to one embodiment, the learning information generation unit (503) provides automatically tagged field operation history information to a manager and allows the manager to review the tags of each event.

[0224] According to one embodiment, the manager can verify the accuracy of the automatically tagged tags and, if there are errors, correct the tags or assign additional tags.

[0225] According to one embodiment, the learning information generation unit (503) can add a verification completion mark to the field operation history information verified by the manager, and store the verification date and time and verifier information as metadata.

[0226] According to one embodiment, when the semi-automatic verification process is completed, the learning information generation unit (503) can convert the field operation history information into an instruction tuning data format that the previously stored artificial intelligence algorithm can learn.

[0227] In relation to the above, the instruction tuning data format is a data format for training a large-scale language model (LLM) to perform a specific task, and may generally be a data format having a structure of instructions, input data, and expected output forms.

[0228] According to one embodiment, the learning information generation unit (503) can analyze verified field operation history information and perform a conversion process that maps field situation information to input data, decision information to expected output, and work instructions to instructions.

[0229] According to one embodiment, when the learning information generation unit (503) completes the conversion into the instruction tuning data format, the converted learning information can be stored in a learning database.

[0230] In relation to the above, the above-mentioned learning database is a database that systematically manages learning information stored in the format of instruction tuning data, and can be utilized for periodic learning and performance improvement of previously stored artificial intelligence algorithms.

[0231] According to one embodiment, when the learning information generation unit (503) stores the learning information in the learning database, it may store the quality of the data, risk level, decision type, result, etc. together as metadata so that data matching specific conditions can be selectively utilized during future learning.

[0232] Accordingly, the aforementioned stored artificial intelligence algorithm can periodically learn from training information in the format of instruction tuning data stored in the training database, thereby continuously improving the ability to identify complex risk situations at construction sites and the ability to generate decisions.

[0233] FIG. 6 is a drawing for explaining an example of the internal configuration of a computing device according to an embodiment of the present invention.

[0234] FIG. 6 illustrates an example of the internal configuration of a computing device according to an embodiment of the present invention. In the following description, descriptions of unnecessary embodiments that overlap with the descriptions of FIG. 1 to 5 described above will be omitted.

[0235] As illustrated in FIG. 6, the computing device (10000) may include at least one processor (11100), memory (11200), peripheral interface (11300), input / output subsystem (I / O subsystem) (11400), power circuit (11500), and communication circuit (11600). In this case, the computing device (10000) may correspond to a user terminal (A) connected to a haptic interface device or the aforementioned computing device (B).

[0236] The memory (11200) may include, for example, high-speed random access memory, a magnetic disk, SRAM, DRAM, ROM, flash memory, or non-volatile memory. The memory (11200) may include software modules, instruction sets, or various other data required for the operation of the computing device (10000).

[0237] At this time, access to memory (11200) from other components, such as the processor (11100) or peripheral device interface (11300), can be controlled by the processor (11100).

[0238] The peripheral device interface (11300) can connect input and / or output peripheral devices of the computing device (10000) to the processor (11100) and memory (11200). The processor (11100) can perform various functions for the computing device (10000) and process data by executing software modules or instruction sets stored in the memory (11200).

[0239] The input / output subsystem (11400) can connect various input / output peripherals to the peripheral interface (11300). For example, the input / output subsystem (11400) may include a controller for connecting peripherals such as a monitor, keyboard, mouse, printer, or, if necessary, a touchscreen or sensor to the peripheral interface (11300). According to another aspect, input / output peripherals may be connected to the peripheral interface (11300) without passing through the input / output subsystem (11400).

[0240] The power circuit (11500) can supply power to all or part of the components of the terminal. For example, the power circuit (11500) may include one or more power sources such as a power management system, a battery or alternating current (AC), a charging system, a power failure detection circuit, a power converter or inverter, a power status indicator, or any other components for power generation, management, and distribution.

[0241] The communication circuit (11600) can enable communication with another computing device using at least one external port.

[0242] Alternatively, as described above, the communication circuit (11600) may enable communication with other computing devices by including an RF circuit and transmitting and receiving an RF signal, also known as an electromagnetic signal.

[0243] The embodiment of FIG. 6 is merely an example of a computing device (10000), and the computing device (11000) may have some components shown in FIG. 6 omitted, additional components not shown in FIG. 6 added, or a configuration or arrangement that combines two or more components. For example, a computing device for a communication terminal in a mobile environment may include, in addition to the components shown in FIG. 6, a touchscreen or a sensor, etc., and the communication circuit (1160) may include a circuit for RF communication of various communication methods (WiFi, 3G, LTE, Bluetooth, NFC, Zigbee, etc.). The components that can be included in the computing device (10000) may be implemented as hardware, software, or a combination of both hardware and software, including one or more integrated circuits specialized for signal processing or applications.

[0244] Methods according to embodiments of the present invention may be implemented in the form of program instructions that can be executed through various computing devices and recorded on a computer-readable medium. In particular, the program according to the present embodiment may be configured as a PC-based program or an application dedicated to a mobile terminal. An application to which the present invention is applied may be installed on a user terminal through a file provided by a file distribution system. For example, the file distribution system may include a file transmission unit (not shown) that transmits the file upon a request from the user terminal.

[0245] The device described above may be implemented as a hardware component, a software component, and / or a combination of a hardware component and a software component. For example, the device and components described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and one or more software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include a plurality of processing elements and / or a plurality of types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are also possible.

[0246] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or command the processing unit independently or collectively. Software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium, or device so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed across networked computing devices and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

[0247] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the medium may be those specifically designed and configured for the embodiment, or they may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

[0248] Although the embodiments have been described above with reference to limited embodiments and drawings, those skilled in the art can make various modifications and variations from the description above. For example, appropriate results may be achieved even if the described techniques are performed in a different order than described, and / or if the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents. Therefore, other implementations, other embodiments, and equivalents to the claims below also fall within the scope of the claims.

Claims

Claim 1 A multimodal artificial intelligence-based construction site integrated operation and risk prediction system implemented by a computing device comprising one or more processors and one or more memories for storing instructions executable by said processors, wherein, upon receiving first multimodal information from a plurality of linked information generation units, a multimodal information synchronization unit maps detailed information included in said first multimodal information to a reference time stamp and a reference coordinate frame, and stores second multimodal information generated by synchronizing to the same time and spatial coordinates in a database; and a decision information generation unit, upon completion of the function of said multimodal information synchronization unit, generates site situation information capable of identifying the current situation of the construction site by reflecting the second multimodal information in previously input construction context information, and generates decision information customized to the current situation of the construction site by analyzing said site situation information through a previously stored artificial intelligence algorithm. A multimodal artificial intelligence-based construction site integrated operation and risk prediction system characterized by including: an operational decision providing unit that, when the generation of the above decision information is completed, compares and analyzes the risk assessment information generated by a worker at the construction site with the above decision information, identifies discrepancies through the analysis results, provides verification result information regarding the discrepancies to a manager, inputs response information in which the manager responds to the verification result information into the above-stored artificial intelligence algorithm to generate final decision information in which the response information is reflected as feedback, and provides this to the manager, thereby enabling the manager to operate the construction site in an integrated manner through the above final decision information. Claim 2 In claim 1, the aforementioned interconnected plurality of information generation units comprises: a weather information generation unit that generates weather information including wind speed, precipitation, temperature, humidity, and atmospheric pressure of an area including a construction site; a BIM (Building Information Modeling) information generation unit that generates BIM information including spatial coordinates, floor zones, material placement, and process plans of a construction structure being constructed at the construction site; an IoT information generation unit that generates sensing information including vibration, deformation, load, gas concentration, and noise through a plurality of IoT sensors installed at the construction site; an image information generation unit that generates image information including real-time captured video captured through a camera module installed at the construction site; a spatial shape generation unit that generates spatial shape information including 3D point cloud data acquired through a LiDAR sensor mounted on a drone flying over the construction site; and a biometric information generation unit that generates biometric information including a worker's heart rate, body temperature, location, and motion patterns measured through a wearable device worn by a worker at the construction site. A multimodal artificial intelligence-based construction site integrated operation and risk prediction system characterized by including: a process management information generation unit that generates process management information including work schedules, pouring plans, personnel inputs, and equipment placement at a construction site. Claim 3 A multimodal artificial intelligence-based construction site integrated operation and risk prediction system according to claim 1, wherein the multimodal information synchronization unit, upon completion of receiving the first multimodal information, maps the detailed information included in the first multimodal information to correspond to a common reference time based on the reference time stamp and a common coordinate system based on the reference coordinate frame in order to align them according to a common standard, and completes noise removal and normalization processing to complete the generation of second multimodal information corresponding to the common standard and stores it in the database. Claim 4 A multimodal artificial intelligence-based construction site integrated operation and risk prediction system according to claim 1, wherein the decision information generation unit comprises: a situation information generation unit that, when the generation of the second multimodal information is completed, reflects detailed information based on the second multimodal information into a background scenario based on the previously entered construction context information to generate site situation information capable of identifying the current situation of the construction site; and a situation analysis unit that, when the generation of the site situation information is completed, analyzes the relationship between detailed contexts based on the site situation information through the previously stored artificial intelligence algorithm to identify a complex risk situation derived from the analysis results, compares the similarity between the identified complex risk situation and a plurality of accident history information, and generates decision information to improve the complex risk situation based on a response plan corresponding to the accident history information with the highest similarity. Claim 5 In claim 4, the situation analysis unit is characterized by, when the identified complex risk situation is identified as a pre-set critical risk situation, immediately dispatching and controlling a drone linked to the construction site before generating the decision information, thereby receiving real-time video of the construction site through the drone and transmitting it to a manager while outputting a warning message to a worker within the construction site, in a multimodal artificial intelligence-based construction site integrated operation and risk prediction system. Claim 6 A multimodal artificial intelligence-based construction site integrated operation and risk prediction system according to claim 1, wherein the decision information generation unit identifies a request intent value based on request information input from a manager, sets the mode of the previously stored artificial intelligence algorithm to one of an inference mode, a response generation mode, and a computational resource allocation mode based on the identified request intent value, and controls the previously stored artificial intelligence algorithm to be executed in an on-device or on-premise manner according to the set mode. Claim 7 A multimodal AI-based construction site integrated operation and risk prediction system according to claim 1, wherein the stored artificial intelligence algorithm is an algorithm that includes a model that periodically learns pattern values ​​derived by analyzing correlations between field situation information, which is past accumulated learning information; complex risk situations based on detailed contexts based on field situation information; final decision information based on response measures corresponding to complex risk situations; and result feedback information based on final decision information; is distributed to a device deployed at a construction site via wireless communication, and is lightweighted through one of model quantization, pruning, and knowledge distillation so that it can be executed on the limited computational resources of the edge device deployed at the construction site; and receives second multimodal information provided to the device and analyzes it in an on-device manner, thereby performing functions independently even in an environment where network connectivity is limited. Claim 8 In claim 1, the operational decision providing unit comprises: a verification result providing unit that, when the generation of the decision information is completed, compares and analyzes the contents of the risk assessment information and the decision information by item, identifies the items corresponding to the inconsistent contents through the analysis results as inconsistent items, and provides verification result information including the contents based on the inconsistent items to the manager; and a final decision providing unit that, when the function of the verification result providing unit is completed and a response information responding to the verification result information is received from the manager, inputs the response information as a weight into the previously stored artificial intelligence algorithm, thereby causing the previously stored artificial intelligence algorithm to generate final decision information in which the response information is reflected as feedback, and provides it to the manager. Claim 9 In claim 1, the multimodal artificial intelligence-based construction site integrated operation and risk prediction system further comprises an information hierarchy management unit; wherein the information hierarchy management unit classifies and stores site operation information, learning analysis information, and personal sensitive information obtained at the construction site based on pre-set classification conditions, and manages them in a hierarchy by classifying them into hot classification information for real-time processing based on the pre-set classification conditions, warm classification information for analysis and learning, and cold classification information for storage and disposal, and in the case of the hot classification information, stores it after de-identification processing or destroys it after a pre-set period if de-identification processing is impossible. Claim 10 In claim 1, the multimodal artificial intelligence-based construction site integrated operation and risk prediction system further comprises: a learning pipeline construction unit; wherein the learning pipeline construction unit comprises: a raw log collection unit that collects site operation history information in the form of raw log data, the second multimodal information, the site situation information, information on complex risk situations, the decision information, the verification result information, the response information, the final decision information, and result feedback information regarding the final decision information; and a learning information generation unit that, upon completion of the collection of the site operation history information, performs a semi-automatic verification process to tag a tag value based on a tagging condition pre-set for the detailed information included in the site operation history information and then allows an administrator to verify the site operation history information, and when the verification of the site operation history information is completed, converts and processes the site operation history information into an instruction tuning data format that the pre-stored artificial intelligence algorithm can learn from and stores it in a learning database.

Images