An engineering safety hidden danger image recognition method and system based on multi-model fusion

Abstract

The application discloses an engineering safety hidden danger image recognition method and system based on multi-model fusion, comprising: acquiring an original image of an engineering site, recognizing a scene type and outputting a scene label, loading a target detection model and a visual language reasoning model from a preset model library according to the scene label; determining whether a detection item corresponding to the scene label belongs to a known category target or an open category target according to a preset scene-detection item configuration table, outputting a boundary box coordinate through the target detection model for the known category target indicated by the scene label, and cutting a target subgraph from the original image, inputting the target subgraph and a preset hidden danger prompt word template into the visual language reasoning model; inputting the original image and the hidden danger prompt word template into the visual language reasoning model for the open category target indicated by the scene label, and obtaining a hidden danger reasoning result; and fusing the boundary box coordinate and the hidden danger reasoning result, and outputting hidden danger data including a target identifier, a coordinate, a hidden danger type and a risk level.

Description

Technical Field

[0001] This invention relates to the field of engineering safety management technology, specifically to an image recognition method and system for engineering safety hazards based on multi-model fusion. Background Technology

[0002] With the continuous expansion of engineering construction scale and the constant improvement of safety production requirements, on-site safety hazard investigation has become a core aspect of construction management. Traditional safety hazard investigation mainly relies on manual visual inspection and paper records, which suffers from low efficiency, high missed detection rates, and delayed hazard feedback. Especially in complex engineering scenarios such as high-altitude work areas, areas with dense pipelines, and the edges of foundation pits, manual investigation struggles to achieve both comprehensiveness and real-time accuracy; the uploading and management of hazard information relies on manual input, which is prone to information distortion and difficulties in traceability. These limitations indicate an urgent need to introduce automated and intelligent technologies to improve the efficiency and accuracy of on-site safety hazard investigation.

[0003] In recent years, deep learning-based object detection technology has been widely applied in smart construction site scenarios. Typical solutions employ convolutional neural network models such as the YOLO series to detect behaviors like workers wearing safety helmets and reflective vests. Meanwhile, visual language models, as a new generation of multimodal understanding technology, possess the ability to perform semantic reasoning on image content based on natural language instructions. Theoretically, this can compensate for the shortcomings of traditional object detection models in non-pre-trained object recognition and image reasoning. Building on this, researchers have further proposed a multi-model scheme combining object detection models and visual language models, aiming to comprehensively utilize the technical advantages of both types of models to achieve more comprehensive identification of safety hazards at construction sites.

[0004] However, the aforementioned technical approaches all face significant limitations in practical applications. Single-object detection models are limited by the coverage of the pre-training dataset, unable to identify open-category hazards such as uncovered bare soil, illegal dumping, and unstable scaffolding, which lack pre-training data. Furthermore, these models only output the target's category label and bounding box coordinates, lacking semantic reasoning capabilities regarding the target's state and thus unable to determine the safety and compliance of detected targets. Visual language models also face technical bottlenecks: general-purpose visual language models lack native pixel-level precise localization capabilities and the coordinate calculation pipeline of bounding box regression and non-maximum suppression found in traditional detection models. This results in label offsets in the output target location information, failing to meet the coordinate accuracy requirements for engineering hazard labeling. For example, Figure 3This diagram illustrates the comparison of multiple rounds of target labeling for the same image using a visual language model. From left to right, it shows the bounding box labeling results of the visual language model for the same person target in three independent inferences. The three labeling results show random offsets, indicating that the visual language model lacks stable pixel-level precise localization capabilities. Furthermore, the visual language model performs inconsistently in filtering small targets; when using prompts to control the filtering of distant, small targets, it exhibits problems such as failure to filter or false filtering. Figure 4 This diagram illustrates the failure of a visual language model's small target filtering. It shows that even with a filter set for targets smaller than 32 pixels in the prompt, the visual language model still marks distant, small targets. This demonstrates that the visual language model lacks precise pixel calculation capabilities and cannot stably perform small target filtering based on pixel thresholds. Most existing multi-model solutions only implement simple model concatenation and stitching, failing to dynamically adapt the combination strategies of detection and inference models for different engineering scenarios. They also lack effective multi-model data fusion mechanisms to handle coordinate alignment and semantic conflict resolution between different model outputs. This results in low overall image understanding accuracy and poor structure of the output results, making it difficult to meet the actual requirements of detection accuracy and response speed for on-site safety hazard investigation. Summary of the Invention

[0005] This invention proposes an image recognition method and system for engineering safety hazards based on multi-model fusion, which solves the problem of the lack of scene-adaptive multi-model collaborative mechanism in existing technologies.

[0006] To address the aforementioned technical problems, this invention provides a method for identifying engineering safety hazards based on multi-model fusion, comprising the following steps: Step S1: Acquire the original image of the engineering site, identify the scene type of the original image and output the scene label, and load the target detection model and visual language reasoning model from the preset model library according to the scene label; Step S2: Based on the preset scene-detection item configuration table, determine that the detection item corresponding to the scene label belongs to a known category target and / or an open category target. For the known category target indicated by the scene label, detect the original image using the target detection model and output the bounding box coordinates. Crop the target sub-image from the original image based on the bounding box coordinates. Input the target sub-image and the preset hazard warning word template into the visual language inference model. For the open category target indicated by the scene label, input the original image and the hazard warning word template into the visual language inference model. The visual language inference model outputs a hazard inference result including hazard type and risk level. Step S3: Perform data fusion between the bounding box coordinates and the hazard inference results, and output structured hazard data including target identifier, coordinate information, hazard type and risk level.

[0007] Preferably, after obtaining the original image in step S1, the original image is subjected to resolution adaptive compression, and the longest side of the original image is compressed to a preset resolution threshold before the scene type recognition is performed.

[0008] Preferably, in step S1, scene type recognition is performed on the original image, including sequentially performing image normality judgment, construction site scene type judgment, and personnel presence judgment; when the result of the image normality judgment is that the image is abnormal, the execution of S2 to S3 is terminated, and the original image is marked as an image that does not require reasoning; when the result of the personnel presence judgment is that no personnel exist, only the detection path for known category targets is terminated, while the reasoning processing of the visual language reasoning model for open category targets is retained.

[0009] Preferably, in step S2, when the scene label indicates a known category target, the detection result output by the target detection model is subjected to dual threshold filtering. The dual threshold filtering includes removing targets with a confidence level lower than a preset confidence threshold and removing targets whose shortest side length of the bounding box coordinates is lower than a preset minimum pixel threshold. Only targets that pass the dual threshold filtering are cropped to obtain the target sub-image.

[0010] Preferably, the hazard warning word template in step S2 is stored with hierarchical coding according to the hazard classification system in the engineering safety hazard investigation standard. The hierarchical coding includes a three-level structure of scene category number, secondary category number, and hazard content number. The scene category number corresponds to the classification system of the scene label. The hierarchical coding adopts a naming rule of fixed prefix characters plus a seven-digit number, where the first two digits are the scene category number, the third and fourth digits are the secondary category number, and the last three digits are the hazard content number. The content structure of the hazard warning word template includes three components: core judgment rules, labeling specifications, and output requirements. The core judgment rules define the positive example judgment conditions and negative example exclusion conditions of the hazard in a conditional listing manner.

[0011] Preferably, the data fusion in step S3 includes the following conflict resolution rules: when the target detection model detects a target but the visual language reasoning model does not identify a hazard, the bounding box coordinates and the target identifier output by the target detection model are retained, and the hazard reasoning result is marked as a hazard-free state; When the visual language reasoning model identifies a hazard but the target detection model does not detect the target, the semantic coordinates output by the visual language reasoning model and the hazard type are used to generate a semantic tag, and no visual detection box is generated. When both the target detection model and the visual language reasoning model output detection results, and there is a contradiction in their hazard judgment conclusions, the hazard reasoning result of the visual language reasoning model shall prevail to determine the hazard type and risk level, and the target location information shall be determined based on the bounding box coordinates output by the target detection model.

[0012] Preferably, the preset confidence threshold and the preset minimum pixel threshold are dynamically configured differently based on the scene label, and different scene labels correspond to different combinations of threshold parameters; the combination of threshold parameters is updated by hot-loading the configuration file.

[0013] Preferably, step S3 further includes hazard classification and determination, calculating the characteristic value of the accident hazard based on the structured hazard data according to the accident hazard classification and determination formula. The formula for classifying and determining potential safety hazards is as follows: ; In the formula, These are characteristic values ​​of potential safety hazards; The accident hazard coefficient; The coefficients represent the location and type of potential safety hazards. This refers to the quantity coefficient of similar potential safety hazards existing within a relative range; Based on the aforementioned accident hazard characteristic values The numerical range is used to determine the level of accident hazard: when When the value exceeds the first threshold, it is determined to be a Level 1 accident hazard. When the value is greater than the second threshold but not exceeding the first threshold, it is judged as a level two accident hazard. When the value is greater than the third threshold but not exceeding the second threshold, it is judged as a level three accident hazard. If the threshold is not exceeded, it is determined to be a level four accident hazard; the first threshold is greater than the second threshold, and the second threshold is greater than the third threshold.

[0014] Preferably, the loading method of the hazard warning word template includes two trigger modes: automatic matching loading and forced specified loading. The automatic matching loading automatically matches the corresponding hazard warning word template according to the scene tag, and the forced specified loading forcibly loads the specified hazard warning word template according to the pre-set detection hazard type.

[0015] This invention also provides an engineering safety hazard image recognition system based on multi-model fusion, implemented based on the above-mentioned engineering safety hazard image recognition method based on multi-model fusion, comprising: The image acquisition module is used to acquire raw images of the engineering site; The scene adaptation module is used to identify the scene type of the original image and output scene labels, and dynamically load the adapted target detection model and visual language inference model from the preset model library according to the scene labels. The target detection module is used to detect the original image and output bounding box coordinates through the target detection model when the scene label indicates a target of known category, and to crop the target sub-image from the original image based on the bounding box coordinates; The hazard reasoning module is used to input the target sub-image or the original image and the hazard prompt word template into the visual language reasoning model, and output the hazard reasoning result including the hazard type and risk level; The fusion output module is used to perform data fusion on the output results of the target detection model and / or the visual language reasoning model, and output structured hazard data including target identification, coordinate information, hazard type and risk level.

[0016] The advantages of this invention include at least the following: 1. By dynamically loading corresponding model combinations based on scene type identification, rather than using a single model, the optimal detection strategy can be flexibly allocated for different engineering scenarios. 2. The target detection model first outputs accurate bounding boxes, and then the cropped sub-image is fed into the visual language model. This avoids the label offset problem that occurs when directly relying on the visual language model for target localization, and ensures coordinate accuracy. 3. By cropping the target sub-image, the target of interest is magnified before inference, which avoids the problem of unstable filtering of small distant targets by the visual language model. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the method flow according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the processing flow for known category targets in an embodiment of the present invention; Figure 3 A comparative illustration of the offset labeling of existing visual language models; Figure 4 A diagram illustrating the failure of small target filtering in existing visual language models; Figure 5 These are the precision-confidence curves and precision-recall curves of the target detection model in this embodiment of the invention. Figure 6The training process curve and F1-confidence curve of the target detection model in this embodiment of the invention are shown. Figure 7 This is a schematic diagram of the system module in an embodiment of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0019] like Figure 1 As shown, this embodiment of the invention provides a method for identifying engineering safety hazards based on multi-model fusion, including the following steps: Step S1: Acquire the original images of the engineering site, identify the scene type of the original images and output scene labels, and load the target detection model and visual language reasoning model from the preset model library according to the scene labels.

[0020] In this step, raw images are acquired using image acquisition devices deployed at the engineering site. These devices include, but are not limited to, fixed high-definition surveillance cameras, wide-angle cameras mounted on wearable AI glasses, and aerial cameras mounted on drones. The raw images support mainstream image encoding formats such as JPEG and PNG. The acquisition resolution is determined based on the on-site equipment conditions, typically ranging from 1080p to 4K.

[0021] After acquiring the original image, resolution-adaptive compression is first performed, compressing the longest side of the original image to the default resolution threshold of 1080p before subsequent scene type recognition. The technical purpose of this preprocessing step is to balance the computational efficiency of subsequent model inference with the degree of image information preservation. Experimental data shows that when the input resolution is reduced from 4K to 1080p, the processing time of the visual encoding stage decreases from approximately 1260 milliseconds to approximately 380 milliseconds, and the total inference time per frame decreases from approximately 1595 milliseconds to approximately 605 milliseconds. This effectively avoids the additional computational cost associated with high-resolution input while preserving key visual feature information.

[0022] After obtaining scene labels, the original image is recompressed according to the preset resolution threshold corresponding to the label for subsequent target detection. The preset resolution threshold is configured differently based on the scene label, rather than using a uniform fixed value. In different engineering scenarios, the typical distance between the camera and the target object varies significantly, resulting in large differences in the pixel proportion of the target in the image. In close-range scenes, such as the entrance / exit area of ​​a construction site, the typical distance between the camera and people is 5 to 15 meters. The pixel height of a person target in a 4K image is usually above 400 pixels. After compressing the image to 1080p, approximately 100 to 200 pixels of target height can still be retained, meeting the needs of subsequent detection and inference. Therefore, the preset resolution threshold is set to 1080 pixels. In long-range scenes, such as high-altitude tower crane operation areas, the typical distance between the camera and personnel on the ground is 50 to 100 meters. The pixel height of personnel targets in a 4K image is usually only 60 to 120 pixels. If the image is compressed to 1080p, the target pixel height will be further reduced to about 15 to 30 pixels, which may be lower than the effective recognition threshold of object detection models and visual language inference models. Therefore, the preset resolution threshold for long-range scenes is set to 1440 pixels or the original resolution is kept uncompressed. The preset resolution threshold for mid-range scenes, such as scaffolding construction areas, is set to 1280 pixels. Like the dual-threshold filtering parameters, the resolution threshold parameters are updated via hot-loading of the configuration file.

[0023] Scene type identification is performed on the original images after resolution compression preprocessing using a scene classifier. The scene classifier is a lightweight classification model independent of the object detection model and the visual language inference model. It resides in memory and does not require dynamic loading. Its input is the original image after resolution compression, and its output is scene labels. The scene classifier can employ lightweight neural networks such as MobileNet and EfficientNet. Scene type identification includes three sequential judgments: image normality judgment, construction site scene type judgment, and personnel presence judgment. The image normality judgment identifies abnormal image states such as lens occlusion, image blur, and complete darkness at night. The construction site scene type judgment determines the type of construction site to which the current image belongs and outputs scene labels, such as construction site entrance / exit, high-altitude tower crane operation area, foundation pit edge area, scaffolding construction area, etc. The personnel presence judgment detects the presence of construction workers in the image. The mapping relationship between scene labels and detection items is shown in Table 1.

[0024] Table 1 Mapping Table of Scene Labels and Detection Items The table above lists only some typical scenario labels and their corresponding detection items. The complete scenario label system is divided according to the 38 major categories in the Engineering Safety Hazard Investigation and Management Standards. Each scenario label can correspond to both known category detection items and open category detection items. The system executes the target detection path and the open category inference path according to the detection item type. The mapping relationship between scenario labels and detection items is stored in the form of a configuration file, which supports dynamic updates via hot loading of the configuration file. When a new hazard detection type is included in the standard system, only the configuration file needs to be updated to expand the system's detection capabilities.

[0025] When the result of the image normality judgment is that the image is abnormal, the system terminates the execution of subsequent steps S2 to S3 and marks the current original image as an image that does not require inference; when the result of the personnel existence judgment is that no personnel exist, only the detection path for known category targets is terminated, while the inference path for open category targets is retained.

[0026] It should be noted that the presence of personnel is only a pre-filter condition for the known category target detection path, not for all hazard detection paths. The known category targets primarily involve construction workers and their protective equipment. When no personnel are present in the image, these detection items are not feasible; therefore, terminating the known category target detection path avoids unnecessary model inference overhead. However, the open category targets include environmental hazards such as uncovered bare soil, illegal dumping, open flames, and smoke. The presence of these hazards is not directly related to the presence of personnel in the image; they may exist even in unmanned scenarios and need to be identified. Therefore, when the presence of personnel is determined to be absent, the system only skips the target detection and sub-image cropping stages for known category targets, but still inputs the original image and the corresponding open category hazard prompt template into the visual language inference model to perform environmental hazard identification. This differentiated filtering strategy retains the computational efficiency improvement effect of pre-filtering while avoiding the systematic missed detection of environmental hazards in unmanned scenarios. Taking 1000 frames of construction site monitoring video as an example, there are about 820 static frames with no one around. Among these 820 frames, about 15% of the frames contain environmental hazards such as uncovered bare soil or illegal dumping. After adopting a differentiated filtering strategy, these hazards can be detected normally, while the original full-frame filtering strategy will cause all of these hazards to be missed.

[0027] Scene labels are output based on scene type identification, and suitable object detection models and visual language inference models are dynamically loaded from a pre-set model library. The object detection model is a pre-trained object detection model based on a convolutional neural network architecture. Specifically, a self-trained YOLOv11 model is used, with training data sourced from a real-time hazard dataset covering 1000 construction sites. This dataset contains approximately 120,000 labeled images, covering five detection categories: safety helmets, not wearing safety helmets, reflective clothing, not wearing reflective clothing, and personnel. The labeling format uses YOLO-formatted normalized coordinates. Data collection covers construction site scenes from different regions, seasons, lighting conditions, and construction stages to ensure the model's generalization ability. Data augmentation strategies include random horizontal flipping, random scaling, data augmentation, and random color space transformation. The training process uses the stochastic gradient descent (SGD) optimizer with an initial learning rate of 0.01, employing a cosine annealing learning rate scheduling strategy. The total training epochs are 30, the batch size is 16, and the input image resolution is 640×640 pixels. The training hardware environment consisted of a server equipped with an NVIDIA A100 GPU. The training and validation sets were randomly divided in an 8:2 ratio.

[0028] The visual language reasoning model is an open-world visual language model based on the Transformer architecture. It does not require special training for specific hazard types and has the ability to perform semantic reasoning on image content based on natural language instructions.

[0029] Step S2: For the known category target indicated by the scene label, the target detection model detects the original image and outputs the bounding box coordinates. Based on the bounding box coordinates, a target sub-image is cropped from the original image. The target sub-image and a preset hazard warning word template are input into the visual language inference model. For the open category target indicated by the scene label, the original image and the hazard warning word template are input into the visual language inference model. The visual language inference model outputs a hazard inference result including the hazard type and risk level.

[0030] Specifically, the system maintains an attribute field for each scene label in the pre-defined model library, marking the hazard detection type under that scene as either a known category or an open category. A known category refers to the detection objects already covered in the pre-training dataset of the target detection model, such as safety helmets, reflective vests, and personnel. An open category refers to detection items not covered by the target detection model's training data but belonging to the engineering safety hazard investigation standards, such as uncovered bare soil, illegal dumping, and open flames. The mapping relationship is stored in the form of a configuration table, with each scene label corresponding to one or more known category detection items and / or open category detection items. The same scene can contain both types of detection items simultaneously, in which case the two paths are executed in parallel.

[0031] Model invocation follows this priority rule: When the scene label indicates that the original image contains a known category target with pre-trained data, such as construction workers, safety helmets, or reflective clothing, the target detection model is invoked first to perform forward inference on the original image, identify the target object in the image, and output the bounding box coordinates of each target object. The bounding box coordinates use a normalized coordinate system, with the format being the top-left x-coordinate. x 1. Top left ordinate y 1. Lower right x-coordinate x 2. Lower right ordinate y 2. Coordinate values ​​are normalized to the range of 0 to 1, unaffected by changes in the input image size. Detection results are output as a structured JSON two-dimensional array, where each element corresponds to a detected target, and the element order corresponds one-to-one with the unique index of the detected target. When the scene label indicates that the original image contains an open-category target without pre-trained data, such as uncovered bare soil, illegal dumping, or open flame hazards, the target detection stage is skipped. The original image and the hazard warning word template are directly input into the visual language inference model, utilizing its open-world semantic reasoning capabilities for hazard identification.

[0032] In the processing path for targets of known categories, after target detection is completed, a dual-threshold filtering is performed on the detection results. The dual-threshold filtering includes two screening mechanisms: the first is confidence filtering, which removes target objects with a confidence level lower than a preset confidence threshold (default confidence threshold is 0.6); the second is small target filtering, which removes target objects whose shortest side of the bounding box coordinates is lower than a preset minimum pixel threshold (default minimum pixel threshold is 32 pixels). Confidence filtering is used to exclude low-quality detection results where the detection model is uncertain about target recognition, reducing the false alarm rate; small target filtering is used to exclude targets with excessively small pixel areas in the distant scene, avoiding the problem of low factor map resolution leading to ineffective recognition by the subsequent visual language inference model. Subsequent cropping operations are performed only on target objects that pass the dual-threshold filtering to obtain target sub-images. Figure 2 This is a schematic diagram of the processing flow for known target categories in an embodiment of the present invention. From top to bottom, it shows the complete processing flow of original image scene recognition, target detection model outputting bounding box coordinates and cropping target sub-image, inputting target sub-image and hazard warning word template into visual language inference model to output hazard type and risk level, and fusion of detection results and inference results to perform hazard annotation visualization on the original image.

[0033] When performing small target filtering, the bounding box size in the normalized coordinate system needs to be converted to absolute pixel values ​​before being compared with a preset minimum pixel threshold. The specific conversion method is as follows: Let the actual width of the original image after resolution adaptive compression be... W pixels, actual height is Hpixels, the normalized bounding box coordinates of a target object are ( x 1, y 1, x 2, y 2), then the absolute pixel width of the bounding box of the target object is ( x 2- x 1) × W The absolute pixel height is ( y 2- y 1) × H The absolute pixel length of the shortest side of the bounding box is the smaller of the two values. When the absolute pixel length of the shortest side is lower than a preset minimum pixel threshold, the target is determined to be a small target and filtered out. It should be noted that... W and H The resolution is adapted to the actual size of the compressed image, rather than the original acquisition resolution before compression, because the forward inference of the object detection model takes the compressed image as input, and its output normalized coordinates are normalized based on the size of the compressed image.

[0034] The preset confidence threshold and preset minimum pixel threshold are dynamically configured differently based on scene labels, with different threshold parameter combinations corresponding to different scene labels. For example, in the high-altitude tower crane operation area, since people are generally small in the image, the minimum pixel threshold can be set to 32 pixels, and the confidence threshold can be appropriately reduced to 0.5 to improve the hazard recall rate in high-risk scenarios. In the personnel entrance and exit area, since people are generally closer and larger in the image, the minimum pixel threshold can be set to 128 pixels to exclude invalid identification of distant targets, while the confidence threshold remains at the standard setting of 0.6. The threshold parameter combinations are updated through hot loading of configuration files, and parameter adjustments can be completed without restarting the inference service, ensuring the system's flexible switching capability between different scenarios.

[0035] Based on the bounding box coordinates of target objects filtered by dual thresholds, target sub-images corresponding to each target object are cropped from the original image according to their coordinate range. Each target sub-image is bound to a unique serial number of its corresponding detected target to enable accurate backtracking and association of results in subsequent processing. The cropped target sub-images are then submitted in batches to the visual language inference model for hazard identification. Each target sub-image input is paired with a hazard warning word template loaded from an engineering hazard knowledge base; together, they constitute the input to the visual language inference model. The visual language inference model performs semantic understanding and inference judgment based on image content and warning word instructions, outputting the hazard inference result for each target object, including the hazard type and risk level.

[0036] The hazard warning word templates in the Engineering Hazard Knowledge Base are stored using hierarchical coding according to the hazard classification system in the Engineering Safety Hazard Investigation Standard. The hierarchical coding includes a three-level structure: scenario category number, secondary category number, and hazard content number. The scenario category number corresponds to the scenario tag classification system. The coding naming rule is a fixed prefix followed by a seven-digit number. The first two digits are the scenario category number, the third and fourth digits are the secondary category number, and the last three digits are the hazard content number in the corresponding hazard classification table. The hazard classification system is categorized according to the 38 major categories in the Engineering Safety Hazard Investigation and Management Standard, covering thousands of specific hazard contents under scenario dimensions such as construction plans, pole foundations, and scaffold stability. The templates are stored in Markdown format and are obfuscated and encrypted using a database approach.

[0037] The hazard warning template comprises three parts: core judgment rules, annotation specifications, and output requirements. The core judgment rules define the positive and negative conditions for hazard identification through a list of conditions, ensuring consistency in the visual language reasoning model's judgment under different visual conditions. Taking safety helmet detection as an example, the core judgment rules explicitly state: when the head area is clearly identifiable as a protective safety helmet conforming to safety standards, it is determined that a safety helmet is being worn; when the head is obscured and cannot be observed, it is also determined that a safety helmet is being worn to avoid false alarms; when the head is uncovered and hair or a bald head is clearly identifiable, it is determined that a safety helmet is not being worn; when the head is covered with non-safety helmet items that are clearly identifiable as a baseball cap, etc., it is also determined that a safety helmet is not being worn. The annotation specifications define the selection requirements for the target area and the coordinate output format. The output requirements specify the JSON data structure of the reasoning results, including the hazard judgment conclusion, confidence level, and target area coordinates.

[0038] The output format of the visual language inference model is guaranteed by embedding strict format constraints and an output template within the output requirements section of the hazard warning word template. The format constraints require that the output of the visual language inference model must contain exactly one valid JSON object, without any prefix text, Markdown code block markers, or postscript supplementary information. The output template defines the name, data type, and value range of each field in the form of a JSON Schema.

[0039] Taking the detection of safety helmets of known target categories as an example, the complete output JSON structure is defined as follows: { "target_id": 1, "hazard_detected": true, "hazard_type": "not wearing a helmet", "confidence": 0.85, "risk_level": "Level 4", "reasoning": "No safety helmet was visible on the target person's head area; bare hair was clearly visible." } Where target_id is an integer, corresponding to the unique index of the target output by the target detection model; hazard_detected is a boolean, indicating whether a hazard has been detected; hazard_type is a string, which takes the value null when hazard_detected is false; confidence is a floating-point number, ranging from 0 to 1, representing the confidence level of the visual language reasoning model in the current judgment; risk_level is a string, ranging from level one to level four; and reasoning is a string used to record the description of the basis for the judgment for subsequent manual review.

[0040] To further ensure the stability of the output format, the hazard warning word template also embeds a set of correct and incorrect output examples. This guides the visual language inference model to strictly adhere to the output format through few-shot learning. After receiving the output from the visual language inference model, the system first performs JSON format validation and handles exceptions for invalid output: when the output contains Markdown code block tags, the tags are automatically removed and the data is re-parsed; when JSON parsing fails, the inference result for that target is marked as inference failure and recorded as a missing value in the final structured hazard data, without affecting the normal processing of other targets in the same frame.

[0041] The following provides examples of the core judgment rules for two hazard warning word templates: reflective clothing detection and bare soil uncovering detection. The former is a reasoning template for known category targets, while the latter is a reasoning template for open category targets.

[0042] Example template for warning messages regarding potential hazards when inspecting reflective clothing: Template code: P0102001, where the prefix P is a fixed prefix character, 01 is the scene category number (personnel protective equipment category), 02 is the secondary category number (upper body protection category), and 001 is the hazard content number (reflective clothing wearing detection).

[0043] Core judgment rules: When a person's upper body area can be clearly identified as wearing a safety reflective vest or reflective clothing with reflective stripe features, it is judged that the person is wearing reflective clothing; when the upper body area of ​​the person is obscured by other objects and the obscured area exceeds 70% of the upper body area, it is judged that it cannot be judged and no hazard marking is made; when the upper body area of ​​the person can be clearly identified as ordinary clothing without reflective stripe features, it is judged that the person is not wearing reflective clothing; when the person is wearing clothing with a small amount of reflective elements but does not meet safety standards, such as only having a small area of ​​reflective logo but no complete reflective stripe, it is judged that the person is not wearing reflective clothing.

[0044] Labeling specifications: The target area is defined by the upper body of the person, and the selected area should include the entire torso area. The coordinate output format is normalized coordinates.

[0045] Output requirements: Output the inference results in JSON format, including the following fields: target_id (unique target number), hazard_detected (hazard detection conclusion), with a value of true or false, hazard_type (hazard type), with a value of "not wearing reflective clothing" or "null", confidence (judgment confidence level), with a value of 0 to 1, and risk_level (risk level), with a value of four levels.

[0046] Example template for warning messages regarding potential hazards from uncovered bare soil: Template code: P0501001, where 05 is the scene category number (environment and site category), 01 is the secondary category number (earthwork and surface category), and 001 is the hazard content number (bare soil uncovering detection).

[0047] Core judgment rules: When there is a clearly visible area of ​​bare soil in the image without dust netting, geotextile, or other covering materials, it is judged as a potential hazard of uncovered bare soil; when green dust netting, black geotextile, or turf are visible above the soil area and the coverage area exceeds 80% of the bare soil area, it is judged as covered and does not constitute a hazard; when the soil area in the image is the face of an excavation pit under construction or a working face where earthwork is being carried out, it is judged as a construction operation and does not constitute a potential hazard of uncovered bare soil; when only a small amount of scattered soil or vehicle tracks are visible in the image, rather than a large area of ​​bare soil, it does not constitute a potential hazard of uncovered bare soil.

[0048] Labeling Guidelines: Since uncovered bare soil is an open category target, it does not require precise localization by the target detection model. The visual language inference model only needs to output the hazard assessment conclusion and a general area description, without requiring precise pixel-level coordinates. The area description is given in natural language, such as "There is an area of ​​approximately 30 square meters of uncovered bare soil in the lower left corner of the image."

[0049] Output requirements: Output the inference results in JSON format, including the following fields: hazard_detected (hazard determination conclusion), hazard_type (values ​​are "bare soil not covered" or "null"), confidence (determination confidence level), risk_level (risk level), and region_description (region description).

[0050] As can be seen from the three examples above, the hazard warning word templates for known category targets and open category targets maintain a consistent three-part structure: core judgment rules + annotation specifications + output requirements. However, there are differences in annotation specifications and output requirements: the template for known category targets requires the output of `target_id` to align with the target detection results, and does not require the output of a region description, as the coordinates are already provided by the target detection model; the template for open category targets does not include a `target_id` field, but requires the output of a region description to provide semantic reference information about the hazard location. Those skilled in the art can refer to the writing paradigm of the above examples to write corresponding hazard warning word templates for other hazard types.

[0051] The loading of hazard warning word templates includes two trigger modes: automatic matching loading and forced specified loading. Automatic matching loading automatically matches the corresponding hazard warning word template based on scene tags, achieving intelligent association between scenes and warning words. Forced specified loading forcibly loads a specified hazard warning word template based on a pre-defined hazard type, suitable for business scenarios requiring targeted detection of specific types of hazards on specific construction site images. Hazard warning word templates support online editing in the background, and the edited template takes effect immediately through hot loading without requiring a restart of the inference service.

[0052] Step S3: Merge the bounding box coordinates with the hazard reasoning results to output structured hazard data including target identifier, coordinate information, hazard type and risk level.

[0053] In this step, the bounding box coordinates output by the target detection model and the hazard inference results output by the visual language inference model are fused to generate structured hazard data containing complete detection information. The coordinate alignment mechanism during the data fusion process is as follows: the final output target coordinates preferentially adopt the bounding box coordinates output by the target detection model, because it has undergone bounding box regression and non-maximum suppression coordinate calculation pipeline processing, and its positioning accuracy is significantly higher than the semantic coordinate output of the visual language inference model. The output of the visual language inference model is only used to supplement semantic information such as hazard type and risk level. Since the output array order of the detection results in step S2 corresponds one-to-one with the unique index of the target, in the input construction stage of batch inference, each target subgraph's inference request carries its corresponding target's unique index as an identifier field. This identifier field is embedded in the input prompt words submitted to the visual language inference model, and the visual language inference model is required to return this identifier field as is in the JSON data structure of the output results. After the inference results are returned, the system compares the unique sequence number field returned in each inference result with the unique sequence number in the target detection result to accurately associate each visual language inference conclusion with the corresponding target detection result, achieving cross-model output alignment and fusion. This alignment mechanism based on explicit identifier fields does not rely on the consistency of the input-output order of batch inference, and can guarantee the correctness of alignment even if the visual language inference model returns results using asynchronous or parallel inference. In the specific implementation, the unique sequence number field is embedded at the end of the prompt word in the form of a key-value pair, with the format target_id:[sequence number]. The corresponding target_id field is also included in the output JSON of the visual language inference model.

[0054] The following conflict resolution rules are set during the data fusion process to handle potential inconsistencies between the outputs of the two models.

[0055] The first scenario: When the target detection model detects a target but the visual language inference model fails to identify a hazard, the bounding box coordinates and target identifier output by the target detection model are retained, and the hazard inference result is marked as hazard-free, completing the current target identification process. The technical consideration behind this rule is that the target detection model may correctly detect a person target, but that person may not actually be engaging in any safety violations. In this case, retaining the target information but marking it as compliant is a reasonable approach.

[0056] The second scenario: When the visual language inference model identifies a hazard but the target detection model fails to detect the target, semantic tags are generated using the semantic coordinates output by the visual language inference model and the hazard type, without generating a visual detection box. The reason for not generating a detection box is that the coordinate output precision of the visual language inference model is insufficient; drawing a detection box based on this might lead to an overly large selection and false alarms. Therefore, hazard information is recorded only in the form of semantic tags.

[0057] The third scenario: When both the target detection model and the visual language reasoning model output detection results, and there is a contradiction between their hazard judgment conclusions, the hazard type and risk level shall be determined based on the hazard reasoning result of the visual language reasoning model, and the target location information shall be determined based on the bounding box coordinates output by the target detection model.

[0058] The structured hazard data output after fusion is organized in JSON format and includes the following fields: the original image file path and a target list array. Each element in the target list array contains three sets of information: detection information (including bounding box coordinates, target category, and detection confidence); inference information (including the judgment conclusion for each hazard inspection item); and cropped sub-image information (including the storage path of the corresponding target sub-image). Subsequent rendering services, based on the configured visualization threshold, perform hazard annotation overlay on the original image for targets that meet the conditions, generating an output image with hazard annotation information.

[0059] After obtaining the structured hazard data, step S3 also includes hazard classification determination. Based on the hazard types and quantities identified in the structured hazard data, the hazard characteristic value L is calculated according to the hazard classification determination formula, which is: ; In the formula, These are the characteristic values ​​of potential safety hazards, used to characterize the overall risk level of the hazard. The accident hazard coefficient represents the severity of the consequences of an accident. Its value is determined based on the table of accident hazard coefficient values ​​in the "Standard for Safety Inspection of Building Construction" (JGJ 59-2011), a standard for investigating and managing safety hazards in engineering projects. The coefficients represent the location and category of potential safety hazards. The coefficient for potential safety hazards in general sub-projects is 1, that for sub-projects with a certain degree of risk is 2, that for sub-projects with a relatively high risk is 4, and that for sub-projects with a relatively high risk exceeding a certain scale is 8. This refers to the quantity coefficient of similar potential hazards existing within a relative range. When the number of similar potential hazards is less than 20% of the number of samples inspected, The value is 1, when the number of similar hidden dangers accounts for 20% to 50% of the number of samples inspected. The value is 1.5, when the number of similar potential hazards exceeds 50% of the number of samples inspected. The value is 2. The statistical period is days. The calculation method is to divide the number of times a certain type of hidden danger is detected at the same monitoring point within a day after deduplication by the total number of times that type of target is detected at the same monitoring point within a day after deduplication. When the same type of hazard is detected repeatedly at the same monitoring point within a preset time window, duplicates are removed and counted as only one hazard detection. Based on statistical analysis of safety hazards at 1000 construction sites, a hazard occurrence rate of less than 20% is considered occasional, 20% to 50% is considered frequent, and higher than 50% is considered widespread. Therefore, the following criteria are used to determine... The grading threshold.

[0060] The numerical values ​​of the accident hazard coefficient are shown in Table 2.

[0061] Table 2. Numerical Table of Accident Hazard Coefficients Based on the characteristic values ​​of accident hazards The numerical range is used to determine the level of accident hazard: when A value greater than 24 is considered a Level 1 accident hazard; when A value greater than 12 but not exceeding 24 is considered a Level II accident hazard; when A hazard level greater than 4 but not exceeding 12 is classified as a Level 3 accident hazard; when If the time limit is less than 4 hours, it is determined to be a level 4 accident hazard.

[0062] To verify the technical effectiveness of the method of the present invention, a systematic experimental evaluation was conducted on the selection of visual language reasoning model, the accuracy of target detection model, and the overall efficiency improvement of the solution.

[0063] 1. Evaluation of Visual Language Reasoning Models The performance of five different visual language models in engineering safety inspection tasks was comprehensively evaluated, focusing on two dimensions: basic performance and safety recognition capability. The results of the basic performance evaluation are shown in Table 3.

[0064] Table 3 Comparison of basic performance of visual language models Regarding safety hazard identification capabilities, recognition rate tests were conducted on six high-frequency safety hazards: not wearing a safety helmet, not wearing a reflective vest, uncovered bare soil, smoking, dangerous intrusion, and presence of smoke or fire. Test results show that uncovered bare soil is an easily identifiable type, with recognition rates for all models consistently between 63% and 68%, indicating that the visual characteristics of this type of hazard are relatively obvious. The presence of smoke or fire and smoking are difficult to identify, with most models achieving near-zero recognition rates, and only a few models exhibiting limited recognition capabilities. Considering both success rate and response time, the Qwen2-VL-8B achieved the best overall performance score with an 88% success rate and an average response time of 2953 milliseconds, making it suitable for monitoring scenarios with high real-time requirements. The qwen-vl-plus has the most comprehensive coverage across all hazard inspection items, making it suitable for key inspection scenarios with high accuracy requirements for hazard identification.

[0065] 2. Target detection model accuracy verification The accuracy of the self-pre-trained YOLOv11 model was validated on a construction site safety dataset, which covers five detection categories: safety helmet, not wearing a safety helmet, not wearing reflective clothing, personnel, and reflective clothing. Figure 5 The figures show the precision-confidence curves and precision-recall curves of the target detection model in this embodiment of the invention. The upper figure shows the precision curves of five detection categories—helmet, no helmet, no reflective vest, person, and reflective vest—as a function of confidence. The lower figure shows the precision-recall curves of each detection category at an IoU threshold of 0.5. In the figures, Precision-Confidence Curve represents the precision-confidence curve, Precision represents precision, Confidence represents confidence, helmet represents a helmet, no-helmet represents no helmet, no-vest represents no reflective vest, person represents a person, vest represents a reflective vest, all classes 1.00 at 0.986 means that the precision of all categories is 1.00 at a confidence level of 0.986, and all classes 0.846 mAP@0.5 means that the mAP of all categories is 0.846 at 0.5. Test results show that the model achieves an overall precision of 1.00 for all categories at a confidence threshold of 0.986. The average precisions for each category at an IoU threshold of 0.5 are as follows: helmet 0.936, no helmet 0.702, no reflective vest 0.776, person 0.942, reflective vest 0.875. The overall average precision (mAP@0.5) for all categories reaches 0.846.

[0066] Figure 6The figures show the training process curves and F1-confidence curves of the target detection model in this embodiment of the invention. The upper part shows the convergence curves of the loss function on the training set and the changing trends of evaluation indicators such as precision, recall, and average precision. The lower part shows the F1 score curves of each detection category as a function of confidence. The overall F1 value of all categories reaches 0.80 when the optimal confidence threshold is 0.203. In the figure, train / box_loss represents training / box loss, train / cls_loss represents training / classification loss, metrics / precision(B) represents metric / precision(B), metrics / recall(B) represents metric / recall(B), val / box_loss represents validation / box loss, val / cls_loss represents validation / classification loss, val / dfl_loss represents validation / distribution focus loss, metrics / mAP50(B) represents metric / mAP@0.5(B), metrics / mAP50-95(B) represents metric / mAP@0.5:0.95(B), results / smooth represents the original result / smoothed curve, and F1-Confidence Curve represents the F1 score-confidence curve. This accuracy level validates the object detection model's ability to locate and classify personnel and protective equipment in a construction site safety scenario. Experiments have shown that the target detection model has a much higher accuracy in selecting bounding boxes than the visual language model. Furthermore, the model can effectively exclude low-confidence targets and small targets through confidence filtering. In contrast, the bounding box markings of the visual language model show random shifts in multiple rounds of testing. This further verifies the rationality of the fusion strategy in this invention, which uses the target detection model coordinates as the primary factor and the visual language inference results as a secondary factor.

[0067] 3. Overall Solution Efficiency Analysis Using 1000 frames of construction site monitoring video as a test sample, the inference efficiency of the baseline solution and the solution of this invention were compared. The baseline solution performs full inference on all frames, that is, object detection and visual language inference are performed sequentially for each frame, with a single frame taking about 1 second, and a total time of about 1000 seconds for 1000 frames. The solution of this invention, through pre-scene filtering, performs only lightweight scene classification processing on 82% of unmanned static frames, with a single frame taking about 0.2 seconds, and only 18% of manned frames enter the complete detection and inference process, with a single frame taking about 1 second, and a total time of about 344 seconds for 1000 frames, resulting in an overall efficiency improvement of about 2.94 times. After further optimization with resolution compression, the overall inference efficiency can be improved by 7 to 8 times, and the inference time is reduced by more than 60%. In terms of computing power cost, compared with the full-frame inference solution, the solution of this invention reduces visual language inference calls by more than 80%, and combined with resolution optimization, the overall computing power cost is reduced by 70% to 87.5%.

[0068] like Figure 7As shown, this embodiment of the invention also provides an engineering safety hazard image recognition system based on multi-model fusion, implemented based on the above method, including the following functional modules: The image acquisition module is used to acquire raw images of the engineering site and supports receiving image data from various acquisition terminals such as fixed monitoring cameras, wearable AI glasses, and drones.

[0069] The scene adaptation module identifies the scene type of the original image and outputs scene labels. It sequentially performs image normality checks, construction site scene type checks, and personnel presence checks. Based on the scene labels, it dynamically loads adapted object detection models and a Transformer-based visual language inference model from a pre-set model library. The scene adaptation module also filters and intercepts abnormal and unattended images, marking images that do not meet the inference conditions as images that do not require inference.

[0070] The object detection module is used to perform forward inference on the original image by loading the object detection model when the scene label indicates a known category object, output the bounding box coordinates of the object, perform dual threshold filtering of confidence filtering and small object filtering, and crop the object sub-image from the original image based on the bounding box coordinates that pass the filtering.

[0071] The hazard reasoning module is used to input the target sub-image or original image and hazard warning word templates loaded from the engineering hazard knowledge base into the visual language reasoning model. It performs semantic understanding of the target state and hazard type determination, outputting hazard reasoning results including hazard type and risk level. The hazard reasoning module has built-in management and loading functions for hazard warning word templates, supporting both automatic matching loading and forced specified loading trigger modes.

[0072] The fusion output module is used to fuse the output results of the target detection model and / or the visual language reasoning model. It handles inconsistencies between the output results of the two models according to preset conflict resolution rules. The final output includes structured hazard data including target identification, coordinate information, hazard type and risk level, for subsequent visualization rendering and safety management platform use.

[0073] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described; only preferred embodiments of the present invention are illustrated. The descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the present invention. As long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0074] It should be noted that those skilled in the art can make various modifications and improvements without departing from the inventive concept, and these all fall within the scope of protection of this invention. Therefore, the scope of protection of this invention should be determined by the appended claims.

Claims

1. A method for image recognition of engineering safety hazards based on multi-model fusion, characterized in that, Includes the following steps: Step S1: Acquire the original image of the engineering site, identify the scene type of the original image and output the scene label, and load the target detection model and visual language reasoning model from the preset model library according to the scene label; Step S2: Based on the preset scene-detection item configuration table, determine that the detection item corresponding to the scene label belongs to a known category target and / or an open category target. For the known category target indicated by the scene label, detect the original image using the target detection model and output the bounding box coordinates. Crop the target sub-image from the original image based on the bounding box coordinates. Input the target sub-image and the preset hazard warning word template into the visual language inference model. For the open category target indicated by the scene label, input the original image and the hazard warning word template into the visual language inference model. The visual language inference model outputs a hazard inference result including hazard type and risk level. Step S3: Perform data fusion between the bounding box coordinates and the hazard inference results, and output structured hazard data including target identifier, coordinate information, hazard type and risk level.

2. The method for identifying engineering safety hazards based on multi-model fusion according to claim 1, characterized in that: After obtaining the original image in step S1, the original image is subjected to resolution adaptive compression, and the longest side of the original image is compressed to a preset resolution threshold before the scene type recognition is performed.

3. The method for image recognition of engineering safety hazards based on multi-model fusion according to claim 1, characterized in that: In step S1, scene type recognition is performed on the original image, including sequentially performing image normality judgment, construction site scene type judgment, and personnel presence judgment. When the result of the image normality judgment is that the image is abnormal, the execution of S2 to S3 is terminated, and the original image is marked as an image that does not require reasoning. When the result of the personnel presence judgment is that no personnel are present, only the detection path for known category targets is terminated, while the reasoning processing of the visual language reasoning model for open category targets is retained.

4. The method for image recognition of engineering safety hazards based on multi-model fusion according to claim 1, characterized in that: In step S2, when the scene label indicates a known category target, the detection results output by the target detection model are subjected to dual threshold filtering. The dual threshold filtering includes removing targets with a confidence level lower than a preset confidence threshold and removing targets whose shortest side length of the bounding box coordinates is lower than a preset minimum pixel threshold. Only targets that pass the dual threshold filtering are cropped to obtain the target sub-image.

5. The method for identifying engineering safety hazards based on multi-model fusion according to claim 1, characterized in that: In step S2, the hazard warning word template is stored using hierarchical coding according to the hazard classification system in the engineering safety hazard investigation standard. The hierarchical coding includes a three-level structure of scene category number, secondary category number, and hazard content number. The scene category number corresponds to the classification system of the scene label. The hierarchical coding adopts a naming rule of fixed prefix characters plus a seven-digit number, where the first two digits are the scene category number, the third and fourth digits are the secondary category number, and the last three digits are the hazard content number. The content structure of the hazard warning word template includes three components: core judgment rules, labeling specifications, and output requirements. The core judgment rules define the positive example judgment conditions and negative example exclusion conditions of the hazard in a conditional listing manner.

6. The method for image recognition of engineering safety hazards based on multi-model fusion according to claim 1, characterized in that: The data fusion in step S3 includes the following conflict resolution rules: when the target detection model detects a target but the visual language reasoning model does not identify a hazard, the bounding box coordinates and the target identifier output by the target detection model are retained, and the hazard reasoning result is marked as a hazard-free state; When the visual language reasoning model identifies a hazard but the target detection model does not detect the target, the semantic coordinates output by the visual language reasoning model and the hazard type are used to generate a semantic tag, and no visual detection box is generated. When both the target detection model and the visual language reasoning model output detection results, and there is a contradiction in their hazard judgment conclusions, the hazard reasoning result of the visual language reasoning model shall prevail to determine the hazard type and risk level, and the target location information shall be determined based on the bounding box coordinates output by the target detection model.

7. The method for image recognition of engineering safety hazards based on multi-model fusion according to claim 4, characterized in that: The preset confidence threshold and the preset minimum pixel threshold are dynamically configured differently based on the scene label, and different scene labels correspond to different combinations of threshold parameters; the combination of threshold parameters is updated by hot-loading the configuration file.

8. The method for image recognition of engineering safety hazards based on multi-model fusion according to claim 1, characterized in that: Step S3 also includes hazard classification and determination, which involves calculating the characteristic value of the accident hazard based on the structured hazard data and the accident hazard classification and determination formula. The formula for classifying and determining potential safety hazards is as follows: ； In the formula, These are characteristic values ​​of potential safety hazards; The accident hazard coefficient; The coefficients represent the location and type of potential safety hazards. This refers to the quantity coefficient of similar potential safety hazards existing within a relative range; Based on the aforementioned accident hazard characteristic values The numerical range is used to determine the level of accident hazard: when When the value exceeds the first threshold, it is determined to be a Level 1 accident hazard. When the value is greater than the second threshold but not exceeding the first threshold, it is judged as a level two accident hazard. When the value is greater than the third threshold but not exceeding the second threshold, it is judged as a level three accident hazard. If the threshold is not exceeded, it is determined to be a level four accident hazard; the first threshold is greater than the second threshold, and the second threshold is greater than the third threshold.

9. The method for image recognition of engineering safety hazards based on multi-model fusion according to claim 5, characterized in that: The loading method of the hazard warning word template includes two trigger modes: automatic matching loading and forced specified loading. The automatic matching loading automatically matches the corresponding hazard warning word template according to the scene tag, while the forced specified loading forcibly loads the specified hazard warning word template according to the pre-set detection hazard type.

10. An engineering safety hazard image recognition system based on multi-model fusion, implemented based on the engineering safety hazard image recognition method based on multi-model fusion as described in any one of claims 1-9, characterized in that, include: The image acquisition module is used to acquire raw images of the engineering site; The scene adaptation module is used to identify the scene type of the original image and output scene labels, and dynamically load the adapted target detection model and visual language inference model from the preset model library according to the scene labels. The target detection module is used to detect the original image and output bounding box coordinates through the target detection model when the scene label indicates a target of known category, and to crop the target sub-image from the original image based on the bounding box coordinates; The hazard reasoning module is used to input the target sub-image or the original image and the hazard prompt word template into the visual language reasoning model, and output the hazard reasoning result including the hazard type and risk level; The fusion output module is used to perform data fusion on the output results of the target detection model and / or the visual language reasoning model, and output structured hazard data including target identification, coordinate information, hazard type and risk level.

Images