A multi-stage identification and structured output method for construction site safety hazards based on a visual language model

Abstract

The present application belongs to the technical field of safety monitoring, and particularly relates to a construction site safety hazard multi-stage identification and structured output method based on a visual language model. In view of the defect that the existing construction safety visual identification does not meet the comprehensive requirements of construction safety inspection, the present application sets a construction visual evidence gating step before hazard analysis, extracts construction visual evidence items through a visual language model, and matches the evidence items with a preset strong construction evidence set, a weak construction clue set and a non-construction exclusion set, determines the image as strong construction evidence, weak construction clue or no construction evidence according to the number and spatial distribution of the evidence items; when the image does not have clear construction visual evidence, stop hazard analysis to reduce false positives of non-construction scenes from the source. At the same time, output structured JSON results, frame-labeled images and rectification suggestions, so that the identification results can enter the artificial review, rectification dispatching, statistical analysis and closed-loop management processes.

Description

Technical Field

[0001] This invention belongs to the field of safety monitoring technology, and in particular relates to a method for multi-stage identification and structured output of safety hazards at construction sites based on a visual language model. Background Technology

[0002] Construction sites are characterized by frequent changes in personnel, machinery, materials, openings, temporary power facilities, and work surfaces. Safety hazards are usually not determined by a single object itself, but by the combined effects of the object's state, spatial relationships, work conditions, and protective measures. For example, whether the edge of a foundation pit constitutes a potential hazard requires a comprehensive assessment of factors such as whether the boundary is clear, accessible, whether there are guardrails, and whether there are personnel or work activities nearby.

[0003] Existing visual safety recognition methods for construction generally fall into the following categories: The first category is based on target detection models for detecting safety helmets, safety belts, reflective vests, personnel, machinery, or components. These methods typically rely on detection frameworks such as YOLO, Faster R-CNN, and DETR, along with manually labeled data, and are suitable for closed-category target recognition. The second category is construction site or construction area recognition methods. Their primary goal is to determine whether an image is within a construction environment. These methods are often used for navigation assistance, scene filtering, or on-site recognition, but usually do not directly output hazard types or rectification suggestions. The third category directly uses visual language models or multimodal large models for hazard identification, requiring the model to describe safety issues in the image through prompts. The fourth category is retrieval enhancement or regulatory knowledge enhancement methods, providing safety regulations, case libraries, or similar image retrieval results to the large model to improve its ability to interpret safety rules. The fifth category combines detectors with visual language models. First, target detectors locate personnel, machinery, or protective facilities, and then the visual language model combines the detection results to make a safety judgment.

[0004] The methods described above address some issues in visual recognition of construction safety, but shortcomings remain. Target detection-based solutions rely on predefined categories, making it difficult to cover open, relational, and state-based hazards on construction sites. Construction scene recognition solutions typically only address whether a site is a construction environment, lacking subsequent hazard evidence assessment and structured report generation. Directly using visual language models to generate hazard reports can lead to problems such as inference based on sign text, over-analysis of weak construction clues, misidentifying normal construction conditions as hazards, mixed descriptions of multiple hazards, or lack of location labeling. Retrieval enhancement solutions can supplement regulatory knowledge, but without image evidence gating and candidate-level verification, they may still output invisible or insufficiently evidenced issues as hazards. Detector-guided solutions can improve local localization, but their effectiveness is still limited by the detection category and typically lack a complete process for outputting multiple hazards, scene evidence explanation, and final subtraction verification. Therefore, relying solely on single target detection, single visual language model question answering, or external knowledge retrieval is insufficient to meet the comprehensive requirements of construction safety inspections for low false alarms, explainability, multiple hazards, localization, and remediability. A multi-stage identification and verification method for construction safety scenarios is needed, which decomposes construction scenario gating, target extraction, risk candidate screening, candidate evidence judgment, structure generation and final verification into reviewable processing steps, and ensures that each hidden danger corresponds to clear image evidence, location and rectification suggestions. Summary of the Invention

[0005] To address the shortcomings of existing construction safety visual recognition systems that fail to meet the comprehensive requirements of low false alarms, explainability, identification of multiple hazards, locationability, and remediability in construction safety inspections, this paper proposes a multi-stage identification and structured output method for construction site safety hazards based on a visual language model, including:

[0006] S1: Obtain the set of images to be recognized;

[0007] S2: Based on the first visual language model, perform construction visual evidence extraction processing on the image set to be identified to obtain N construction visual evidence items;

[0008] Based on N construction visual evidence items, the first judgment process is performed on the image set to be identified to obtain the image set to be identified that passes the first judgment, and then proceeds to S3;

[0009] When no image to be identified passes the first judgment process, the set of images to be identified in S1 is determined to be in the case of no strong construction, and subsequent analysis is stopped;

[0010] S3: Input the image to be identified through S2 gate, N construction visual evidence items and a preset construction object vocabulary into the second visual language model for construction target extraction processing, and obtain A construction area recognition results and M target recognition results;

[0011] S4: Based on the preset construction safety risk rule base, rule matching processing is performed on the identification results of each target to be detected obtained in S3 to obtain an initial risk candidate set;

[0012] Based on the third visual language model with preset construction status filtering rules, the initial risk candidate set is filtered to obtain the key risk candidate set.

[0013] S5: Based on the fourth visual language model, image evidence judgment processing is performed on the key risk candidate set to obtain K risks that are judged to be valid;

[0014] S6, based on the risks that K judgments are valid, generate K structured hazard records;

[0015] S7: Based on the fifth visual language model, the K structured hazard records obtained in S6 are reviewed to obtain a set of qualified structured hazard records;

[0016] S8: Based on the verified and qualified structured hazard record set, output the final hazard set, structured JSON results, boxed annotation diagram, and text report.

[0017] Furthermore, the structured JSON result of this invention includes all fields of each potential hazard record, facilitating data exchange between systems;

[0018] The framed annotation diagram of the present invention is a normalized annotation box and its corresponding hazard category label for each hazard record retained in S7 superimposed on the original image;

[0019] The written report lists the on-site phenomena, location, risk level, consequences, and rectification suggestions for each hazard, grouped by hazard category, for safety management personnel to read and use directly.

[0020] The images to be identified in S1 include: screenshots of construction site monitoring, mobile phone inspection photos, drone images or other fixed images; the image source can be real-time monitoring video frames transmitted over the network, or on-site photos uploaded by inspection personnel through mobile terminals;

[0021] In step S2, the visual language model is used to extract construction visual evidence from the images to be identified, resulting in N construction visual evidence items. Based on these N construction visual evidence items, a first judgment process is performed on the set of images to be identified to obtain a set of images that pass the first judgment. The specific process is as follows:

[0022] S2.1: Input natural language prompts containing the construction scene evidence classification system into the first visual language model.

[0023] The specific method for the visual language model to extract visual evidence items of construction is as follows: input prompt words containing the classification system of construction scene evidence into the visual language model, and require the model to traverse the image content and output the category, position coordinates in the image, and visibility description of each identifiable object or scene element.

[0024] Furthermore, the first visual language model selected in this invention can be a multimodal large model deployed in the cloud (such as GPT-4V, Gemini Pro Vision, etc.), or a locally deployed open-source visual language model, or a multimodal inference model composed of an image encoder and a large language model.

[0025] The natural language prompts list complete definitions for the following sets of strong construction evidence (foundation pits, trenches, excavation work surfaces, scaffolding, formwork supports, edge openings, construction machinery, hoisting operations, pouring operations, rebar processing, welding and cutting operations, construction area workers, temporary construction power facilities, and work surfaces within enclosures), weak construction clues (scattered bricks, pipes, bagged materials, construction waste, ordinary building entrances, ordinary building facade maintenance traces, and small amounts of stockpiled materials), and non-construction exclusions (ordinary roads, green spaces, private vehicles, electric vehicles, ordinary pedestrians, and the everyday building environment).

[0026] S2.2: Input the task description text into the first visual language model: "It is required to traverse the image content and output the following for each identifiable object or scene element: evidence category, normalized bounding box coordinates in the image, visible state description text, and whether the recognition result comes only from the text content in the image"; (i.e. whether the recognition is based solely on text information such as sign text, project name, and monitoring title, rather than on the visual features of the object itself).

[0027] Furthermore, the first visual language model outputs N construction visual evidence items, each of which includes: evidence category, normalized bounding box, and visibility state feature; wherein, the i-th construction visual evidence item output by the first visual language model is denoted as... Includes evidence categories Normalized x-coordinate and normalized y-coordinate of the annotation box Visible state characteristics This can be expressed as a formula:

[0028] ;

[0029] Then, for each piece of evidence, the evidence category is matched with the set of strong construction evidence, the set of weak construction clues, and the set of non-construction exclusions using string or semantic matching, respectively. The number of strong construction evidence items is then counted. And the number of weak construction clues And calculate the percentage r of the image area covered by all evidence item annotation boxes.

[0030] The decision logic is: when When ≥1 and r>0, it is considered strong construction evidence; when =0 and When ≥1, it is judged as a weak construction clue; when =0 and If the value is 0, or all evidence items only match the non-construction exclusion set, or only text, signs, project names, and monitoring titles exist without any physical construction evidence, it is determined that there is no construction evidence. Only when it is determined to be strong construction evidence will the analysis continue flag be set to true, triggering subsequent steps S3 to S8; otherwise, the analysis continue flag will be set to false, terminating the entire analysis process.

[0031] S2.3: Match the evidence categories in the N construction visual evidence items with the preset strong construction evidence set, weak construction clue set, and non-construction exclusion set, and count the number of strong construction evidence items and the number of weak construction clue items.

[0032] The set of strong construction evidence includes at least one of the following: foundation pit, trench, excavation working face, scaffolding, formwork support, edge openings, construction machinery, hoisting operations, pouring operations, steel bar processing, welding and cutting operations, construction area workers, temporary construction power facilities, and working face within the enclosure;

[0033] The set of weak construction clues includes at least one of the following: scattered bricks, pipes, bags, construction waste, ordinary building entrances, traces of maintenance on the exterior facade of ordinary buildings, and small amounts of stockpiled materials;

[0034] The non-construction exclusion set includes at least one of the following: ordinary roads, green spaces, private vehicles, electric vehicles, ordinary pedestrians, and everyday building environments;

[0035] When the image to be identified contains only categories not included in the construction exclusion set, or only text, signs, project names, or monitoring titles, it is determined that there is no construction evidence.

[0036] S2.4: Calculate the percentage of the image area covered by the bounding boxes of the evidence items;

[0037] S2.5: Based on the number of statistically strong construction evidence items and the proportion of image areas covered by the evidence item annotation boxes, perform a first judgment process on each image to be identified to obtain a set of images to be identified that pass the first judgment. The specific process is as follows:

[0038] When the number of strong construction evidence items corresponding to any image to be identified is greater than or equal to 1 and the proportion of the image area to be identified is greater than zero, the image to be identified is determined to pass the first judgment; otherwise, it is determined to fail the first judgment condition.

[0039] If there is strong evidence of construction and subsequent steps S3 to S8 are triggered; otherwise, the analysis is terminated and subsequent steps are not executed.

[0040] The beneficial effects of this invention are as follows:

[0041] (1) Before the hazard analysis, the present invention sets a construction visual evidence gating step, extracts construction visual evidence items through a visual language model, and matches the evidence items with a preset set of strong construction evidence, weak construction clues, and non-construction exclusion set. Based on the number and spatial distribution of the evidence items, the image is judged as strong construction evidence, weak construction clues, or no construction evidence. When the image does not have clear construction visual evidence, the hazard analysis is stopped, thereby reducing false alarms in non-construction scenarios from the source.

[0042] (2) This invention breaks down the identification of construction safety hazards into eight steps: target extraction, risk candidate screening, candidate-level evidence judgment, structured hazard generation, and subtraction verification. Each step satisfies a progressively narrowing constraint relationship, rather than generating the final conclusion all at once by a visual language model. The output of each step serves only as a subset of the input for the next specific step, forming an auditable processing chain.

[0043] (3) In the S5 candidate-level image evidence determination step of the present invention, the visual language model only observes and judges the current candidate annotation box area, and verifies one by one the five conditions: the existence of the target object, the abnormal state is clearly visible, the dangerous spatial position relationship is clear, the lack of protection or management measures is visible and does not belong to the reasonable construction state; the determination is based on the visual evidence that can be directly observed in the image, and does not use the confidence value output by the model as the determination condition.

[0044] (4) The S7 subtraction verification step of the present invention only performs deletion, downgrading or retention operations, without adding new hidden danger records. It is used to filter results with weak image evidence, blurred targets, inference based solely on text, misjudgment of normal construction status and inconsistent field sources. For hidden dangers involving the edge, a special inspection of the clarity, stability and visibility of the danger boundary is also performed to make the final report more stable and verifiable.

[0045] (5) This invention outputs multiple hazards in the form of a hazard set. Each hazard includes a hazard category, a normalized label box, a risk level, a location description, on-site phenomena, a consequence description, and a rectification suggestion field. The on-site phenomena field is directly derived from the visible evidence output of S5 and is used to describe the hazard phenomena that can be directly observed in the image, so that safety management personnel can trace the visual basis of each judgment in the image.

[0046] (6) The present invention simultaneously outputs structured JSON results, boxed annotation diagrams and rectification suggestions, enabling the identification results to enter the manual review, rectification order assignment, statistical analysis and closed-loop management process. Attached Figure Description

[0047] Figure 1 A flowchart for a multi-stage identification and structured output method for safety hazards at construction sites;

[0048] Figure 2 Example diagram illustrating the results with boxed annotations and the field descriptions of on-site phenomena. Detailed Implementation

[0049] Specific Implementation Method 1: A method for multi-stage identification and structured output of construction site safety hazards based on a visual language model according to the present invention includes:

[0050] S1: Obtain the set of images to be recognized;

[0051] S2: Based on the first visual language model, perform construction visual evidence extraction processing on the image set to be identified to obtain N construction visual evidence items;

[0052] Based on N construction visual evidence items, the first judgment process is performed on the image set to be identified to obtain the image set to be identified that passes the first judgment, and then proceeds to S3;

[0053] When no image to be identified passes the first judgment process, the set of images to be identified in S1 is determined to be in the case of no strong construction, and subsequent analysis is stopped;

[0054] S3: Input the image to be identified through S2 gate, N construction visual evidence items and a preset construction object vocabulary into the second visual language model for construction target extraction processing, and obtain A construction area recognition results and M target recognition results;

[0055] S4: Based on the preset construction safety risk rule base, rule matching processing is performed on the identification results of each target to be detected obtained in S3 to obtain an initial risk candidate set;

[0056] Based on the third visual language model with preset construction status filtering rules, the initial risk candidate set is filtered to obtain the key risk candidate set.

[0057] S5: Based on the fourth visual language model, image evidence judgment processing is performed on the key risk candidate set to obtain K risks that are judged to be valid;

[0058] S6: Based on the K risks that meet the criteria, generate K structured hazard records;

[0059] S7: Based on the fifth visual language model, the K structured hazard records obtained in S6 are reviewed to obtain a set of qualified structured hazard records;

[0060] S8: Based on the verified and qualified structured hazard record set, output the final hazard set, structured JSON results, boxed annotation diagram, and text report.

[0061] Furthermore, the structured JSON result of this invention includes all fields of each potential hazard record, facilitating data exchange between systems;

[0062] The framed annotation diagram of the present invention is a normalized annotation box and its corresponding hazard category label for each hazard record retained in S7 superimposed on the original image;

[0063] The written report lists the on-site phenomena, location, risk level, consequences, and rectification suggestions for each hazard, grouped by hazard category, for safety management personnel to read and use directly.

[0064] Specific Implementation Method Two: The difference between this implementation method and Specific Implementation Method One is that:

[0065] The images to be identified in S1 include: screenshots of construction site monitoring, mobile phone inspection photos, drone images or other fixed images; the image source can be real-time monitoring video frames transmitted over the network, or on-site photos uploaded by inspection personnel through mobile terminals;

[0066] In step S2, the visual language model is used to extract construction visual evidence from the images to be identified, resulting in N construction visual evidence items. Based on these N construction visual evidence items, a first judgment process is performed on the set of images to be identified to obtain a set of images that pass the first judgment. The specific process is as follows:

[0067] S2.1: Input natural language prompts containing the construction scene evidence classification system into the first visual language model.

[0068] The specific method for the visual language model to extract visual evidence items of construction is as follows: input prompt words containing the classification system of construction scene evidence into the visual language model, and require the model to traverse the image content and output the category, position coordinates in the image, and visibility description of each identifiable object or scene element.

[0069] Furthermore, the first visual language model selected in this invention can be a multimodal large model deployed in the cloud (such as GPT-4V, Gemini Pro Vision, etc.), or a locally deployed open-source visual language model, or a multimodal inference model composed of an image encoder and a large language model.

[0070] The natural language prompts list complete definitions for the following sets of strong construction evidence (foundation pits, trenches, excavation work surfaces, scaffolding, formwork supports, edge openings, construction machinery, hoisting operations, pouring operations, rebar processing, welding and cutting operations, construction area workers, temporary construction power facilities, and work surfaces within enclosures), weak construction clues (scattered bricks, pipes, bagged materials, construction waste, ordinary building entrances, ordinary building facade maintenance traces, and small amounts of stockpiled materials), and non-construction exclusions (ordinary roads, green spaces, private vehicles, electric vehicles, ordinary pedestrians, and the everyday building environment).

[0071] S2.2: Input the task description text into the first visual language model: "It is required to traverse the image content and output the following for each identifiable object or scene element: evidence category, normalized bounding box coordinates in the image, visible state description text, and whether the recognition result comes only from the text content in the image"; (i.e. whether the recognition is based solely on text information such as sign text, project name, and monitoring title, rather than on the visual features of the object itself).

[0072] Furthermore, the first visual language model outputs N construction visual evidence items, each of which includes: evidence category, normalized bounding box, and visibility state feature; wherein, the i-th construction visual evidence item output by the first visual language model is denoted as... Includes evidence categories Normalized x-coordinate and normalized y-coordinate of the annotation box Visible state characteristics This can be expressed as a formula:

[0073] ;

[0074] Then, for each piece of evidence, the evidence category is matched with the set of strong construction evidence, the set of weak construction clues, and the set of non-construction exclusions using string or semantic matching, respectively. The number of strong construction evidence items is then counted. And the number of weak construction clues And calculate the percentage r of the image area covered by all evidence item annotation boxes.

[0075] The decision logic is: when When ≥1 and r>0, it is considered strong construction evidence; when =0 and When ≥1, it is judged as a weak construction clue; when =0 and If the value is 0, or all evidence items only match the non-construction exclusion set, or only text, signs, project names, and monitoring titles exist without any physical construction evidence, it is determined that there is no construction evidence. Only when it is determined to be strong construction evidence will the analysis continue flag be set to true, triggering subsequent steps S3 to S8; otherwise, the analysis continue flag will be set to false, terminating the entire analysis process.

[0076] S2.3: Match the evidence categories in the N construction visual evidence items with the preset strong construction evidence set, weak construction clue set, and non-construction exclusion set, and count the number of strong construction evidence items and the number of weak construction clue items.

[0077] The set of strong construction evidence includes at least one of the following: foundation pit, trench, excavation working face, scaffolding, formwork support, edge openings, construction machinery, hoisting operations, pouring operations, steel bar processing, welding and cutting operations, construction area workers, temporary construction power facilities, and working face within the enclosure;

[0078] The set of weak construction clues includes at least one of the following: scattered bricks, pipes, bags, construction waste, ordinary building entrances, traces of maintenance on the exterior facade of ordinary buildings, and small amounts of stockpiled materials;

[0079] The non-construction exclusion set includes at least one of the following: ordinary roads, green spaces, private vehicles, electric vehicles, ordinary pedestrians, and everyday building environments;

[0080] When the image to be identified contains only categories not included in the construction exclusion set, or only text, signs, project names, or monitoring titles, it is determined that there is no construction evidence.

[0081] S2.4: Calculate the percentage of the image area covered by the bounding boxes of the evidence items;

[0082] S2.5: Based on the number of statistically strong construction evidence items and the proportion of image areas covered by the evidence item annotation boxes, perform a first judgment process on each image to be identified to obtain a set of images to be identified that pass the first judgment. The specific process is as follows:

[0083] When the number of strong construction evidence items corresponding to any image to be identified is greater than or equal to 1 and the proportion of the image area is greater than zero, the image to be identified is determined to pass the first judgment; otherwise, it is determined to fail the first judgment condition.

[0084] If there is strong evidence of construction, subsequent steps S3 to S8 will be triggered; otherwise, the analysis will be terminated and subsequent steps will not be executed; other steps and parameters are the same as in Specific Implementation Method 1.

[0085] Specific Implementation Method Three: This implementation method differs from Specific Implementation Methods One and Two in that: the specific process in S3 of inputting the image to be identified, N construction visual evidence items, and a preset construction object vocabulary into the second visual language model for construction target extraction processing, and obtaining A construction area identification results and M target identification results, is as follows:

[0086] S3.1: The image to be identified, which is gated by S2, N construction visual evidence items, and a preset construction object vocabulary are used as joint inputs and loaded into the second visual language model;

[0087] The second visual language model has a built-in vocabulary of construction objects and region-target level prompt word templates;

[0088] S3.2: Input the text describing the construction area recognition task into the second visual language model;

[0089] Preferably, the description text for the construction area identification task of the present invention is "requiring the identification of construction-related areas (such as foundation pit area, scaffolding area, material storage area, etc.) in the image";

[0090] The second visual language model outputs A recognition results for construction areas.

[0091] Each construction area identification result includes: area number, area type, and label box; the identification result of the j-th construction area is expressed by the formula:

[0092] ;

[0093] In the formula, j represents the region number. Indicates the region type. Indicates the x-coordinate of the center of the region label box. Indicates the ordinate of the center of the region label box;

[0094] S3.3: Input the target recognition task description text into the second visual language model;

[0095] Preferably, the target recognition task description text of the present invention is as follows: "In each region, enumerate the targets to be detected belonging to the vocabulary of construction objects, and output a description of the spatial positional relationship between each target and its region and other targets. The targets to be detected are limited to the types of personnel, machinery and equipment, edges, openings, scaffolding, formwork supports, working surfaces, temporary power facilities, material storage areas, passages, hoisting areas, and construction waste, and delete ordinary pedestrians, social vehicles, electric vehicles, greening and ordinary road objects in non-construction related areas, and normalize all the coordinates of the annotation boxes to integer coordinates from 0 to 1000." The second visual language model outputs M target recognition results.

[0096] Each target identification result includes: the target category, normalized bounding box, visible state features, spatial position relationship description text (such as "Target A is located directly above Target B", "Target C is less than 1 meter away from adjacent target D"), and the construction area number to which it belongs.

[0097] The recognition result of the m-th target to be detected is expressed by the formula:

[0098] ;

[0099] In the formula, Indicates the target category. This represents the x-coordinate of the center of the target's normalized bounding box. The ordinate represents the center of the target's normalized annotation box; F represents the target's visibility status feature; P represents the spatial location relationship text; and j represents the construction area number to which the target belongs.

[0100] Furthermore, the target categories are limited to: personnel, machinery and equipment, edges, openings, scaffolding, formwork supports, working surfaces, temporary power facilities, material storage areas, passageways, hoisting areas, and types of construction waste;

[0101] Furthermore, the target visibility state features include:

[0102] Visible characteristics of the "edge" or "opening" include: "complete protection", "no protection" or "incomplete protection".

[0103] The visible status characteristics of "mechanical equipment" include: "not in operation" and "in operation";

[0104] Visible status characteristics of "temporary power facilities" include: "cables are normal", "cables are dragging on the ground", "cables are exposed" or "no leakage protection".

[0105] Visible characteristics of the "scaffolding" or "working surface" include: "with guardrails", "with safety nets", "without guardrails" or "without safety nets".

[0106] Visible characteristics of scaffolding include: "scaffolding is normal", "wall ties are missing", "scaffold boards are not fully laid" or "scaffolding is tilted".

[0107] Visible characteristics of the "material storage area" or "construction waste" include: "unoccupied passageway", "occupied passageway", "covered with dust netting", or "uncovered with dust netting".

[0108] The visible status characteristics of the "personnel" include: "wearing a safety helmet", "not wearing a safety helmet", "wearing a reflective vest", "not wearing a reflective vest", "not in a danger zone", or "in a danger zone". Other steps and parameters are the same as in one of the specific implementation methods one or two.

[0109] Specific Implementation Method Four: This implementation method differs from Specific Implementation Methods One to Three in that, in that,

[0110] The pre-set construction safety risk rule base in S4 includes: edge rules, machinery rules, temporary power rules, high-altitude operation rules, scaffolding rules, civilized construction rules, material stacking rules, and worker rules;

[0111] Each risk rule predefines matching conditions for target category, visible status, and spatial relationship;

[0112] Furthermore, in S4, based on the preset construction safety risk rule base, rule matching processing is performed on the identification results of each target to be detected obtained in S3 to obtain an initial risk candidate set;

[0113] The specific process of performing rule matching on the identification result of a target to be detected, based on a pre-set construction safety risk rule base, is as follows: rule matching is performed on the target category, bounding box position, visibility features, and spatial positional relationships in the identification result of the target to be detected.

[0114] The specific method of rule matching is as follows: compare the target category with the triggering conditions of eight rules, and compare the spatial relationship with the predefined dangerous space patterns in the rules (e.g., whether the spatial distance between the personnel target and the adjacent target is less than a preset threshold, whether the temporary power cable target has spatial overlap with the water accumulation area target). When the matching condition of any rule is met, the target to be inspected and its associated spatial relationship are taken as an initial risk candidate.

[0115] Furthermore, a pre-built construction safety risk rule base includes the following eight categories of rules, each with its own specific matching conditions:

[0116] Edge Protection Rule: When the target to be inspected contains the categories of "edge" or "opening", and its visible status is "no protection" or "incomplete protection", and the spatial relationship shows that there are "personnel" targets around the edge target, this rule is matched to generate a candidate with the risk direction of "lack of edge protection".

[0117] Mechanical rule: When the target to be inspected contains the category of "mechanical equipment", its visible status is "operating", and the spatial relationship shows that the distance between the mechanical target and the "personnel" target is less than the preset safe distance threshold, this rule is matched;

[0118] Temporary power supply rule: If the target to be inspected includes the category of "temporary power supply facilities" and its visible status is "cables dragging on the ground", "cables exposed" or "no leakage protection", this rule applies;

[0119] High-altitude operation rule: When the target to be inspected contains the category of "scaffolding" or "work surface", and the spatial relationship shows that its height is greater than the preset threshold, and its visibility status is "no guardrail" or "no safety net", this rule is matched;

[0120] External scaffolding rule: When the target to be inspected contains the "scaffolding" category, and its visible status is "missing wall ties", "scaffold boards not fully laid" or "scaffolding tilted", this rule is matched;

[0121] Civilized construction rules: When the target to be inspected contains the category of "material storage area" or "construction waste", and its visibility status is "obstructing passage" or "without dust net", this rule applies;

[0122] Material stacking rule: When the target to be inspected contains the "material stacking area" category, and the spatial relationship shows that it is located at the edge of the edge or opening, this rule is matched;

[0123] Worker rule: When the target to be inspected contains the category of "personnel" and its visible status is "not wearing a safety helmet", "not wearing a reflective vest" or "in a dangerous area", this rule is matched.

[0124] A third-party visual language model based on pre-defined construction status filtering rules is used to filter the initial risk candidate set to obtain a key risk candidate set; the construction status filtering rules are as follows:

[0125] Candidates falling under the following categories will be removed: those under excavation with unstable boundaries, those with a small amount of temporarily stored materials, those with a small amount of standing water, those with localized mud and water, and those with blurred figures in the distance.

[0126] The specific method for filtering construction status is as follows: The third-party visual language model performs a second observation of the labeled area of ​​each initial risk candidate to determine whether the status involved in the candidate belongs to a normal, reasonable, or unavoidable temporary status in the construction process. Initial risk candidates that belong to normal, reasonable, or unavoidable temporary status in the construction process are deleted, thereby obtaining B key risk candidates to form a key risk candidate set, and the reasons for retention are output.

[0127] In summary, the detection system based on the method of this invention iterates through each target output by S3, comparing its target category, visible state features, and spatial position relationship with the matching conditions of the eight rules mentioned above. An initial risk candidate is generated when any rule is satisfied. Then, a secondary filtering is performed using reasonable construction state filtering rules: the bounding box area of ​​each initial risk candidate is observed by a visual language model to determine whether it belongs to one of the following reasonable construction states—an excavated pit area with unstable boundaries, a temporary stockpile of a small amount of construction materials, unavoidable small amounts of water accumulation or localized mud during construction, or a distance too great for clear identification of personnel features. Candidates falling under these conditions are deleted. The remaining candidates after filtering are the B key risk candidates, forming a key risk candidate set. Each key risk candidate includes a candidate number, risk direction, bounding box coordinates, and retention reason. Other steps and parameters are the same as in one of the specific implementation methods one to three.

[0128] Specific Implementation Method Five: The difference between this implementation method and Specific Implementation Methods One to Four is that:

[0129] In step S5, based on the fourth visual language model, image evidence judgment processing is performed on the key risk candidate set to obtain K risks that are deemed valid. The specific process is as follows:

[0130] S5.1: Construct the evidence input set for each key risk candidate;

[0131] Among them, the evidence determination input set for the b-th key risk candidate This includes: the image to be identified corresponding to the b-th key risk candidate, the coordinates of the bounding box of the b-th key risk candidate, the risk category of the b-th key risk candidate, and the reason for retaining the b-th key risk candidate;

[0132] Expressed as a formula:

[0133] ;

[0134] In the formula, This represents the image to be identified corresponding to the b-th key risk candidate. This indicates the x-coordinate of the center of the bounding box for the b-th key risk candidate. This indicates the ordinate of the center of the bounding box for the b-th key risk candidate. This represents the risk category of the b-th key risk candidate. This indicates the reason for retaining the b-th key risk candidate;

[0135] Furthermore, the reasons for retaining key risk candidates in this invention are obtained from the output of the S4 third visual language model; the risk category is obtained from S4, and whichever matching rule is satisfied determines the corresponding risk category. The risk categories include: edge risk, mechanical risk, temporary power supply risk, high-altitude operation risk, scaffolding risk, civilized construction risk, material stacking risk, and worker risk.

[0136] S5.2: Input the evidence judgment input set for each key risk candidate into the fourth visual language model for visible visual evidence judgment processing to obtain K risks with valid judgments; the specific process is as follows:

[0137] S5.2.1: Input the evidence judgment input set of the b-th key risk candidate into the fourth visual language model, perform target object existence verification judgment processing, and obtain the target object existence verification judgment result of the b-th key risk candidate;

[0138] S5.2.2: Input the evidence judgment input set of the b-th key risk candidate into the fourth visual language model, perform the verification judgment process of the abnormal state being clearly visible in the image, and obtain the verification judgment result of the abnormal state of the b-th key risk candidate being clearly visible in the image.

[0139] S5.2.3: Input the evidence judgment input set of the b-th key risk candidate into the fourth visual language model, perform the verification and judgment process of the dangerous spatial location relationship, and obtain the verification and judgment result of the dangerous spatial location relationship of the b-th key risk candidate;

[0140] S5.2.4: Input the evidence judgment input set of the b-th key risk candidate into the fourth visual language model, perform the verification judgment of the lack of protective or management measures, and obtain the verification judgment result of the lack of protective or management measures of the b-th key risk candidate.

[0141] S5.2.5: Input the evidence judgment input set of the b-th key risk candidate into the fourth visual language model, perform the verification judgment process for not belonging to the reasonable construction state, and obtain the verification judgment result of the b-th key risk candidate not belonging to the reasonable construction state.

[0142] S5.2.6: Based on the verification results of the existence of the target object of the b-th key risk candidate, the verification results of the abnormal state being clearly visible in the image, the verification results of the clear spatial relationship of the danger, the verification results of the visible absence of protection or management measures, and the verification results of the unreasonable construction state, the b-th key risk candidate is finally judged.

[0143] When all verification results are valid, the b-th key risk candidate is determined to be a valid risk.

[0144] If any verification result is invalid, the b-th key risk candidate is determined to be a risk that is invalid.

[0145] S5.2.7: Repeat S5.2.1 to S5.2.6 until all key risk candidates have been finally determined, resulting in K risks that have been determined to be valid;

[0146] Furthermore, the fourth visual language model is configured to observe and judge only within the current candidate bounding box region and the corresponding risk category, without adding any target objects or risk categories outside of this candidate region. The prompts require the visual language model to observe only the image region defined by the bounding box coordinates, to judge only within this risk category, and not to pay attention to content outside this region or raise new potential hazards outside of this risk category.

[0147] Furthermore, the fourth visual language model of the present invention verifies whether the following five conditions are simultaneously met in the image: the target object exists (the candidate construction object is indeed visible within the marked area), the abnormal state is clearly visible in the image (abnormal states such as non-standard, damaged, and missing can be directly observed from the image pixels, rather than inferred from text or common sense), the dangerous spatial positional relationship is clear (the dangerous distance or positional relationship between objects can be directly identified in the image), the lack of protective or management measures is visible (the absence of protective facilities that should exist can be directly observed in the image), and it does not belong to a reasonable construction state (normal construction situations excluding those covered by the S4 reasonable construction state filter).

[0148] If all five conditions are met, the candidate is deemed valid; if there is clear counter-evidence (such as an image clearly showing that the protective measures are intact and the danger has been eliminated), it is deemed invalid; if the target is blurry, obscured, too far away to identify key details, or key parts are not in the frame, or the possibility of reasonable construction conditions cannot be ruled out, it is deemed insufficient evidence.

[0149] Furthermore, for candidates whose hazard assessment is established, the fourth visual language model of the present invention outputs a visible evidence field. This field only records visual evidence content that can be directly observed in the image and supports the hazard assessment (e.g., "No guardrails are seen at the edge of the foundation pit, the boundary is clear, and there are people standing 1 meter from the edge" and "No guardrails are seen within about 3 meters of the eastern edge of the foundation pit, the boundary line is clear, and two construction workers can be seen standing at the edge"). It does not contain speculative language such as "there may be a risk of falling".

[0150] The other steps and parameters are the same as those in one of the specific implementation methods one to four.

[0151] Specific Implementation Method Six: The difference between this implementation method and Specific Implementation Methods One through Five is that:

[0152] Each structured hazard record in S6 includes: hazard category field, normalized annotation box field, risk level field, location description field, on-site phenomenon field, consequence description field, rectification suggestion field, and case reference field;

[0153] Furthermore, the structured generation of potential hazards can be achieved by configuring structured output prompts using a visual language model, or by filling in preset templates. The core requirement is that only candidates that S5 determines as "valid" are processed, and the source of each field can be traced back to the specific output of the aforementioned steps.

[0154] The source of each field follows the following tracing constraints:

[0155] The hazard category field is obtained from the risk type field of the candidate in S4 through a preset category mapping table. For example, "lack of edge protection" is mapped to "edge protection", and "cable dragging on the ground" is mapped to "temporary power supply".

[0156] The normalized annotation box field prioritizes using the annotation box coordinates of the candidate in S4; if the candidate involves multiple targets to be inspected (such as the combined hidden danger of personnel + adjacent edge), the annotation boxes of the corresponding targets to be inspected in S3 are combined into a single annotation box.

[0157] The risk level field is generated by querying a preset risk consequence level rule table based on the hazard category. The risk consequence level rule table predefines the default risk level for various types of hazards. For example, hazards related to edge protection and temporary power supply are high-risk by default, while hazards related to civilized construction are medium-risk by default.

[0158] The location description field is generated by combining the region type description of the construction-related area in S3 and the orientation description of the candidate area in the image in S4; for example, the region type of the construction-related area to which the candidate belongs in S3 (such as "excavation pit area") and the orientation description of the candidate annotation box in the image in S4 (such as "center right of image") are concatenated to generate the location description field "excavation pit area (center right of image)".

[0159] The field of on-site phenomena is directly filled by the content of the visible evidence field corresponding to the candidate in S5;

[0160] Consequence Description Field: The hazard structure generation module retrieves the corresponding consequence description text from a preset consequence description library based on the hazard category. The consequence description library is a key-value pair structure, where the key is the hazard category and the value is a description of the accident consequences that the hazard of that category may cause.

[0161] The rectification suggestion field: The hazard structure generation module retrieves the corresponding rectification measure text from the preset rectification suggestion library according to the hazard category;

[0162] Case reference fields: The hazard structure generation module uses the hazard category and on-site phenomenon fields as query conditions to perform similarity matching in the historical hazard case database. When a match is successful, it returns the number and summary of the most similar case. If no matching case is found, "None" is entered.

[0163] The structured hazard record fields are shown in the table below:

[0164]

[0165] The other steps and parameters are the same as those in any of the specific implementation methods one to five.

[0166] Specific Implementation Method Seven: The difference between this implementation method and Specific Implementation Methods One through Six is ​​that:

[0167] In step S7, the K structured hazard records obtained in step S6 are reviewed based on the fifth visual language model to obtain a set of qualified structured hazard records. The specific process is as follows:

[0168] S7.1: Construct the input set for each structured hazard record; where the input set for the k-th structured hazard record includes:

[0169] The kth structured hazard record, the original image to be identified corresponding to the kth structured hazard record, and the evidence determination result corresponding to the kth structured hazard record;

[0170] S7.2: Input the structured hazard record set into the fifth visual language model for review, and obtain a qualified structured hazard record set; the specific process is as follows:

[0171] S7.2.1: Input the input set of the kth structured hidden danger record into the fifth visual language model to perform a verification process to check whether the image evidence is sufficient (based on whether the content of the visible evidence field in S5 is based on the visual information that can be identified in the image).

[0172] If the review result indicates that the image evidence is sufficient, proceed to S7.2.2; otherwise, proceed to S7.2.8.

[0173] S7.2.2: Input the input set of the kth structured hazard record into the fifth visual language model to perform a verification process on whether the target is clear and unobstructed (based on the degree of recognizability of the target in the image).

[0174] If the verification result shows that the target is cleaned without obstruction, proceed to S7.2.3; otherwise, proceed to S7.2.8.

[0175] S7.2.3: Input the input set of the kth structured hazard record into the fifth visual language model to verify whether the key areas are fully displayed in the image.

[0176] If the verification result shows that the key area is fully displayed in the image, proceed to S7.2.4; otherwise, proceed to S7.2.8.

[0177] S7.2.4: Input the input set of the kth structured hazard record into the fifth visual language model to verify whether it belongs to a normal construction state and has been misjudged.

[0178] If the review result indicates that the construction status is not normal and has been mistakenly judged, proceed to S7.2.5; otherwise, proceed to S7.2.8.

[0179] S7.2.5: Input the input set of the kth structured hazard record into the fifth visual language model to verify whether the position of the annotation box matches the hazard description.

[0180] If the verification result shows that the position of the marked box is consistent with the description of the hidden danger, proceed to S7.2.6; otherwise, proceed to S7.2.8.

[0181] S7.2.6: Input the input set of the kth structured hazard record into the fifth visual language model to verify whether the source of the hazard record's fields is consistent.

[0182] If the review result shows that the fields of the hidden danger record are consistent, proceed to S7.2.7; otherwise, proceed to S7.2.8.

[0183] S7.2.7: Input the input set of the kth structured hazard record into the fifth visual language model to perform a verification process to determine whether the edge-specific evidence is qualified.

[0184] When the review result shows that the evidence for the edge-specific project is qualified, the kth structured hidden danger is judged to be qualified for review and proceeds to S7.2.9;

[0185] Otherwise, delete the kth structured hazard record and proceed to S7.2.8;

[0186] The specific process for reviewing the eligibility of evidence from marginal projects is as follows:

[0187] S7.2.6.1: Identify whether the kth structured hazard record involves edges, pits, trenches, openings, or the edges of high-altitude operations.

[0188] When the kth structured hazard record does not involve edges, foundation pits, trenches, openings, or the edge of high-altitude operations, the review result is that the evidence of high-altitude operations is qualified;

[0189] When the kth structured hazard record does not involve edges, pits, trenches, openings, or the edges of high-altitude operations, further determine whether the hazard boundary is clearly visible in the image, whether the boundary is stable and accessible, and whether the protective damage is clearly visible in the image.

[0190] When the dangerous boundary is clearly visible in the image, the boundary is unstable and accessible, and the damage to the protective barrier is clearly visible in the image, the verification result is that the evidence for the operation near the airspace is qualified.

[0191] If any one of the following three conditions is not met: whether the dangerous boundary is clearly present in the image, whether the boundary is unstable and accessible, or whether the protective damage is clearly visible in the image, the review result is that the evidence for the operation in the airspace is unqualified.

[0192] S7.2.8: Determine the kth structured hazard as unqualified for review, delete the kth structured hazard record, and proceed to S7.2.9;

[0193] S7.2.9: Repeat steps S7.2.1 to S7.2.8 for the (k+1)th structured hazard record; when all K structured hazard records have been reviewed, a set of qualified structured hazard records is obtained;

[0194] If none of the retention conditions are met, the final review module will reject all entries and output an empty set of potential hazards.

[0195] The other steps and parameters are the same as those in any of the specific implementation methods one to six.

[0196] Specific Implementation Method Eight: The difference between this implementation method and Specific Implementation Methods One through Seven is that:

[0197] The first, second, third, fourth, and fifth visual language models are all visual language models based on the Transformer architecture.

[0198] Furthermore, the optional visual language model can be a cloud-deployed multimodal large model (such as GPT-4V, Gemini Pro Vision, etc.), a locally deployed open-source visual language model, or a multimodal inference model composed of an image encoder and a large language model.

[0199] The other steps and parameters are the same as those in any of the specific implementation methods one to seven.

[0200] Specific Implementation Method Nine: This implementation method describes a construction site safety hazard identification system based on a visual language model. The system includes:

[0201] (1) Image input module: used to receive the image to be recognized and complete format reading, size adaptation and image encoding. It supports common image formats such as JPEG, PNG, BMP and supports extracting images frame by frame from the monitoring video stream;

[0202] (2) Construction Evidence Gating Module: Built-in visual language model calling interface and evidence classification prompt word template, used to execute the S2 construction visual evidence gating steps, output the judgment result of strong construction evidence / weak construction clues / no construction evidence and analysis continuation mark;

[0203] (3) Construction target extraction module: It has a built-in construction object vocabulary and area-target level prompt word template, which is used to execute the S3 construction target extraction step and output the construction-related area list and the target list to be inspected;

[0204] (4) Risk candidate screening module: It has eight built-in construction safety risk rules and reasonable construction status filtering rule set, which are used to execute the S4 key risk candidate screening steps, output the key risk candidate list and the risk direction and retention reason of each candidate;

[0205] (5) Candidate Evidence Judgment Module: Built-in candidate-level evidence judgment prompt word template, used to execute the S5 candidate-level image evidence judgment step, outputting a three-value judgment result of being true / false / insufficient evidence and visible evidence fields for each candidate;

[0206] (6) Hazard structure generation module: It has a built-in category mapping table, risk consequence level rule table, consequence description library, rectification suggestion library and case matching engine, which are used to execute the S6 hazard structure generation steps and output a list of structured hazard records.

[0207] (7) Final review module: Built-in review inspection rule set and edge special inspection rule set, used to execute the S7 subtraction review step and output the final hidden danger record list after review;

[0208] (8) Result output module: used to execute the S8 result output step, generate structured JSON file, boxed annotation diagram and text report.

[0209] Specific Embodiment Ten: This embodiment describes a computer-readable storage medium storing at least one instruction, which is loaded and executed by a processor to implement the method described in any one of Specific Embodiments One to Three. The computer storage medium includes, but is not limited to, magnetic storage media, optical storage media, magneto-optical storage media, read-only memory (ROM), random access memory (RAM), erasable programmable memory (EPROM and EEPROM), and flash memory.

[0210] Specific Implementation Method Eleven: This implementation method describes a multi-stage identification and structured output device for safety hazards at construction sites based on a visual language model. The device includes a processor and a memory. The memory stores at least one instruction, which is loaded and executed by the processor to implement the method described in any one of Specific Implementation Methods One to Three. This device can be an edge computing device, a remote server, or a cloud computing platform deployed at a construction site.

[0211] In optional implementations, the visual language model can be a cloud-based multimodal large model, a locally deployed visual language model, or a multimodal inference model composed of an image encoder and a language model. This invention does not limit the specific model structure, only requiring that it be able to output structured text results based on the input image and natural language prompts.

[0212] In an optional implementation, the eight categories of rules in the construction safety risk rule base can be expanded or replaced according to the project type, regional safety regulations, or enterprise safety inspection standards. The hazard categories can also be expanded to include fire lanes, dust control, deep foundation pit monitoring, hoisting and lifting, confined space operations, and other categories.

[0213] Test results

[0214] This invention was tested and verified on 100 construction image samples. The test samples covered various construction stages, including foundation pit construction, main structure construction, and decoration and finishing construction, as well as various types of potential hazards such as lack of edge protection, non-standard temporary power supply, illegal stacking of materials, and personnel not wearing safety protective equipment. The system achieved high image-level recognition accuracy on 82 images with clear manual annotations, and demonstrated good structured output capabilities in hazard-level statistics: the accuracy of hazard category classification, annotation box positioning accuracy, and accuracy of on-site phenomenon field description all reached practical levels. The test results verify the practical feasibility of this solution.

[0215] Experiment Example 1: Identification of Hidden Dangers from Construction Site Photos

[0216] The system receives a construction site image containing two types of hazards: temporary power cables dragging on the ground and missing edge protection. The S2 construction evidence gating module identifies strong construction evidence items in the image, such as construction machinery, workers, and edge areas, and determines them as strong construction evidence, triggering subsequent analysis. The S3 target extraction module identifies two construction-related areas: the foundation pit area and the material storage area, as well as personnel, edge areas, temporary power facilities, and machinery as pending inspection targets. The S4 risk candidate screening module matches edge protection rules and temporary power rules, generating two key risk candidates. The S5 candidate evidence judgment module verifies five conditions for each of the two candidates, determining that both are valid and outputting visible evidence descriptions for the dragging cables and missing guardrails, respectively. The S6 hazard structure generation module generates two structured hazard records. The S7 subtraction verification module confirms that the two records are sufficient evidence and have consistent fields, and retains them. The S8 output includes a structured result containing the two hazard records, a labeled flowchart, and a rectification suggestion report. The above description only illustrates preferred embodiments of the present invention. It should be understood that the present invention is not limited to the specific embodiments described above. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the disclosed technical content without departing from the scope of the present invention, forming equivalent embodiments with equivalent changes. Any simple modifications, equivalent substitutions, and improvements made to the above embodiments without departing from the scope of the present invention, based on the technical essence of the present invention and within the spirit and principles of the present invention, shall still fall within the protection scope of the present invention.

Claims

1. A method for multi-stage identification and structured output of safety hazards at construction sites based on a visual language model, characterized in that, include: S1: Obtain the set of images to be recognized; S2: Based on the first visual language model, perform construction visual evidence extraction processing on the image set to be identified to obtain N construction visual evidence items; Based on N construction visual evidence items, the first judgment process is performed on the image set to be identified to obtain the image set to be identified that passes the first judgment, and then proceeds to S3; When no image to be identified passes the first judgment process, the set of images to be identified in S1 is determined to be in the case of no strong construction, and subsequent analysis is stopped; S3: Input the image to be identified through S2 gate, N construction visual evidence items and a preset construction object vocabulary into the second visual language model for construction target extraction processing, and obtain A construction area recognition results and M target recognition results; S4: Based on the preset construction safety risk rule base, rule matching processing is performed on the identification results of each target to be detected obtained in S3 to obtain an initial risk candidate set; Based on the third visual language model with preset construction status filtering rules, the initial risk candidate set is filtered to obtain the key risk candidate set. S5: Based on the fourth visual language model, image evidence judgment processing is performed on the key risk candidate set to obtain K risks that are judged to be valid; S6: Based on the K risks that meet the criteria, generate K structured hazard records; S7: Based on the fifth visual language model, the K structured hazard records obtained in S6 are reviewed to obtain a set of qualified structured hazard records; S8: Based on the verified and qualified structured hazard record set, output the final hazard set, structured JSON results, boxed annotation diagram, and text report.

2. The method for multi-stage identification and structured output of construction site safety hazards based on a visual language model according to claim 1, characterized in that, The images to be identified in S1 include: screenshots of construction site monitoring, mobile phone inspection photos, drone images or other fixed images; In step S2, the visual language model is used to extract construction visual evidence from the images to be identified, resulting in N construction visual evidence items. Based on these N construction visual evidence items, a first judgment process is performed on the set of images to be identified to obtain a set of images that pass the first judgment. The specific process is as follows: S2.1: Input natural language prompts containing the construction scene evidence classification system into the first visual language model. S2.2: Input the task description text into the first visual language model; The first visual language model outputs N construction visual evidence items, each of which includes: evidence category, normalized bounding box, and visibility state features; where the i-th construction visual evidence item output by the first visual language model is denoted as... Includes evidence categories Normalized x-coordinate and normalized y-coordinate of the annotation box Visible state characteristics , S2.3: Match the evidence categories in the N construction visual evidence items with the preset strong construction evidence set, weak construction clue set, and non-construction exclusion set, and count the number of strong construction evidence items and the number of weak construction clue items. The set of strong construction evidence includes at least one of the following: foundation pit, trench, excavation working face, scaffolding, formwork support, edge openings, construction machinery, hoisting operations, pouring operations, steel bar processing, welding and cutting operations, construction area workers, temporary construction power facilities, and working face within the enclosure; The set of weak construction clues includes at least one of the following: scattered bricks, pipes, bags, construction waste, ordinary building entrances, traces of maintenance on the exterior facade of ordinary buildings, and small amounts of stockpiled materials; The non-construction exclusion set includes at least one of the following: ordinary roads, green spaces, private vehicles, electric vehicles, ordinary pedestrians, and everyday building environments; When the image to be identified contains only categories not included in the construction exclusion set, or only text, signs, project names, or monitoring titles, it is determined that there is no construction evidence. S2.4: Calculate the percentage of the image area covered by the bounding boxes of the evidence items; S2.5: Based on the number of statistically strong construction evidence items and the proportion of image areas covered by the evidence item annotation boxes, perform a first judgment process on each image to be identified to obtain a set of images to be identified that pass the first judgment. The specific process is as follows: When the number of strong construction evidence items corresponding to any image to be identified is greater than or equal to 1 and the proportion of the image area is greater than zero, the image to be identified is determined to pass the first judgment; otherwise, it is determined to fail the first judgment condition.

3. The method for multi-stage identification and structured output of construction site safety hazards based on a visual language model according to claim 2, characterized in that, The specific process in S3, where the image to be identified, gated by S2, N construction visual evidence items, and a preset construction object vocabulary are input into the second visual language model for construction target extraction, to obtain A construction area identification results and M target identification results, is as follows: S3.1: The image to be identified, which is gated by S2, N construction visual evidence items, and a preset construction object vocabulary are used as joint inputs and loaded into the second visual language model; The second visual language model has a built-in vocabulary of construction objects and region-target level prompt word templates; S3.2: Input the text describing the construction area recognition task into the second visual language model; The second visual language model outputs A recognition results for construction areas. Each construction area identification result includes: area number, area type, and label box; the identification result of the j-th construction area is expressed by the formula: ； In the formula, j represents the region number. Indicates the region type. Indicates the x-coordinate of the center of the region label box. Indicates the ordinate of the center of the region label box; S3.3: Input the target recognition task description text into the second visual language model; the second visual language model outputs the recognition results of M targets to be detected; Each target identification result includes: the target category of the target, the normalized bounding box, the visible state features, the spatial location relationship description text, and the construction area number to which the target belongs; The recognition result of the m-th target to be detected is expressed by the formula: ； In the formula, Indicates the target category, This represents the x-coordinate of the center of the target's normalized bounding box. The value represents the center ordinate of the target's normalized annotation box; F represents the target's visibility status feature; P represents the spatial location relationship text; and j represents the construction area number to which the target belongs.

4. The method for multi-stage identification and structured output of construction site safety hazards based on a visual language model according to claim 3, characterized in that, The pre-set construction safety risk rule base in S4 includes: edge rules, machinery rules, temporary power rules, high-altitude operation rules, scaffolding rules, civilized construction rules, material stacking rules, and worker rules; Each risk rule predefines matching conditions for target category, visible state, and spatial relationship.

5. The method for multi-stage identification and structured output of construction site safety hazards based on a visual language model according to claim 4, characterized in that, In step S5, based on the fourth visual language model, image evidence judgment processing is performed on the key risk candidate set to obtain K risks that are deemed valid. The specific process is as follows: S5.1: Construct the evidence input set for each key risk candidate; Among them, the evidence determination input set for the b-th key risk candidate This includes: the image to be identified corresponding to the b-th key risk candidate, the coordinates of the bounding box of the b-th key risk candidate, the risk category of the b-th key risk candidate, and the reason for retaining the b-th key risk candidate; Expressed as a formula: ； In the formula, This represents the image to be identified corresponding to the b-th key risk candidate. This indicates the x-coordinate of the center of the bounding box for the b-th key risk candidate. This indicates the ordinate of the center of the bounding box for the b-th key risk candidate. This represents the risk category of the b-th key risk candidate. This indicates the reason for retaining the b-th key risk candidate; S5.2: Input the evidence judgment input set for each key risk candidate into the fourth visual language model for visible visual evidence judgment processing to obtain K risks with valid judgments; the specific process is as follows: S5.2.1: Input the evidence judgment input set of the b-th key risk candidate into the fourth visual language model, perform target object existence verification judgment processing, and obtain the target object existence verification judgment result of the b-th key risk candidate; S5.2.2: Input the evidence judgment input set of the b-th key risk candidate into the fourth visual language model, perform the verification judgment process of the abnormal state being clearly visible in the image, and obtain the verification judgment result of the abnormal state of the b-th key risk candidate being clearly visible in the image. S5.2.3: Input the evidence judgment input set of the b-th key risk candidate into the fourth visual language model, perform the dangerous spatial location relationship verification and judgment process, and obtain the dangerous spatial location relationship verification and judgment result of the b-th key risk candidate; S5.2.4: Input the evidence judgment input set of the b-th key risk candidate into the fourth visual language model, perform the verification judgment of the lack of protective or management measures, and obtain the verification judgment result of the lack of protective or management measures of the b-th key risk candidate. S5.2.5: Input the evidence judgment input set of the b-th key risk candidate into the fourth visual language model, perform the verification judgment process for not belonging to the reasonable construction state, and obtain the verification judgment result of the b-th key risk candidate not belonging to the reasonable construction state. S5.2.6: Based on the verification results of the existence of the target object of the b-th key risk candidate, the verification results of the abnormal state being clearly visible in the image, the verification results of the clear spatial relationship of the danger, the verification results of the visible absence of protection or management measures, and the verification results of the unreasonable construction state, the b-th key risk candidate is finally judged. When all verification results are valid, the b-th key risk candidate is determined to be a valid risk. If any verification result is invalid, the b-th key risk candidate is determined to be a risk that is invalid. S5.2.7: Repeat S5.2.1 to S5.2.6 until all key risk candidates have been finally determined, resulting in K risks that have been determined.

6. The method for multi-stage identification and structured output of construction site safety hazards based on a visual language model according to claim 5, characterized in that, Each structured hazard record in S6 includes: hazard category field, normalized annotation box field, risk level field, location description field, on-site phenomenon field, consequence description field, rectification suggestion field, and case reference field.

7. The method for multi-stage identification and structured output of construction site safety hazards based on a visual language model according to claim 6, characterized in that, In step S7, the K structured hazard records obtained in step S6 are reviewed based on the fifth visual language model to obtain a set of qualified structured hazard records. The specific process is as follows: S7.1: Construct the input set for each structured hazard record; where the input set for the k-th structured hazard record includes: The kth structured hazard record, the original image to be identified corresponding to the kth structured hazard record, and the evidence determination result corresponding to the kth structured hazard record; S7.2: Input the structured hazard record set into the fifth visual language model for review, and obtain a qualified structured hazard record set; the specific process is as follows: S7.2.1: Input the input set of the kth structured hidden danger record into the fifth visual language model to perform a verification process to determine whether the image evidence is sufficient; If the review result indicates that the image evidence is sufficient, proceed to S7.2.2; otherwise, proceed to S7.2.

8. S7.2.2: Input the input set of the kth structured hazard record into the fifth visual language model to perform a verification process to check whether the target is clear and unobstructed; If the verification result shows that the target is clear and unobstructed, proceed to S7.2.3; otherwise, proceed to S7.2.

8. S7.2.3: Input the input set of the kth structured hazard record into the fifth visual language model to verify whether the key areas are fully displayed in the image. If the verification result shows that the key area is fully displayed in the image, proceed to S7.2.4; otherwise, proceed to S7.2.

8. S7.2.4: Input the input set of the kth structured hazard record into the fifth visual language model to verify whether it belongs to a normal construction state and has been misjudged. If the review result indicates that the construction status is not normal and has been mistakenly judged, proceed to S7.2.5; otherwise, proceed to S7.2.

8. S7.2.5: Input the input set of the kth structured hazard record into the fifth visual language model to verify whether the position of the annotation box matches the hazard description. If the verification result shows that the position of the marked box is consistent with the description of the hidden danger, proceed to S7.2.6; otherwise, proceed to S7.2.

8. S7.2.6: Input the input set of the kth structured hazard record into the fifth visual language model to verify whether the source of the hazard record's fields is consistent. If the review result shows that the fields of the hidden danger record are consistent, proceed to S7.2.7; otherwise, proceed to S7.2.

8. S7.2.7: Input the input set of the kth structured hazard record into the fifth visual language model to perform a verification process to determine whether the edge-specific evidence is qualified. When the review result shows that the evidence for the edge-specific project is qualified, the kth structured hidden danger is judged to be qualified for review and proceeds to S7.2.9; Otherwise, delete the kth structured hazard record and proceed to S7.2.8; S7.2.8: Determine the kth structured hazard as unqualified for review, delete the kth structured hazard record, and proceed to S7.2.9; S7.2.9: Repeat steps S7.2.1 to S7.2.8 for the (k+1)th structured hazard record; when all K structured hazard records have been reviewed, a set of qualified structured hazard records is obtained.

8. The method for multi-stage identification and structured output of construction site safety hazards based on a visual language model according to claim 7, characterized in that, The first visual language model, the second visual language model, the third visual language model, the fourth visual language model, and the fifth visual language model are all visual language models based on the Transformer architecture.

9. A computer storage medium, characterized in that, The storage medium stores at least one instruction, which is loaded and executed by a processor to implement the method for multi-stage identification and structured output of construction site safety hazards based on a visual language model as described in any one of claims 1 to 8.

10. A device for multi-stage identification and structured output of safety hazards at construction sites based on a visual language model, characterized in that, The device includes a processor and a memory, the memory storing at least one instruction, which is loaded and executed by the processor to implement a method for multi-stage identification and structured output of construction site safety hazards based on a visual language model as described in any one of claims 1 to 8.

Images