A method for spatial conflict detection and early warning in construction scenes based on semantic networks

Abstract

The present invention provides a method for detecting and warning spatial conflicts in construction scenes based on semantic networks. The method comprises the following steps: capturing construction scene images; extracting construction machinery, vehicles, and personnel from the images; extracting hazardous areas; and performing semantic reasoning. This method enables dynamic monitoring of the spatial areas of various construction elements within a construction scene and provides early warnings of potential accidents. Compared to existing technologies, the method is not limited to identifying hazardous element features after an accident has occurred; it also integrates the recognition results with natural human reasoning, avoiding the high errors and low accuracy of traditional purely machine vision detection methods.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to a construction scene spatial conflict detection and early warning method based on a semantic network. Background Art

[0002] The construction industry, due to its inherently high risk, complex and dynamic environment, and labor-intensive nature, has long experienced a high rate of casualties. To address these casualties, new safety measures, including regulations, protective equipment, and monitoring and early warning systems, are constantly being proposed and developed. Regarding monitoring and pre-tensioning measures, current research focuses on simple safety target feature recognition, such as using neural network-based image recognition technology to detect the presence of safety equipment and identify dangerous movements. However, spatial factors are also a key factor directly impacting safety on construction sites, yet existing monitoring and early warning technologies have paid relatively little attention to this aspect.

[0003] Space is considered a critical resource on construction sites. On sites with diverse construction elements and complex tasks, the spatial demands of various parties can lead to conflicts in workspace. In recent years, visual inspection-based spatial hazard detection methods have been developed and applied to construction safety supervision. However, these methods typically only define static spaces for specific recognition tasks. These static spaces are often constructed around buildings, thus lacking monitoring of other elements.

[0004] Heinrich's Law divides the causes of collision accidents into three groups: equipment factors, human factors, and environmental factors. Based on this, safety warnings for construction site environments should comprehensively consider the spatial interactions between construction equipment, workers, and the environment. This requires that spatial monitoring of construction scenes must not only identify people and objects, but also detect potential risk events between people and objects, such as people not wearing safety equipment and moving in areas without guardrails. These risk events represent high-level semantic information that cannot be directly captured from image sources using deep learning algorithms alone. Summary of the Invention

[0005] In response to the shortcomings of the existing technology, the present invention provides a construction scene spatial conflict detection and early warning method based on semantic networks. It integrates semantic reasoning under the framework of computer vision technology, bridges the "semantic gap" between underlying image features and high-level semantics of event descriptions, and solves the problem of safety early warning in construction sites under complex dynamic environments.

[0006] The present invention achieves the above technical objectives through the following technical means.

[0007] A construction scene spatial conflict detection and early warning method based on semantic network includes the following steps:

[0008] Step 1: Collect construction scene images;

[0009] Step 2: extract the targets in the image, including the hazard source subject and its status, the hazard source object and its status;

[0010] Step 3: Extract the hazard source area; perform image space transformation on the hazard source subject and the hazard source object, so that the image position under the original camera perspective changes to a two-dimensional plane image under a top-down perspective, and then display the hazard source subject and the hazard source object in a primitive manner. Then, based on the primitive information, extract the spatial position distance information between the hazard source subject and the hazard source object;

[0011] Step 4, semantic reasoning: Based on pre-established semantic rules, semantic reasoning is performed on the information extracted in steps 2 and 3 to determine and identify dangerous information.

[0012] Furthermore, the main body of the hazard source is construction machinery, and the status of the construction machinery is divided into active and stationary; the objects of the hazard source are loading vehicles and personnel, among which the status of loading vehicles is divided into active and stationary, and the status of personnel is divided into wearing safety protective gear and not wearing safety protective gear.

[0013] Furthermore, step 3 includes:

[0014] Step 3.1: Perform a plane transformation on the construction scene image to convert the image plane to the ground plane. The transformation formula between the image coordinates and the ground coordinates is:

[0015]

[0016]

[0017] Where u and v are the pixel coordinates of the original image, m is the width of the original image, n is the height of the original image, X and Y are the ground coordinates after the change, α r is 1 / 2 the vertical field of view of the camera, α c is 1 / 2 the horizontal field of view of the camera, θ0 is the tilt angle of the camera, and h is the height of the camera from the ground;

[0018] Step 3.2: Convert the objects into primitives. First, grayscale and binarize the image transformed in step 3.1. Then, based on the outlines of each object extracted in step 2, primitives are drawn and replaced with primitives. For construction machinery, its minimum enclosing circle is used as the primitive; for loader trucks, its minimum enclosing rectangle is used as the primitive; and for personnel, the center point of their minimum enclosing circle is used as the primitive.

[0019] Step 3.3: Extract the spatial position distance information between each target based on the primitive information.

[0020] Furthermore, the step 4 is implemented using Protege reasoning software, wherein before reasoning, the software is first subjected to ontology construction and semantic rule formulation operations.

[0021] Furthermore, the semantic rules are written according to the spatial conflict rules, and the spatial conflict rules are:

[0022] For construction machinery:

[0023] The risk areas are delineated with the center point of the engineering machinery element as the base point. The area within a radius of a meter is designated as the danger area, the area from a to a+b meters is designated as the warning area, and the area from a+b to a+b+c meters is designated as the alert area.

[0024] For loader vehicles:

[0025] If the loading vehicle is not in the risk area, it is considered safe;

[0026] A stationary loader in any risk area is considered safe;

[0027] If an active loader is in the alert area and the construction machinery is in an active state, it is considered to be in alert state;

[0028] If an active loader is in the warning area and the construction machinery is in an active state, it is considered a warning;

[0029] For personnel:

[0030] If personnel do not wear safety gear, it is considered dangerous;

[0031] If the person wearing safety gear is not in the risk area, it is considered safe;

[0032] If the personnel wearing safety gear are in any risk area and the construction machinery is stationary, it is considered safe;

[0033] If personnel wearing safety gear are in the alert area and the construction machinery is in an active state, it is considered to be on alert;

[0034] If a person wearing safety gear is in the warning area and the construction machinery is in an active state, it is considered a warning.

[0035] Furthermore, in step 2, the Yolo X target detection algorithm is used to extract the target in the image.

[0036] Furthermore, in step 1, the Mosiac data augmentation method is used to expand the dataset, and the image quality is enhanced through light uniformity processing and denoising processing.

[0037] Furthermore, the status of the construction machinery and the status of the loader are identified and judged by the difference between target frames in the video stream.

[0038] The beneficial effects of the present invention are:

[0039] (1) The present invention provides a construction scene spatial conflict detection and early warning method based on a semantic network, which can achieve dynamic monitoring of the spatial areas of various construction elements in a construction scene and provide early warning of accident hazards. Compared with existing technologies, the identification targets are not limited to the identification of dangerous element features after an accident has occurred; the identification results are combined with human natural reasoning language, avoiding the large errors and low accuracy problems of traditional pure machine vision detection methods.

[0040] (2) In the present invention, the collected image is converted from the image plane under the camera perspective to a two-dimensional plane under the top perspective, and each target is simplified and represented in the form of graphic elements to achieve visual and flat monitoring of the spatial situation of the construction site. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 Flowchart of the construction scene spatial conflict detection and early warning method based on semantic network of the present invention;

[0042] Figure 2 This is a schematic diagram of the construction scene graphic element coding of the present invention;

[0043] Figure 3 This is a schematic diagram of the knowledge graph elements under the construction scene spatial conflict semantic framework of the present invention;

[0044] Figure 4 It is the knowledge graph under the construction scene spatial conflict semantic framework of the present invention;

[0045] Figure 5 A schematic diagram of the risk area delineation surrounding the main hazard source in the present invention;

[0046] Figure 6a An example of identifying objects and their states in an image;

[0047] Figure 6b An example of visualizing two-dimensional spatial relationships;

[0048] Figure 6c It is a diagram of the mapping between the risk rule base and the image semantic consistency;

[0049] Figure 6d This is an example of scene reasoning results. DETAILED DESCRIPTION

[0050] The following describes embodiments of the present invention in detail. Examples of the illustrated embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and are not to be construed as limiting the present invention.

[0051] Figure 1 The figure shows a flowchart of a semantic network-based construction scene spatial conflict detection and early warning method, which includes construction site image acquisition, hazard source subject state identification and analysis, object state attribute extraction, hazard source area extraction, semantic rule formulation (rule base), and semantic reasoning. The specific steps are as follows:

[0052] Step 1: Collect construction scene images

[0053] On-site monitoring equipment is used to collect images of construction sites. Mosiac data augmentation is used to expand the dataset, and the quality of the image data obtained within the construction site is enhanced through light leveling and denoising.

[0054] Step 2: Target extraction

[0055] The Yolo X target detection algorithm is used to extract the corresponding targets from the construction site images collected in step 1, where the targets to be extracted include the hazard source subject and the hazard source object.

[0056] a. Extract the main body of the hazard source and its motion state

[0057] In this embodiment, the hazard source is set as earthmoving construction machinery (forklifts, excavators, etc.); the motion state of the hazard source is divided into ① active and ② stationary. The motion state of the hazard source (construction machinery) is identified and determined by the difference between the target frames in the video stream. Specifically, the motion state is determined by setting a threshold for the difference between the inter-frame coordinates of the corner points of the recognition frame, and then determining the target motion state based on the threshold overflow within a few frames. The determination threshold is obtained through clustering, and in this embodiment, the threshold is ≤100 DPI.

[0058] b. Extract the hazardous source object and its attribute status

[0059] In this embodiment, there are two types of dangerous source objects, among which object 1 is set as a loader and object 2 is set as a person; the attribute status of object 1 loader is divided into ① active and ② stationary, and the attribute status of object 2 person is divided into ① wearing a safety helmet and ② not wearing a safety helmet (the safety helmet here can also be expanded to other safety protective gear). The attribute status of the loader is identified and judged by the difference between the target frames in the video stream. Specifically, the motion state is determined by setting the difference threshold between the coordinates of the weight center point of the recognition box between frames, and then determining the target motion state based on the threshold overflow within a few frames; the determination threshold is obtained by clustering, and specifically in this embodiment, the threshold is set to ≦100DPI.

[0060] Step 3: Extraction of hazardous areas

[0061] Based on the captured construction site images, the extracted hazard sources (construction machinery) and hazard objects (loaders and personnel) are transformed into image space. The original camera-perspective image positions are transformed into a two-dimensional plan view from a bird's-eye view. The hazard sources and hazard objects are then displayed as primitives. Finally, the spatial distance information between the construction machinery, loader, and personnel is extracted based on the primitive information.

[0062] Step 3.1, perform a plane transformation on the captured image to convert the image plane to the real ground plane. The corresponding transformation formula between image coordinates and ground coordinates is:

[0063]

[0064]

[0065] Where u and v are the pixel coordinates of the original image, m is the width of the original image, n is the height of the original image, X and Y are the ground coordinates after the change, α r is 1 / 2 the vertical field of view of the camera, α c is 1 / 2 the horizontal field of view of the camera, θ0 is the tilt angle of the camera, and h is the height of the camera from the ground.

[0066] Step 3.2: Convert the targets into primitives. First, grayscale and binarize the image to convert it into a 0-1 pixel value format. Then, based on the outlines of each target (construction machinery, loader, and personnel) detected and extracted in step 2, primitives are drawn and replaced with primitives. Construction machinery is represented by a circular frame, loader by a rectangular frame, and personnel by a point. Specifically, the minimum enclosing circle of the construction machinery's outline is used as the primitive; the minimum enclosing rectangle of the loader's outline is used as the primitive; and the center point of the minimum enclosing circle of the personnel's outline is used as the primitive.

[0067] In step 3.3, based on the primitive information, the spatial position distance information between the construction machinery, the loading vehicle, and the personnel is extracted, and the primitive information of different target types is encoded, where Ti maps the object target 1 loading vehicle (rectangle), Mi maps the hazard source body (circle), and Pi maps the object target 2 personnel (point). Figure 2 The figure shows a schematic diagram of the construction scene element coding.

[0068] Step 4: Semantic Reasoning

[0069] Step 4.1, ontology construction

[0070] To implement spatial conflict risk reasoning, Protege reasoning software will be used. The reasoning process relies on the Ontology Description Language (OWL). First, an ontology class (owl:thing) is constructed, which contains multiple subclasses: the hazard source subject class and the hazard source object class. This establishes the ontology class structure. Within Protege reasoning software, corresponding object properties are created for each subclass. For the hazard source subject class, the object properties include motion and stationary; for the hazard source object class, the object properties include motion and stationary; and for the person object, the properties include wearing a helmet and not wearing a helmet. Within Protege, a data property is created for each subclass, declaring each data property to be spatial distance (Distance). Individuals whose spatial distance values ​​are considered are declared to have the Distance data property. Different data property categories are defined for different values. Specifically, attribute categories are defined in descending order of value, with the four major categories being "Safety," "Alert," "Warning," and "Danger."

[0071] The above steps together constitute the semantic reasoning elements of subject class, attribute class, object class and spatial value class. The specific elements are as follows: Figure 3 shown.

[0072] Step 4.2, semantic rule formulation

[0073] Formulate spatial conflict rules and build a SWRL (Semantic Web Rule Language, a language that presents rules in a semantic way) risk rule library based on the spatial conflict rules. Figure 4 The figure shows the knowledge graph under the semantic framework of spatial conflict in construction scenes of the present invention. For example, the spatial conflict rules are:

[0074] (1) For the main hazardous source engineering machinery

[0075] Risk areas are delineated based on the center point of the engineering machinery element, including danger zone, warning zone, and alert zone. The area within a radius of 10m is designated as the danger zone, the area within a radius of 10m to 12m is designated as the warning zone, and the area within a radius of 12m to 13m is designated as the alert zone. Figure 5 Shown is a schematic diagram of the scope of each risk area surrounding the main source of danger.

[0076] (2) For object 1 loading vehicle

[0077] If the loading vehicle is not in the risk area, it is considered safe;

[0078] A stationary loader in any risk area is considered safe;

[0079] If an active loader is in the alert area and the construction machinery is in an active state, it is considered to be in alert state;

[0080] If an active loader is in the warning area and the construction machinery is in an active state, it is considered a warning;

[0081] (3) For Object 2 personnel

[0082] If the personnel are not wearing a helmet, it is considered dangerous;

[0083] If the person wearing the helmet is not in the risk area, it is judged to be safe;

[0084] If the person wearing the helmet is in any risk area and the construction machinery is stationary, it is considered safe;

[0085] If the person wearing a helmet is in the alert area and the construction machinery is in an active state, it is considered to be on alert;

[0086] If the person wearing a safety helmet is in the warning area and the construction machinery is in an active state, it is judged as a warning.

[0087] Note: The above rules do not determine the relevant dangerous sign information of the dangerous area, because the purpose of setting the alert area and warning area and identifying and determining them in this embodiment is to take proactive warning.

[0088] Step 4.3, Inference

[0089] According to the established spatial conflict rules, the information extracted in steps 2 and 3 is subjected to semantic reasoning. That is, the extracted target information of the hazard source subject and object, their respective attribute status information, and the spatial position relationship information between them are first semantically transcoded. Then, as the input of the spatial conflict semantic reasoning network, reasoning is performed under the preset dangerous omen rules (spatial conflict rules) to determine and identify various types of dangerous omen information. The specific operation is as follows:

[0090] The OWL application programming interface (API) is used to read the semantically transcoded image semantics of the test image and create an ontology instance. The extracted image semantics are mapped to the classes, objects, and data attributes in the constructed ontology description rules. Based on the mapping relationship between image semantics and ontology descriptions, the image semantics are mapped to the ontology classes. The reasoning element relationship structure between object types (subclasses) and between classes and attributes is established. The semantic information in the ontology instance is logically integrated and written into an OWL file.

[0091] Pellet, part of the Protege reasoning software, was used as the inference engine for the spatial conflict reasoning task. The inference process was implemented by matching the OWL semantics file of the ontology instance mapped to the image semantics using a SWRL rule base. The Pellet engine was used to verify the consistency between the rule base's axioms and constraints and the OWL semantics file of the mapped ontology instance, determining whether any critical factors were present.

[0092] Figure 6a to Figure 6d The following is a specific example test using the above method, where Figure 6a The following shows the recognition of each target and its status in the image. Figure 6b Shown Figure 6a A visualization diagram of the two-dimensional plane relationship between the targets in the image. Figure 6c For the mapping of risk rule base and image semantic consistency, Figure 6d The figure shows the final warning result.

[0093] The present invention is not limited to the above-mentioned embodiments. Any obvious improvement, replacement or modification that can be made by those skilled in the art without departing from the essence of the present invention shall fall within the scope of protection of the present invention.

Claims

1. A construction scene spatial conflict detection and early warning method based on semantic network, characterized in that: The steps include: Step 1: Collect construction scene images; Step 2: Extract the targets in the image, including the main hazard source and its state, and the object hazard source and its state. The main hazard source is engineering machinery, and the state of the engineering machinery is divided into active and stationary. The object hazard source is a loader and a person, where the state of the loader is divided into active and stationary, and the state of the person is divided into wearing safety gear and not wearing safety gear. Step 3: Extract the hazard source area; perform image space transformation on the hazard source subject and the hazard source object, so that the image position under the original camera perspective changes to a two-dimensional plane image under a top-down perspective, and display the hazard source subject and the hazard source object in a primitive manner. Then, based on the primitive information, extract the spatial position distance information between the hazard source subject and the hazard source object; including: Step 3.1: Perform a plane transformation on the construction scene image to convert the image plane to the ground plane. The transformation formula between the image coordinates and the ground coordinates is: ； ； Where u and v are the pixel coordinates of the original image, m is the width of the original image, n is the height of the original image, X and Y are the ground coordinates after the change, α r is 1 / 2 the vertical field of view of the camera, α c is 1 / 2 the horizontal field of view of the camera, θ0 is the tilt angle of the camera, and h is the height of the camera from the ground; Step 3.2: Convert the objects into primitives. First, grayscale and binarize the image transformed in step 3.

1. Then, based on the outlines of each object extracted in step 2, primitives are drawn and replaced with primitives. For construction machinery, its minimum enclosing circle is used as the primitive; for loader trucks, its minimum enclosing rectangle is used as the primitive; and for personnel, the center point of their minimum enclosing circle is used as the primitive. Step 3.3, extracting the spatial position distance information between each target based on the primitive information; Step 4, semantic reasoning: Based on pre-established semantic rules, semantic reasoning is performed on the information extracted in steps 2 and 3 to determine and identify dangerous information.

2. The early warning method according to claim 1, characterized in that: The step 4 is implemented using Protege reasoning software, wherein before reasoning, the software is first subjected to ontology construction and semantic rule formulation operations.

3. The early warning method according to claim 2, characterized in that: The semantic rules are written according to the spatial conflict rules, which are: For construction machinery: The risk areas are delineated with the center point of the engineering machinery element as the base point. The area within a radius of a meter is designated as the danger area, the area from a to a+b meters is designated as the warning area, and the area from a+b to a+b+c meters is designated as the alert area. For loader vehicles: If the loading vehicle is not in the risk area, it is considered safe; A stationary loader in any risk area is considered safe; If an active loader is in the alert area and the construction machinery is in an active state, it is considered to be in alert state; If an active loader is in the warning area and the construction machinery is in an active state, it is considered a warning; For personnel: If personnel do not wear safety gear, it is considered dangerous; If the person wearing safety gear is not in the risk area, it is considered safe; If the personnel wearing safety gear are in any risk area and the construction machinery is stationary, it is considered safe; If personnel wearing safety gear are in the alert area and the construction machinery is in an active state, it is considered to be on alert; If a person wearing safety gear is in the warning area and the construction machinery is in an active state, it is considered a warning.

4. The early warning method according to claim 1, characterized in that: In step 2, the Yolo X target detection algorithm is used to extract the target in the image.

5. The early warning method according to claim 1, characterized in that: In step 1, the Mosiac data augmentation method is used to expand the dataset, and the image quality is enhanced through light uniformity processing and denoising processing.

6. The early warning method according to claim 1, characterized in that: The status of the construction machinery and the loader are identified and judged by the difference between target frames in the video stream.

Images