Construction site illegal behavior detection management method and system, terminal and medium

Abstract

The invention relates to the field of construction site illegal behavior detection, and particularly discloses a construction site illegal behavior detection management method and system, a terminal and a medium. Processing the to-be-detected image by using a pre-trained target detection model, and extracting a target physical entity in the to-be-detected image; retrieving violation management information corresponding to the target physical entity according to the graph knowledge base, and recording the violation management information as first violation management information; processing the to-be-detected image by using a pre-trained deep learning convolutional neural network model, and extracting a target feature vector of the to-be-detected image; retrieving violation management information corresponding to the target feature vector according to an image-text vector library, and recording the violation management information as second violation management information; and processing the first violation management information and the second violation management information by using a multi-modal large model to obtain final violation management information. According to the invention, the accuracy of illegal behavior mining is improved, matching of safety management requirements is carried out, the working efficiency is improved, and the safety risk is reduced.

Description

Technical Field

[0001] The present invention relates to the field of construction site violation detection, and in particular to a construction site violation detection management method, system, terminal and medium. Background Art

[0002] Safety management of construction sites is of vital importance, but traditional methods of safety management of construction sites have many shortcomings. On the one hand, the detection of violations on construction sites often relies on manual inspections, which is not only inefficient but also prone to omissions and misjudgments. Manual inspections make it difficult to monitor every corner of the construction site in real time and fail to detect potential safety violations in a timely manner. At the same time, the coverage of manual inspections is limited, making it difficult to achieve full coverage of safety risk points on construction sites. On the other hand, when matching safety management requirements, managers usually need to rely on experience and memory to find the corresponding regulations. This process is time-consuming and labor-intensive, and accuracy is difficult to guarantee. Different violations may correspond to different safety management requirements, and manual matching is prone to errors, resulting in the inability to accurately implement safety management measures.

[0003] With the continuous development of science and technology, some image recognition technologies have been applied to construction site safety management. However, single image feature extraction technology can only identify some obvious violations and is difficult to detect potential safety violations. At the same time, existing technologies lack efficient and accurate methods for matching detected violations with safety management requirements, and cannot meet the growing needs of construction site safety management. Summary of the invention

[0004] To solve the above problems, the present invention provides a construction site violation detection and management method, system, terminal and medium, which fully mines potential violations in construction site pictures from both image entities and image feature vectors, improves the accuracy of violation mining, and simultaneously matches safety management requirements, thereby improving work efficiency and reducing safety risks.

[0005] In a first aspect, the technical solution of the present invention provides a construction site violation detection and management method, comprising the following steps: Construct a graph knowledge base and a graph-text vector library. The graph knowledge base stores the relationship between physical entities and violation management information, and the graph-text vector library stores the relationship between image feature vectors and violation management information. Use the pre-trained target detection model to process the image to be tested and extract the target physical entity in the image to be tested; Retrieving the violation management information corresponding to the target physical entity according to the graph knowledge base, and recording it as the first violation management information; Use the pre-trained deep learning convolutional neural network model to process the image to be tested and extract the target feature vector of the image to be tested; Retrieving the violation management information corresponding to the target feature vector according to the image-text vector library, and recording it as the second violation management information; The first violation management information and the second violation management information are processed using a multimodal large model to obtain final violation management information.

[0006] In an optional implementation, constructing a graph knowledge base includes: Constructing a construction site map knowledge graph, wherein the construction site map knowledge graph includes nodes and edge relationships, wherein the nodes include physical entity nodes and violation management information nodes, wherein the physical entity nodes include entity attributes and names of the physical entities, and the violation management information nodes include file names; Store the construction site map knowledge graph in the Neo4j database.

[0007] In an optional embodiment, the edge relationships of the construction site map knowledge graph include spatial relationships and semantic relationships. The spatial relationships are used to describe the relative positions, size ratios, and color similarities between physical entity nodes, and the semantic relationships are used to describe the text descriptions and semantic information between violation management information nodes.

[0008] In an optional implementation, a pre-trained target detection model is used to process the image to be tested, and the target physical entity is extracted from the image to be tested, specifically including: Using the pre-trained Grounding DINO model to process the image to be tested, and extracting the first physical entity in the image to be tested; The image to be tested is processed using a pre-trained Faster RCNN neural network model to extract a second physical entity from the image to be tested; The image to be tested is processed using the YOLO algorithm to extract the third physical entity in the image to be tested; The first physical entity, the second physical entity, and the third physical entity are merged to obtain a target physical entity.

[0009] In an optional implementation, the pre-trained Grounding DINO model is used to process the image to be tested, and the first physical entity is extracted from the image to be tested, specifically including: Input the image to be tested into the image backbone network for feature extraction to obtain image features; Input relevant text description into the text backbone network for feature extraction to obtain text features; The text guide is constructed by the following formula,

[0010] In the formula, Refers to the text guide, Refers to the text-guided network, Refers to the k similar image feature vectors, Refers to image features, Refers to text features; The image features, text features, and text guidance are input into a multimodal encoder to obtain a first physical entity.

[0011] In an optional embodiment, the pre-trained deep learning convolutional neural network model includes 13 convolutional layers, 4 pooling layers, 1 fully connected layer and 1 Softmax activation layer.

[0012] In an optional implementation, the first violation management information and the second violation management information are processed using a multimodal large model to obtain final violation management information, specifically including: Form a candidate set by multi-channel recall of the first violation management information and the second violation management information; Construct Prompt prompt information; According to the prompt information, the GraphRAG model is used to retrieve and refine the candidate set to obtain the final violation management information.

[0013] In a second aspect, the technical solution of the present invention provides a construction site violation detection and management system, comprising: Database construction module: construct a graph knowledge base and a graph-text vector library. The graph knowledge base stores the relationship between physical entities and violation management information, and the graph-text vector library stores the relationship between image feature vectors and violation management information. Target physical entity extraction module: uses the pre-trained target detection model to process the image to be tested and extracts the target physical entity in the image to be tested; The first information retrieval module: retrieves the violation management information corresponding to the target physical entity according to the graph knowledge base, and records it as the first violation management information; Target feature vector extraction module: uses a pre-trained deep learning convolutional neural network model to process the image to be tested and extracts the target feature vector of the image to be tested; The second information retrieval module: retrieves the violation management information corresponding to the target feature vector according to the image-text vector library, and records it as the second violation management information; Violation management information acquisition module: uses a multimodal large model to process the first violation management information and the second violation management information to obtain the final violation management information.

[0014] In a third aspect, the technical solution of the present invention provides a terminal, including: A memory, used for storing a construction site violation detection management program; A processor is used to implement the steps of the construction site violation detection management method as described in any of the above items when executing the construction site violation detection management program.

[0015] In a fourth aspect, the technical solution of the present invention provides a computer-readable storage medium, on which is stored a construction site violation detection management program, which, when executed by a processor, implements the steps of the construction site violation detection management method as described in any one of the above items.

[0016] The present invention provides a method, system, terminal and medium for detecting and managing construction site violations, which have the following beneficial effects compared to the prior art: first, a graph knowledge base and a graph-text vector base are constructed, and then a pre-constructed model is used to extract the target physical entity and the target feature vector, and corresponding violation management information is obtained based on the construction of the graph knowledge base and the graph-text vector base, and finally the final violation management information is obtained through a multimodal large model. The present invention fully mines potential violations in construction site images from both image entities and image feature vectors, improves the accuracy of violation mining, and matches safety management requirements at the same time, improves work efficiency, and reduces safety risks. BRIEF DESCRIPTION OF THE DRAWINGS In order to more clearly illustrate the embodiments of the present invention or the technical solutions of the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 It is a schematic diagram of the architecture of a construction site violation detection and management method provided by an embodiment of the present invention.

[0018] Figure 2 It is a flow chart of a construction site violation detection and management method provided by an embodiment of the present invention.

[0019] Figure 3 It is a flowchart of using the pre-trained Grounding DINO model to process the image to be tested.

[0020] Figure 4 The present invention provides a schematic block diagram of a construction site violation detection and management system.

[0021] Figure 5 It is a schematic diagram of the structure of a terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to enable those skilled in the art to better understand the scheme of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0023] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0024] Figure 1 is a schematic diagram of the architecture of a construction site violation detection and management method provided by an embodiment of the present invention, such as Figure 1 As shown in the figure, this method mainly includes three aspects: image entity extraction and knowledge graph construction, image feature vectorization and image-text vector library construction, and multimodal large model GraphRAG. It uses entity relationship extraction technology, image feature extraction technology and multi-way recall retrieval technology to dig out potential safety violations in the image, and achieve full coverage detection of safety risk points on the construction site and management requirement reminders. This method integrates heterogeneous information data such as entity relationship graphs and image feature vectors in the image to detect violations, inputs image data, model prompts, potential violations and candidate requirement files into the multimodal large model for reasoning and matching, and outputs accurate safety management requirements. The above method can realize the detection of construction site violations in the image to be tested and the matching with relevant safety regulations, effectively improve the work efficiency of the construction site, and greatly reduce safety risks.

[0025] Figure 2 This is a flow chart of a construction site violation detection and management method provided by an embodiment of the present invention. Figure 2 The execution subject may be a construction site violation detection and management system. The construction site violation detection and management method provided in the embodiment of the present invention is executed by a computer device, and accordingly, the construction site violation detection and management system runs in the computer device. According to different requirements, the order of the steps in the flowchart can be changed, and some can be omitted.

[0026] like Figure 2 As shown, the method includes the following steps.

[0027] S1, build graph knowledge base and image-text vector library.

[0028] In this embodiment, the graph knowledge base stores the relationship between physical entities and violation management information, which is used to match the violation management information according to the physical entities. The graph vector library stores the relationship between image feature vectors and violation management information, which is used to match the violation management information according to the image feature vectors. The violation management information may include violation behaviors and management requirements.

[0029] S2, using the pre-trained target detection model to process the image to be tested and extract the target physical entity in the image to be tested.

[0030] The purpose of this step is to extract the target physical entity in the image to be tested, pre-train the target detection model, and process the image to be tested according to the model to extract the target physical entity.

[0031] S3, searching the violation management information corresponding to the target physical entity according to the graph knowledge base, and recording it as the first violation management information.

[0032] The purpose of this step is to match the violation management information corresponding to the target physical entity extracted in step S2 from the graph knowledge base.

[0033] S4, uses the pre-trained deep learning convolutional neural network model to process the image to be tested and extracts the target feature vector of the image to be tested.

[0034] The purpose of this step is to extract the target feature vector of the image to be tested, pre-train the deep learning convolutional neural network model, and use the model to process the image to be tested to extract the target feature vector.

[0035] S5, searching the violation management information corresponding to the target feature vector according to the image-text vector library, and recording it as the second violation management information.

[0036] The purpose of this step is to match the violation management information corresponding to the target feature vector extracted in step S4 from the image-text vector library.

[0037] S6, using a multimodal large model to process the first violation management information and the second violation management information to obtain final violation management information.

[0038] This embodiment first constructs a graph knowledge base and a graph-text vector library, then uses a pre-built model to extract the target physical entity and target feature vector, obtains the corresponding violation management information based on the construction of the graph knowledge base and the graph-text vector library, and finally obtains the final violation management information through a multimodal large model. This embodiment fully mines potential violations in construction site images from both image entities and image feature vectors, improves the accuracy of violation mining, and matches safety management requirements at the same time, improves work efficiency, and reduces safety risks. This embodiment pre-builds a graph knowledge base to match the violation management information after the physical entity is extracted. This embodiment builds the graph knowledge base including building a construction site graph knowledge graph and storing the construction site graph knowledge graph in a Neo4j database.

[0039] Knowledge graph is a structured knowledge representation method that can effectively integrate and manage various entities and relationship information. By constructing a knowledge graph of construction site violation image entities, we can achieve a deep understanding of the image content and improve the accuracy of related detection.

[0040] Knowledge graph modeling consists of three parts: entities, relationships, and attributes. In the entity extraction stage, the entity information in the image to be inspected is obtained, and the corresponding attribute values ​​can be obtained by annotating the entity. Each entity corresponds to a node in the knowledge graph, and each violation management information corresponds to a node in the knowledge graph, including a physical entity node and a violation management information node. The physical entity node contains the entity attributes and name of the physical entity, and the violation management information node contains the file name, which is obtained through the construction site safety management regulations, and can also contain information such as the file release time.

[0041] In this embodiment, the relationship edges of the image knowledge graph are defined as two categories: spatial relationship and semantic relationship, which can be expressed as The relationship extraction between entities can be based on multiple factors such as the visual features, spatial layout, and semantic information of the image. By analyzing the visual features such as the relative position, size ratio, and color similarity between entities, the spatial relationship between physical entity nodes can be inferred; by analyzing the text description and semantic information of the violation management information, the semantic relationship between the violation management information nodes can be inferred.

[0042] Due to the graphical characteristics of the knowledge graph, the table structure of the structured database cannot meet its storage requirements, so the Neo4j database is used as a knowledge graph storage tool. The obtained image entities, security management requirements entities, spatial semantic relationships and related entity attributes are used to form a knowledge graph, and it is stored in the database to complete the construction of the graph database.

[0043] The images to be tested in the construction site scene often contain rich entity information, such as building materials, construction workers, safety equipment, etc. How to accurately extract all entity information from the image is of great significance for preventing the problem of missed detection of violations. This embodiment uses a pre-trained target detection model to process the image to be tested, and extracts the target physical entity in the image to be tested, including extracting the entity information contained in the image by target detection technology based on deep learning. The text-assisted target detection model Grounding DINO model is used, supplemented by the convolutional neural network architecture Faster RCNN and YOLO series algorithms to perform target detection on the input image. These models are pre-trained with large-scale image data sets and learn the characteristic patterns of different objects, so that they can accurately identify the entities in the image to be tested. For example, for images with incomplete inspection records of fire extinguishers on the construction site, the target detection model can detect the location and entity category of each fire extinguisher, inspection record card, guardrail, and ground crack. The pre-trained target detection model is used to process the image to be tested and extract the target physical entity in the image to be tested, which specifically includes the following steps.

[0044] Step 1: Use the pre-trained Grounding DINO model to process the image to be tested and extract the first physical entity in the image to be tested.

[0045] Figure 3 It is a flowchart of using the pre-trained Grounding DINO model to process the image to be tested, including the following steps.

[0046] Step 1.1: Input the image to be tested into the image backbone network for feature extraction to obtain image features.

[0047] Step 1.2: Input the relevant text description into the text backbone network for feature extraction to obtain text features.

[0048] Step 1.3, construct the text guide by the following formula,

[0049] In the formula, Refers to the text guide, Refers to the text-guided network, Refers to the k similar image feature vectors, Refers to image features, Refers to text features.

[0050] Step 1.4, input the image features, text features, and text guidance into the multimodal encoder to obtain the first physical entity.

[0051] First, the text description and the image to be tested are input into the corresponding backbone network for feature extraction, and then feature enhancement is performed to obtain text features. and image features In order to increase the weight of text description in the model, a text guide is added before the relevant features are input into the multimodal encoder. The construction process.

[0052] Step 2: Use the pre-trained Faster RCNN neural network model to process the image to be tested and extract the second physical entity in the image to be tested.

[0053] Step 3: Use the YOLO algorithm to process the image to be tested and extract the third physical entity in the image to be tested.

[0054] Step 4: merge the first physical entity, the second physical entity, and the third physical entity to obtain a target physical entity.

[0055] This embodiment improves the accuracy and robustness of entity extraction by adopting a multi-scale feature fusion method during the image entity extraction process. By fusing convolutional features at different levels, the model can capture global and local information in the image at the same time, thereby better adapting to entities of different sizes and shapes. By fusing high-level semantic features with low-level detail features, the detection capability of small target entities can be enhanced.

[0056] In an optional implementation, the quality of entity extraction is improved by post-processing and optimizing the target detection results, including removing duplicate entities, adjusting the size of the bounding box, filtering low-confidence detection results, etc.

[0057] In order to achieve efficient retrieval and analysis of a large number of images and safety management regulations, it is necessary to establish an image-text vector library to uniformly store and manage image feature vectors and related text information. This embodiment uses the Elasticsearch vector database to build a table, which may contain fields such as image ID, feature vector, image description, and related safety management regulations. In this way, the image feature vector, content description text, and safety management regulations are associated and stored.

[0058] It should be noted that with the introduction and abolition of construction site safety management regulations, it is necessary to regularly update and maintain the image and text vector library to ensure the accuracy and completeness of the data. Regularly update construction site violations and safety management regulations through image feature vector matching, delete expired or invalid data, etc.

[0059] Feature extraction and vectorization of images can realize the digital representation of image features, which is convenient for subsequent storage and retrieval operations. This embodiment uses a deep learning convolutional neural network model to extract features from input images. Convolutional neural networks have powerful image feature learning capabilities and can automatically extract multi-level feature information from images. After several layers of convolution, pooling and activation function operations, a high-dimensional feature vector is obtained, which contains all the feature information of the image.

[0060] The network architecture for image feature extraction in this embodiment is a VGG-like architecture. The entire network contains 13 convolutional layers, 4 pooling layers, 1 fully connected layer, and 1 Softmax activation layer. It can perform multi-level feature mining on the construction site images to be inspected, ensuring that the model output vector contains all important information in the image.

[0061] This embodiment uses a multimodal large model to process the first violation management information and the second violation management information to obtain the final violation management information, so as to obtain the violation management information based on physical entities and feature vectors, and realize the deep mining of violation behaviors. First, the first violation management information and the second violation management information are recalled in multiple ways to form a candidate set; Prompt prompt information is constructed; according to the Prompt prompt information, the GraphRAG model is used to retrieve and sort the candidate set to obtain the final violation management information.

[0062] This embodiment combines the respective advantages of the constructed knowledge graph database and the image-text vector library by using multi-way recall retrieval sorting technology to further improve the accuracy of violation detection. After obtaining the candidate set through multi-way recall, the results are processed by retrieval sorting. The retrieval sorting process mainly screens and sorts the candidate set, and quickly excludes obviously irrelevant results according to relatively simple rules or conditions, reducing the amount of calculation and time cost in the model reasoning stage. Graph Retrieval Enhanced Generation (GraphRAG) can more effectively handle complex relationships and contexts by utilizing graph structured data, thereby providing more accurate and in-depth answers. By retrieving data elements such as entities, triples, paths, and subgraphs related to the image to be inspected in the graph database, the complex relationships and contextual information between entities can be better captured. This method is suitable for scenarios where it is necessary to understand and reason about complex relationships between entities, and can achieve effective retrieval of violation management information in the task of detecting violations on construction sites.

[0063] An embodiment of a construction site violation detection and management method is described in detail above. Based on the construction site violation detection and management method described in the above embodiment, an embodiment of the present invention further provides a construction site violation detection and management system corresponding to the method.

[0064] Figure 4The present invention provides a schematic block diagram of a construction site violation detection and management system. The construction site violation detection and management system 400 can be divided into multiple functional modules according to the functions it performs. The module referred to in the present invention refers to a series of computer program segments that can be executed by at least one processor and can complete fixed functions, which are stored in a memory.

[0065] Database construction module 410: constructs a graph knowledge base and a graph-text vector library, wherein the graph knowledge base stores the relationship between physical entities and violation management information, and the graph-text vector library stores the relationship between image feature vectors and violation management information.

[0066] The target physical entity extraction module 420 uses a pre-trained target detection model to process the image to be tested, and extracts the target physical entity in the image to be tested.

[0067] The first information retrieval module 430: retrieves the violation management information corresponding to the target physical entity according to the graph knowledge base, and records it as the first violation management information.

[0068] Target feature vector extraction module 440: uses a pre-trained deep learning convolutional neural network model to process the image to be tested and extracts the target feature vector of the image to be tested.

[0069] The second information retrieval module 450: retrieves the violation management information corresponding to the target feature vector according to the image-text vector library, and records it as the second violation management information.

[0070] The violation management information acquisition module 460 uses a multimodal large model to process the first violation management information and the second violation management information to obtain final violation management information.

[0071] The construction site violation detection and management system of this embodiment is used to implement the aforementioned construction site violation detection and management method. Therefore, the specific implementation method of this system can be seen in the implementation example part of the construction site violation detection and management method in the previous text. Therefore, its specific implementation method can refer to the description of the corresponding each part of the implementation example, which will not be elaborated here.

[0072] In addition, since the construction site violation detection and management system of this embodiment is used to implement the aforementioned construction site violation detection and management method, its function corresponds to that of the aforementioned method and will not be repeated here.

[0073] Figure 5 A schematic diagram of the structure of a terminal 500 provided in an embodiment of the present invention includes: a processor 510, a memory 520 and a communication unit 530. The processor 510 is used to implement the following steps when implementing the construction site violation detection management program stored in the memory 520: Construct a graph knowledge base and a graph-text vector library. The graph knowledge base stores the relationship between physical entities and violation management information, and the graph-text vector library stores the relationship between image feature vectors and violation management information. Use the pre-trained target detection model to process the image to be tested and extract the target physical entity in the image to be tested; Retrieving the violation management information corresponding to the target physical entity according to the graph knowledge base, and recording it as the first violation management information; Use the pre-trained deep learning convolutional neural network model to process the image to be tested and extract the target feature vector of the image to be tested; Retrieving the violation management information corresponding to the target feature vector according to the image-text vector library, and recording it as the second violation management information; The first violation management information and the second violation management information are processed using a multimodal large model to obtain final violation management information.

[0074] The present invention also provides a computer storage medium, wherein the storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM).

[0075] The computer storage medium stores a construction site violation detection management program, and when the construction site violation detection management program is executed by the processor, the following steps are implemented: Construct a graph knowledge base and a graph-text vector library. The graph knowledge base stores the relationship between physical entities and violation management information, and the graph-text vector library stores the relationship between image feature vectors and violation management information. Use the pre-trained target detection model to process the image to be tested and extract the target physical entity in the image to be tested; Retrieving the violation management information corresponding to the target physical entity according to the graph knowledge base, and recording it as the first violation management information; Use the pre-trained deep learning convolutional neural network model to process the image to be tested and extract the target feature vector of the image to be tested; Retrieving the violation management information corresponding to the target feature vector according to the image-text vector library, and recording it as the second violation management information; The first violation management information and the second violation management information are processed using a multimodal large model to obtain final violation management information.

[0076] The above disclosure is only a preferred embodiment of the present invention, but the present invention is not limited thereto. Any non-creative changes that can be thought of by a person skilled in the art, as well as several improvements and modifications made without departing from the principle of the present invention, should fall within the protection scope of the present invention.

Claims

1. A construction site violation detection and management method, characterized in that: The following steps are involved: Construct a graph knowledge base and a graph-text vector library. The graph knowledge base stores the relationship between physical entities and violation management information, and the graph-text vector library stores the relationship between image feature vectors and violation management information. Use the pre-trained target detection model to process the image to be tested and extract the target physical entity in the image to be tested; Retrieving the violation management information corresponding to the target physical entity according to the graph knowledge base, and recording it as the first violation management information; Use the pre-trained deep learning convolutional neural network model to process the image to be tested and extract the target feature vector of the image to be tested; Retrieving the violation management information corresponding to the target feature vector according to the image-text vector library, and recording it as the second violation management information; The first violation management information and the second violation management information are processed using a multimodal large model to obtain final violation management information.

2. The construction site violation detection and management method according to claim 1, characterized in that: Building a graph knowledge base includes: Constructing a construction site map knowledge graph, wherein the construction site map knowledge graph includes nodes and edge relationships, wherein the nodes include physical entity nodes and violation management information nodes, wherein the physical entity nodes include entity attributes and names of the physical entities, and the violation management information nodes include file names; Store the construction site map knowledge graph in the Neo4j database.

3. The construction site violation detection and management method according to claim 2, characterized in that: The edge relationships of the construction site map knowledge graph include spatial relationships and semantic relationships. Spatial relationships are used to describe the relative position, size ratio, and color similarity between physical entity nodes, and semantic relationships are used to describe the text description and semantic information between violation management information nodes.

4. The construction site violation detection and management method according to claim 1, characterized in that: Use the pre-trained target detection model to process the image to be tested and extract the target physical entity in the image to be tested, including: Using the pre-trained Grounding DINO model to process the image to be tested, and extracting the first physical entity in the image to be tested; The image to be tested is processed using a pre-trained Faster RCNN neural network model to extract a second physical entity from the image to be tested; The image to be tested is processed using the YOLO algorithm to extract the third physical entity in the image to be tested; The first physical entity, the second physical entity, and the third physical entity are merged to obtain a target physical entity.

5. The construction site violation detection and management method according to claim 4, characterized in that: The pre-trained Grounding DINO model is used to process the image to be tested and extract the first physical entity in the image to be tested, specifically including: Input the image to be tested into the image backbone network for feature extraction to obtain image features; Input relevant text description into the text backbone network for feature extraction to obtain text features; The text guide is constructed by the following formula, In the formula, Refers to the text guide, Refers to the text-guided network, Refers to the k similar image feature vectors, Refers to image features, Refers to text features; The image features, text features, and text guidance are input into a multimodal encoder to obtain a first physical entity.

6. The construction site violation detection and management method according to claim 1, characterized in that: The pre-trained deep learning convolutional neural network model contains 13 convolutional layers, 4 pooling layers, 1 fully connected layer, and 1 Softmax activation layer.

7. The construction site violation detection and management method according to claim 1, characterized in that: The first violation management information and the second violation management information are processed using a multimodal large model to obtain final violation management information, specifically including: Form a candidate set by multi-channel recall of the first violation management information and the second violation management information; Construct Prompt prompt information; According to the prompt information, the GraphRAG model is used to retrieve and refine the candidate set to obtain the final violation management information.

8. A construction site violation detection and management system, characterized in that: include, Database construction module: construct a graph knowledge base and a graph-text vector library. The graph knowledge base stores the relationship between physical entities and violation management information, and the graph-text vector library stores the relationship between image feature vectors and violation management information. Target physical entity extraction module: uses the pre-trained target detection model to process the image to be tested and extracts the target physical entity in the image to be tested; The first information retrieval module: retrieves the violation management information corresponding to the target physical entity according to the graph knowledge base, and records it as the first violation management information; Target feature vector extraction module: uses a pre-trained deep learning convolutional neural network model to process the image to be tested and extracts the target feature vector of the image to be tested; The second information retrieval module: retrieves the violation management information corresponding to the target feature vector according to the image-text vector library, and records it as the second violation management information; Violation management information acquisition module: uses a multimodal large model to process the first violation management information and the second violation management information to obtain the final violation management information.

9. A terminal, characterized in that: include: A memory, used for storing a construction site violation detection management program; A processor is used to implement the steps of the construction site violation detection management method as described in any one of claims 1-7 when executing the construction site violation detection management program.

10. A computer-readable storage medium, characterized in that: The readable storage medium stores a construction site violation detection management program, which, when executed by a processor, implements the steps of the construction site violation detection management method as described in any one of claims 1-7.

Images