A smart construction site safety monitoring method and system integrating multi-modal large models

Abstract

The present invention discloses a smart construction site safety monitoring method and system integrating a multimodal large model, and relates to the technical field of building construction. The method includes: based on a Linux system platform, using an Nginx server as a static website server in combination with a camera to request video stream data, a client establishes a connection with the Nginx server, configures according to a configuration file, sends response data according to the HTTP request of the client, and performs video online monitoring and image recognition real-time processing on the construction area. The present invention can accurately identify different scenes and ensure that monitoring meets actual needs through detailed scene division, targeted safety guidelines, and a few-sample scene recognition method based on a visual language model; through the coordination of a personnel and item wearing detection module and an item attribute detection module, a deep and comprehensive monitoring of personnel protective equipment and behavior is achieved; depth estimation and three-dimensional collision detection ensure full coverage of monitoring scenes.

Description

Technical Field

[0001] The present invention relates to the technical field of building construction, and in particular to a smart construction site safety monitoring method and system integrating a multi-modal large model. Background Art

[0002] The rapid development of artificial intelligence technology has promoted the process of intelligence in all walks of life, and the intelligence of construction sites is also gradually becoming popular; in the actual working environment of construction sites, the safety of construction workers is of vital importance, however, unsafe behavior has become a key factor in causing safety accidents. Although many safety regulations have been formulated and corresponding inspections have been arranged, due to the high complexity of the construction site scene, which covers a variety of construction areas, complex building structures, frequently changing personnel and equipment layouts, and many other factors, it is difficult for manual inspections to ensure safety around the clock and in all directions.

[0003] Although some existing technical solutions, such as: China Patent Publication No. CN112784821A discloses a construction site behavior safety detection and identification method and system based on YOLOv5, China Patent Publication No. CN114677640A discloses a smart construction site safety monitoring system and method based on machine vision, and China Patent Publication No. CN112200043B discloses a hazard source intelligent identification system and method for outdoor construction sites, etc., have utilized deep learning and computer vision technology to a certain extent, but there are still problems such as relatively single detection direction and failure to fully consider the complex and changeable construction site scenes; in actual construction sites, the equipment worn by workers will vary due to changes in tasks and environments, and the safety monitoring requirements and methods in different scenarios are also different, and existing methods are difficult to effectively adapt to this diversity; in addition, for temporary added specific scenes and equipment, the small number of samples makes identification difficult, and it is difficult to achieve comprehensive safety compliance monitoring;

[0004] Therefore, a smart construction site safety monitoring method and system integrating multi-modal large models are proposed. Summary of the invention

[0005] The purpose of the present invention is to provide a smart construction site safety monitoring method and system integrating a multi-modal large model to solve one of the problems raised in the above background technology.

[0006] In order to solve the above technical problems, a technical solution adopted in this application is: a smart construction site safety monitoring method integrating a multi-modal large model, comprising the following steps:

[0007] Step 1: Based on the Linux system platform, use the Nginx server as a static website server to combine the camera to request video stream data. The client establishes a connection with the Nginx server, configures it according to the configuration file, sends response data according to the client's HTTP request, and performs video online monitoring and image recognition real-time processing of the construction area;

[0008] Step 2: Divide the construction area into multiple scenarios and formulate safety guidelines for different scenarios;

[0009] Step 3: A few-sample scene recognition method based on the CLIP visual language model is used, and low-rank adaptive LoRA is used to improve scene recognition performance and perform scene recognition;

[0010] Step 4: Generate safety items according to the scenario based on the LLM large language model, and fill the semantic and reasoning gap between fine-grained requirements and safety items to obtain the visual attributes of safety items;

[0011] Step 5: Based on the YOLO-World object detection model, generate bounding boxes for people in the scene, extract person image blocks, pass them and object prompt text to the VLM pre-trained visual language model to generate text and image embeddings, calculate the affinity matrix, and set the threshold;

[0012] Step 6: In response to the visual attributes of the security project of the LLM large language model, the YOLO-World target detection model is used to generate the object bounding box and extract the image block, and the object prompt text is passed to the VLM pre-trained visual language model to generate an embedding vector, calculate the cosine similarity, and set the threshold;

[0013] Step 7: Perform a few-sample depth estimation based on the teacher network and student network method to obtain a depth estimation result;

[0014] Step 8: Based on the Yolov7 detector, detect and locate the target in each frame, obtain the target frame and category probability, and classify it into falling and not falling to obtain the target detection result;

[0015] Step 9: Generate a 3D detection frame based on the depth estimation results and target detection results, and analyze the projection status of the two cuboids in the XOY plane and Z axis direction.

[0016] As a further preferred embodiment of the present technical solution: in step 1, the configuration according to the configuration file includes setting of the listening port, configuration of the virtual host and optimization adjustment of data transmission parameters according to the network bandwidth and server performance.

[0017] As a further preferred embodiment of the present technical solution: in step three, the sample data used by the few-sample scene recognition method based on the CLIP visual language model is screened and preprocessed, including removing blurred images with severe noise interference and normalizing the images.

[0018] As a further preferred embodiment of the present technical solution: in step four, the LLM large language model is a model that has been pre-trained and fine-tuned and optimized for the field of smart construction site safety, and the safety items generated by it are highly correlated with the actual construction site safety risks; the prompt template that guides the LLM large language model to summarize the visual attributes required for safety items is designed according to the characteristics of different safety items and scenarios.

[0019] As a further preferred embodiment of the present technical solution: in step five, the YOLO-World target detection model uses a data set containing rich category labels of common personnel, equipment, tools and safety protection supplies on construction sites during training, and the training parameters of the model are adjusted and optimized according to actual monitoring scenarios and needs.

[0020] As a further preferred embodiment of the present technical solution: in step six, the visual attributes of the security items responsive to the LLM large language model include color, shape, size, and texture features; when the YOLO-World target detection model is used to generate the object bounding box and extract the image blocks, an adaptive image segmentation algorithm is used to ensure that the extracted image blocks accurately contain the target objects and are representative.

[0021] As a further preferred embodiment of the present technical solution: in step seven, the method based on the teacher network and the student network is used to perform few-sample depth estimation, and the ratio of the labeled image set and the unlabeled image set is reasonably allocated according to the actual number of samples and the complexity of the scene, and the network structure of the teacher network and the student network is designed and optimized according to the characteristics of the depth estimation task, and the depth estimation is assisted by combining the semantic knowledge and spatial relationship information of the construction site scene.

[0022] As a further preferred embodiment of the present technical solution: in step eight, the model parameters of the Yolov7 detector are adjusted and optimized according to the scale distribution and appearance characteristics of the target objects on the construction site, including the adjustment of the size and proportion of the anchor frame, the optimization of the number of feature extraction network layers, and the threshold of the non-maximum suppression (NMS) is dynamically adjusted according to the accuracy and recall requirements of the target detection to balance the detection performance.

[0023] As a further preferred embodiment of the present technical solution: the method has the ability to adaptively adjust under different climatic conditions, and obtains ambient temperature, humidity, and light intensity information through sensors. If the environmental conditions change, the camera parameters, model detection thresholds, and data transmission parameters are automatically adjusted; the method also supports data interaction and integration with other construction site management systems, information sharing and collaborative work, transmits safety monitoring data to other systems through data interfaces, and receives relevant data from other systems at the same time, optimizing safety monitoring strategies and resource allocation.

[0024] To solve the above technical problems, another technical solution adopted by this application is: a smart construction site safety monitoring system integrating a multi-modal large model, including a video stream acquisition module, a scene division and safety guide module, a scene recognition module, a safety project generation module, a personnel and item wearing detection module, an item attribute detection module, a depth estimation module, a target detection and anomaly judgment module, a collision detection module and a system configuration and optimization module;

[0025] The video stream acquisition module is used to use the Nginx server as a static website server based on the Linux system platform to request video stream data in combination with a camera, and convert the RTSP original video stream into the HLS format through the FFmpeg audio and video processing framework and output the m3u8 format file and upload it to the Nginx server;

[0026] The scenario division and safety guide module is used to divide the construction area into multiple scenarios and formulate safety guides for different scenarios;

[0027] The scene recognition module is used to improve the scene recognition performance by using a few-sample scene recognition method based on the CLIP visual language model and combining it with low-rank adaptive LoRA;

[0028] The safety project generation module is used to generate necessary safety projects according to the scenario using the LLM large language model, and to fill the semantic and reasoning gap between fine-grained requirements and safety items;

[0029] The person-item wearing detection module is used in the YOLO-World target detection model to generate a bounding box for the person in the scene, extract the person image block, and determine whether the person is wearing the required items through the VLM pre-trained visual language model;

[0030] The item attribute detection module is used to obtain the visual attributes of safety products from the LLM large language model response, and use the YOLO-World target detection model and the VLM pre-trained visual language model to check whether the items worn by the person meet the attribute requirements;

[0031] The depth estimation module is used to perform few-sample depth estimation based on the teacher network and student network methods, and perform depth estimation with the help of semantic information-assisted perception;

[0032] The target detection and abnormality judgment module is used to detect and locate the target in each frame based on the Yolov7 detector, filter the target frame with high confidence through non-maximum suppression (NMS), and classify it into falling and not falling, and judge whether the operator is abnormal;

[0033] The collision detection module is used to generate a 3D detection frame based on the depth estimation and target detection results to determine whether the target object collides;

[0034] The system configuration and optimization module is used to set the configuration file of the Nginx server in the video stream acquisition module, including the setting of the listening port, the configuration of the virtual host and the optimization adjustment of the data transmission parameters, and optimize the model parameters and detection thresholds in other modules.

[0035] Advantages of the present invention:

[0036] 1. The present invention divides scenes into detailed categories, such as ordinary ground construction, aerial work, large equipment operation, material stacking and transportation, and formulates targeted safety guidelines. Combined with the few-sample scene recognition method based on the CLIP visual language model, the present invention can accurately identify different scenes, ensure that monitoring meets actual needs, and effectively solve the monitoring problems caused by complex scenes;

[0037] 2. The present invention uses a pre-trained and fine-tuned LLM large language model to generate highly relevant safety items according to the scenario, and summarizes visual attributes by designing prompt templates, thus bridging the semantic and reasoning gap between safety requirements and actual monitoring, and can adapt to changes in safety monitoring requirements in various scenarios;

[0038] 3. The present invention collaborates with the person-item wearing detection module and the item attribute detection module, based on technologies such as the YOLO-World target detection model and the VLM pre-trained visual language model, to not only determine whether a person is wearing items, but also check the attributes of the items worn, thereby achieving in-depth and comprehensive monitoring of personnel protective equipment and behaviors, avoiding the limitations of a single detection direction, and in addition, for dangerous areas of large equipment, using depth estimation and three-dimensional collision detection to achieve safety monitoring of construction personnel, ensuring full coverage of monitoring scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 A schematic diagram of a flow chart of a smart construction site safety monitoring method integrating a multi-modal large model according to the present invention;

[0041] Figure 2 This is a schematic diagram of the functional modules of a smart construction site safety monitoring system that integrates a multi-modal large model according to the present invention. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] Example

[0044] Figure 1 is a flow chart of a smart construction site safety monitoring method integrating a multi-modal large model according to an embodiment of the present invention. It should be noted that if there are substantially the same results, the method of the present application is not based on Figure 1 The process sequence shown is limited. Figure 1 As shown: A smart construction site safety monitoring method integrating a multi-modal large model includes the following steps:

[0045] Step 1. Based on the Linux system platform, use the Nginx server as a static website server to request video stream data in combination with the camera; use the FFmpeg audio and video processing framework to obtain the stream from the camera, convert the RTSP original video stream into HLS format, output the m3u8 format file and upload it to the Nginx server; the client establishes a connection with the Nginx server, configures it according to the configuration file, sends response data according to the client's HTTP request, and performs video online monitoring and image recognition real-time processing; among them, the configuration file (located in / etc / nginx / nginx.conf) can be used to set the listening port, virtual host, etc.; during configuration, the data transmission parameters can also be optimized and adjusted according to the network bandwidth and server performance, such as appropriately reducing the video resolution when the network bandwidth is low to ensure smooth data transmission.

[0046] Step 2: Divide the construction area into multiple scenarios, including ordinary ground construction scenarios, aerial work scenarios, large equipment operation scenarios, material stacking and transportation scenarios, and formulate safety guidelines for different scenarios, including personnel protective equipment requirements, equipment operation specifications, and environmental safety precautions; among them, in ordinary construction work areas, provide safety guidelines for wearing safety helmets and reflective clothing, as well as prohibiting irregular behaviors such as smoking and making phone calls; provide safety buckles, safety belts and other protective equipment testing requirements in aerial work areas; large equipment dangerous areas warn construction workers in dangerous areas such as directly under the equipment; material stacking and transportation scenarios clearly define safety specifications such as material stacking height, spacing, and transportation vehicle routes, speed limits, etc.

[0047] Step 3: Use the few-shot scene recognition method based on the CLIP visual language model and apply low-rank adaptive LoRA to improve scene recognition performance. The CLIP visual language model consists of an image and text encoder. The test image is embedded in the visual features. The text feature representation is obtained by appending the category name in the dataset to a specific prompt and inputting it into the language encoder. The cosine similarity score between the image and text features is calculated to determine the probability of the category to which the image belongs, and perform scene recognition.

[0048] Specifically, the CLIP visual language model consists of an image encoder and a text encoder, both of which are jointly trained through a large number of image-text pairs so that they can be well aligned in the feature space, where the image encoder converts images into visual feature vectors, while the text encoder converts text (such as category names) into text feature vectors; the test image is input into the image encoder to obtain its visual feature vector; the category names in the dataset (such as "high-altitude work scene", "ordinary ground construction scene", etc.) are combined with specific prompts (such as "a photo of...") to form a complete text description (such as "a photo of high-altitude work"), and these text descriptions are input into the text encoder to obtain its text feature vector; for each test image, the cosine similarity score between its visual feature vector and all category text feature vectors is calculated, and the probability of the category to which the test image belongs is determined based on the cosine similarity score, and the category with the highest similarity is selected as the recognition result; the performance of the model on the scene recognition task is evaluated using the validation set, including indicators such as accuracy and recall, and according to the evaluation results, the fine-tuning parameters of the low-rank adaptive LoRA (such as learning rate, number of iterations, etc.) are adjusted to further optimize the model performance;

[0049] Among them, low-rank adaptive LoRA is a low-rank adaptive fine-tuning technology, which fine-tunes the model by adding a low-rank matrix based on the original model weights. This method can significantly reduce computing resources and storage requirements;

[0050] The specific implementation method of low-rank adaptive LoRA is: apply low-rank adaptive LoRA on the image encoder or text encoder (or both) of the CLIP visual language model; fine-tune the low-rank adaptive LoRA using a small amount of scene recognition task data to optimize the performance of the model on specific tasks; since low-rank adaptive LoRA only adjusts the low-rank matrix, most of the weights of the original model can be kept unchanged, thereby improving performance without significantly increasing the computational burden.

[0051] Step 4: Generate necessary safety items according to the scenario based on the LLM large language model, fill the semantic and reasoning gap between fine-grained requirements and safety items, and obtain the visual attributes of safety items; the specific process is to receive scene information from the image description model, query the LLM large language model to obtain the response containing safety items, and then guide the LLM large language model to summarize the visual attributes required for safety items;

[0052] Specifically, first, use the trained image description model (such as deep learning-based image classification or YOLO-World target detection model) to analyze the input scene image and extract scene information. This information can include scene types (such as high-altitude work areas, ordinary ground construction areas, material storage areas, etc.);

[0053] Then, the extracted scene information is used as input to make a query request to the LLM large language model. The LLM large language model generates a response containing necessary safety items based on the input scene information, combined with its internal knowledge base and semantic understanding capabilities. These safety items may include hard hats, safety belts, material stacking labels, etc. A list of safety items is extracted from the response of the LLM large language model, which will serve as the basis for subsequent steps;

[0054] In order to guide the LLM large language model to summarize the visual attributes required for security projects, a series of prompt templates need to be designed. These templates should be customized according to the characteristics of different security projects and scenarios to ensure that the LLM large language model can generate accurate and specific visual attribute requirements;

[0055] For example, for a hard hat, the prompt template might include;

[0056] “What are the common color requirements for hard hats?”;

[0057] “Does a helmet need specific logos or patterns in different scenarios?”;

[0058] The designed prompt template is used as input and a query request is made to the LLM large language model again. The LLM large language model generates the visual attribute requirements required for the safety project based on the prompt template and its internal semantic reasoning ability. Specific visual attribute requirements are extracted from the response of the LLM large language model, such as the color, logo, pattern, etc. of the helmet;

[0059] The extracted safety items and their visual attribute requirements are integrated to form a complete list of safety items and their visual attribute requirements, and the integrated information is verified to ensure that the safety items and their visual attribute requirements meet the needs of the actual scenario. If necessary, the response of the LLM large language model can be adjusted or supplemented.

[0060] Step 5: Based on the YOLO-World target detection model, generate bounding boxes for people in the scene, extract person image blocks, pass them and item prompt text to the VLM pre-trained visual language model to generate text and image embedding, calculate affinity matrix, and set thresholds; determine whether the person is wearing the required items by setting the threshold;

[0061] Specifically, the trained YOLO-World target detection model is used to detect targets on the input scene images. During the training process, the model uses a dataset containing rich category labels of common construction site personnel, equipment, tools, and safety protection products, and adjusts and optimizes the training parameters according to the actual monitoring scene and needs. The YOLO-World target detection model generates bounding boxes for the people in the scene. These bounding boxes define the position and size of each person in the image. According to the generated bounding boxes, the corresponding person image blocks are extracted from the original image.

[0062] Prepare a text list containing the names of the categories of objects to be detected, such as helmets, seat belts, etc.; use the pre-trained visual language model VLM, take the extracted person image blocks and object prompt text as input, and generate corresponding text and image embeddings. These embeddings are high-dimensional vectors that can capture the semantic relationship between images and texts;

[0063] Suppose the character image block list is ( is the number of people in the image), the item prompt text list is ( is the number of item categories to be detected), the process of calculating the affinity matrix is ​​as follows:

[0064] ;

[0065] in, , , represents the vector dot product, represents the vector norm;

[0066] By setting the threshold Determine whether the person is wearing the required items:

[0067] ;

[0068] in, Indicates Does the individual wear Predictions for items.

[0069] Step 6: In response to the visual attributes of the safety products from the LLM large language model, the YOLO-World target detection model is used to generate the object bounding box and extract the image block, and the object prompt text is combined and passed to the VLM pre-trained visual language model to generate an embedding vector, calculate the cosine similarity, and set the threshold; by setting the threshold, check whether the items worn by the person meet the attribute requirements;

[0070] Specifically, the LLM large language model is used to generate the visual attributes of safety products according to scene information or safety requirements, including color, shape, size, texture features, etc.; the YOLO-World target detection model is used to detect targets on the input scene images and generate bounding boxes for the detected objects; when extracting image blocks, an adaptive image segmentation algorithm is used to ensure that the extracted image blocks accurately contain the target objects and are representative, which may require considering factors such as the size and shape of the object and its position in the image;

[0071] Suppose the list of item image blocks is ( is the number of items detected), the attribute prompt text list is ( is the number of item attribute categories), calculate the cosine similarity The process is as follows:

[0072] ;

[0073] in, , ;

[0074] Setting the Threshold Check whether the items worn by the person meet the property requirements:

[0075] ;

[0076] in, Indicates Predict whether an item meets the jth attribute requirement.

[0077] Step 7: Perform a few-sample depth estimation based on the teacher network and student network method to obtain a depth estimation result; wherein, the teacher network is trained on a labeled image set, the depth value is converted to a disparity space and normalized, and multiple data sets are jointly trained using an affine invariance loss. After the teacher model is trained, an unlabeled image set is predicted, and a student model is trained using a combination of labeled and unlabeled images, and depth estimation is performed using semantic information to assist perception;

[0078] Specifically, first, prepare a set of images containing labels (i.e., with depth information). These labels can be obtained through laser scanners, stereo vision or other depth sensing technologies; convert the depth values ​​in the image set into disparity space, which describes the difference in relative position between two points in the scene and is inversely proportional to the depth value. Then, normalize the depth value to the range of 0-1 on each depth map for subsequent processing; in order to improve the generalization ability of the model on different data sets, use affine invariance loss for joint training of multiple data sets. Affine invariance loss can capture the geometric transformation relationship between images, thereby enhancing the robustness of the model to image deformation;

[0079] The teacher network is trained on a set of labeled images, and the depth values ​​are obtained by Convert to disparity space and then normalize to 0-1 on each depth map, using affine invariance loss Perform joint training on multiple data sets, and the calculation formula is:

[0080] ;

[0081] in, is the sample size, is the number of image pixels, is the predicted depth value, is the real depth value;

[0082] After the teacher network is trained, it is used to predict the depth values ​​of the unlabeled image set. These unlabeled image sets can be images from different scenes or different devices to increase the generalization ability of the model; the labeled image set (and its true depth values) and the unlabeled image set (and its depth values ​​predicted by the teacher network) are combined into a new training set, and then the student model is trained using this combined set; in order to further improve the accuracy of depth estimation, semantic information can be used to assist perception. For example, a semantic segmentation model can be used to obtain object category information in the image, and this information can be used as additional input or constraints to guide the training of the student model; the student model can be a deep convolutional neural network that is lighter or more efficient than the teacher network. Through training, the student model can learn the mapping relationship from image to depth value and can perform accurate depth estimation with few samples.

[0083] Step 8: Based on the Yolov7 detector, detect and locate the target in each frame, obtain the target frame and category probability, and classify it into falling and not falling to obtain the target detection result;

[0084] Specifically, the Yolov7 detector is selected as the target detector because it is efficient and accurate and is suitable for real-time target detection tasks. The Yolov7 detector is used to detect targets in each frame of a video or image sequence to obtain the target box (i.e., the rectangular box surrounding the target) and the category probability (i.e., the probability that the target belongs to each category). NMS is used to solve the common overlapping box problem in target detection, that is, when multiple detection boxes overlap, only one optimal detection box is retained.

[0085] Among them, the NMS calculation function is optimized as follows:

[0086] ;

[0087] in, Indicates A judgment on whether a detection box is retained (1 means retention, 0 means suppression), For the The score of the detection box, is the score of the detection box with the highest score in the current frame, is the NMS threshold;

[0088] The specific implementation process of NMS is as follows:

[0089] Sort all detection boxes by score from high to low;

[0090] Starting from the detection box with the highest score, traverse each detection box in turn;

[0091] For the current detection box, calculate its overlap with the retained detection box (usually using the intersection over union (IoU) as a metric);

[0092] If the overlap between the current detection frame and the retained detection frame exceeds the NMS threshold, the current detection frame is suppressed (i.e., it is set not to be retained);

[0093] Otherwise, keep the current detection frame;

[0094] Repeat the above process until all detection boxes are traversed.

[0095] Finally, based on the detection boxes and category probabilities retained after NMS screening, the targets are classified as fallen or not fallen. This is usually achieved by setting a category probability threshold, that is, when the probability of a certain category exceeds the threshold, the target is classified as this category; based on the classification results, it is determined whether the operator is abnormal. If a target of the fall category is detected, the operator is judged to be abnormal; otherwise, it is judged to be normal.

[0096] Step 9: Generate a 3D detection frame based on the depth estimation and target detection results, and analyze the projection status of the two cuboids in the XOY plane and Z axis direction;

[0097] Specifically, first, the depth information of each point in the scene is obtained by using a depth estimation algorithm (as described in step 7), and the 2D bounding box (i.e., target box) and category information of the target object are obtained by combining with a target detection algorithm (such as the Yolov7 detector described in step 8);

[0098] Then, based on the depth estimation result and the 2D coordinates of the target box, the target box is transformed into a cuboid in 3D space, which usually involves converting 2D pixel coordinates into 3D world coordinates and taking into account the intrinsic and extrinsic parameters of the camera;

[0099] Next, based on the converted 3D coordinates, a 3D detection box of the target object is constructed. Each 3D detection box is defined by the coordinates of the eight vertices of a cuboid.

[0100] Finally, calculate the XOY plane projections of the two cuboids, that is, their rectangular areas on the horizontal plane, and calculate the intersection area and union area of ​​the two rectangles; if the ratio of the intersection area to the union area is greater than or equal to a certain intersection threshold (such as 0.1, the specific value can be adjusted according to the actual situation), it is considered that the projections of the two cuboids on the XOY plane intersect. At this time, it is necessary to continue to determine whether their projections in the Z-axis direction intersect;

[0101] Z-axis projection intersection judgment:

[0102] According to the depth estimation result, the interval of the two cuboids on the Z axis (that is, the minimum and maximum values ​​of the cuboids on the Z axis) is calculated; if the two intervals overlap, that is, the minimum value of one cuboid is less than the maximum value of the other cuboid, and the maximum value of one cuboid is greater than the minimum value of the other cuboid, then it is determined that the projections on the Z axis intersect; if the projections of the two cuboids on the XOY plane intersect, and the projections on the Z axis also intersect, then it is determined that the two target objects collide;

[0103] Among them, the intersection of the XOY plane projection can be determined by calculating the ratio of the intersection area of ​​the two rectangles to the union area ,like ( is the intersection threshold), it is considered that the projections on the XOY plane intersect, and the Z-axis projections are judged to be intersected; the Z-axis projection intersection judgment can be calculated based on the depth estimation result to calculate whether the intervals of the two cuboids on the Z axis have overlapping parts. If there is overlap, it is judged as a collision.

[0104] In this embodiment, specifically: in step 1, configuration is performed according to the configuration file, including setting of the listening port, configuration of the virtual host, and optimization and adjustment of the data transmission parameters according to the network bandwidth and server performance;

[0105] The configuration of the listening port includes port selection and port listening range;

[0106] Port selection: Select a suitable listening port for the system. This port will serve as the entrance for the system to communicate with the outside world. Therefore, choosing a port that is not commonly used and not easily guessed by attackers can improve the security of the system.

[0107] Port monitoring range: According to actual needs, configure whether the system monitors a specific IP address or all available network interfaces;

[0108] The configuration of the virtual host includes domain name and IP mapping, document root directory and access control;

[0109] Domain name and IP mapping: In the configuration file, set the corresponding domain name and IP address for each virtual host. This helps the system to correctly route the request to the corresponding virtual host according to the requested domain name or IP address.

[0110] Document root directory: Specify a document root directory for each virtual host, which will be the default location where the server responds to requests;

[0111] Access control: configure the access rights of the virtual host, including allowing or prohibiting access by specific IP addresses or domain names, and setting access authentication methods, etc.

[0112] Optimization and adjustment of data transmission parameters include network bandwidth considerations, server performance optimization, caching mechanisms, and compression and decompression;

[0113] Network bandwidth considerations: Adjust the data transmission rate and number of concurrent connections based on the actual network bandwidth. If the network bandwidth is limited, reducing the transmission rate or limiting the number of concurrent connections can avoid network congestion and data loss.

[0114] Server performance optimization: Optimize the relevant parameters of data transmission according to the server's hardware configuration and performance. For example, on a high-performance server, the transmission rate and number of concurrent connections can be appropriately increased to fully utilize the server's processing power.

[0115] Cache mechanism: Configure the cache mechanism to reduce the transmission of duplicate data. By caching static resources or frequently accessed data, you can reduce the server load and network bandwidth usage.

[0116] Compression and decompression: Enable data compression to reduce the size of transmitted data, thereby speeding up data transmission and reducing network latency. At the same time, ensure that the server can correctly decompress the received data to restore the original content.

[0117] In this embodiment, specifically: in step three, the sample data used in the few-sample scene recognition method based on the CLIP visual language model is screened and preprocessed, including removing blurred images and images with severe noise interference, and normalizing the images; due to factors such as shooting equipment, ambient light or motion blur, the image may become blurred. These blurred images are useless for scene recognition and may even mislead the model. Therefore, these blurred images need to be removed; the noise in the image may come from various sources, such as device noise, transmission errors or errors in the image processing process. The noise will interfere with the model's understanding of the image content, so it also needs to be removed; in order to enable the model to process image data more effectively, it is usually necessary to normalize the image. Normalization is to convert the pixel values ​​of the image into a standard range (such as 0 to 1 or -1 to 1), which helps the model converge faster and improve recognition performance.

[0118] Specific prompts are customized according to different scene categories and are targeted and representative. Specific prompts are customized according to different scene categories. These prompts are designed to help the model better understand the image content and distinguish different scene categories. By providing key information related to the scene, the prompts can guide the model to focus on important features in the image. Among them, customized prompts should be representative and accurately reflect the core features of each scene category. For example, for the "high-altitude work" scene, prompts may include keywords such as "scaffolding", "elevator", "ladder", "hanging basket", "unloading platform" and "blue sky". These keywords are not only closely related to the high-altitude work scene, but also can help the model distinguish high-altitude work from other similar scenes (such as ordinary ground construction areas or dangerous areas of large equipment).

[0119] In this embodiment, specifically: in step 4, the LLM large language model is a model that has been pre-trained and fine-tuned and optimized for the field of smart construction site safety, and the safety items generated by the model are highly correlated with the actual safety risks of the construction site; the LLM large language model is first pre-trained with large-scale text data to capture the statistical laws and semantic information of the language. In order to apply the LLM large language model to the field of smart construction site safety, it is necessary to fine-tune and optimize it, which usually involves further training the model with text data related to smart construction site safety so that the model can better understand the professional terms and contextual information in this field; the fine-tuned and optimized LLM large language model can generate safety items that are highly correlated with the safety risks of smart construction sites. These safety items may include various safety measures, hidden danger inspection points, emergency plans, etc., aiming to improve the safety management level of the construction site; according to the characteristics and needs of the specific construction site, the LLM large language model can be guided to generate customized safety items, which helps to ensure that the generated safety items can effectively solve the safety problems faced by the construction site;

[0120] The prompt templates that guide the LLM large language model to summarize the visual attributes required for safety items are designed according to the characteristics of different safety items and scenarios; In order to guide the LLM large language model to summarize the visual attributes required for safety items, corresponding prompt templates need to be designed. These templates are intended to help the model understand the relationship between safety items and visual features, thereby generating more specific and practical suggestions; Prompt templates should be designed according to the characteristics of different safety items and scenarios. For example, for high-altitude work safety items, the template may include "Please describe the main visual features of the high-altitude work area, such as edge protection, safety net setting, etc."; For electrical safety items, the template may emphasize "Please list the key points of visual inspection of electrical equipment and its surrounding environment, such as cable laying, grounding devices, etc."

[0121] In this embodiment, specifically: in step 5, the YOLO-World target detection model uses a data set containing common construction site personnel, equipment, tools and safety protection products with rich category labels during training, and the training parameters of the model are adjusted and optimized according to the actual monitoring scenario and needs;

[0122] Specifically, in order to train the YOLO-World target detection model to identify common personnel, equipment, tools, and safety protection equipment on the construction site, it is necessary to select a dataset that contains these elements. This dataset should be diverse and representative enough to cover various situations that may occur on the construction site. Each image in the dataset should be carefully annotated to include all relevant category labels. These labels should accurately reflect the objects in the image and describe their properties and characteristics as detailed as possible. By providing rich category labels, the model can be helped to better learn the differences between different objects and improve recognition accuracy.

[0123] According to the actual monitoring scenarios and requirements, the training parameters of the YOLO-World target detection model need to be adjusted and optimized. For example, if the sizes of objects in the monitoring scenarios vary greatly, the size of the model's anchor boxes may need to be adjusted to accommodate objects of different sizes. The selection of training parameters should also take into account the performance requirements of the model. For example, if the model needs to be deployed in real-time applications, the model's inference speed may need to be optimized to reduce latency, which can be achieved by adjusting the complexity of the model, using more efficient algorithms, or using hardware acceleration technology. In order to prevent the model from overfitting during training, regularization techniques (such as L1 or L2 regularization) can be used to limit the complexity of the model. In addition, data enhancement techniques (such as rotation, scaling, flipping, etc.) can be used to increase the diversity of the data set, thereby improving the generalization ability of the model.

[0124] During the training process, the performance of the model needs to be evaluated regularly to understand whether it meets actual needs. Common evaluation indicators include accuracy, recall rate, F1 score, etc. These indicators can help discover problems and deficiencies in the model and provide direction for its improvement; based on the evaluation results, the model can be iteratively optimized, which may include adjusting training parameters, increasing the size or diversity of the data set, using more advanced algorithms, etc. Through continuous iteration and optimization, the performance and accuracy of the model can be gradually improved.

[0125] In this embodiment, specifically: in step six, the visual attributes of the safety items in response to the LLM large language model include color, shape, size, and texture features; this information is crucial for subsequent target detection and image block extraction because they help the model to more accurately identify image content related to safety products. In order to ensure the accuracy of the visual attribute information extracted from the LLM large language model response, the LLM large language model needs to be fully trained and verified. In addition, the output of the LLM large language model can be verified and corrected using manually annotated data sets to improve the accuracy of the attributes.

[0126] When using the YOLO-World object detection model to generate object bounding boxes and extract image blocks, an adaptive image segmentation algorithm is used to ensure that the extracted image blocks accurately contain the target objects and are representative.

[0127] The YOLO-World object detection model is used to detect target objects related to safety products in images. The model can use previously learned features to identify objects in images and generate corresponding bounding boxes. In order to improve the accuracy of bounding boxes, the YOLO-World object detection model can be fine-tuned to adapt to object detection tasks in specific scenarios. After generating bounding boxes, an adaptive image segmentation algorithm is used to extract image patches containing target objects. This algorithm can automatically adjust the segmentation strategy based on the position and size of the bounding box, as well as the shape and texture features of the target object, to ensure that the extracted image patches accurately contain the target object and are representative. In order to ensure that the extracted image patches are representative, it is necessary to ensure that they can fully reflect the key visual features of the target object, which can be achieved by considering factors such as the integrity, clarity, and contrast of the object when extracting patches. The extracted image patches can be used for a variety of subsequent applications, such as automatic identification, classification, and counting of safety products, which can further improve the efficiency and accuracy of construction site safety management. In practical applications, the performance of the LLM large language model, the YOLO-World object detection model, and the adaptive image segmentation algorithm can be continuously iterated and optimized as needed, which can include collecting more training data, adjusting model parameters, and introducing more advanced algorithms.

[0128] In this embodiment, specifically: in step seven, a few-sample depth estimation is performed based on the teacher network and the student network method, the ratio of the labeled image set and the unlabeled image set is reasonably allocated according to the actual number of samples and the complexity of the scene, and the network structure of the teacher network and the student network is designed and optimized according to the characteristics of the depth estimation task, and the depth estimation is assisted by combining the semantic knowledge and spatial relationship information of the construction site scene;

[0129] Specifically, the proportion of labeled image sets and unlabeled image sets needs to be reasonably planned according to the actual number of samples and scene complexity. When samples are scarce or the labeling cost is high, a reasonable allocation of labeled and unlabeled data can maximize the use of limited resources while maintaining the stability and accuracy of model training. As the training process progresses, the proportion of labeled and unlabeled image sets can be dynamically adjusted according to the performance feedback and actual needs of the model to further optimize the model performance.

[0130] The teacher network is usually a trained model with good performance, which is used to provide high-quality pseudo labels or guidance information. The student network is a model that is being trained and uses the guidance information provided by the teacher network for learning. In order to meet the challenges of depth estimation tasks, the network structures of the teacher network and the student network need to be designed and optimized according to the characteristics of the task, which may include using a deeper network architecture, introducing an attention mechanism, and adopting a specific loss function to improve the depth estimation ability of the model.

[0131] In the depth estimation process, combining the semantic knowledge of the construction site scene (such as object category, shape, spatial relationship, etc.) can significantly improve the model's perception ability. This semantic knowledge can help the model better understand the image content and thus more accurately estimate the depth information. In addition to semantic knowledge, spatial relationship information can also be used to assist depth estimation. For example, by analyzing the relative position and size relationship between objects in the image, the depth relationship between them can be inferred, thereby improving the accuracy of depth estimation.

[0132] In this embodiment, specifically: in step eight, the model parameters of the Yolov7 detector are adjusted and optimized according to the scale distribution and appearance characteristics of the target object on the construction site, including the adjustment of the size and proportion of the anchor frame, the optimization of the number of feature extraction network layers, and the threshold of the non-maximum suppression NMS is dynamically adjusted according to the accuracy and recall requirements of the target detection to balance the detection performance;

[0133] Specifically, the anchor box is the initial reference box used by the model to generate the bounding box in the prediction stage. Its size and proportion directly affect the model's detection accuracy of the target object. In addition to scale distribution, it is also necessary to consider the appearance features of the target object, such as shape, texture, etc., and extract them into high-dimensional semantic information through a deep network. This information can help the model locate the target object more accurately and reduce the possibility of false detection; according to the complexity of the target objects on the construction site and the requirements of the detection task, select the appropriate number of feature extraction network layers. A deeper network can extract richer feature information, but it may also lead to increased computation and the risk of overfitting. Therefore, it is necessary to reasonably control the depth of the network while ensuring detection accuracy; in addition to adjusting the number of network layers, feature fusion strategies can also be used to improve the detection performance of the model. By fusing feature information at different levels, target object information of different scales can be fully utilized to improve the robustness of the model;

[0134] Non-maximum suppression (NMS) is a commonly used post-processing step in target detection, which is used to eliminate overlapping bounding boxes and retain the best detection results. The threshold of NMS directly affects the precision and recall rate of the model. A higher threshold can reduce false detections but may lead to missed detections; a lower threshold may increase false detections but improve the recall rate. In order to balance the detection performance, the threshold of NMS can be dynamically adjusted according to the needs of the actual scenario and the performance of the model. This can be achieved by monitoring the precision and recall rate indicators of the model during training and adjusting the threshold of NMS according to the changing trends of these indicators. After adjusting the model parameters, the model needs to be evaluated to verify whether its detection accuracy, recall rate and other indicators meet the actual needs. This can be achieved by performing target detection tasks on an independent test set. According to the evaluation results, the model can be further optimized and iterated, including adjusting the model parameters, introducing new feature extraction methods, improving the NMS strategy, etc., to improve the detection performance of the model.

[0135] In this embodiment, specifically: the method has the ability to adaptively adjust under different climatic conditions, obtains ambient temperature, humidity, and light intensity information through sensors, and automatically adjusts camera parameters, model detection thresholds, and data transmission parameters if environmental conditions change; the method also supports data interaction and integration with other construction site management systems, information sharing and collaborative work, transmits safety monitoring data to other systems through data interfaces, and simultaneously receives relevant data from other systems to optimize safety monitoring strategies and resource allocation.

[0136] In summary, this solution has built a comprehensive, intelligent, efficient and adaptable smart construction site safety monitoring system through the deep integration and fine optimization of innovative technologies, providing a practical solution to solve the problems of traditional construction site safety management. At the technical level, the coordinated use of multimodal large models realizes the seamless connection of visual information and language information. The performance of the CLIP visual language model in few-sample scene recognition, after fine-tuning the low-rank adaptive LoRA technology for the engineering field, achieves high-precision scene classification at a low computing cost, laying the foundation for subsequent precise monitoring. The LLM large language model, with its powerful semantic understanding and generation capabilities, generates key safety items and detailed visual attributes based on the scene, further enriches the monitoring dimension, and effectively responds to the diversity and complexity of construction site safety needs. In terms of target detection models, the YOLO-World target detection model has more accurate detection of construction site personnel and objects under rich data set training and parameter optimization. In conjunction with the VLM pre-trained visual language model, both the judgment of the wearing of personnel protective equipment and the detailed inspection of object attributes show high accuracy, greatly reducing the risk of visual omissions in manual monitoring. Depth estimation and collision detection technology are applied to dangerous areas of large equipment. The teacher-student network architecture rationally allocates data resources and combines the semantic knowledge of construction site scenes to achieve reliable depth estimation under few sample conditions. The targeted optimization of the Yolov7 detector and the dynamically adjusted NMS threshold ensure accurate and efficient target detection. Finally, the collision risk is accurately judged through the 3D detection frame, which effectively guarantees the safety of construction workers working near large equipment. The adaptive adjustment capability of the system enables it to operate stably under different climatic conditions, automatically optimize camera parameters, model thresholds and data transmission, and ensure the continuity and reliability of monitoring. Data interaction and integration with other construction site management systems break down information barriers, realize resource sharing and collaborative operations, and further improve the overall safety management efficiency. This solution not only effectively improves the accuracy, comprehensiveness and real-time performance of smart construction site safety monitoring, but also shows significant advantages in model optimization, sample processing, adaptive adjustment, etc., promoting the development of smart construction site safety management towards a more intelligent and efficient direction. It has broad application prospects and significant social benefits, and is expected to become a key technical support in the field of smart construction site construction, helping to continuously improve the safety level of the construction industry.

[0137] Figure 2 is a functional module diagram of a smart construction site safety monitoring system integrating a multi-modal large model according to an embodiment of the present application, such as Figure 2As shown, a smart construction site safety monitoring system integrating a multi-modal large model includes a video stream acquisition module, a scene division and safety guide module, a scene recognition module, a safety project generation module, a personnel and item wearing detection module, an item attribute detection module, a depth estimation module, a target detection and anomaly judgment module, a collision detection module and a system configuration and optimization module;

[0138] The video stream acquisition module is used based on the Linux system platform, using the Nginx server as a static website server to request video stream data in combination with the camera, and converting the RTSP original video stream into HLS format through the FFmpeg audio and video processing framework and outputting the m3u8 format file and uploading it to the Nginx server;

[0139] The scenario division and safety guide module is used to divide the construction area into multiple scenarios and formulate safety guidelines for different scenarios;

[0140] The scene recognition module is used to improve the scene recognition performance by using the few-shot scene recognition method based on the CLIP visual language model and combining it with low-rank adaptive LoRA;

[0141] The safety project generation module is used to generate necessary safety projects according to the scenario using the LLM large language model and to fill the semantic and reasoning gap between fine-grained requirements and safety items;

[0142] The person and item wearing detection module is used to generate bounding boxes for people in the scene based on the open vocabulary YOLO-World object detection model, extract person image blocks, and determine whether people are wearing the required items through the pre-trained LLM large language model;

[0143] The item attribute detection module is used to obtain the visual attributes of the security item from the LLM large language model response, and use the YOLO-World target detection model and the LLM large language model to check whether the items worn by the person meet the attribute requirements;

[0144] Depth estimation module, which is used to perform few-sample depth estimation based on teacher network and student network methods, and to perform depth estimation with the help of semantic information-assisted perception;

[0145] The target detection and anomaly judgment module is used to detect and locate the target in each frame based on the Yolov7 detector, filter the target frame with high confidence through non-maximum suppression (NMS), and classify it into falling or not falling, and judge whether the operator is abnormal;

[0146] The collision detection module is used to generate a 3D detection frame based on the depth estimation and target detection results to determine whether the target object collides;

[0147] The system configuration and optimization module is used to set the configuration file of the Nginx server in the video stream acquisition module, including the setting of the listening port, the configuration of the virtual host, and the optimization and adjustment of the data transmission parameters, and optimize the model parameters and detection thresholds in other modules.

[0148] The embodiment of the present invention provides a smart construction site safety monitoring system that integrates a multimodal large model. Through the close collaboration of multiple functional modules, it forms an organic whole, providing a comprehensive, multi-level, and intelligent solution for the safety management of smart construction sites. The system integrates cutting-edge technologies such as multimodal large models, deep learning algorithms, and computer vision technology to achieve comprehensive perception, real-time monitoring, accurate early warning, and efficient management of the safety status of smart construction sites, effectively solving many problems existing in traditional construction site safety management, greatly improving the safety level and management efficiency of smart construction sites, and has significant innovation, practicality, and promotion value, providing strong technical support for the safe production and intelligent development of the construction industry.

[0149] For other details about the technical solutions for implementing each module in the system of the above-mentioned embodiment, please refer to the description of a smart construction site safety monitoring method integrating a multi-modal large model in the above-mentioned embodiment, which will not be repeated here.

[0150] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0151] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A smart construction site safety monitoring method integrating a multi-modal large model, characterized in that: The following steps are involved: Step 1: Based on the Linux system platform, use the Nginx server as a static website server to combine the camera to request video stream data. The client establishes a connection with the Nginx server, configures it according to the configuration file, sends response data according to the client's HTTP request, and performs video online monitoring and image recognition real-time processing of the construction area; Step 2: Divide the construction area into multiple scenarios and formulate safety guidelines for different scenarios; Step 3: A few-sample scene recognition method based on the CLIP visual language model is used, and low-rank adaptive LoRA is used to improve scene recognition performance and perform scene recognition; Step 4: Generate safety items according to the scenario based on the LLM large language model, and fill the semantic and reasoning gap between fine-grained requirements and safety items to obtain the visual attributes of safety items; Step 5: Based on the YOLO-World object detection model, generate bounding boxes for people in the scene, extract person image blocks, pass them and object prompt text to the VLM pre-trained visual language model to generate text and image embeddings, calculate the affinity matrix, and set the threshold; Step 6: In response to the visual attributes of the security project of the LLM large language model, the YOLO-World target detection model is used to generate the object bounding box and extract the image block, and the object prompt text is passed to the VLM pre-trained visual language model to generate an embedding vector, calculate the cosine similarity, and set the threshold; Step 7: Perform a few-sample depth estimation based on the teacher network and student network method to obtain a depth estimation result; Step 8: Based on the Yolov7 detector, detect and locate the target in each frame, obtain the target frame and category probability, and classify it into falling and not falling to obtain the target detection result; Step 9: Generate a 3D detection frame based on the depth estimation results and target detection results, and analyze the projection status of the two cuboids in the XOY plane and Z axis direction.

2. According to claim 1, a smart construction site safety monitoring method integrating a multi-modal large model is characterized in that: In step one, the configuration according to the configuration file includes setting of the listening port, configuration of the virtual host, and optimization adjustment of data transmission parameters according to the network bandwidth and server performance.

3. According to claim 1, a smart construction site safety monitoring method integrating a multi-modal large model is characterized in that: In step three, the sample data used by the few-sample scene recognition method based on the CLIP visual language model is screened and preprocessed, including removing blurry and noisy images and normalizing the images.

4. According to claim 1, a smart construction site safety monitoring method integrating a multi-modal large model is characterized in that: In step 4, the LLM large language model is a pre-trained model optimized for the field of smart construction site safety, and the safety items generated by it are related to the actual construction site safety risks; The prompt template that guides the LLM large language model to summarize the visual attributes required for the safety project is designed according to the characteristics of different safety projects and scenarios.

5. According to claim 1, a smart construction site safety monitoring method integrating a multi-modal large model is characterized in that: In step five, the YOLO-World target detection model uses a data set containing labels of common personnel, equipment, tools and safety protection equipment on the construction site during the training process, and the training parameters of the model are adjusted and optimized according to the actual monitoring scenarios and needs.

6. The method for intelligent construction site safety monitoring integrating multi-modal large models according to claim 1 is characterized in that: In step six, the visual attributes of the security item responsive to the LLM large language model include color, shape, size, and texture features; The YOLO-World target detection model is used to generate object bounding boxes and extract image blocks to achieve fine-grained attribute verification.

7. The method for intelligent construction site safety monitoring integrating multi-modal large models according to claim 1 is characterized in that: In step seven, the teacher network and student network based method is used to perform few-sample depth estimation. The ratio of labeled image sets and unlabeled image sets is allocated according to the actual number of samples and the complexity of the scene. The network structures of the teacher network and the student network are designed and optimized according to the characteristics of the depth estimation task, and depth estimation is assisted by combining the semantic knowledge and spatial relationship information of the construction site scene.

8. The method for intelligent construction site safety monitoring integrating multi-modal large models according to claim 1 is characterized in that: In step eight, the model parameters of the Yolov7 detector are adjusted and optimized according to the scale distribution and appearance characteristics of the target objects on the construction site, including the adjustment of the size and proportion of the anchor box and the optimization of the number of feature extraction network layers.

9. The method for intelligent construction site safety monitoring integrating multi-modal large models according to claim 1 is characterized in that: This method has the ability to adaptively adjust under different climatic conditions. It obtains ambient temperature, humidity, and light intensity information through sensors. If the environmental conditions change, the camera parameters, model detection thresholds, and data transmission parameters are automatically adjusted.

10. A smart construction site safety monitoring system integrating a multi-modal large model, applied to a smart construction site safety monitoring method integrating a multi-modal large model according to any one of claims 1 to 9, characterized in that: It includes video stream acquisition module, scene division and safety guide module, scene recognition module, safety project generation module, personnel and item wearing detection module, item attribute detection module, depth estimation module, target detection and abnormality judgment module, collision detection module and system configuration and optimization module; The video stream acquisition module is used to use the Nginx server as a static website server based on the Linux system platform to request video stream data in combination with a camera, and convert the RTSP original video stream into the HLS format through the FFmpeg audio and video processing framework and output the m3u8 format file and upload it to the Nginx server; The scenario division and safety guide module is used to divide the construction area into multiple scenarios and formulate safety guides for different scenarios; The scene recognition module is used to improve the scene recognition performance by using a few-sample scene recognition method based on the CLIP visual language model and combining it with low-rank adaptive LoRA; The safety project generation module is used to generate necessary safety projects according to the scenario using the LLM large language model, and to fill the semantic and reasoning gap between fine-grained requirements and safety items; The person-item wearing detection module is used in the YOLO-World target detection model to generate a bounding box for the person in the scene, extract the person image block, and determine whether the person is wearing the required items through the VLM pre-trained visual language model; The item attribute detection module is used to obtain the visual attributes of the security item from the LLM large language model response, and use the YOLO-World target detection model and the VLM pre-trained visual language model to check whether the items worn by the person meet the attribute requirements; The depth estimation module is used to perform few-sample depth estimation based on the teacher network and student network methods, and perform depth estimation with the help of semantic information-assisted perception; The target detection and abnormality judgment module is used to detect and locate the target in each frame based on the Yolov7 detector, filter the target frame with high confidence through non-maximum suppression (NMS), and classify it into falling and not falling, and judge whether the operator is abnormal; The collision detection module is used to generate a 3D detection frame based on the depth estimation and target detection results to determine whether the target object collides; The system configuration and optimization module is used to set the configuration file of the Nginx server in the video stream acquisition module, including the setting of the listening port, the configuration of the virtual host and the optimization adjustment of the data transmission parameters, and optimize the model parameters and detection thresholds in other modules.

Images