Worker safety protective equipment absence detection method based on multi-modal large model

Abstract

The application relates to the technical field of computer vision, and discloses a worker safety protective equipment missing detection method based on a multi-modal large model, which comprises the following steps: acquiring an original construction site image, performing worker target detection and candidate region extraction to obtain a worker information image set and an original target boundary box coordinate set; inputting the worker information image set into the multi-modal large model to perform safety equipment missing identification, and obtaining an answer list; further performing semantic analysis and classification on the answer list, screening out negative sample indexes with violations; and finally, based on the negative sample indexes and the original boundary box coordinates, performing violation event positioning and visual alarm generation identification on the original image. In this way, by converting the visual recognition task into a multi-modal inference task and utilizing the generalization and inference ability of the large model, the problem of low detection accuracy and poor generalization ability of traditional methods in complex and occluded scenes is effectively solved, and the robustness and reliability of detection are improved.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and more specifically, to a method for detecting missing worker safety protective equipment based on a multimodal large model. Background Technology

[0002] In construction site safety systems, safety helmets and reflective vests are fundamental protective equipment for ensuring workers' lives, and their importance cannot be ignored. If workers fail to wear safety helmets or reflective vests as required, they will be directly exposed to the risks of potential physical impacts and insufficient traffic warnings, which can easily lead to serious consequences. Therefore, real-time and effective automatic detection of workers' safety equipment wearing status has become a key aspect of smart construction site safety management.

[0003] Currently, methods for detecting the lack of safety protective equipment for workers mainly rely on computer vision technology, particularly schemes based on fine-tuning of pre-trained object detection models (such as the YOLO series models). These methods typically train the model on specific datasets collected at construction sites to enable it to identify workers in images and whether they are wearing safety equipment. However, this traditional detection method based on a single-modal vision model has revealed several drawbacks in practical applications. First, to learn comprehensive visual features, these methods often require massive amounts of labeled data for fine-tuning, which is not only time-consuming and labor-intensive but also introduces a high application cost barrier. Second, the generalization ability of these methods is relatively poor. When real-world scenarios present safety helmet colors, reflective vest styles, or worker postures that have not been seen or are only seen in limited quantities in the training data, the model's recognition performance will significantly decrease, leading to a high false positive rate. More importantly, real construction environments are often complex, and key parts of the worker's body (such as the head and upper torso) are frequently partially obscured by equipment, materials, or other personnel, resulting in incomplete samples. In situations where direct judgment is difficult, traditional small models struggle to learn and reason effectively, resulting in severely insufficient detection accuracy in complex occlusion scenarios, making it difficult to meet the needs of actual security supervision. Summary of the Invention

[0004] To address the aforementioned technical problems, this application is proposed. Embodiments of this application propose a method for detecting missing worker safety protective equipment based on a multimodal large model, overcoming the problems of high fine-tuning costs, insufficient generalization ability, and poor detection accuracy in complex construction environments inherent in traditional detection methods described in the background.

[0005] According to one aspect of this application, a method for detecting missing worker safety protective equipment based on a multimodal large model is provided, comprising the following steps: acquiring original construction site images; performing worker target detection and candidate region extraction on the original construction site images to obtain a set of worker information images and a set of original target bounding box coordinates; performing multimodal recognition of missing safety equipment on the set of worker information images to obtain a list of responses indicating missing worker safety protective equipment; performing semantic parsing and classification of the recognition results on the list of responses indicating missing worker safety protective equipment to obtain a set of negative sample indices; and, based on the set of negative sample indices and the set of original target bounding box coordinates, performing violation event localization and visual alarm on the original construction site images to obtain a worker safety protective equipment missing recognition map.

[0006] In one possible implementation, worker target detection and candidate region extraction are performed on the original construction site image to obtain a set of worker information images and an original target bounding box coordinate set. This includes: inputting the original construction site image into a pre-trained worker detection model to obtain an initial target bounding box coordinate set as the original target bounding box coordinate set; and based on the initial target bounding box coordinate set, acquiring candidate region images from the original construction site image to obtain the worker information image set.

[0007] In one possible implementation, candidate region images are acquired from the original construction site image based on the initial target bounding box coordinate set to obtain a set of worker information images. This includes: contextually expanding each initial target bounding box in the initial target bounding box coordinate set based on the size information of the original construction site image to obtain an expanded target bounding box coordinate set; and acquiring candidate region images from the original construction site image based on the expanded target bounding box coordinate set to obtain a set of worker information images.

[0008] In one possible implementation, based on the size information of the original construction site image, contextual expansion is performed on each initial target bounding box in the initial target bounding box coordinate set to obtain an expanded target bounding box coordinate set. This includes: contextual expansion of each initial target bounding box in the initial target bounding box coordinate set using the following formula: ;in, These are the coordinates of the top left and bottom right corners of the adjusted target bounding box. It is the minimum value along the X-axis of the image. It is the minimum value in the Y-axis direction of the image. The maximum value along the X-axis of the image is the width of the image. The maximum value along the Y-axis of the image is the height of the image.

[0009] In one possible implementation, performing multimodal recognition of missing safety equipment on a set of worker information images to obtain a list of responses indicating missing worker safety protective equipment includes: constructing multimodal input pairs based on the set of worker information images to obtain a batch of multimodal large model inputs; and inputting the batch of multimodal large model inputs into a fine-tuned multimodal large model to obtain the list of responses indicating missing worker safety protective equipment.

[0010] In one possible implementation, multimodal input pairs are constructed based on a set of worker information images to obtain a large batch of multimodal model inputs, including pairing each worker information image with a preset question text for asking whether the worker is missing safety protective equipment.

[0011] In one possible implementation, the list of responses to the test worker's missing safety protective equipment is semantically parsed and classified to obtain a negative sample index set. This includes: performing semantic parsing on each response text in the list of responses to the test worker's missing safety protective equipment to obtain a classification result list; and filtering the classification result list using negative sample indexes to obtain a negative sample index set.

[0012] In one possible implementation, based on the negative sample index set and the original target bounding box coordinate set, violation event localization and visualization alarm are performed on the original construction site image to obtain a worker safety protection equipment missing identification map, including: localizing the violation target based on the negative sample index set and the original target bounding box coordinate set to obtain the violation target bounding box coordinate set; and performing a drawing operation on the original construction site image based on the violation target bounding box coordinate set to obtain the worker safety protection equipment missing identification map.

[0013] Compared with existing technologies, the worker safety protection equipment missing detection method based on a multimodal large model provided in this application first benefits from the massive pre-training knowledge of the multimodal large model. This method has strong generalization ability and is more adaptable to changes in safety helmet color, reflective vest style, and worker posture that have not appeared in the training data, thus effectively overcoming the high false alarm rate problem caused by scene changes in traditional methods. Secondly, when dealing with complex scenes, especially when the worker's body is partially occluded and the information is incomplete, this method can make more reasonable judgments by utilizing the model's reasoning ability, such as generating conclusions that cannot be judged, rather than forcibly making incorrect classifications. This greatly improves the detection accuracy and reliability in complex construction environments. In addition, compared with traditional single-modal models that require a large amount of labeled data for fine-tuning, this method utilizes the few-sample learning ability of the large model, significantly reducing the fine-tuning cost for specific tasks. Attached Figure Description

[0014] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0015] Figure 1 This is a schematic flowchart illustrating a method for detecting missing worker safety protective equipment based on a multimodal large model, according to an embodiment of this application.

[0016] Figure 2 This is a schematic diagram of the identification and detection process in a real-time scenario according to an embodiment of this application.

[0017] Figure 3 This is a schematic diagram of the worker detection module structure in an embodiment of this application.

[0018] Figure 4 This is a schematic diagram of the structure of the multimodal large model prediction module in an embodiment of this application.

[0019] Figure 5 This is a schematic diagram illustrating the process of fine-tuning a multimodal large model in an embodiment of this application. Detailed Implementation

[0020] Hereinafter, exemplary embodiments according to this application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein.

[0021] Figure 1 This is a schematic flowchart illustrating a method for detecting missing worker safety protective equipment based on a multimodal large model, according to an embodiment of this application. Figure 1 As shown, this application provides a method for detecting missing worker safety protection equipment based on a multimodal large model, including the following steps: S1, acquiring original construction site images; S2, performing worker target detection and candidate region extraction on the original construction site images to obtain a set of worker information images and a set of original target bounding box coordinates; S3, performing multimodal recognition of missing safety equipment on the set of worker information images to obtain a list of responses to detect missing worker safety protection equipment; S4, performing semantic parsing and classification of the recognition results on the list of responses to detect missing worker safety protection equipment to obtain a set of negative sample indices; S5, based on the set of negative sample indices and the set of original target bounding box coordinates, performing violation event localization and visual alarm on the original construction site images to obtain a worker safety protection equipment missing recognition map.

[0022] Please also see Figure 2The figure is a schematic diagram of the identification and detection process in a real-time scene according to an embodiment of this application. The process mainly consists of two core parts: a worker detection module and a multimodal large model prediction module.

[0023] For example, in step S1, the original construction site image is acquired. It should be understood that in actual monitoring scenarios, video stream data from construction site cameras is acquired via the RTSP (Real-Time Streaming Protocol), and the real-time video stream is captured to obtain the construction site image to be detected. This process samples continuous video signals at specific points in time, capturing a single frame as input for subsequent analysis, ensuring both real-time detection and the authenticity of the scene.

[0024] In one specific embodiment, to ensure image clarity and detail, a high-definition camera is typically used for data acquisition. For example, the acquired construction site image has a width and height of 1920*1080 pixels. Specifically, the display format of the construction site camera video stream is 1080P, and the size of the construction site image to be detected obtained through the above image capture processing is also 1920*1080 pixels. This resolution is sufficient to support subsequent algorithms to clearly and effectively identify individual workers and their safety protective equipment (such as safety helmets and reflective vests).

[0025] For example, in step S2, worker target detection and candidate region extraction are performed on the original construction site images to obtain a set of worker information images and a set of original target bounding box coordinates, such as... Figure 3 As shown, this step is performed by the worker detection module. It should be understood that raw construction site images are typically wide-angle, complex scenes that may contain a large amount of background information, such as building structures, construction equipment, material stacks, and multiple workers. Directly inputting such unprocessed raw images into a multimodal large model for detailed analysis presents two main technical problems: First, the model's attention will be distracted by irrelevant background information, making it difficult to focus on the clothing of individual workers, thus affecting the accuracy of the judgment; second, multimodal large models are typically designed to process and understand a relatively focused subject, and processing multiple workers in an image simultaneously can lead to ambiguous referencing, making it impossible for the model to accurately associate the analysis conclusions with a specific worker. Therefore, to address these problems, this application first performs worker target detection and candidate region extraction on the raw construction site images to obtain a set of worker information images and a set of original target bounding box coordinates.

[0026] In one embodiment, worker target detection and candidate region extraction are performed on the original construction site image to obtain a set of worker information images and an original target bounding box coordinate set, including: inputting the original construction site image into a pre-trained worker detection model to obtain an initial target bounding box coordinate set as the original target bounding box coordinate set; and acquiring candidate region images from the original construction site image based on the initial target bounding box coordinate set to obtain the worker information image set.

[0027] Specifically, the first step is worker target detection. This involves inputting the original construction site image into a pre-trained worker detection model. This model identifies and detects only humans as targets, thus locating the positions of all workers in the image. The output of this process is an initial set of target bounding box coordinates, where each bounding box coordinate corresponds to a worker in the image, and this set is used as the original set of target bounding box coordinates for the output. If no worker target is detected in this sub-step, the detection process for the current image is terminated.

[0028] Next, based on the initial target bounding box coordinate set, candidate region images are acquired from the original construction site image. The core of this process is context-enlargement of the initially detected bounding boxes. Specifically, based on the initial target bounding box coordinate set, candidate region images are acquired from the original construction site image to obtain a set of worker information images. This includes: First, based on the size information of the original construction site image, context-enlargement is performed on each initial target bounding box in the initial target bounding box coordinate set to obtain an enlarged target bounding box coordinate set. This operation aims to incorporate the environmental information surrounding the worker's body, allowing for better fine-tuning of the multimodal large model. Specifically, context-enlargement is performed on each initial target bounding box in the initial target bounding box coordinate set using the following formula: ;in, These are the coordinates of the top left and bottom right corners of the adjusted target bounding box. It is the minimum value along the X-axis of the image. It is the minimum value in the Y-axis direction of the image. The maximum value along the X-axis of the image is the width of the image. The maximum value along the Y-axis of the image is the height of the image.

[0029] Next, using the expanded target bounding box coordinate set, candidate region images are acquired from the original construction site images to obtain a set of worker information images. This involves precisely cropping out the corresponding rectangular image regions, resulting in the final set of worker information images. Finally, to ensure data integrity and consistency, the cropped set of worker information images is strictly correlated with the original target bounding box coordinate set obtained in the detection stage (without expansion). These two sets, as the final output of this step, are passed to the subsequent multimodal recognition module.

[0030] In one specific embodiment, YOLO11x is selected as the worker detection model, and the construction site image to be detected is identified only as a human, with a confidence threshold set to 0.1. If one or more workers are detected, one or more target bounding box coordinates are output, denoted as (x1, y1, x2, y2). Subsequently, the aforementioned context expansion formula is applied to each output target bounding box. After processing, the construction site image is cropped according to the adjusted target bounding box coordinates to obtain one or more worker information images. Finally, the generated set of worker information images and the corresponding set of original target bounding box coordinates before adjustment are used as the output of this step, ensuring that the elements in the two sets correspond one-to-one in order.

[0031] For example, in step S3, multimodal recognition of missing safety equipment is performed on the worker information image set to obtain a list of responses indicating missing worker safety protective equipment. Step S3 and subsequent steps S4 and S5 are handled by... Figure 4The multimodal large-scale model prediction module shown is executed. It should be understood that traditional detection methods treat protective equipment detection as an image classification problem, directly mapping image pixels to discrete labels such as compliance or non-compliance. This approach has significant limitations when facing complex and ever-changing scenarios with incomplete information at construction sites. For example, when a worker's body is partially obscured, making it impossible to definitively determine whether they are wearing a safety helmet, traditional classification models often force a low-confidence incorrect judgment, failing to express the more realistic intermediate state of "insufficient information, unable to make a judgment." Furthermore, traditional models have limited generalization ability and are highly sensitive to changes in safety equipment styles, colors, or worker postures not present in the training set, easily leading to false alarms. This step introduces multimodal recognition, elevating the task from classification to visual question answering and reasoning. This paradigm leverages the powerful pre-trained knowledge and logical reasoning capabilities of multimodal large-scale models, enabling them to, like humans, combine visual information (worker images) and context (pre-set questions) to comprehensively analyze the scene and generate a natural language response containing judgment criteria and conclusions. This kind of logical and context-rich response can not only accurately handle clearly identifiable scenarios, but also effectively deal with difficult situations such as ambiguity and obstruction, and provide an interpretable and more reliable basis for subsequent decision-making.

[0032] In one embodiment, performing multimodal identification of missing safety equipment on a set of worker information images to obtain a list of responses indicating missing worker safety protective equipment includes: constructing multimodal input pairs based on the set of worker information images to obtain a batch of multimodal large model inputs; and inputting the batch of multimodal large model inputs into a fine-tuned multimodal large model to obtain a list of responses indicating missing worker safety protective equipment.

[0033] Specifically, firstly, multimodal input pairs are constructed based on the worker information image set obtained in the previous step to form a large multimodal model input batch. In practice, each worker information image in the set is traversed, and a preset question text is paired with each worker information image. The question text is, "This is a photo taken at a construction site. Please answer whether the worker in the center of the photo is missing safety equipment?" Of course, in other embodiments of this application, more detailed questions can be set. For example, the question text could also be set as: "This is a photo taken at a construction site. Examine the photo carefully. Please answer whether the worker in the center of the photo (if any) is wearing a safety helmet? Is he wearing a reflective vest? Finally, analyze whether there is a lack of safety equipment?" By combining each worker image with its corresponding question text into a data pair, a large multimodal model input batch containing multiple image-text pairs is finally formed, preparing for batch inference of the model.

[0034] Subsequently, the constructed multimodal large-scale model is input batches into a multimodal large-scale model that has been fine-tuned for a specific task for inference and response generation. Upon receiving each image-text pair, the model performs joint understanding and analysis of the image content and the text question, and based on its powerful visual analysis and language generation capabilities, produces a descriptive response text for the question. The output of this process is a list of responses to the detection of missing worker safety protective equipment. Each item in this list corresponds strictly in sequence to the worker information images in the input batch, ensuring a one-to-one correspondence between the images and the analysis results.

[0035] In one specific embodiment, the large model used in this step is the finely tuned multimodal large model described in the previous process, such as the DeepSeek Janus Pro 1B visual language large model. During prediction, a set of worker information images is input into this large model, which automatically pairs each image with a uniformly set, unconfigured question text (i.e., the aforementioned fixed question text). The large model then performs inference to obtain its output, which is a list of responses indicating missing worker safety equipment. Each item in this list corresponds sequentially to one image in the input set of worker information images, thus completing the transformation from image input to natural language analysis output.

[0036] For example, in step S4, the identification results of the list of responses regarding missing safety protective equipment for inspected workers are semantically parsed and classified to obtain a negative sample index set. It should be understood that although the list of responses regarding missing safety protective equipment for inspected workers is rich in content and interpretable, it is essentially a text string and cannot be directly understood or executed by the subsequent visualization alarm module. The subsequent module requires a clear, discrete signal to indicate which workers have been judged to be in violation. Therefore, by performing semantic parsing and classification on the identification results of the list of responses regarding missing safety protective equipment for inspected workers, the most core judgment conclusions are extracted and converted into a precise and operable data format, namely, the negative sample index set. This not only realizes the transformation from descriptive information to instructional information but also filters out compliant samples and undetermined samples through a classification mechanism, ensuring that the final alarm operation only targets confirmed violations, thereby avoiding unnecessary operations and false alarms.

[0037] In one embodiment, the process of semantically parsing and classifying the results of the identification of the list of responses indicating missing safety protective equipment for inspecting workers to obtain a negative sample index set includes: performing semantic parsing on each response text in the list of responses indicating missing safety protective equipment for inspecting workers to obtain a classification result list; and filtering the classification result list using negative sample indexes to obtain a negative sample index set.

[0038] Specifically, firstly, semantic parsing is performed on each response text in the list of responses regarding missing safety protective equipment for inspected workers to obtain a classification result list. Specifically, each response text in the input list is traversed, and keyword matching or a rule-based semantic analysis engine is used to check for conclusive phrases within the text. Based on whether the text contains key phrases such as "missing safety protective equipment exists," "missing safety protective equipment does not exist," or "cannot be determined," each response text is categorized as a "negative sample," "positive sample," or "other sample," and the original index of each text and its corresponding category are stored as a tuple in the classification result list.

[0039] After generating the classification result list, the list is then filtered by negative sample index to obtain the final negative sample index set. This sub-step iterates through the classification result list, extracts all tuples that are classified as "negative samples," and collects their corresponding original index numbers. All collected index numbers together constitute the final output of this step, namely the negative sample index set. If this set is empty, it means that no violations were found in the currently detected image frame, and the process can terminate early, waiting for the input of the next image frame.

[0040] In one specific embodiment, this step classifies the list of response texts from the multimodal large model into three categories: positive samples, negative samples, and other samples, resulting in a classification set: a positive sample index set, a negative sample index set, and an other sample index set. Specifically, the classification rule is as follows: responses with the analysis result "No safety protection equipment is missing" correspond to positive samples; responses with the analysis result "Safety protection equipment is missing" correspond to negative samples; and responses with the analysis result "It cannot be determined whether safety protection equipment is missing" correspond to other samples. This rule allows for the accurate selection of all negative sample indices, forming a negative sample index set for use in subsequent steps.

[0041] For example, in step S5, based on the negative sample index set and the original target bounding box coordinate set, violation events are located and visualized in the original construction site image to obtain a worker safety protection equipment missing identification map. It should be understood that while the negative sample index set precisely indicates which individuals being analyzed have violated regulations, it is merely an internal data identifier, lacking spatial location information and scene context, and cannot directly inform safety management personnel of the specific location of the violation event. Therefore, by locating and visualizing violation events in the original construction site image based on the negative sample index set and the original target bounding box coordinate set, the abstract violation index can be precisely bound to the specific physical location in the original image, ultimately generating an image with clear warning markers.

[0042] In one embodiment, based on the negative sample index set and the original target bounding box coordinate set, violation event localization and visualization alarm are performed on the original construction site image to obtain a worker safety protection equipment missing identification map, including: localizing the violation target based on the negative sample index set and the original target bounding box coordinate set to obtain the violation target bounding box coordinate set; and performing a drawing operation on the original construction site image based on the violation target bounding box coordinate set to obtain the worker safety protection equipment missing identification map.

[0043] Specifically, first, a violation target localization operation is performed, based on a negative sample index set and an original target bounding box coordinate set. Specifically, each index number in the negative sample index set is used as a query criterion to precisely retrieve the corresponding original target bounding box coordinates from the original target bounding box coordinate set. By traversing all negative sample indices, a new set is constructed—the violation target bounding box coordinate set—which contains only the precise location information of all workers identified as having equipment missing issues within the original images.

[0044] After obtaining the set of bounding box coordinates for the violation targets, the next step is to visualize the alarm information. This operation is based on the set of bounding box coordinates for the violation targets and the original construction site image. Specifically, the drawing operation is performed on a copy of the original construction site image. It iterates through each bounding box coordinate in the set of bounding box coordinates for the violation targets and uses these coordinates to draw a bright alarm box on the image copy, such as a prominent red rectangle, to clearly indicate the location of the violating worker. Depending on the actual needs, text descriptions such as missing equipment can be added near the alarm box to further enhance the readability and clarity of the alarm information. The final output of this drawing operation is the worker safety protection equipment missing identification image obtained by this invention.

[0045] In a specific embodiment, the processing logic of this step is as follows: First, it is determined whether negative samples exist. No processing is performed on positive samples and other samples; if no negative samples exist, the process ends, waiting for the next detection input. If negative samples exist, the corresponding set of negative sample target bounding box coordinates is obtained through the negative sample index. Subsequently, this coordinate set is used to mark the original construction site image, i.e., to draw the worker safety protection equipment missing identification box, ultimately obtaining the worker safety protection equipment missing identification map. This identification map serves not only as the output of the multimodal large model prediction module but also as the final output result of the entire real-world scene recognition and detection process.

[0046] Furthermore, it should be understood that before deploying the worker safety protection equipment missing detection method based on multimodal large model described in this application, it is necessary to first prepare and fine-tune the multimodal large model. Figure 5 This is a schematic diagram illustrating the process of fine-tuning a multimodal large model according to an embodiment of this application. Figure 5 As shown, the fine-tuning process of the multimodal large model includes: Step 1: Constructing the fine-tuning dataset. First, construction site images are collected, with the image dimensions being 1920*1080 pixels in width and height, forming the construction site dataset.

[0047] Step 2: Preprocess the construction site dataset. Use a YOLO11x pre-trained object detection model as the worker detection model, with the detection category set to 0 (Person), and a confidence threshold of 0.1. For a single construction site image, if the worker detection model identifies one or more workers, the output will be one or more target bounding boxes with coordinates (x1, y1, x2, y2), where (x1, y1) is the coordinate of the top-left corner of the target bounding box, and (x2, y2) is the coordinate of the bottom-right corner of the target bounding box.

[0048] Step 3: Expand and crop the target bounding box. The target bounding box output by the detection model is expanded. The expanded and adjusted target bounding box coordinates are (x'1, y'1, x'2, y'2). The adjusted target bounding box can contain a certain amount of information about the worker's surroundings, allowing for better fine-tuning of the multimodal large model. Based on the adjusted target bounding box coordinate set, the corresponding construction site images are cropped to obtain one or more images of construction workers, thus forming a worker dataset.

[0049] Step 4: Data Labeling. The worker dataset is divided into 8 categories, and corresponding descriptive text (response text) and a unified question text are set for each of these 8 categories. This process labels the worker dataset, resulting in a fine-tuned dataset. Detailed descriptions of the 8 categories and their corresponding descriptive texts (response texts) are shown in the table below:

[0050] Table 1:

[0051]

[0052] When fine-tuning the multimodal large model in this application, the question text used was uniformly set as follows: "This is a photo taken at a construction site. Please examine the photo carefully. Please answer whether the worker in the center of the photo (if any) is wearing a hard hat? Is he wearing a reflective vest? Finally, analyze whether there is a lack of safety protective equipment?"

[0053] Step 5: Model Fine-tuning. The DeepSeek Janus Pro 1B Visual Language Model (VLM) is selected as the multimodal large model for this method. Fine-tuning is performed using the aforementioned dataset. The multimodal large model fine-tuning framework uses SWIFT (Scalable lightWeight Infrastructure for Fine-Tuning), with the following parameters: LoRA fine-tuning method is used, LoRA Rank is set to 8, LoRA scaling factor (LoRA Alpha) is set to 32, LoRA fine-tuning is applied to all linear layers of the multimodal large model, the parameters of the ViT (Vision Transformer) part are frozen and not updated, the maximum length of the input sequence is set to 2048, the batch size is set to 1, the number of gradient accumulation steps is set to 8, the number of fine-tuning iterations (train epochs) is set to 5, the loss function is the cross-entropy loss function, the optimizer is AdamW, the maximum learning rate of AdamW is set to 0.0001, the learning rate warm-up ratio is set to 5%, and the floating-point data type used in PyTorch is specified as bfloat16.

[0054] The fine-tuned environment is as follows: operating system is Ubuntu 22.04, Python 3.10, PyTorch 2.1.0, computing resources (graphics card) are NVIDIA RTX 4090D (24GB), CUDA version is 12.1.

[0055] The data format used in this application for fine-tuning is a multimodal format, which uses not only image features but also text features to fine-tune the multimodal large model. Furthermore, only the linear layer is fine-tuned, and the sample size used for fine-tuning is much smaller than that used in traditional methods for fine-tuning single-modal models, thereby significantly controlling the fine-tuning cost of this task.

[0056] Because the pre-trained large model has a large parameter base, fine-tuning only the linear layers of the large model can still demonstrate excellent generalization ability. Compared with traditional methods, it can better perform the task of detecting missing safety protective equipment for workers in complex construction site scenarios. After fine-tuning, a fine-tuned multimodal large model is obtained, which is suitable for the task of detecting missing safety protective equipment for construction workers.

[0057] Furthermore, this application may include the following improvements: In another embodiment, negative samples can be classified more finely. For example, based on key information in the response text such as "he was not wearing a safety helmet" or "he was not wearing a reflective vest," negative samples can be further classified into two subcategories: "missing safety helmet" and "missing reflective vest." This generates two different worker safety equipment missing identification maps: a safety helmet missing identification map and a reflective vest missing identification map, providing managers with more accurate alarm information. In yet another embodiment, more comprehensive detection results can be output. In addition to generating worker safety equipment missing identification maps, the descriptive information of the negative samples (i.e., the corresponding multimodal large model response text) and the corresponding target bounding box coordinate information can also be included, along with the identification... Figure 1 This serves as the final output. In this way, users can not only see the location of the violation, but also directly obtain a detailed description of the reason for the violation, resulting in a more comprehensive and easily understandable identification and detection result.

[0058] In summary, the method for detecting missing worker safety protective equipment based on a multimodal large model provided in this application has been elucidated. First, it uses object detection technology to accurately separate candidate regions of various worker subjects from complex original construction site images, i.e., a set of worker information images, while preserving their positional information in the original images, i.e., a set of original target bounding box coordinates. Subsequently, instead of performing simple feature matching and classification on these images, this method combines them with pre-defined natural language questions and inputs them into a multimodal large model fine-tuned for a specific task for in-depth analysis. This multimodal identification step for missing safety equipment elevates the simple image recognition task to a combined image-text visual question answering and reasoning task. Utilizing the powerful world knowledge and logical reasoning capabilities of the large model, it generates a list of responses containing detailed descriptions and clear conclusions for detecting missing worker safety protective equipment. Then, through semantic parsing and classification of these natural language responses, it accurately filters out a set of negative sample indices judged as violations, realizing the transformation from unstructured text to structured decision information. Finally, based on the index set and the original target bounding box coordinates, the violation event is accurately located and visualized as an alarm on the original image. The final output is an intuitive image of missing worker safety equipment, thus constructing a complete and closed-loop intelligent detection system from raw data collection to final alarm presentation.

[0059] This application also provides a worker safety protection equipment missing detection device based on a multimodal large model, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the worker safety protection equipment missing detection method based on a multimodal large model provided in the above embodiments.

[0060] This application also provides a computer-readable storage medium storing computer program code. When the computer program code is run on a computer, the computer executes the above-described related method steps to implement the method for detecting missing worker safety protective equipment based on a multimodal large model provided in the above embodiments.

[0061] This application also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned steps to implement the method for detecting missing worker safety protective equipment based on a multimodal large model provided in the above embodiments.

[0062] The apparatus, computer-readable storage medium, or computer program product provided in the embodiments of this application are all used to execute the corresponding methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0063] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments.

[0064] The processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are possible or may be advantageous. The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments.

[0065] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A worker safety personal protective equipment absence detection method based on a multi-modal large model, characterized in that, The process includes the following steps: acquiring original construction site images; performing worker target detection and candidate region extraction on the original construction site images to obtain a set of worker information images and a set of original target bounding box coordinates; and performing multimodal recognition of missing safety equipment on the set of worker information images to obtain a list of responses indicating missing safety protective equipment for workers. The process involves semantic parsing and classification of the results from the identification of the list of missing safety protective equipment for detected workers to obtain a negative sample index set. Based on the negative sample index set and the original target bounding box coordinate set, violation events are located and visualized in the original construction site images to obtain a worker safety protective equipment missing identification map. Specifically, a multimodal identification of missing safety equipment is performed on the worker information image set to obtain the list of missing worker safety protective equipment, including: constructing multimodal input pairs based on the worker information image set to obtain a multimodal large model input batch; and inputting the multimodal large model input batch into a fine-tuned multimodal large model to obtain the list of missing worker safety protective equipment.

2. The worker safety PPE absence detection method based on a multi-modal large model according to claim 1, wherein, The process involves detecting worker targets and extracting candidate regions from original construction site images to obtain a set of worker information images and a set of original target bounding box coordinates. This includes: inputting the original construction site images into a pre-trained worker detection model to obtain an initial set of target bounding box coordinates as the original target bounding box coordinate set; and acquiring candidate region images from the original construction site images based on the initial set of target bounding box coordinates to obtain a set of worker information images. 3.The worker safety PPE absence detection method based on a multi-modal large model according to claim 2, wherein, Based on the initial target bounding box coordinate set, candidate region images are acquired from the original construction site image to obtain a set of worker information images. This includes: based on the size information of the original construction site image, contextual expansion is performed on each initial target bounding box in the initial target bounding box coordinate set to obtain an expanded target bounding box coordinate set; and candidate region images are acquired from the original construction site image based on the expanded target bounding box coordinate set to obtain a set of worker information images.

4. The worker safety PPE absence detection method based on a multi-modal large model according to claim 3, wherein, Based on the size information of the original construction site construction image, the context of each initial target bounding box in the initial target bounding box coordinate set is enlarged to obtain an enlarged target bounding box coordinate set, including: the context of each initial target bounding box in the initial target bounding box coordinate set is enlarged by the following formula: ; wherein, are the left upper corner and the right lower corner coordinates of the adjusted target bounding box, is the minimum value of the X-axis direction of the image, is the minimum value of the Y-axis direction of the image, is the maximum value of the X-axis direction of the image, and is the width of the image, is the maximum value of the Y-axis direction of the image, and is the height of the image. 5.The worker safety PPE absence detection method based on a multi-modal large model according to claim 1, wherein, Multimodal input pairs are constructed based on a set of worker information images to obtain a batch of multimodal large model inputs, including pairing each worker information image with a preset question text to ask whether the worker is missing safety protective equipment. 6.The worker safety PPE absence detection method based on a multi-modal large model according to claim 1, wherein, The results of semantic analysis and classification of the list of responses indicating missing safety protective equipment for testing workers are used to obtain a negative sample index set. This includes: semantic analysis of each response text in the list of responses indicating missing safety protective equipment for testing workers to obtain a classification result list; and filtering of the classification result list using the negative sample index to obtain a negative sample index set.

7. The worker safety PPE absence detection method based on multi-modal large model according to claim 1, wherein, Based on the negative sample index set and the original target bounding box coordinate set, violation event localization and visualization alarm are performed on the original construction site image to obtain the worker safety protection equipment missing identification map. This includes: localizing the violation target based on the negative sample index set and the original target bounding box coordinate set to obtain the violation target bounding box coordinate set; and performing a drawing operation on the original construction site image based on the violation target bounding box coordinate set to obtain the worker safety protection equipment missing identification map.

Images