Power construction site operator safety equipment wearing compliance detection method based on multi-modal hybrid model

Abstract

An electric power construction site operator safety equipment wearing compliance detection method based on a multi-modal hybrid model comprises the following steps: firstly, performing scene recognition on an input image by using a multi-modal large model, and outputting an operation scene type; secondly, inputting the scene type into a large-scale language model, and generating a safety equipment name list required in the scene and fine-grained visual attribute description of each type of equipment through two times of natural language query; then, positioning a person by utilizing an open vocabulary object detection model, and judging whether the person wears required safety equipment or not through a visual language model; and finally, carrying out secondary positioning on the worn safety equipment, calculating the semantic similarity between the image blocks of the worn safety equipment and the fine-grained attribute description, and verifying whether the wearing accords with the specification or not. According to the method, fine-grained, interpretable and cross-scene compliance detection of the wearing condition of the safety equipment can be realized, and the defects of the prior art in the aspects of normative recognition, scene adaptability and detection efficiency are overcome.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of equipment wearing monitoring, and particularly relates to a safety equipment wearing compliance detection method for electric power construction site workers based on a multi-modal hybrid model. BACKGROUND

[0002] Safety compliance of an electric power construction site is a key link to ensure production safety. Non-standard wearing of personal protective equipment may not only lead to serious safety accidents, but also cause legal liability, economic penalties, and damage to the reputation of an enterprise. Therefore, there are not only wearing requirements for protective equipment, but also detailed regulations on wearing specifications at an electric power construction site.

[0003] At present, safety compliance detection at an electric power construction site mainly relies on manual inspection, a sensor-based system, and a visual monitoring system. The manual inspection method has high labor intensity and cost, and is easily affected by the experience level and subjective judgment of an inspector, and has problems such as missed detection and misjudgment. The sensor-based system (such as RFID and Zigbee) can only determine the presence of equipment, but cannot identify whether the wearing is standard, for example, cannot determine whether a safety helmet is correctly worn, whether a safety belt is cross-worn, or whether a safety rope is "low hanging and high use", and has high system deployment cost and complex maintenance. The visual method based on a convolutional neural network (such as Faster R-CNN and YOLO) has low cost, but is usually limited to object detection, lacks fine-grained judgment ability on wearing specifications and scene applicability, and has low precision and insufficient generalization ability in complex environments (such as occlusion and light changes). In addition, the method directly using a visual language model has slow reasoning speed, cannot meet real-time requirements, and lacks accurate discrimination ability for structured safety rules.

[0004] Therefore, the existing technology still has obvious deficiencies in detection efficiency, specification identification accuracy, scene adaptability, and real-time performance, and it is difficult to realize comprehensive, reliable, and intelligent compliance detection of safety equipment wearing of workers at an electric power construction site. SUMMARY

[0005] The technical problem to be solved by the present application is to provide a safety equipment wearing compliance detection method for electric power construction site workers based on a multi-modal hybrid model, which can realize fine-grained, interpretable, and cross-scene compliance detection of safety equipment wearing, and solve the deficiencies of the existing technology in specification identification, scene adaptability, and detection efficiency.

[0006] To solve the above technical problems, the technical solution adopted by the present application is: A safety equipment wearing compliance detection method for electric power construction site workers based on a multi-modal hybrid model, comprising the following steps: S1: generating a scene description caption for the input image using a multi-modal large model, and fine-tuning the multi-modal large model through a low-rank adaptation method to identify the type of the current power construction site, and outputting a scene type identification result; S2: inputting the scene type identification result output by S1 into a large language model, and performing two natural language queries through the large language model; S3: detecting all personnel in the input image using an open-vocabulary object detection model to generate a personnel bounding box, extracting a personnel image block based on the personnel bounding box, inputting the personnel image block and the safety equipment name list output by S2 into a visual language model, calculating the similarity between the text embedding of the personnel image block and each safety equipment name in the safety equipment name list, and determining whether each personnel wears the required safety equipment according to the similarity, and outputting a safety equipment wearing detection result; S4: positioning the safety equipment determined to be worn in the safety equipment wearing detection result output by S3 using an open-vocabulary object detection model, extracting a safety equipment image block, inputting the safety equipment image block and the fine-grained visual attribute description output by S2 into a visual language model, calculating the cosine similarity between the visual embedding of the safety equipment image block and the text embedding of the fine-grained visual attribute description, and determining whether the wearing of the safety equipment meets the standard requirements according to the cosine similarity, and outputting a fine-grained compliance verification result.

[0007] Preferably, the sub-steps of S1 are: S1.1: generating an image caption containing semantic information of the power construction site using a multi-modal large model, the image caption content including scene features of aerial work, ladder work or underground work; S1.2: updating the parameters of the multi-modal large model through a low-rank adaptation method, adjusting only the low-rank trainable parameters corresponding to the query weight matrix and the key weight matrix of the multi-head self-attention mechanism in the multi-modal large model, and keeping the original weight matrix of the multi-modal large model unchanged; S1.3: fine-tuning the multi-modal large model using power construction scene image and text paired data, the text paired data adopting a three-part text format starting with the word “in”, ending with “scene”, and filling in the specific work scene name in the middle, so that the scene description caption output by the multi-modal large model can accurately reflect the actual work environment.

[0008] Preferably, the two natural language queries include: The first query generates a safety equipment name list required to be worn by the work personnel in the current work scene type; The second query generates a fine-grained visual attribute description corresponding to each safety equipment in the safety equipment name list, and outputs the safety equipment name list and the fine-grained visual attribute description.

[0009] Preferably, the sub-steps of S2 are: S2.1: Combining the scene type recognition result output by S1 as a dynamic scene variable with a first fixed semantic segment and a second fixed semantic segment to form a first natural language query instruction, wherein the first fixed semantic segment is used to express a safety equipment query intention, and the second fixed semantic segment is used to limit the query object to safety equipment required to be worn by the worker, inputting the first natural language query instruction into a large language model to generate a safety equipment name list, the safety equipment name list including safety helmets, safety belts, connecting ropes, safety glasses, high-visibility clothing, and safety boots; S2.2: Combining the scene type recognition result output by S1 as a dynamic scene variable with a third fixed semantic segment, a fourth fixed semantic segment, the safety equipment name list generated by S2.1, and a fifth fixed semantic segment to form a second natural language query instruction, wherein the third fixed semantic segment is used to express an attribute summary intention, the fourth fixed semantic segment is used to introduce an equipment list, and the fifth fixed semantic segment is used to limit the query content to visual features, inputting the second natural language query instruction into the large language model to generate three levels of fine-grained visual attribute descriptions for each safety equipment; S2.3: Classifying the generated fine-grained visual attribute descriptions to obtain directly observable attributes, context observable attributes, and inferred observable attributes; S2.4: Performing direct observable attribute recognition, the direct observable attributes including color, material, and structural components; performing context observable attribute recognition, the context observable attributes including position relationship, posture, and connection mode; and performing inferred observable attribute recognition, the inferred observable attributes including energy absorption capacity, impact resistance level, and anti-slip performance.

[0010] Preferably, the sub-steps of S3 are: S3.1: Detecting all “person” class targets in the input image using an open vocabulary object detection model to generate a personnel bounding box; S3.2: Cutting out personnel image blocks from the input image based on the personnel bounding box to form a personnel image block list; S3.3: Inputting the personnel image block list and the safety equipment name list output by S2 into a visual language model, and encoding the personnel image block list into a personnel visual embedding matrix and the safety equipment name list into an equipment text embedding matrix through the visual language model; S3.4: Calculating the matrix multiplication of the personnel visual embedding matrix and the equipment text embedding matrix to generate a similarity matrix; S3.5: Setting a judgment threshold, if an element value in the similarity matrix is greater than or equal to the judgment threshold, it is determined that the corresponding personnel wears the corresponding safety equipment; otherwise, it is determined that the corresponding personnel does not wear the corresponding safety equipment.

[0011] Preferably, the sub-steps of S4 are: S4.1: performing secondary detection on the safety equipment judged as worn in the safety equipment wearing detection result output by S3 using an open-vocabulary object detection model to generate a safety equipment bounding box; S4.2: cutting out safety equipment image blocks from the input image based on the safety equipment bounding box to form a safety equipment image block list; S4.3: inputting the safety equipment image block list and the fine-grained visual attribute description output by S2 into a visual language model, encoding the safety equipment image block list into a safety equipment visual embedding and the fine-grained visual attribute description into an attribute text embedding through the visual language model; S4.4: calculating the cosine similarity between the safety equipment visual embedding and the corresponding attribute text embedding for each safety equipment image block; S4.5: setting a verification threshold, if the cosine similarity is greater than or equal to the verification threshold, it is judged that the safety equipment meets the corresponding fine-grained attribute requirement; otherwise, it is judged that the safety equipment does not meet the corresponding fine-grained attribute requirement; S4.6: inputting the fine-grained compliance verification result into a compliance judgment and output module, if there is safety equipment that does not meet the fine-grained attribute requirement, generating an alarm signal through the compliance judgment and output module.

[0012] A power construction site worker safety equipment wearing compliance detection system based on a multi-modal hybrid model adopts the power construction site worker safety equipment wearing compliance detection method based on the multi-modal hybrid model, and the system comprises: A scene recognition module is configured to generate a scene description caption for an input image using a multi-modal large model, and fine-tune the multi-modal large model through a low-rank adaptation method to identify the type of the current power construction site, and output a scene type recognition result. A visual prompt generation module is configured to input the scene type recognition result output by the scene recognition module into a large language model, and generate a list of required safety equipment names and fine-grained visual attribute descriptions corresponding to each safety equipment under the current work scene type through the large language model. A safety equipment detection module is configured to detect all personnel in an input image using an open-vocabulary object detection model to generate a personnel bounding box, extract a personnel image block, input the personnel image block and the list of safety equipment names output by the visual prompt generation module into a visual language model, calculate the similarity, determine whether the personnel are wearing the required safety equipment, and output a safety equipment wearing detection result. The fine-grained verification module is configured to locate the safety equipment determined to be worn in the safety equipment wearing detection result by using the open vocabulary object detection model, extract a safety equipment image block, input the safety equipment image block and the fine-grained visual attribute description output by the visual prompt generation module into a visual language model, calculate a cosine similarity, determine whether the safety equipment wearing meets the standard requirement, and output a fine-grained compliance verification result. The compliance determination and output module is configured to receive the fine-grained compliance verification result output by the fine-grained verification module, and generate an alarm signal if the safety equipment does not meet the fine-grained attribute requirement.

[0013] Preferably, the visual prompt generation module comprises: The equipment query sub-module is configured to combine the job scene type output by the scene recognition module as a dynamic variable with the first fixed semantic segment and the second fixed semantic segment to form a first natural language query instruction, and send the first natural language query instruction to the large language model to obtain a safety equipment name list. The attribute generation sub-module is configured to combine the job scene type, the safety equipment name list, the third fixed semantic segment, the fourth fixed semantic segment, and the fifth fixed semantic segment to form a second natural language query instruction, and send the second natural language query instruction to the large language model to obtain a fine-grained visual attribute description. The data organization sub-module is configured to store the safety equipment name list and the fine-grained visual attribute description in a structured data format for calling by the safety equipment detection module and the fine-grained verification module.

[0014] Preferably, the visual language model in the fine-grained verification module is a pre-trained image-text alignment model, which is trained by a contrast learning method and is used to calculate the semantic matching degree between the safety equipment image block and the fine-grained visual attribute description.

[0015] A computer device comprises one or more processors, and one or more executable programs are stored on the processors, and when the one or more executable programs are executed by the one or more processors, a method for detecting the safety equipment wearing compliance of power construction site workers based on a multi-modal hybrid model is implemented.

[0016] The present application can achieve the following beneficial effects: 1. The present application provides an interpretable and fine-grained detection capability, which can verify the wearing condition of personal protective equipment in detail, not only detecting whether necessary safety equipment exists, but also verifying whether it meets specific fine-grained attribute requirements. This detailed analysis helps to provide a more in-depth safety assessment.

[0017] 2、The application has processing capability for diversified scenarios. By introducing a scene recognition part, different working environments such as high-altitude work, ladder work and underground work can be identified. Combined with the visual prompt part and the large language model, the specific safety equipment list and detailed visual attribute description required in the environment can be automatically generated according to the identified scene. This dynamic adaptability makes the detection method have strong universality, without the need to retrain the model for the safety requirements of each new scene. For the following important scenarios: (1) High-altitude work: Traditional methods cannot identify whether the safety belt is correctly worn or whether there are high-risk behaviors such as "low hanging high use". The application generates corresponding detection elements by scene recognition to realize targeted verification.

[0018] (2) Ladder work: Due to the variable posture of personnel and frequent occlusion, traditional visual models are prone to misjudgment or omission of key equipment (such as anti-skid shoes and waist belts). The application uses visual prompts and language descriptions to improve the equipment positioning and attribute discrimination ability in occluded scenes.

[0019] (3) Underground work / enclosed space: The light in such scenes is extremely poor, and the types of safety equipment (such as gas masks and headlamps) are special. The application can automatically adjust the detection target list according to the scene semantics to identify and verify special equipment, making up for the shortcomings of the existing model's insufficient generalization.

[0020] (4) Temporary work site or non-standard environment: such as outdoor repair, transformer station fence edge, etc. The safety requirements of these places often do not have a universal template. The application realizes dynamic mapping of "scene-equipment" by integrating the semantic reasoning ability of the language model, without the need to manually set rules or retrain the model for each new scene.

[0021] 3、The application shows higher computing efficiency and significantly faster reasoning speed. In comparison with existing mainstream visual question answering models, the average reasoning time of this method is 0.2 seconds, while the average reasoning time of LLaVA-1.6-7b is 4.2 seconds. This high efficiency makes it suitable for real-time safety monitoring applications that require fast response.

[0022] 4、The application shows comparable or higher level than existing technologies on the multi-scene dataset of power operation site. In the complex safety equipment detection task, the accuracy of the application reaches 76.8%, which is about 5.3 percentage points higher than that of LLaVA-1.5-7b (71.5%). In the aspect of fine-grained attribute verification, especially in the identification accuracy of directly observable attributes, it reaches 76.9%, which is 8.7 percentage points higher than that of LLaVA-1.5-7b (68.2%).

[0023] 5、The application can recognize new object categories outside the training data, handle diversified safety equipment problems, and effectively overcome the challenges of traditional methods in data sparsity and class imbalance, by using an open vocabulary object detection model and a pre-trained visual language model. BRIEF DESCRIPTION OF DRAWINGS

[0024] The application will be further described below in conjunction with the accompanying drawings and embodiments: Figure 1 A flowchart of a power construction site safety equipment compliance detection method based on a multi-modal hybrid model; Figure 2 A structural block diagram of a power construction site safety equipment wearing compliance detection system. DETAILED DESCRIPTION

[0025] The preferred scheme is as shown in Figures 1-2 A safety equipment wearing compliance detection method for power construction site workers based on a multi-modal hybrid model is provided to solve the technical problems of low efficiency of traditional manual inspection, high cost of sensor solutions, poor generalization ability of visual methods, and weak adaptability to diversified scenes. This method combines computer vision and lightweight multi-modal visual language models, and realizes accurate identification and state judgment of various types of protective equipment (such as safety helmets, protective clothing, and insulating gloves) through collaborative understanding of images and texts. The system has strong scalability and real-time processing capability, can adapt to differences in equipment attributes and changes in scenes under complex working conditions, and provides interpretable detection results for easy safety supervision and traceability. This method not only ensures the personal safety of workers, but also improves the intelligent safety management level of power construction sites, and has wide application prospects and practical value.

[0026] Referring to Figure 1 , the application provides an interpretable fine-grained detection method for personal protective equipment safety compliance in power construction sites, including the following steps: P1-1: Generate high-quality image captions to provide concise summaries of visual content. Considering the limitation of computing resources and the need to generate more accurate scene descriptions, the Qwen-VL-7B model is selected as the image captioning model. This model integrates multi-modal capabilities of visual and language understanding.

[0027] P1-2: To further improve performance and comply with computing resource limitations, the LoRA (Low Rank Adaptation) technique is used to fine-tune the model to provide accurate scene recognition. The text prompt "in XXX scene" is paired with the corresponding image as text input. LoRA is implemented by adding small trainable weight matrices (A and B) to the existing weight matrix, only updating the weights of A and B while keeping the original weights W unchanged. The updated matrix is where . This applies to the query ( ) and key ( ) weight matrices in the multi-head self-attention part. The fine-tuning process aims to minimize the negative log-likelihood loss.

[0028] P2-1: Receive the scene information generated by the image captioning model, query the LLM, for example: "List the safety equipment that workers need to wear in the XXX scene". The LLM generates a list of safety items.

[0029] P2-2: Based on the response of the LLM, make a second query, for example: "Summarize the visual features of these XXX safety items in the XXX scene". This prompt guides the LLM to summarize the specific visual attributes of the required safety accessories; P2-3: Obtain the visual attributes that each safety equipment should have from the LLM response. These attributes usually include color, material and function, and three attributes are generated for each category.

[0030] P3-1: Utilize the open vocabulary object detection model (choose YOLO-World) to generate the bounding boxes (Npbbox) of all personnel in the scene.

[0031] P3-2: Extract personnel image blocks based on the personnel bounding box coordinates; input the list of N personnel image blocks (Lpbbox) and the list of LLM-generated Ngear equipment prompt texts (Lgear) into the pre-trained VLM (such as OpenAI CLIP) to generate text and image embeddings. Calculate the matrix multiplication of the personnel image block visual embedding matrix (Eperson) and the equipment text embedding matrix (Egear) to generate the predicted similarity matrix (Apredict), which has a shape of Npbbox x Ngear.

[0032] P3-3: Set a threshold , if an item in the similarity matrix is greater than or equal to , it is predicted that the personnel is wearing the equipment.

[0033] P4-1: Utilize the same open vocabulary object detection model as the safety equipment detection part to generate the bounding boxes of the detected safety equipment and extract the equipment image blocks.

[0034] P4-2: Input the list of equipment image blocks (Libbox) and the list of attribute prompt texts generated by the visual prompt part (Lattribute) into the pre-trained VLM (such as OpenAI CLIP) to generate equipment visual embeddings (Egear) and attribute text embeddings (Eattribute).

[0035] P4-3: Extract embedding vectors (Vgear and Vattribute) for each equipment image block and its corresponding attribute prompt text. Calculate the cosine similarity ω(Vgear, Vattribute) between Vgear and Vattribute to measure their alignment.

[0036] P4-4: Set another threshold ( ), if the cosine similarity is greater than or equal to , it is predicted that the equipment meets the attribute requirements.

[0037] Embodiment 1: An image containing workers performing aerial work (e.g., maintenance of power transmission line towers) at a power construction site is detected to verify whether their personal safety equipment is worn in compliance. The test process is as follows: Step 1-1: Use the image captioning model to generate descriptive captions for the input image.

[0038] Model selection: Choose the Qwen-VL-7B model as the image captioning model. This model is chosen because it has advanced multi-modal capabilities, integrating visual and language understanding, and can generate more detailed and contextually appropriate descriptions.

[0039] Effect: The generated captions provide a concise summary of the visual content, for example, may contain information such as "power transmission line", "power transmission tower", "aerial work personnel", etc.

[0040] Step 1-2: Improve scene recognition accuracy through LoRA fine-tuning.

[0041] In order to improve the performance of the scene recognition module while adhering to the constraints of computing resources, accurate scene recognition is provided. LoRA is applied to the query weight and key weight matrices in the multi-head self-attention module of the Transformer architecture. Fine-tuning uses text prompts "in XXX scene" paired with corresponding images as text input. For the power construction site aerial work scene, the text input is constructed as a prompt similar to "in the power construction site aerial work scene". After LoRA fine-tuning, the image captioning model can more accurately recognize specific scenes (such as power construction site aerial work).

[0042] Step 2-1: Query the necessary safety equipment list from Qwen-VL-7B.

[0043] To generate a list of safety equipment that workers need to wear in the identified scenario using Qwen-VL-7B. Input is set as: scenario information obtained from image captioning model (“power construction site aerial work”). Input the scenario information into Qwen-VL-7B and conduct the first query. For example, query example: “list the safety equipment that workers need to wear in the power construction site aerial work”. Output is: the list of safety items that should be worn in the scenario generated by Qwen-VL-7B. For power construction site aerial work, the list includes: hard hat, safety harness, lanyard, safety glasses / goggles, work gloves, high-visibility clothing, safety boots. Step 2-2: Query Qwen-VL-7B for visual attribute description of safety equipment.

[0044] To guide Qwen-VL-7B to summarize the specific visual attribute description that each piece of equipment should have based on the safety equipment list generated in the previous step and scenario information. This is because even the same safety equipment may have different fine-grained requirements in different scenarios. Input is set as: safety equipment list generated in the previous step and scenario information. Based on the response of Qwen-VL-7B, conduct the second query. Query example: “summarize the visual features of these [safety equipment list] in the power construction site aerial work scenario”. Output is: the visual attribute description that each piece of safety equipment should have generated by Qwen-VL-7B.

[0045] Step 2-3: Obtain fine-grained visual attributes of safety equipment.

[0046] Detailed visual attribute descriptions that each safety equipment should have are obtained from the responses of Qwen-VL-7B. Attribute classification: These attributes are classified according to their ease of observability. Directly observable: attributes that can be directly observed from visual features. For example: hard shell of safety helmet, specific color. Lanyard material of safety belt, D-ring, visible fastener. Lens type, frame, side shield of safety glasses. Fluorescent color, reflective strip of high-visibility clothing. Material, coverage of work gloves. Contextually observable: attributes that require some scene understanding, but still have a significant visual component. For example: whether the safety helmet has a chin strap. Specific color of safety belt, whether it is connected to a lanyard. Material of work gloves. Coverage of high-visibility clothing. Ankle support of safety boots. Inferred observable: attributes inferred based on context, use case, or additional knowledge. For example: whether the lanyard contains an energy absorber. Impact resistance, UV protection of safety glasses. Grip, cut resistance of work gloves. Steel toe, non-slip sole, electric shock protection, waterproof of safety boots. Output: a list of fine-grained visual attribute descriptions classified by category, related to the scene and equipment.

[0047] Step 3-1: Detect personnel in the scene.

[0048] In order to locate all personnel in the original image, the open vocabulary object detection model YOLO-World is used to process the original image, detect all objects of the "person" category, and generate their bounding boxes (Npbbox). YOLO-World is pre-trained on a large dataset and has strong open vocabulary detection capabilities, which can be generalized to new environments and new types of objects without retraining, which is very advantageous for diverse industrial environments. Output: bounding boxes of all detected personnel in the image and their number (Npbbox).

[0049] Step 3-2: Generate embeddings using CLIP and calculate similarity matrix.

[0050] In order to compare the personnel image blocks with the necessary safety equipment text prompts and determine whether the personnel are wearing the required equipment. Input settings: personnel bounding boxes generated in step 3-1; list of necessary safety equipment names generated in step 2-1 (text prompts). Based on the personnel bounding box coordinates, extract the image blocks of each personnel from the original image. These image blocks form the personnel image block list (Lpbbox). Input the personnel image block list (Lpbbox) and the safety equipment name list (as the equipment prompt text list (Lgear)) into the pre-trained visual language model CLIP.

[0051] The CLIP generates a matrix of visual embeddings (Eperson) and a matrix of text embeddings (Egear) for the person image patches and the equipment text prompts, respectively. The matrix multiplication of (Eperson) and (Egear) generates a predicted similarity or affinity matrix (A predict) with shape N pbbox x Ngear. The output is an affinity matrix (A predict) that reflects the similarity between each person image patch and each equipment text prompt.

[0052] Step 3-3: Predict the safety equipment wearing situation.

[0053] According to the similarity matrix, determine whether each person wears each type of safety equipment in the list. The input is set as: The affinity matrix (A predict) generated in step 3-2. Set a threshold ( ). Traverse each item in the similarity matrix, if an item (corresponding to the similarity score of a specific person and a specific equipment) is greater than or equal to , it is predicted that the person wears the equipment. The output is a binary prediction result (yes / no) for each person in the image about whether they wear each safety equipment in the step 2-1 list.

[0054] Step 4-1: Detect the safety equipment worn and extract the image patches.

[0055] Locate the safety equipment itself that is determined to be "worn" in step 3. The input is set as the detection results of step 3 (which personnel wear which equipment). For the safety equipment that is detected to be worn in step 3, use the same open vocabulary object detection model (YOLO-World) as step 3-1 to detect the equipment itself and generate their bounding boxes. Extract the image patches (Libbox) of the equipment from these bounding boxes. The output is a list of image patches (Libbox) of the safety equipment that has been detected.

[0056] Step 4-2: Generate equipment and attribute embeddings using CLIP.

[0057] Compare the equipment image patches with the corresponding fine-grained visual attribute description text. The input is set as the list of equipment image patches (Libbox) generated in step 4-1; the list of fine-grained visual attribute descriptions (text prompts) related to these equipment generated in step 2-3. Input the list of equipment image patches (Libbox) and the list of attribute prompt text (Lattribute) into CLIP. The CLIP generates a matrix of equipment visual embeddings (Egear) and a matrix of attribute text embeddings (Eattribute) for the equipment image patches and the attribute prompt text, respectively. The output is the visual embedding of the detected equipment and the text embedding of its corresponding attribute description.

[0058] Step 4-3: Calculate the similarity between the equipment image block and the attribute text embedding.

[0059] Measure the degree of matching between each equipment image block and its corresponding fine-grained attribute description. The input is set as: the equipment visual embedding generated in Step 4-2 and the attribute text embedding. Extract the embedding vector (Vgear) of each equipment image block and the embedding vector (Vattribute) of its corresponding attribute prompt text. Calculate the cosine similarity ω(Vgear, Vattribute) between the two vectors to measure their alignment. The output is: the cosine similarity score between each detected equipment and its corresponding fine-grained attribute description.

[0060] Step 4-4: Predict whether the equipment meets the attribute requirements (fine-grained verification).

[0061] Determine whether the detected safety equipment meets its specific fine-grained attribute requirements according to the cosine similarity score. The input is set as: the list of cosine similarity scores generated in Step 4-3. Set another threshold ( ). If the cosine similarity between a certain equipment image block and its corresponding attribute prompt text is greater than or equal to , it is predicted that the equipment meets the attribute requirements. The output is: a binary prediction result (yes / no) for each detected safety equipment on whether it meets the specific fine-grained attribute requirements in Step 2-3. The verification results will be evaluated for different types of attributes.

[0062] Example 2: Detect an image containing a ladder work site to verify whether there is a violation of the escalator without a person. According to the safety operation specification, when working on the ladder, a person must be at the bottom of the ladder to monitor the escalator and ensure the safety of the workers. The detection process is as follows: Step 1-1: Use the image captioning model to generate descriptive captions for the input image.

[0063] Model selection: Choose the Qwen-VL-7B model as the image captioning model. This model is chosen because it has advanced multi-modal capabilities, integrating visual and language understanding, and can generate more detailed and contextually appropriate descriptions.

[0064] Function: The generated captions provide a concise summary of the visual content, such as possible information including "ladder", "high work", "construction site", "workers", etc.

[0065] Step 1-2: Improve scene recognition accuracy through LoRA fine-tuning.

[0066] To improve the performance of the scene recognition module while adhering to computational resource constraints, accurate scene recognition is provided. LoRA is applied to the query weight and key weight matrices in the multi-head self-attention module of the Transformer architecture. Fine-tuning uses text prompts "in the XXX scene" paired with corresponding images as text input.

[0067] For the ladder work scene, the text input is constructed as a prompt similar to "in the ladder work construction scene."

[0068] Output: An image captioning model fine-tuned with LoRA that can more accurately recognize specific scenes, such as a ladder work construction site.

[0069] Step 2-1: Query Qwen-VL-7B for ladder work safety requirements.

[0070] Objective: Based on the identified scene, generate safety work requirements in that scene using Qwen-VL-7B.

[0071] Input settings: Scene information obtained from the image captioning model ("ladder work construction scene").

[0072] Query example: "List the safety requirements and regulations that must be followed in ladder work."

[0073] Output: Qwen-VL-7B generated list of ladder work safety requirements, including: (1) Ladder holder at bottom. (2) Safety supervisor at base. (3) Proper ladder angle. (4) Secure ladder top. (5) Worker wearing safety harness. (6) Clear area around ladder.

[0074] Step 2-2: Query Qwen-VL-7B for visual recognition features of ladder personnel. Objective: Based on the ladder work safety requirements, guide Qwen-VL-7B to summarize specific visual recognition features that ladder personnel should possess.

[0075] Input settings: Safety requirement list and scene information generated in the previous step.

[0076] Query example: "Describe the typical visual features and positional characteristics of ladder personnel at the base of the ladder in the ladder work scene."

[0077] Output: Qwen-VL-7B generated escalator personnel visual recognition feature description.

[0078] Step 2-3: Obtain fine-grained recognition attributes of escalator personnel; Obtain detailed visual recognition attributes that escalator personnel should have from Qwen-VL-7B's response.

[0079] Directly observable attributes include: (1) Personnel position near the bottom of the ladder. (2) Both hands touching or close to the ladder. (3) Stand facing the direction of the ladder. (4) Wear work clothes.

[0080] Contextually observable attributes include: (1) Relative distance between personnel and ladder. (2) Body posture (standing, supporting posture). (3) Attention orientation (observing work on the ladder).

[0081] (4) Concentrated monitoring state.

[0082] Inferred observable scenarios include: (1) Awareness of monitoring responsibility. (2) Ready to support state at any time. (3) Cooperative relationship with personnel on the ladder.

[0083] Output: Escalator personnel fine-grained visual recognition attribute description list classified by category.

[0084] Step 3-1: Detect ladders and personnel in the scene; Purpose: Locate all ladder and personnel objects in the original image.

[0085] Use the open vocabulary object detection model YOLO-World to process the original image, detect all "ladder" and "person" category objects, and generate their bounding boxes.

[0086] Output: Bounding boxes of all detected ladders in the image and their number (Nlbbox). Bounding boxes of all detected personnel in the image and their number (Npbbox).

[0087] Step 3-2: Analyze the spatial relationship between ladders and personnel using CLIP; Purpose: Analyze whether there are escalator personnel in the ladder bottom area.

[0088] Input settings: Ladder bounding boxes and personnel bounding boxes generated in step 3-1. Ladder bottom area definition (specific range around the bottom of the ladder).

[0089] Processing process: 1. Based on the ladder bounding box, calculate the ladder bottom area range (a rectangular area extending downward from the bottom of the ladder).

[0090] 2. Check if the personnel bounding box overlaps or is adjacent to the ladder bottom area.

[0091] 3. Extract the image patch of the escalator bottom area and the image patch of the adjacent person.

[0092] 4. Input the image patch into the pre-trained CLIP model.

[0093] Output: Similarity matrix between the image patch of the escalator bottom area and the text description of "escalator person".

[0094] Step 3-3: Predict the presence of escalator personnel; Purpose: Determine whether each escalator has corresponding escalator personnel according to the similarity matrix.

[0095] Input settings: Similarity matrix generated in step 3-2.

[0096] Processing process: 1. Set threshold θ1; 2. Traverse the similarity matrix, if the similarity score between the escalator bottom area and "escalator personnel" is greater than or equal to θ1, it is predicted that the escalator has escalator personnel.

[0097] 3. Count the number of escalators without personnel; Output: Binary prediction results (yes / no) for the presence of escalator personnel for each escalator in the image.

[0098] Step 4-1: Detect escalator personnel and extract image patches; Purpose: For escalators determined to have "escalator personnel", further locate the specific position of the escalator personnel.

[0099] Input settings: Detection results of step 3 (which escalators have escalator personnel).

[0100] Use YOLO-World to detect "person" objects in the escalator bottom area, generate accurate bounding boxes of the escalator personnel, and extract corresponding image patches.

[0101] Output: List of detected escalator personnel image patches (Lhbbox).

[0102] Step 4-2: Generate escalator personnel behavior embedding using CLIP; Purpose: Compare the escalator personnel image patch with the standard escalator behavior description.

[0103] Input settings: (1) List of escalator personnel image patches (Lhbbox) generated in step 4-1. (2) List of escalator personnel behavior feature descriptions (text prompts) generated in step 2-3.

[0104] Processing process: 1. Input the list of escalator people image patches (Lhbbox) and the list of behavior feature texts (Lbehavior) into CLIP.

[0105] 2. CLIP generates the matrix of escalator people visual embeddings (Eholder) and the matrix of behavior text embeddings (Ebehavior) respectively.

[0106] Output: The visual embeddings of detected escalator people and the text embeddings of their corresponding behavior features.

[0107] Step 4-3: Calculate the escalator people behavior compliance similarity; Purpose: Measure the degree of matching between the actual behavior of each escalator person and the standard escalator behavior.

[0108] Input settings: The escalator people visual embeddings and behavior text embeddings generated in step 4-2.

[0109] Processing process: 1. Extract the embedding vector (Vholder) of each escalator people image patch; 2. Extract the embedding vector (Vbehavior) of the corresponding standard behavior feature; 3. Calculate the cosine similarity ω(Vholder, Vbehavior); Output: The cosine similarity score between each detected escalator person and the standard escalator behavior.

[0110] Step 4-4: Predict escalator behavior compliance (fine-grained verification).

[0111] Purpose: Determine whether the behavior of the escalator person meets the safety specification requirements according to the cosine similarity score.

[0112] Input settings: The list of cosine similarity scores generated in step 4-3.

[0113] Processing process: 1. Set the threshold θ2.

[0114] 2. If the cosine similarity between the escalator people image patch and the standard behavior feature is greater than or equal to θ2, predict that the escalator behavior is compliant.

[0115] Output: The binary prediction result of behavior compliance (compliant / non-compliant) for each detected escalator person.

[0116] The above embodiments are only preferred technical solutions of the present application, and should not be regarded as a limitation of the present application. The protection scope of the present application should be the technical solutions recited in the claims, including equivalent replacement solutions of the technical features recited in the claims. That is, equivalent replacement improvements within this scope are also within the protection scope of the present application.

Claims

1. A method for detecting the compliance of safety equipment wearing by workers at power construction sites based on a multimodal hybrid model, characterized in that, Includes the following steps: S1: Use a multimodal large model to generate scene description captions for the input image, and fine-tune the multimodal large model through a low-rank adaptation method to identify the operation scene type of the current power construction site and output the scene type identification result; S2: Input the scene type recognition result output by S1 into the large language model, and perform two natural language queries through the large language model; S3: Use the open vocabulary object detection model to detect all people in the input image, generate personnel bounding boxes, extract personnel image patches based on the personnel bounding boxes, input the personnel image patches and the list of safety equipment names output by S2 into the visual language model, calculate the similarity between the text embeddings of the personnel image patches and each safety equipment name in the list of safety equipment names, determine whether each person is wearing the required safety equipment based on the similarity, and output the safety equipment wearing detection result. S4: Using an open vocabulary object detection model, locate the safety equipment identified as being worn in the safety equipment wearing detection results output by S3, extract the safety equipment image blocks, input the safety equipment image blocks and the fine-grained visual attribute description output by S2 into the visual language model, calculate the cosine similarity between the visual embedding of the safety equipment image blocks and the text embedding of the fine-grained visual attribute description, determine whether the wearing of the safety equipment complies with the specifications based on the cosine similarity, and output the fine-grained compliance verification result.

2. The method for detecting the compliance of safety equipment wearing by workers at power construction sites based on a multimodal hybrid model according to claim 1, characterized in that: The sub-steps of S1 are: S1.1: Use a multimodal large model to generate image captions containing semantic information of power construction sites. The image captions include scene features of high-altitude operations, ladder operations, or underground operations. S1.2: Update the parameters of the multimodal large model by means of the low-rank adaptation method. Only adjust the low-rank trainable parameters corresponding to the query weight matrix and key weight matrix of the multi-head self-attention mechanism in the multimodal large model, and keep the original weight matrix of the multimodal large model unchanged. S1.3: Fine-tune the multimodal large model by using image and text pairing data of power construction scene. The text pairing data adopts a three-segment text format so that the scene description subtitles output by the multimodal large model can accurately reflect the actual working environment.

3. The method for detecting the compliance of safety equipment wearing by workers at power construction sites based on a multimodal hybrid model according to claim 1, characterized in that: The two natural language queries include: The first query generates a list of safety equipment names that workers need to wear under the current work scenario type. The second query generates a list of security equipment names with detailed visual attribute descriptions for each type of security equipment, and outputs the list of security equipment names and the detailed visual attribute descriptions.

4. The method for detecting the compliance of safety equipment wearing by workers at power construction sites based on a multimodal hybrid model according to claim 1, characterized in that: The sub-steps of S2 are: S2.1: The scene type recognition result output by S1 is used as a dynamic scene variable and combined with the first fixed semantic segment and the second fixed semantic segment to form the first natural language query instruction. The first fixed semantic segment is used to express the intention of querying safety equipment, and the second fixed semantic segment is used to limit the query object to the safety equipment that workers need to wear. The first natural language query instruction is input into the large language model to generate a list of safety equipment names. The list of safety equipment names includes safety helmet, safety belt, connecting rope, safety glasses, high-visibility clothing, and safety boots. S2.2: The scene type recognition result output by S1 is used as a dynamic scene variable and combined with the third fixed semantic segment, the fourth fixed semantic segment, the list of security equipment names generated by S2.1 and the fifth fixed semantic segment to form a second natural language query instruction. The third fixed semantic segment is used to express the intention of attribute summary, the fourth fixed semantic segment is used to introduce the equipment list, and the fifth fixed semantic segment is used to limit the query content to visual features. The second natural language query instruction is input into a large language model to generate a three-level fine-grained visual attribute description for each security equipment. S2.3: Classify the generated fine-grained visual attribute descriptions to obtain directly observable attributes, context-observable attributes, and inferred observable attributes; S2.4: Perform direct observable attribute recognition, including color, material, and structural components; perform context-observable attribute recognition, including positional relationships, posture, and connection methods; perform inferred observable attribute recognition, including energy absorption capacity, impact resistance level, and anti-slip performance.

5. The method for detecting the compliance of safety equipment wearing by workers at power construction sites based on a multimodal hybrid model according to claim 1, characterized in that: The sub-steps of S3 are: S3.1: Utilize an open-vocabulary object detection model to detect all "person" class targets in the input image and generate person bounding boxes; S3.2: Based on the person bounding box, crop out the person image blocks from the input image to form a list of person image blocks; S3.3: Input the list of personnel image blocks and the list of safety equipment names output by S2 into the visual language model. The visual language model encodes the list of personnel image blocks into a personnel visual embedding matrix and the list of safety equipment names into an equipment text embedding matrix. S3.4: Calculate the matrix multiplication of the personnel visual embedding matrix and the equipment text embedding matrix to generate a similarity matrix; S3.5: Set a judgment threshold. If the value of an element in the similarity matrix is ​​greater than or equal to the judgment threshold, then it is determined that the corresponding person is wearing the corresponding safety equipment. Otherwise, it will be determined that the corresponding personnel are not wearing the corresponding safety equipment.

6. The method for detecting the compliance of safety equipment wearing by workers at power construction sites based on a multimodal hybrid model according to claim 1, characterized in that: The sub-steps of S4 are: S4.1: Use the open vocabulary object detection model to perform secondary detection on the safety equipment identified as being worn in the safety equipment wearing detection results output by S3, and generate safety equipment bounding boxes; S4.2: Based on the bounding box of the safety equipment, crop out the safety equipment image blocks from the input image to form a list of safety equipment image blocks; S4.3: Input the list of safety equipment image blocks and the fine-grained visual attribute description output by S2 into the visual language model. The visual language model encodes the list of safety equipment image blocks into a safety equipment visual embedding and encodes the fine-grained visual attribute description into an attribute text embedding. S4.4: For each security equipment image block, calculate the cosine similarity between the security equipment visual embedding and the corresponding attribute text embedding; S4.5: Set a verification threshold. If the cosine similarity is greater than or equal to the verification threshold, the security equipment is determined to meet the corresponding fine-grained attribute requirements; otherwise, the security equipment is determined not to meet the corresponding fine-grained attribute requirements. S4.6: Input the fine-grained compliance verification results into the compliance judgment and output module. If there are safety devices that do not meet the fine-grained attribute requirements, an alarm signal will be generated through the compliance judgment and output module.

7. A system for detecting the compliance of safety equipment wearing by workers at power construction sites based on a multimodal hybrid model, characterized in that, The system employs a method for detecting the compliance of safety equipment wearing by power construction site workers based on a multimodal hybrid model, as described in any one of claims 1-6. The system comprises: The scene recognition module is used to generate scene description captions for the input image using a multimodal large model, and to fine-tune the multimodal large model through a low-rank adaptation method to identify the operation scene type of the current power construction site and output the scene type recognition result. The visual cue generation module is used to input the scene type recognition results output by the scene recognition module into a large language model, and generate a list of safety equipment names required for the current work scene type and a fine-grained visual attribute description for each safety equipment through the large language model. The safety equipment detection module is used to detect all personnel in the input image using an open vocabulary object detection model, generate personnel bounding boxes, extract personnel image patches, input the personnel image patches and the list of safety equipment names output by the visual cue generation module into the visual language model, calculate similarity, determine whether the personnel are wearing the required safety equipment, and output the safety equipment wearing detection result. The fine-grained verification module is used to locate the safety equipment identified as being worn in the safety equipment wearing detection results using the open vocabulary object detection model, extract the safety equipment image blocks, input the safety equipment image blocks and the fine-grained visual attribute descriptions output by the visual cue generation module into the visual language model, calculate the cosine similarity, determine whether the wearing of safety equipment complies with the specifications, and output the fine-grained compliance verification results. The compliance determination and output module is used to receive the fine-grained compliance verification results output by the fine-grained verification module. If any security equipment does not meet the fine-grained attribute requirements, an alarm signal is generated.

8. The safety equipment wearing compliance detection system for power construction site workers according to claim 7, characterized in that: The visual cues generation module includes: The equipment query submodule is used to take the operation scenario type output by the scenario recognition module as a dynamic variable, combine it with the first fixed semantic segment and the second fixed semantic segment to form the first natural language query command, and send it to the large language model to obtain the list of safety equipment names. The attribute generation submodule is used to combine the job scenario type, the list of safety equipment names with the third, fourth and fifth fixed semantic segments to form a second natural language query command, and send it to the large language model to obtain fine-grained visual attribute descriptions. The data organization submodule is used to store the list of safety equipment names and fine-grained visual attribute descriptions in a structured data format for use by the safety equipment testing module and the fine-grained verification module.

9. The safety equipment wearing compliance detection system for power construction site workers according to claim 7, characterized in that: The visual language model in the fine-grained verification module is a pre-trained image-text alignment model. The image-text alignment model is trained through contrastive learning and is used to calculate the semantic matching degree between security equipment image patches and fine-grained visual attribute descriptions.

10. A computer device, characterized in that: The device includes one or more processors, on which one or more executable programs are stored. When the one or more executable programs are executed by the one or more processors, they are used to implement the method for detecting the compliance of safety equipment wearing by power construction site workers based on a multimodal hybrid model, as described in any one of claims 1-6.

Images