A system for identifying illegal actions at power operation sites based on multimodal large models

Abstract

The present invention discloses a system for identifying illegal actions at power operation sites based on a multimodal large model, and relates to the technical field of power operation site supervision; the system comprises: a data acquisition module for acquiring monitoring videos and task information; a quality analysis module for calculating the quality score of video frames; an image enhancement module for enhancing the target image if the quality score is less than a first preset threshold; a violation recognition module for taking the video frame sequence and the task information as inputs of a pre-trained multimodal large model to obtain a violation action recognition result; and a violation alarm module for sending the violation action recognition result to a handheld terminal of a manager. Through the pre-trained multimodal large model, it is possible to judge whether the behavior of the operator is illegal based on the video data and task text data at the operation site, thereby achieving real-time monitoring and immediate warning, and improving the efficiency and timeliness of supervision.

Description

Technical Field

[0001] The present invention relates to the technical field of power operation site supervision, and in particular to a power operation site illegal action recognition system based on a multi-modal large model. Background Art

[0002] With the popularization of monitoring equipment, although many power operation sites have installed video monitoring systems, manual inspection is still required, which cannot achieve real-time supervision, but is used for post-investigation and evidence collection. For a long time, the safety management of power operation sites has mainly relied on manual inspections and on-site supervision. This method is not only time-consuming and labor-intensive, but also easily affected by human factors, such as fatigue and negligence. This leads to low efficiency and error-prone manual monitoring, making it difficult to detect and deal with violations in a timely manner, which may lead to serious safety accidents. Summary of the invention

[0003] The purpose of the present invention is to solve the problem of low efficiency and easy error of manual monitoring mentioned in the above background technology, and to propose a system for identifying illegal actions at power operation sites based on a multi-modal large model.

[0004] In a first aspect of the present invention, there is provided a system for identifying illegal actions at power operation sites based on a multimodal large model, the system comprising:

[0005] A data acquisition module is used to acquire monitoring video and task information of the power operation site during the target period, and extract a video frame sequence according to the monitoring video and a preset step length;

[0006] A quality analysis module, used for calculating the quality score of a target image; the target image is any video frame in the video frame sequence;

[0007] An image enhancement module, configured to enhance the target image if the quality score is less than a first preset threshold;

[0008] A traffic violation recognition module, used to use the video frame sequence and the task information as inputs of a pre-trained multimodal large model to obtain a traffic violation action recognition result;

[0009] The violation alarm module is used to send the violation action recognition result to the handheld terminal of the manager.

[0010] Optionally, the quality analysis module includes:

[0011] The brightness calculation module is used to calculate the brightness value as the quality score based on the red, green and blue channel values ​​of the target image:

[0012] ;

[0013] Where Sc is the quality score required; the size of the target image is M×N; R ij , G ij and B ij are respectively the red, green and blue channel values ​​of the pixel in the i-th row and j-th column in the target image.

[0014] Optionally, the image enhancement module includes:

[0015] A normalization module, used for normalizing the pixel values ​​of the target image to obtain a first feature tensor;

[0016] an enhancement parameter calculation module, configured to use the first feature tensor as an input of a pre-trained image enhancement model to obtain an enhancement parameter tensor; the size of the enhancement parameter tensor is the same as the size of the first feature tensor;

[0017] A feature updating module, configured to obtain a second feature tensor according to the enhanced parameter tensor and the first feature tensor:

[0018] ;

[0019] Wherein, E(p) is the element value of the second feature tensor at position p; T(p) is the element value of the first feature tensor at position p; A(p) is the element value of the enhancement parameter tensor at position p;

[0020] An image restoration module, used for obtaining a temporary image according to the second feature tensor;

[0021] The iterative enhancement module is used to use the temporary image as the target image, return to the normalization module, perform iterative enhancement until a preset iteration stop condition is reached, and use the final temporary image as the target image.

[0022] Optionally, the image enhancement model includes a first convolutional network layer, a multi-scale feature fusion layer, a weight calculation layer, a weighted fusion layer and an output layer; the multi-scale feature fusion layer includes a plurality of convolution kernels of different sizes and a summation layer;

[0023] in:

[0024] The first convolutional network layer is used to receive the first feature tensor as input and output a third feature tensor;

[0025] The multi-scale feature fusion layer is used to extract features from the third feature tensor using multiple convolution kernels to obtain multiple fourth feature tensors, and sum the multiple fourth feature tensors to obtain a fifth feature tensor;

[0026] The weight calculation layer is used to receive the fifth feature tensor as input and output a fusion weight vector corresponding to multiple fourth feature tensors;

[0027] The weighted fusion layer is used to receive a plurality of fourth feature tensors and a plurality of corresponding fusion weight vectors as input, and output a sixth feature tensor;

[0028] The output layer is used to receive the sixth feature tensor as input and output an enhanced parameter tensor.

[0029] Optionally, the multimodal large model includes a rule determination sub-model, a behavior analysis sub-model and a comprehensive judgment sub-model; wherein:

[0030] The rule determination sub-model is used to obtain a target rule set according to the task information;

[0031] The behavior analysis sub-model is used to obtain behavior labels of multiple categories according to the video frame sequence;

[0032] The comprehensive judgment sub-model is used to obtain the illegal action recognition result according to the target rule set and multiple categories of behavior labels.

[0033] Optionally, the rule determination sub-model includes:

[0034] The information matching module is used to use the task information as a keyword to search in a preset knowledge base to obtain multiple on-site operation rules, and incorporate the multiple on-site operation rules into a target rule set.

[0035] Optionally, the behavior analysis sub-model includes an entity detection module, a skeleton extraction module and a plurality of behavior qualitative modules; wherein:

[0036] The entity detection module is used to obtain entity detection results according to the target image; the entity detection results include entity category and entity location;

[0037] The skeleton extraction module is used to extract the skeleton information of the operator in the target image;

[0038] The target behavior qualitative module is used to determine the behavior label of the corresponding category according to the entity detection result and the skeleton information; the target behavior qualitative module is any one of the multiple behavior qualitative modules.

[0039] Optionally, the multiple behavior qualitative modules include a helmet wearing qualitative module; the multiple categories of behavior tags include a helmet wearing tag; the helmet wearing tag includes safely wearing a helmet and not wearing a helmet;

[0040] Wherein, the helmet wearing qualitative module is used for:

[0041] Obtaining a head region according to the skeleton information;

[0042] According to the location information of the helmet entity, the helmet area is obtained;

[0043] According to the head area and the helmet area, the wearing estimation value is obtained:

[0044] ;

[0045] Among them, SHW is the required wearing estimate; HE is the set of pixels in the head area; SH is the set of pixels in the helmet area; |HE| represents the number of elements in the set HE;

[0046] If the wearing estimation value is greater than a second preset threshold, the helmet wearing tag of the target image is determined as a helmet being safely worn; otherwise, the helmet wearing tag is determined as a helmet not being worn;

[0047] The helmet wearing label of the video frame sequence is determined according to the helmet wearing label of each image in the video frame sequence.

[0048] Optionally, the entity detection module embeds a YOLOv8 model; and the skeleton extraction module embeds an OpenPose model.

[0049] Beneficial effects of the present invention:

[0050] The present invention proposes a system for identifying illegal actions at an electric power operation site based on a multimodal large model. The system comprises: a data acquisition module, which is used to acquire monitoring video and task information at the electric power operation site in a target period, and extract a video frame sequence according to the monitoring video and a preset step length; a quality analysis module, which is used to calculate the quality score of a target image; the target image is any video frame in the video frame sequence; an image enhancement module, which is used to enhance the target image if the quality score is less than a first preset threshold; a violation recognition module, which is used to use the video frame sequence and task information as inputs of a pre-trained multimodal large model to obtain a violation action recognition result; and a violation alarm module, which is used to send the violation action recognition result to a handheld terminal of a manager.

[0051] Through the pre-trained multimodal large model, it is possible to judge whether the operator's behavior is illegal based on the video data and task text data of the work site, realizing real-time monitoring and immediate warning, and improving the efficiency and timeliness of supervision. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The present invention will be further described below in conjunction with the accompanying drawings.

[0053] Figure 1An architecture diagram of a system for identifying illegal actions at power operation sites based on a multimodal large model is provided for an embodiment of the present invention;

[0054] Figure 2 A framework diagram of an image enhancement model is provided for an embodiment of the present invention. DETAILED DESCRIPTION

[0055] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] The embodiment of the present invention provides a system for identifying illegal actions at power operation sites based on a multi-modal large model. Figure 1 , Figure 1 The present invention provides an architecture diagram of a system for identifying illegal actions at power operation sites based on a multimodal large model. The system includes:

[0057] The data acquisition module is used to obtain the monitoring video and task information of the power operation site during the target period, and extract the video frame sequence according to the monitoring video and the preset step length.

[0058] The quality analysis module is used to calculate the quality score of the target image.

[0059] The image enhancement module is used to enhance the target image if the quality score is less than a first preset threshold.

[0060] The violation recognition module is used to take the video frame sequence and task information as the input of the pre-trained multimodal large model to obtain the violation action recognition result.

[0061] The violation alarm module is used to send the violation action recognition results to the manager's handheld terminal.

[0062] The target image is any video frame in the video frame sequence.

[0063] A system for identifying illegal actions at power operation sites based on a multimodal large model is provided in an embodiment of the present invention. Through a pre-trained multimodal large model, it is possible to determine whether the behavior of the operator is illegal based on the video data and task text data of the operation site, thereby achieving real-time monitoring and immediate warning, and improving the efficiency and timeliness of supervision.

[0064] In one implementation, in actual monitoring, due to factors such as light, weather, and equipment aging, the quality of the monitoring video may be poor, affecting the recognition effect. By enhancing the low-quality video frames, the quality of the video frames is ensured, and the misjudgment caused by poor image quality is reduced, thereby improving the accuracy and reliability of the recognition results.

[0065] In one implementation, the surveillance video can be acquired by a fixed-position camera on site, or by a handheld terminal of a supervisor. When in use, all video frames are adjusted to the same size.

[0066] In one implementation, the task information may include task scenario, specific work content of the task, start time of the task, task location, operator information, equipment information used, environmental conditions of the work site, and other information.

[0067] In one implementation, the target time period may be a period of time before the current moment, such as the previous minute. The preset step size may be set to 2 seconds, and a video frame is taken every 2 seconds. At this time, there are 30 images in the video frame sequence.

[0068] In one embodiment, the quality analysis module includes:

[0069] The brightness calculation module is used to calculate the brightness value as the quality score based on the red, green and blue channel values ​​of the target image:

[0070] ;

[0071] Where Sc is the desired quality score; the size of the target image is M×N; R ij , G ij and B ij are the red, green, and blue channel values ​​of the pixel in the i-th row and j-th column of the target image, respectively.

[0072] In one implementation, brightness is an important factor in evaluating image quality. Under low-light conditions, details in the image may be lost, making it difficult to identify the edges and features of the target. The model may miss some darker targets, reducing the detection accuracy. By combining the information of the three channels, the overall brightness of the image can be more accurately reflected, avoiding the deviation caused by the brightness calculation of a single channel.

[0073] In one embodiment, the image enhancement module includes:

[0074] The normalization module is used to normalize the pixel values ​​of the target image to obtain a first feature tensor.

[0075] The enhancement parameter calculation module is used to use the first feature tensor as an input of a pre-trained image enhancement model to obtain an enhancement parameter tensor.

[0076] The feature update module is used to obtain the second feature tensor according to the enhanced parameter tensor and the first feature tensor:

[0077] ;

[0078] Among them, E(p) is the element value of the second feature tensor at position p; T(p) is the element value of the first feature tensor at position p; A(p) is the element value of the enhancement parameter tensor at position p.

[0079] The image restoration module is used to obtain a temporary image according to the second feature tensor.

[0080] The iterative enhancement module is used to use the temporary image as the target image, return to the normalization module, perform iterative enhancement until a preset iteration stop condition is reached, and use the final temporary image as the target image.

[0081] Among them, the size of the enhanced parameter tensor is the same as the size of the first feature tensor.

[0082] In one implementation, enhancement parameters are dynamically calculated based on a pre-trained model, and the enhancement strategy can be automatically adjusted for different images. Moreover, enhancement is performed separately at the pixel level, so that good enhancement effects can be obtained in different environments and conditions.

[0083] In one implementation, the pixel values ​​of the target image are normalized to obtain the first feature tensor. The RGB channels of the image can be normalized separately. For example, if a pixel at a certain point is (r 0 , g 0 , b 0 ), after normalization, it is ( ).

[0084] In one implementation, the preset iteration stop condition may be reaching a fixed number of iterations.

[0085] In one embodiment, the image enhancement model includes a first convolutional network layer, a multi-scale feature fusion layer, a weight calculation layer, a weighted fusion layer, and an output layer; the multi-scale feature fusion layer includes a plurality of convolution kernels of different sizes and a summation layer;

[0086] in:

[0087] The first convolutional network layer is used to receive the first feature tensor as input and output the third feature tensor.

[0088] The multi-scale feature fusion layer is used to extract features from the third feature tensor using multiple convolution kernels to obtain multiple fourth feature tensors, and sum the multiple fourth feature tensors to obtain a fifth feature tensor.

[0089] The weight calculation layer is used to receive the fifth feature tensor as input and output a fusion weight vector corresponding to multiple fourth feature tensors.

[0090] The weighted fusion layer is used to receive multiple fourth feature tensors and multiple corresponding fusion weight vectors as input, and output a sixth feature tensor.

[0091] The output layer is used to receive the sixth feature tensor as input and output the enhanced parameter tensor.

[0092] In one implementation, see Figure 2 , Figure 2 A framework diagram of an image enhancement model provided in an embodiment of the present invention.

[0093] The first convolutional network layer can contain 7 convolutional layers, and a RELU is placed after each convolutional layer as an activation function;

[0094] The multi-scale feature fusion layer may include three convolution kernels of sizes 3, 5, and 7, respectively, to process the third feature tensor, respectively, to obtain three fourth feature tensors (called U41, U42, and U43); the three are added together to obtain the fifth feature tensor (called U5);

[0095] The weight calculation layer includes a global average pooling layer (used to obtain embedded global information S1 through U5), a fully connected layer 1 (used to obtain compact global information S2 through S1), a fully connected layer 2 (used to obtain the fusion weight vector of U41 through S2), a fully connected layer 3 (used to obtain the fusion weight vector of U42 through S2), and a fully connected layer 4 (used to obtain the fusion weight vector of U43 through S2);

[0096] The weighted fusion layer includes weighted layer 1, weighted layer 2, weighted layer 3 and summation layer 2, which are used to perform weighted summation on U1, U2 and U3 to obtain the sixth feature tensor;

[0097] The output layer can be a depthwise separable convolutional network, and the activation function can be Tanh.

[0098] By making full use of spatial features and using receptive fields of different scales for multi-scale fusion, the interference of noise on image enhancement can be reduced.

[0099] In one embodiment, the multimodal large model includes a rule determination sub-model, a behavior analysis sub-model and a comprehensive judgment sub-model; wherein:

[0100] The rule determination sub-model is used to obtain the target rule set based on the task information.

[0101] The behavior analysis sub-model is used to obtain behavior labels of multiple categories based on video frame sequences.

[0102] The comprehensive judgment sub-model is used to obtain the violation action recognition results based on the target rule set and multiple categories of behavior labels.

[0103] In one implementation, different work scenarios have different work rules, such as wearing a safety belt for high-altitude work, but not for ground maintenance. Based on the task information, the rules that need to be followed in the current task scenario can be determined to prevent misjudgment.

[0104] In one embodiment, the rule determination sub-model includes:

[0105] The information matching module is used to use the task information as a keyword to search in a preset knowledge base, obtain multiple on-site operation rules, and incorporate the multiple on-site operation rules into the target rule set.

[0106] In one implementation, the preset knowledge base can be a pre-established knowledge graph, and the corresponding entity and attribute relationship can be quickly found through the task information, so that the target rules can be quickly obtained from a large amount of information. The automated information matching process reduces the burden of manual search and screening, reduces the occurrence of mismatches and missed matches, and makes the processing of task information more efficient and accurate.

[0107] In one embodiment, the behavior analysis sub-model includes an entity detection module, a skeleton extraction module and a plurality of behavior characterization modules; wherein:

[0108] The entity detection module is used to obtain entity detection results based on the target image; the entity detection results include entity category and entity position.

[0109] The skeleton extraction module is used to extract the skeleton information of the operator in the target image.

[0110] The target behavior qualitative module is used to determine the behavior label of the corresponding category based on the entity detection results and skeleton information.

[0111] The target behavior qualitative module is any one of a plurality of behavior qualitative modules.

[0112] In one implementation, extracting the skeleton information of the operator can further refine the granularity of behavior analysis. The skeleton information contains the key postures and movement trends of the human body, which helps to identify complex behaviors, such as climbing over a fence.

[0113] In one implementation, multiple behavior qualitative modules may include a helmet wearing qualitative module, a safety belt wearing qualitative module, a shielding suit wearing qualitative module, a protective glove wearing qualitative module, a protective fence establishment qualitative module, a fence climbing qualitative module, a dangerous area stay qualitative module, and the like. The behavior analysis sub-model adopts a modular design so that each module can be flexibly combined and replaced. This design approach is not only convenient for maintenance and upgrading, but also enables the system to be customized and optimized according to different application scenarios and needs. Among them, behaviors such as wearing safety belts, wearing shielding suits, and setting up protective fences can be judged by rule-based methods; behaviors such as climbing over fences can be detected by targeted training of long short-term memory network models.

[0114] In one embodiment, the plurality of behavior qualitative modules include a helmet wearing qualitative module; the plurality of categories of behavior tags include a helmet wearing tag; the helmet wearing tag includes safely wearing a helmet and not wearing a helmet;

[0115] Among them, the helmet wearing qualitative module is used to:

[0116] According to the skeleton information, the head area is obtained;

[0117] According to the location information of the helmet entity, the helmet area is obtained;

[0118] Based on the head area and helmet area, get the estimated value of wearing:

[0119] ;

[0120] Among them, SHW is the required wearing estimate; HE is the set of pixels in the head area; SH is the set of pixels in the helmet area; |HE| represents the number of elements in the set HE;

[0121] If the estimated wearing value is greater than a second preset threshold, the helmet wearing label of the target image is determined as a helmet being safely worn; otherwise, the helmet wearing label is determined as a helmet not being worn;

[0122] The helmet wearing label of the video frame sequence is determined according to the helmet wearing label of each image in the video frame sequence.

[0123] In one implementation, by identifying the head area and the helmet area and calculating the relative position relationship between them (i.e., the wearing estimate), it is possible to more accurately determine whether the helmet is actually worn on the head. This method is more reliable than relying solely on physical detection.

[0124] In one implementation, the helmet wearing label of the video frame sequence is determined based on the helmet wearing label of each image in the video frame sequence. If the ratio of the number of images showing the helmet being safely worn to the entire video frame sequence is greater than a third preset threshold, the helmet wearing label of the video frame sequence is determined to be the helmet being safely worn.

[0125] In one embodiment, the entity detection module embeds a YOLOv8 model; and the skeleton extraction module embeds an OpenPose model.

[0126] In one implementation, YOLOv8 has a fast detection speed and can process high-resolution images while maintaining high accuracy. OpenPose is an advanced deep learning framework for real-time human key point detection and full-body posture estimation. It can simultaneously identify and locate multiple key points of the human body, including the face, hands, feet, etc., so as to accurately capture the posture changes of the human body. In a video surveillance system, YOLOv8 can be used to detect the operator first, and then OpenPose can be used to extract its skeleton information for behavioral analysis.

[0127] In one embodiment, after a traffic violation is detected, the traffic violation video is saved to a database, and the type of violation and the rule basis are added for subsequent query and evidence.

[0128] The above is a detailed description of an embodiment of the present invention, but the content is only a preferred embodiment of the present invention and cannot be considered to limit the scope of implementation of the present invention. All equivalent changes and improvements made within the scope of the present invention should still fall within the scope of the patent coverage of the present invention.

Claims

1. A system for identifying illegal actions at power operation sites based on a multimodal large model. It is characterized in that The system comprises: A data acquisition module is used to acquire monitoring video and task information of the power operation site during the target period, and extract a video frame sequence according to the monitoring video and a preset step length; A quality analysis module, used for calculating the quality score of a target image; the target image is any video frame in the video frame sequence; An image enhancement module, configured to enhance the target image if the quality score is less than a first preset threshold; A traffic violation recognition module, used to use the video frame sequence and the task information as inputs of a pre-trained multimodal large model to obtain a traffic violation action recognition result; A violation alarm module, used to send the violation action recognition result to the handheld terminal of the manager; The quality analysis module includes: The brightness calculation module is used to calculate the brightness value as the quality score based on the red, green and blue channel values ​​of the target image: ； Where Sc is the quality score required; the size of the target image is M×N; R ij , G ij and B ij are the red, green and blue channel values ​​of the pixel in the i-th row and j-th column in the target image, respectively; The image enhancement module comprises: A normalization module, used for normalizing the pixel values ​​of the target image to obtain a first feature tensor; an enhancement parameter calculation module, configured to use the first feature tensor as an input of a pre-trained image enhancement model to obtain an enhancement parameter tensor; the size of the enhancement parameter tensor is the same as the size of the first feature tensor; A feature updating module, configured to obtain a second feature tensor according to the enhanced parameter tensor and the first feature tensor: ； Wherein, E(p) is the element value of the second feature tensor at position p; T(p) is the element value of the first feature tensor at position p; A(p) is the element value of the enhancement parameter tensor at position p; An image restoration module, used for obtaining a temporary image according to the second feature tensor; An iterative enhancement module, used to use the temporary image as a target image, return to the normalization module, perform iterative enhancement until a preset iteration stop condition is reached, and use the final temporary image as the target image; The image enhancement model includes a first convolutional network layer, a multi-scale feature fusion layer, a weight calculation layer, a weighted fusion layer and an output layer; the multi-scale feature fusion layer includes a plurality of convolution kernels of different sizes and a summation layer; in: The first convolutional network layer is used to receive the first feature tensor as input and output a third feature tensor; The multi-scale feature fusion layer is used to extract features from the third feature tensor using multiple convolution kernels to obtain multiple fourth feature tensors, and sum the multiple fourth feature tensors to obtain a fifth feature tensor; The weight calculation layer is used to receive the fifth feature tensor as input and output a fusion weight vector corresponding to multiple fourth feature tensors; The weighted fusion layer is used to receive a plurality of fourth feature tensors and a plurality of corresponding fusion weight vectors as input, and output a sixth feature tensor; The output layer is used to receive the sixth feature tensor as input and output an enhanced parameter tensor.

2. According to claim 1, a system for identifying illegal actions at power operation sites based on a multimodal large model, It is characterized in that The multimodal large model includes a rule determination sub-model, a behavior analysis sub-model and a comprehensive judgment sub-model; wherein: The rule determination sub-model is used to obtain a target rule set according to the task information; The behavior analysis sub-model is used to obtain behavior labels of multiple categories according to the video frame sequence; The comprehensive judgment sub-model is used to obtain the illegal action recognition result according to the target rule set and multiple categories of behavior labels.

3. According to claim 2, a system for identifying illegal actions at power operation sites based on a multimodal large model, It is characterized in that The rule determination sub-model includes: The information matching module is used to use the task information as a keyword to search in a preset knowledge base to obtain multiple on-site operation rules, and incorporate the multiple on-site operation rules into a target rule set.

4. According to claim 2, a system for identifying illegal actions at power operation sites based on a multimodal large model, It is characterized in that The behavior analysis sub-model includes an entity detection module, a skeleton extraction module and multiple behavior qualitative modules; wherein: The entity detection module is used to obtain entity detection results according to the target image; the entity detection results include entity category and entity location; The skeleton extraction module is used to extract the skeleton information of the operator in the target image; The target behavior qualitative module is used to determine the behavior label of the corresponding category according to the entity detection result and the skeleton information; the target behavior qualitative module is any one of the multiple behavior qualitative modules.

5. According to claim 4, a system for identifying illegal actions at power operation sites based on a multimodal large model, It is characterized in that The multiple behavior qualitative modules include a helmet wearing qualitative module; the multiple categories of behavior labels include a helmet wearing label; the helmet wearing label includes safely wearing a helmet and not wearing a helmet; Wherein, the helmet wearing qualitative module is used for: Obtaining a head region according to the skeleton information; According to the position information of the helmet entity, the helmet area is obtained; According to the head area and the helmet area, the wearing estimation value is obtained: ； Among them, SHW is the required wearing estimate; HE is the set of pixels in the head area; SH is the set of pixels in the helmet area; |HE| represents the number of elements in the set HE; If the wearing estimation value is greater than a second preset threshold, the helmet wearing tag of the target image is determined as a helmet being safely worn; otherwise, the helmet wearing tag is determined as a helmet not being worn; The helmet wearing label of the video frame sequence is determined according to the helmet wearing label of each image in the video frame sequence.

6. According to claim 4, a system for identifying illegal actions at power operation sites based on a multimodal large model, It is characterized in that The entity detection module embeds a YOLOv8 model; the skeleton extraction module embeds an OpenPose model.

Images