Power construction detection method based on multi-task learning and multi-modal data

Abstract

The invention relates to a power construction detection method based on multi-task learning and multi-modal data, and the method comprises the following steps: setting a deploy and control ball to collect image data, carrying out the normalization and noise removal processing of the image data, and carrying out the alignment processing of the processed image data; using the image data after the alignment processing to train a YOLOv8 model; carrying out abnormal behavior detection on image data acquired by the deploy and control ball based on an LSTM model in combination with a sliding window technology; the abnormal behaviors comprise the behavior of not wearing a safety helmet, the behavior of not fastening a safety belt and the preset unsafe behavior; for the abnormal behavior segments, further determining whether the abnormal behavior segments are abnormal or not by using the trained YOLOv8 model; alarm mechanisms of different levels are designed according to different abnormal behaviors, the abnormal behaviors are alarmed, and detection of the power construction site is achieved.

Description

Technical Field

[0001] The invention relates to a power construction detection method based on multi-task learning and multi-modal data, and belongs to the field of power construction safety detection. Background Art

[0002] In the field of power construction, construction safety is a key factor in ensuring the smooth progress of the project and protecting the safety of construction workers. With the continuous expansion of the scale of power construction and the increasing complexity of the construction environment, traditional safety detection methods have gradually revealed their limitations, such as low efficiency, strong subjectivity and difficulty in real-time monitoring of manual inspections. Therefore, a new type of safety detection technology is urgently needed to improve the safety and efficiency of power construction.

[0003] The YOLOv8 model is the latest version of the YOLO object detection and image segmentation model developed by Ultralytics. The LSTM model, short for Long Short-Term Memory, is a special variant of the recurrent neural network (RNN) that excels in processing sequence data, especially in capturing long-term dependencies.

[0004] In recent years, the rapid development of multi-task learning (MTL) and multimodal data processing technology has provided new solutions for power construction safety detection. Multi-task learning is a method to improve model performance by sharing learned features on multiple related tasks. This method can share knowledge between different tasks, thereby improving the model's ability to handle complex situations. In power construction safety detection, multiple tasks may include real-time monitoring of the construction site, detection of dangerous behaviors, equipment status assessment, etc. These tasks are interrelated to some extent.

[0005] Existing technologies such as the Chinese patent number "CN117854110A" disclose a construction site safety helmet strap recognition system and method based on YOLOv5. The system includes a recognition system including a USB camera, an edge device, a display screen, and a data storage card; the recognition method includes step S1, image acquisition; step S2, data set annotation; step S3, building a safety helmet strap data set; step S4, a safety helmet strap recognition model; step S5, intelligent graded warning; step S6, field deployment and application; the present invention applies advanced artificial intelligence recognition algorithms to the field of construction site safety, and through automatic detection of the individual safety helmet strap status, ensures the standardization of safety helmet wearing, and further strengthens the construction safety of construction sites.

[0006] The problem with the above-mentioned existing technology is that it mainly relies on the YOLOv5 model to identify the status of the helmet strap, which may lead to limited generalization ability of the model in processing complex scenes or extreme lighting conditions, and lack of more complex abnormal behavior detection capabilities. Summary of the invention

[0007] In order to solve the above problems existing in the prior art, the present invention proposes a power construction detection method based on multi-task learning and multi-modal data.

[0008] The technical solution of the present invention is as follows:

[0009] On the one hand, the present invention provides a power construction detection method based on multi-task learning and multi-modal data, comprising the following steps:

[0010] Set the control ball to collect image data, normalize and remove noise from the image data, and align the processed image data;

[0011] Use the aligned image data to train the YOLOv8 model;

[0012] Based on the LSTM model combined with the sliding window technology, abnormal behavior detection is performed on the image data collected by the control ball; the abnormal behavior includes not wearing a safety helmet, not wearing a seat belt, and preset unsafe behaviors;

[0013] For abnormal behavior clips, use the trained YOLOv8 model to further confirm whether they are abnormal;

[0014] Design different levels of alarm mechanisms based on different abnormal behaviors and issue alarms for abnormal behaviors.

[0015] As a preferred implementation, the method for normalizing the image data is:

[0016]

[0017] Where I represents the original image, I′ represents the normalized image, min(I) represents the minimum value of the original image, and max(I) represents the maximum value of the original image;

[0018] The method of removing noise from image data uses Gaussian filtering formula for processing, and the processing method is:

[0019]

[0020] Among them, I(x*y) represents the original image pixel value, I′(x,y) represents the denoised image pixel value, σ is the standard deviation of the Gaussian kernel, * represents the convolution operation, and x 2 +y 2Represents the weight of the original image pixel, Indicates the distance between the pixel and the center point;

[0021] The image data alignment process uses mutual information-based registration technology to align the panoramic image and depth image in the image data:

[0022]

[0023] Where A represents the panoramic image, B represents the depth image, p(a) represents the grayscale value probability distribution of the panoramic image, p(b) represents the grayscale value probability distribution of the depth image, and p(a,b) represents the joint probability distribution. It represents the information difference between the joint distribution and independent distribution of the pixel grayscale values ​​of the panoramic image and the depth image. MI(A,B) represents the degree of correlation between the panoramic image and the depth image. a represents the pixel grayscale value of the panoramic image, and b represents the pixel grayscale value of the depth image.

[0024] As a preferred implementation, the training process of the YOLOv8 model is:

[0025] Use the CSPDarknet53 deep convolutional neural network model as the feature extraction network to extract the features of the panoramic image and depth image in the aligned image data;

[0026] The features of the panoramic image and the depth image are fused to obtain a unified feature representation:

[0027] F fused =Concat(F panoram ,F Depth );

[0028] Among them, F panoram represents the characteristics of the panoramic image, F Depth Represents the features of the depth image, Concat(F panoram ,F Depth ) represents the feature fusion function, F fused Represents the fused features;

[0029] Design detection tasks based on abnormal behavior:

[0030] {B helmet ,B harness ,B hole}=YOLOv8(F fused );

[0031] Among them, B helmet Indicates that the helmet is not worn. B harness Indicates not wearing seat belts, B holeIndicates the preset unsafe behavior detection, F fused Represents the fused features, YOLOv8(F fused ) indicates that the YOLOv8 model is used to perform target detection on the fused features;

[0032] The loss function of the YOLOv8 model is expressed as a weighted loss function L:

[0033] L=αL helmet +βL harness +γL hole ;

[0034] Among them, L helmet Indicates that the helmet is not worn. L harness Indicates not wearing seat belts, L hole represents the preset unsafe behavior detection, α represents the weight coefficient of the detection of not wearing a helmet, β represents the weight coefficient of the detection of not wearing a seat belt, and γ represents the weight coefficient of the preset unsafe behavior detection;

[0035] The loss function L is decomposed into positioning loss and classification loss:

[0036] L=L loc +L cls ;

[0037]

[0038] Among them, L loc represents the target positioning loss, L cls represents the classification loss of the target, SmoothL1() represents the smooth L1 loss function, B i represents the bounding box coordinates of the i-th target predicted by the model, represents the true bounding box coordinates of the i-th target, y i represents the true category label of the target in the i-th sample, represents the model's predicted probability for the target in the i-th sample, represents the confidence of predicting the negative class, Represents the confidence of predicting the positive class.

[0039] As a preferred implementation, the method of using the LSTM model in combination with the sliding window technology to detect abnormal behavior of the image data collected by the control ball is as follows:

[0040] The image data is sliced ​​and processed according to the sliding window technology, divided into several time windows, and each time window is detected using the LSTM model:

[0041] Identify the behavioral characteristics of the time window and use the threshold judgment method to identify whether there is abnormal behavior:

[0042]

[0043] Among them, A nomaly Indicates whether there is abnormal behavior, 1 indicates that there is an abnormality, and 0 indicates that there is no abnormality; S behavior represents the behavior feature score, T represents the score threshold, otherwise represents that the behavior feature score is less than the score threshold;

[0044] If the judgment result is 1, the YOLOv8 model is connected to further confirm the abnormal behavior; if the judgment result is 0, the next time window is detected.

[0045] On the other hand, the present invention also provides a power construction safety detection system based on multi-task learning and multi-modal data, comprising:

[0046] Data acquisition and processing module: set the control ball to collect image data, normalize and remove noise from the image data, and align the processed image data;

[0047] Model training module: Use the aligned image data to train the YOLOv8 model;

[0048] Anomaly detection module: Based on the LSTM model combined with sliding window technology, abnormal behavior detection is performed on the image data collected by the control ball; the abnormal behavior includes not wearing a safety helmet, not wearing a seat belt, and preset unsafe behaviors;

[0049] Abnormal behavior confirmation module: For abnormal behavior fragments, use the trained YOLOv8 model to further confirm whether it is abnormal;

[0050] Alarm module: Design different levels of alarm mechanisms according to different abnormal behaviors and issue alarms for abnormal behaviors.

[0051] On the other hand, the present invention further provides an electronic device having a computer program stored thereon, wherein when the computer program is executed by a processor, the method for detecting electric power construction based on multi-task learning and multi-modal data as described in any embodiment of the present invention is implemented.

[0052] On the other hand, the present invention also provides a computer-readable medium for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the power construction detection method based on multi-task learning and multimodal data as described in any embodiment of the present invention.

[0053] The present invention has the following beneficial effects:

[0054] The present invention normalizes and denoises the image data collected by the surveillance ball, which can significantly improve the quality of the image data. Normalization helps the model converge faster, thereby improving training efficiency; while denoising can effectively reduce noise interference in the image and improve the accuracy of subsequent image analysis and recognition. After completing the normalization and denoising processes, the image data is aligned to ensure the consistency of the image input into the YOLOv8 model in space and direction. This processing step helps the model to more accurately capture the features in the image and improve the accuracy of target detection. The high-quality image data after these optimization processes is used to train the YOLOv8 model, which can give full play to the advantages of the model in target detection. YOLOv8 is known for its faster speed and higher accuracy. Combined with the optimized image data, it can further improve the accuracy and efficiency of target detection.

[0055] In addition, the LSTM model is combined with the sliding window technology to detect abnormal behavior of image data. The LSTM model is good at processing long-term dependencies in sequence data, while the sliding window technology can effectively capture local features in the image. The combination of the two makes abnormal behavior detection more comprehensive and accurate. For the detected abnormal behavior fragments, the trained YOLOv8 model is used for further confirmation. This step ensures the accuracy and reliability of abnormal behavior detection and reduces the possibility of false positives and false negatives. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 The present invention is a flowchart for implementing the method. DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] It should be understood that the step numbers used in this document are only for convenience of description and are not intended to limit the order in which the steps are executed.

[0059] It should be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0060] The terms “include” and “comprising” indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0061] The term "and / or" means and includes any and all possible combinations of one or more of the associated listed items.

[0062] Embodiment 1:

[0063] See also Figure 1 The present invention provides a power construction detection method based on multi-task learning and multi-modal data, comprising the following steps:

[0064] Set the control ball to collect image data, normalize and remove noise from the image data, and align the processed image data;

[0065] Use the aligned image data to train the YOLOv8 model;

[0066] Based on the LSTM model combined with the sliding window technology, abnormal behavior detection is performed on the image data collected by the control ball; the abnormal behavior includes not wearing a safety helmet, not wearing a seat belt, and preset unsafe behaviors;

[0067] For abnormal behavior clips, use the trained YOLOv8 model to further confirm whether they are abnormal;

[0068] Design different levels of alarm mechanisms based on different abnormal behaviors and issue alarms for abnormal behaviors.

[0069] As a preferred implementation, the method for normalizing the image data is:

[0070]

[0071] Where I represents the original image, I′ represents the normalized image, min(I) represents the minimum value of the original image, and max(I) represents the maximum value of the original image;

[0072] The method of removing noise from image data uses Gaussian filtering formula for processing, and the processing method is:

[0073]

[0074] Among them, I(x*y) represents the original image pixel value, I′(x,y) represents the denoised image pixel value, σ is the standard deviation of the Gaussian kernel, * represents the convolution operation, and x 2 +y 2 Represents the weight of the original image pixel, Indicates the distance between the pixel and the center point;

[0075] The image data alignment process uses mutual information-based registration technology to align the panoramic image and depth image in the image data:

[0076]

[0077] Where A represents the panoramic image, B represents the depth image, p(a) represents the grayscale value probability distribution of the panoramic image, p(b) represents the grayscale value probability distribution of the depth image, and p(a,b) represents the joint probability distribution. It represents the information difference between the joint distribution and independent distribution of the pixel grayscale values ​​of the panoramic image and the depth image. MI(A,B) represents the degree of correlation between the panoramic image and the depth image. a represents the pixel grayscale value of the panoramic image, and b represents the pixel grayscale value of the depth image.

[0078] By normalizing or standardizing the data, data of different modalities (such as depth images and image pixel values) can be converted to the same scale range to avoid imbalanced model learning due to numerical scale differences during training. Denoising, enhancement and other steps help the model better learn the true characteristics of the data, while enhancing the diversity of the data set and reducing overfitting.

[0079] When processing multimodal data, panoramic images and depth images may be spatially misaligned due to different shooting angles, resolutions, or sensor characteristics. Alignment can ensure that multimodal data has similar spatial positions and semantic information at the same pixel points, reducing confusion when the model learns different modal features. After alignment, the mutual information between panoramic images and depth images increases, which means that more information is shared between the two, which can help the model better integrate multimodal information and improve decision accuracy.

[0080] As a preferred implementation, the training process of the YOLOv8 model is:

[0081] Use the CSPDarknet53 deep convolutional neural network model as the feature extraction network to extract the features of the panoramic image and depth image in the aligned image data;

[0082] The features of the panoramic image and the depth image are fused to obtain a unified feature representation:

[0083] F fused =Concat(F panotam ,F Depth );

[0084] Among them, F panoram represents the characteristics of the panoramic image, F DepthRepresents the features of the depth image, Concat(F panoram ,F Depth ) represents the feature fusion function, F fused Represents the fused features;

[0085] Design detection tasks based on abnormal behavior:

[0086] {B helmet ,B harness ,B hole}=YOLOv8(F fused );

[0087] Among them, B helmet Indicates that the helmet is not worn. B harness Indicates not wearing seat belts, B hole Indicates the preset unsafe behavior detection, F fused Indicates the fused features, YOLOv8(F fused ) indicates that the YOLOv8 model is used to perform target detection on the fused features;

[0088] The loss function of the YOLOv8 model is expressed as a weighted loss function L:

[0089] L=αL helmet +βL harness +γL hole ;

[0090] Among them, L helmet Indicates that the helmet is not worn. L harness Indicates not wearing seat belts, L hole represents the preset unsafe behavior detection, α represents the weight coefficient of the detection of not wearing a helmet, β represents the weight coefficient of the detection of not wearing a seat belt, and γ represents the weight coefficient of the preset unsafe behavior detection;

[0091] The loss function L is decomposed into positioning loss and classification loss:

[0092] L=L loc +L cls ;

[0093]

[0094] Among them, L loc represents the target positioning loss, L cls represents the classification loss of the target, SmoothL1() represents the smooth L1 loss function, B i represents the bounding box coordinates of the i-th target predicted by the model, represents the true bounding box coordinates of the i-th target, y irepresents the true category label of the target in the i-th sample, represents the model's predicted probability for the target in the i-th sample, represents the confidence of predicting the negative class, Represents the confidence of predicting the positive class.

[0095] YOLOv8 performs well in detecting targets of different sizes and is suitable for a variety of scenarios in power construction. Whether it is a larger human posture or smaller details such as a helmet or tools, the model can accurately identify and locate it, thereby ensuring that all types of safety hazards can be fully monitored. YOLOv8 has good generalization capabilities in different scenarios and backgrounds, and can adapt to different lighting conditions, weather changes and complex environments on the construction site. This means that whether it is day or night, sunny or rainy, the model can maintain a high detection accuracy. YOLOv8 supports the detection of multiple targets at the same time and can identify multiple different types of abnormal behaviors in the same picture. This is especially useful at power construction sites with a large number of people and complex mechanical equipment, ensuring that different types of potential safety hazards can be discovered in a timely manner.

[0096] As a preferred implementation, the method of using the LSTM model in combination with the sliding window technology to detect abnormal behavior of the image data collected by the control ball is as follows:

[0097] The image data is sliced ​​and processed according to the sliding window technology, divided into several time windows, and each time window is detected using the LSTM model:

[0098] Identify the behavioral characteristics of the time window and use the threshold judgment method to identify whether there is abnormal behavior:

[0099]

[0100] Among them, A nomaly Indicates whether there is abnormal behavior, 1 indicates that there is an abnormality, and 0 indicates that there is no abnormality; S behavior represents the behavior feature score, T represents the score threshold, otherwise represents that the behavior feature score is less than the score threshold;

[0101] If the judgment result is 1, the YOLOv8 model is connected to further confirm the abnormal behavior; if the judgment result is 0, the next time window is detected.

[0102] The LSTM model combined with sliding window technology can quickly process continuous image data and preliminarily screen out potential abnormal behavior fragments. This fast preliminary detection saves time in processing all image data. The preliminary screening of the LSTM model can reduce the number of fragments that need further confirmation, so only a small number of high-risk fragments need to be run on the more complex YOLOv8 model, thereby reducing the overall consumption of computing resources. The YOLOv8 model further confirms abnormal behavior based on the abnormal fragments screened by the LSTM model, which can effectively reduce false positives. The high-precision target detection function of the YOLOv8 model can more accurately determine the actual abnormal behavior, thereby improving the accuracy of detection.

[0103] Alarm method for not wearing a safety helmet: issue a warning sound and text prompt, mark it on the monitoring interface, and push notification to the terminal device of the safety manager.

[0104] Embodiment 2:

[0105] The present invention also provides a power construction safety detection system based on multi-task learning and multi-modal data, comprising:

[0106] Data acquisition and processing module: set the control ball to collect image data, normalize and remove noise from the image data, and align the processed image data;

[0107] Model training module: Use the aligned image data to train the YOLOv8 model;

[0108] Anomaly detection module: Based on the LSTM model combined with sliding window technology, abnormal behavior detection is performed on the image data collected by the control ball; the abnormal behavior includes not wearing a safety helmet, not wearing a seat belt, and preset unsafe behaviors;

[0109] Abnormal behavior confirmation module: For abnormal behavior fragments, use the trained YOLOv8 model to further confirm whether it is abnormal;

[0110] Alarm module: Design different levels of alarm mechanisms according to different abnormal behaviors and issue alarms for abnormal behaviors.

[0111] Embodiment three:

[0112] This embodiment provides an electronic device having a computer program stored thereon, and when the computer program is executed by a processor, the method for detecting electric power construction based on multi-task learning and multi-modal data as described in any embodiment of the present invention is implemented.

[0113] Embodiment 4:

[0114] This embodiment provides a computer-readable medium for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the power construction detection method based on multi-task learning and multimodal data as described in any embodiment of the present invention.

[0115] In the embodiments of the present application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. Among them, A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can be represented by: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, c can be single or multiple.

[0116] Those of ordinary skill in the art will appreciate that the various units and algorithm steps described in the embodiments disclosed herein can be implemented in a combination of electronic hardware, computer software, and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0117] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0118] In several embodiments provided in the present application, if any function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory; hereinafter referred to as: ROM), random access memory (Random Access Memory; hereinafter referred to as: RAM), disk or optical disk, and other media that can store program codes.

[0119] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A power construction detection method based on multi-task learning and multi-modal data, characterized in that: The following steps are involved: Set the control ball to collect image data, normalize and remove noise from the image data, and align the processed image data; Use the aligned image data to train the YOLOv8 model; Based on the LSTM model combined with the sliding window technology, abnormal behavior detection is performed on the image data collected by the control ball; the abnormal behavior includes not wearing a safety helmet, not wearing a seat belt, and preset unsafe behaviors; For abnormal behavior clips, use the trained YOLOv8 model to further confirm whether they are abnormal; Design different levels of alarm mechanisms based on different abnormal behaviors and issue alarms for abnormal behaviors.

2. The electric power construction detection method based on multi-task learning and multi-modal data according to claim 1 is characterized in that: The method for normalizing the image data is: Where I represents the original image, I′ represents the normalized image, min(I) represents the minimum value of the original image, and max(I) represents the maximum value of the original image; The method of removing noise from image data uses Gaussian filtering formula for processing, and the processing method is: Among them, I(x*y) represents the original image pixel value, I′(x,y) represents the denoised image pixel value, σ is the standard deviation of the Gaussian kernel, * represents the convolution operation, and x 2 +y 2 Represents the weight of the original image pixel, Indicates the distance between the pixel and the center point; The image data alignment process uses mutual information-based registration technology to align the panoramic image and depth image in the image data: Where A represents the panoramic image, B represents the depth image, p(a) represents the grayscale value probability distribution of the panoramic image, p(b) represents the grayscale value probability distribution of the depth image, and p(a,b) represents the joint probability distribution. It represents the information difference between the joint distribution and independent distribution of the pixel grayscale values ​​of the panoramic image and the depth image. MI(A,B) represents the degree of correlation between the panoramic image and the depth image. a represents the pixel grayscale value of the panoramic image, and b represents the pixel grayscale value of the depth image.

3. The electric power construction detection method based on multi-task learning and multi-modal data according to claim 2 is characterized in that: The training process of the YOLOv8 model is: Use the CSPDarknet53 deep convolutional neural network model as the feature extraction network to extract the features of the panoramic image and depth image in the aligned image data; The features of the panoramic image and the depth image are fused to obtain a unified feature representation: F fused =Concat(F panoram ,F Depth ); Among them, F panoram represents the characteristics of the panoramic image, F Depth Represents the features of the depth image, Concat(F panoram ,F Depth ) represents the feature fusion function, F fused Represents the fused features; Design detection tasks based on abnormal behavior: {B helmet ,B harness ,B hole }=YOLOv8(F fused ); Among them, B helmet Indicates that the helmet is not worn. B harness Indicates not wearing seat belts, B hole Indicates the preset unsafe behavior detection, F fused Indicates the fused features, YOLOv8(F fused ) indicates that the YOLOv8 model is used to perform target detection on the fused features; The loss function of the YOLOv8 model is expressed as a weighted loss function L: L=αL helmet +βL harness +γL hole ; Among them, L helmet Indicates that the helmet is not worn. L harness Indicates not wearing seat belts, L hole represents the preset unsafe behavior detection, α represents the weight coefficient of the detection of not wearing a helmet, β represents the weight coefficient of the detection of not wearing a seat belt, and γ represents the weight coefficient of the preset unsafe behavior detection; The loss function L is decomposed into positioning loss and classification loss: L=L loc +L cls ; Among them, L loc represents the target positioning loss, L cls represents the classification loss of the target, SmoothL1() represents the smooth L1 loss function, B i represents the bounding box coordinates of the i-th target predicted by the model, represents the true bounding box coordinates of the i-th target, y i represents the true category label of the target in the i-th sample, represents the model's predicted probability for the target in the i-th sample, represents the confidence of predicting the negative class, Represents the confidence of predicting the positive class.

4. The electric power construction detection method based on multi-task learning and multi-modal data according to claim 1 is characterized in that: The method of using the LSTM model combined with the sliding window technology to detect abnormal behavior of the image data collected by the control ball is as follows: The image data is sliced ​​and processed according to the sliding window technology, divided into several time windows, and each time window is detected using the LSTM model: Identify the behavioral characteristics of the time window and use the threshold judgment method to identify whether there is abnormal behavior: Among them, A nomaly Indicates whether there is abnormal behavior, 1 indicates that there is an abnormality, and 0 indicates that there is no abnormality; S behavior represents the behavior feature score, T represents the score threshold, otherwise, it means that the behavior feature score is less than the score threshold; If the judgment result is 1, the YOLOv8 model is connected to further confirm the abnormal behavior; if the judgment result is 0, the next time window is detected.

5. A power construction safety detection system based on multi-task learning and multi-modal data, characterized in that: include: Data acquisition and processing module: set the control ball to collect image data, normalize and remove noise from the image data, and align the processed image data; Model training module: Use the aligned image data to train the YOLOv8 model; Anomaly detection module: Based on the LSTM model combined with sliding window technology, abnormal behavior detection is performed on the image data collected by the control ball; the abnormal behavior includes not wearing a safety helmet, not wearing a seat belt, and preset unsafe behaviors; Abnormal behavior confirmation module: For abnormal behavior fragments, use the trained YOLOv8 model to further confirm whether it is abnormal; Alarm module: Design different levels of alarm mechanisms according to different abnormal behaviors and issue alarms for abnormal behaviors.

6. The electric power construction safety detection system based on multi-task learning and multi-modal data according to claim 5 is characterized in that: The method for normalizing image data in the data acquisition and processing module is as follows: Where I represents the original image, I′ represents the normalized image, min(I) represents the minimum value of the original image, and max(I) represents the maximum value of the original image; The method of removing noise from image data uses Gaussian filtering formula for processing, and the processing method is: Among them, I(x*y) represents the original image pixel value, I′(x,y) represents the denoised image pixel value, σ is the standard deviation of the Gaussian kernel, * represents the convolution operation, and x 2 +y 2 Represents the weight of the original image pixel, Indicates the distance between the pixel and the center point; The image data alignment process uses mutual information-based registration technology to align the panoramic image and depth image in the image data: Where A represents the panoramic image, B represents the depth image, p(a) represents the grayscale value probability distribution of the panoramic image, p(b) represents the grayscale value probability distribution of the depth image, and p(a,b) represents the joint probability distribution. It represents the information difference between the joint distribution and independent distribution of the pixel grayscale values ​​of the panoramic image and the depth image. MI(A,B) represents the degree of correlation between the panoramic image and the depth image. a represents the pixel grayscale value of the panoramic image, and b represents the pixel grayscale value of the depth image.

7. The electric power construction safety detection system based on multi-task learning and multi-modal data according to claim 5 is characterized in that: The model training module, the training process of the YOLOv8 model is: Use the CSPDarknet53 deep convolutional neural network model as the feature extraction network to extract the features of the panoramic image and depth image in the aligned image data; The features of the panoramic image and the depth image are fused to obtain a unified feature representation: F fused =Concat(F panoram ,F Depth ); Among them, F panoram represents the characteristics of the panoramic image, F Depth Represents the features of the depth image, Concat(F panoram ,F Depth ) represents the feature fusion function, F fused Represents the fused features; Design detection tasks based on abnormal behavior: {B helmet ,B harness ,B hole }=YOLOv8(F fused ); Among them, B helmet Indicates that the helmet is not worn. B harness Indicates not wearing seat belts, B hole Indicates the preset unsafe behavior detection, F fused Indicates the fused features, YOLOv8(F fused ) indicates that the YOLOv8 model is used to perform target detection on the fused features; The loss function of the YOLOv8 model is expressed as a weighted loss function L: L=αL helmet +βL harness +γL hole ; Among them, L helmet Indicates that the helmet is not worn. L harness Indicates not wearing seat belts, L hole represents the preset unsafe behavior detection, α represents the weight coefficient of the detection of not wearing a helmet, β represents the weight coefficient of the detection of not wearing a seat belt, and γ represents the weight coefficient of the preset unsafe behavior detection; The loss function L is decomposed into positioning loss and classification loss: L=L loc +L cls ; Among them, L loc represents the target positioning loss, L cls represents the classification loss of the target, SmoothL1() represents the smooth L1 loss function, B i represents the bounding box coordinates of the i-th target predicted by the model, represents the true bounding box coordinates of the i-th target, y i represents the true category label of the target in the i-th sample, represents the model's predicted probability for the target in the i-th sample, represents the confidence of predicting the negative class, Represents the confidence of predicting the positive class.

8. The electric power construction safety detection system based on multi-task learning and multi-modal data according to claim 5 is characterized in that: The abnormal behavior confirmation module uses the LSTM model combined with the sliding window technology to detect abnormal behavior of the image data collected by the control ball: The image data is sliced ​​and processed according to the sliding window technology, divided into several time windows, and each time window is detected using the LSTM model: Identify the behavioral characteristics of the time window and use the threshold judgment method to identify whether there is abnormal behavior: Among them, A nomaly Indicates whether there is abnormal behavior, 1 indicates that there is an abnormality, and 0 indicates that there is no abnormality; S behavior represents the behavior feature score, T represents the score threshold, otherwise represents that the behavior feature score is less than the score threshold; If the judgment result is 1, the YOLOv8 model is connected to further confirm the abnormal behavior; if the judgment result is 0, the next time window is detected.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the power construction detection method based on multi-task learning and multi-modal data as described in any one of claims 1 to 4 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the power construction detection method based on multi-task learning and multi-modal data as described in any one of claims 1 to 4 is implemented.

Images