Method and device for determining damage information of iron tower structure and electronic equipment
The tower video was taken by a drone and the structure damage detection was performed using the YOLO v4 model. Combined with the marked sample set and the actual environmental sample set, the accuracy of the tower structure damage judgment was solved, and efficient and accurate tower damage detection was achieved.
Patent Information
- Application Number
- CN202410968858.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-18
- Publication Date
- 2024-11-12
AI Technical Summary
It is difficult for the prior art to accurately judge structural damage caused by wind loads, earthquakes and other factors during service.
The tower video was shot by drones, and the structure damage was detected through the YOLO v4 target detection model. The marked tower image sample set and the actual environmental tower video sample set were used to train the model to achieve accurate judgment of tower structural damage.
It improves the accuracy of tower structure damage detection, can effectively identify damage location and type in actual engineering environment, and improves detection efficiency and accuracy.
Smart Images

Figure CN118941951A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the technical field of tower structure damage detection, and in particular to a method, device and electronic equipment for determining tower structure damage information. Background Art
[0002] During their service, steel towers may be affected by environmental factors such as wind loads and earthquakes and suffer structural damage. However, there is currently a lack of effective means to determine the structural damage of steel towers, making it impossible to accurately judge the structural damage of steel towers. Summary of the invention
[0003] The embodiments of the present application provide a method, device and electronic device for determining structural damage information of an iron tower, which are used to make an accurate judgment on the structural damage of the iron tower.
[0004] The present application embodiment adopts the following technical solutions:
[0005] In a first aspect, a method for determining damage information of a tower structure is provided, the method comprising:
[0006] Get the video of the First Tower taken by the drone;
[0007] Input the images contained in the video frame by frame into a target detection model, wherein the target detection model refers to a general term for a model that can realize the task of detecting structural damage of an iron tower, and the target detection model is used to detect and annotate structural damage information in the input iron tower image. The target model is obtained by training based on a training data set, and the training data set includes a first sample set and a second sample set. The first sample set includes images of the iron tower annotated with structural damage information, and the second sample set includes a video of the second iron tower taken by a drone;
[0008] Receive structural damage information of the first tower output by the target detection model.
[0009] In a second aspect, a device for determining damage information of a tower structure is provided, the device comprising:
[0010] An image acquisition module, used to acquire the video of the first tower taken by the drone;
[0011] An image input module is used to input the images contained in the video into a target detection model frame by frame, wherein the target detection model is a general term for a model that can realize the task of detecting damage to the iron tower structure, and the target detection model is used to detect and annotate the structural damage information in the input iron tower image, and the training data set includes a first sample set and a second sample set, wherein the first sample set includes the iron tower image with annotated structural damage information, and the second sample set includes the video of the second iron tower taken by a drone;
[0012] A result receiving module is used to receive the structural damage information of the first tower output by the target detection model.
[0013] In a third aspect, an electronic device is provided, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the steps of the method described in the first aspect are implemented.
[0014] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in the first aspect are implemented.
[0015] In a fifth aspect, a computer program product is provided. The program product is stored in a storage medium and is executed by at least one processor to implement the method as described in the first aspect.
[0016] The embodiment of the present application proposes a method for determining the structural damage information of a tower. The target detection model is trained in advance based on a first sample set of tower images containing labeled structural damage information and a video of the second tower taken by a drone as a second sample set. Then, during the detection process, the structural damage information of the first tower can be obtained by inputting the video of the first tower image captured by the drone into the target detection model. In addition to using the first sample set to train the target detection model, the method also uses the tower video in the actual environment - the second sample set to train the target detection model. Therefore, the complex uncertain factors in the actual project can be grasped, and accurate tower structure damage detection results can be given, thereby effectively improving the accuracy of tower structure damage detection. On the other hand, an efficient and accurate target detection model is used, so the accuracy of tower structure damage detection can be further improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0018] Figure 1 It is a flow chart of a method for determining damage information of a tower structure provided in one embodiment of the present application.
[0019] Figure 2 It is a flow chart of a method for determining damage information of a tower structure provided in another embodiment of the present application.
[0020] Figure 3 It is a flow chart of a method for determining damage information of a tower structure provided in another embodiment of the present application.
[0021] Figure 4 It is a structural schematic diagram of a tower structure damage information determination device provided in one embodiment of the present application.
[0022] Figure 5 It is a structural schematic diagram of a tower structure damage information determination device provided by another embodiment of the present application.
[0023] Figure 6 It is a structural schematic diagram of a tower structure damage information determination device provided by another embodiment of the present application.
[0024] Figure 7 It is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION
[0025] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.
[0026] In order to make an accurate judgment on the structural damage of the iron tower, the embodiment of the present application provides a method and device for determining the structural damage information of the iron tower. The method can be executed by an electronic device, such as a terminal device or a server, or the method can be executed by software installed in the electronic device.
[0027] The method for determining the damage information of a tower structure proposed in the embodiment of the present application includes two stages:
[0028] Phase 1: Training the target detection model;
[0029] Phase 2: Apply the target detection model to detect tower structure damage.
[0030] The first stage is described below.
[0031] like Figure 1 As shown, a method for determining damage information of a tower structure provided in an embodiment of the present application may include:
[0032] Step 101: train an object detection model based on a training data set.
[0033] The target detection model mentioned in this application refers to a general term for models that can achieve the task of tower structure damage detection. As an example, the target detection model can be YOLO v4 (You Only Look Once Version 4).
[0034] In the embodiment of the present application, the iron tower may include but is not limited to a single-tube communication tower.
[0035] In some embodiments, the training data set includes a first sample set, and the first sample set includes tower images with labeled structural damage information.
[0036] In some embodiments, the training data set includes a second sample set, and the second sample set includes a video of a second iron tower taken by a drone. The second iron tower may be any iron tower that has been confirmed to have structural damage, and the number of the second iron towers may be multiple. Optionally, the second sample set is a labeled sample set.
[0037] In some other embodiments, the training data set includes a first sample set and a second sample set.
[0038] 1) In the case where the training data set includes the first sample set, before step 101, the method for determining damage information of a tower structure proposed in the embodiment of the present application may further include:
[0039] Acquire images of several towers with structural damage;
[0040] Preprocessing the plurality of iron tower images;
[0041] marking structural damage information of the preprocessed images of the plurality of iron towers;
[0042] The labeled images of the several iron towers are used as the first sample set.
[0043] As an example, the above-mentioned acquisition of several tower images with structural damage may include: several common tower structural damage images collected through relevant channels, wherein the relevant channels may include but are not limited to manual on-site photography, Internet websites, relevant documents, and the like.
[0044] Taking YOLO v4 as an example of the target detection model mentioned in this application, the above-mentioned preprocessing of the several iron tower images may include: adjusting the size of the several iron tower images with structural damage to a size suitable for input to YOLOv4, specifically, adjusting the size of the above-mentioned several iron tower images to 608×608 pixels.
[0045] The structural damage information of the tower may include but is not limited to at least one of the structural damage location and the structural damage type. The structural damage type may be set in advance according to actual needs. For example, the structural damage type may include but is not limited to corrosion, deformation, etc. Furthermore, for different types, the levels may be divided according to the degree of damage.
[0046] As an example, the above-mentioned labeling of structural damage information of the preprocessed images of the several towers may include: setting common structural damage types of towers in Labeling software, and using a label selection box to label the structural damage location and the corresponding structural damage type in the image in response to the user's labeling operation.
[0047] Furthermore, in the case where the training data set includes a first sample set, step 101 may specifically include: using the tower image in the first sample set as input to train the target detection model.
[0048] 2) When the training data set also includes a second sample set, before step 101, the method for determining damage information of a tower structure proposed in the embodiment of the present application may further include:
[0049] Get the video of the second tower taken by the drone;
[0050] The video of the second tower is used as the second sample set.
[0051] It should be noted that the second iron tower may include the first iron tower or may not include it.
[0052] Furthermore, in the case where the training data set also includes a second sample set, step 101 may also include: using the video in the second sample set as input to train the object detection model.
[0053] YOLO v4 is used as an example of the target detection model mentioned in this application. Specifically, each frame of the video of the second tower taken by the drone is used as the input of YOLO v4, and the image with structural damage information such as the location and type of structural damage is used as the output to build, train and optimize the hyperparameters of the YOLO v4 model.
[0054] In the embodiment of the present application, YOLO v4 is used as an example of the target detection model mentioned in the present application. Among them, YOLO v4 as a target detection model includes a backbone network (Backbone), a neck network (Neck) and a prediction network (Head), and the neck network connects the backbone network and the prediction network, wherein:
[0055] The backbone network is used to extract features from the input tower image;
[0056] The neck network is used to further extract features from the features extracted by the backbone network;
[0057] The prediction network is used to predict the target detection result based on the features extracted by the neck network.
[0058] The backbone network, neck network and prediction network are described in detail below.
[0059] 1) Backbone
[0060] As an example, this application uses the CSPDarknet53 network as the backbone network in the YOLO v4 model to extract features from the input tower image. After the tower image with a size of 608×608 pixels is input into the YOLO v4 model, the CSPDarknet53 network is first used to extract the features of the tower damage target.
[0061] In an embodiment of the present application, the backbone network adopts the Mish activation function instead of the original RELU activation function, and adds a Dropblock block to further enhance the generalization ability of the backbone network.
[0062] In an embodiment of the present application, the backbone network includes 5 residual block modules, and the input tower image is downsampled through a convolution kernel of size 3×3 in each residual block module. After the backbone network performs feature extraction on the input tower detection image, the feature map size changes five times, and finally changes from 608×608 to a feature map of size 19×19, thereby achieving rapid dimensionality reduction of the tower image. The feature map output by the residual block module is fused in the neck network of the YOLO v4 model.
[0063] Among them, the Mish activation function formula is as follows:
[0064] y = x*tanh(x*ln(1+e x )) (1)
[0065] In the formula (1), tanh() represents a hyperbolic function.
[0066] 2) Neck network
[0067] The neck network (Neck) connects the backbone network (Backbone) and the prediction network (Head) to further improve the diversity and robustness of features.
[0068] The neck network includes a spatial pyramid pooling (SPP) module and a path aggregation network (PAN).
[0069] The SPP module concatenates the max-pooling outputs of low-resolution feature maps to extract the most representative features. The SPP module performs max pooling operations using kernels of size 1×1, 5×5, 9×9, and 13×13. The concatenation of feature maps increases the receptive field of features extracted by the backbone network, thereby improving the accuracy of the entire network in detecting small objects.
[0070] The feature maps spliced by the SPP module are fused with the high-resolution feature maps through PAN. PAN sets up bottom-up and top-down paths through upsampling and downsampling operations to achieve the combination of low-level and high-level features. The PAN module outputs a set of aggregated features for prediction.
[0071] 3) Prediction Network (Head)
[0072] In the embodiment of the present application, YOLO v4 uses the prediction network of YOLO v3 for multi-scale prediction. The prediction network uses three different levels of feature maps for fusion, which are located in the middle layer, the lower middle layer, and the bottom layer. The sizes of the three feature layers are different, which improves the detection performance of targets of different sizes.
[0073] Furthermore, the structural damage information includes the structural damage location and the structural damage type, and the prediction network includes a first detection layer, a second detection layer and a third detection layer, wherein:
[0074] The first detection layer is used to predict a bounding box;
[0075] The second detection layer is used to predict the type of structural damage;
[0076] The third detection layer is used to predict structural damage.
[0077] That is to say, in the embodiment of the present application, the prediction network has three detection layers. Each detection layer is a YOLO v3 that calculates the final prediction. The YOLO v4 model outputs feature maps of size 19×19, 38×38, and 76×76, which are used to predict bounding boxes, structural damage type scores, and structural damage scores, respectively. YOLO v4 uses the complete intersection over union (CIoU) as the loss function. The CIoU loss function calculation formula is as follows:
[0078]
[0079]
[0080] In the above formula (2), A represents the actual marked target box area; B represents the model's prediction of the target box area; b and bgt represent the center points of the predicted target box and the real box respectively, ρ represents the Euclidean distance between the two center points; c represents the diagonal distance of the minimum closure area that can contain both the predicted box and the real box.
[0081] In the above formula (3), w represents the width of the target box predicted by the model, h represents the height of the target box predicted by the model, wgt represents the width of the actual marked target box, and hgt represents the height of the actual marked target box.
[0082] In a method for determining tower structure damage information proposed in an embodiment of the present application, on the one hand, in addition to using the first sample set to train the target detection model, a second sample set of tower videos in an actual environment is also used to train the target detection model. Therefore, the complex uncertain factors in the actual project can be grasped, and accurate tower structure damage detection results can be given, thereby effectively improving the accuracy of tower structure damage detection; on the other hand, the efficient and accurate YOLO v4 is used as the target detection model, so the accuracy of tower structure damage detection can be further improved.
[0083] Next, the second stage will be described.
[0084] like Figure 2 As shown, a method for determining damage information of a tower structure provided in an embodiment of the present application may include:
[0085] Step 102, obtaining a video of the first tower taken by a drone.
[0086] Step 103: Input the images contained in the video frame by frame into the target detection model.
[0087] The target detection model refers to a general term for models that can realize the task of detecting structural damage of iron towers. The target detection model is used to detect and annotate the structural damage information in the input iron tower image. The target model is obtained by training based on a training data set. The training data set includes a first sample set and a second sample set. The first sample set includes the iron tower image with annotated structural damage information, and the second sample set includes the video of the second iron tower taken by a drone. Specifically, the target detection model is obtained by the training in the first stage mentioned above. For the specific training process, please refer to the description of the first stage in the above question.
[0088] It can be understood that if the images contained in the video of the first tower captured by the drone are input into the target detection model frame by frame for damage detection, the structural damage information of the first tower can be output in real time, that is, the structural damage marking and classification of the first tower can be completed in real time, which can improve the efficiency of structural damage detection of the first tower.
[0089] Step 103: receiving structural damage information of the first tower output by the target detection model.
[0090] As mentioned above, the structural damage information includes but is not limited to the structural damage location and the structural damage type.
[0091] Specifically, the target detection model outputs an image of the first tower annotated with structural damage information, wherein the image includes the location and type of the structural damage.
[0092] A method for determining tower structure damage information proposed in an embodiment of the present application uses a target detection model. On the one hand, during training, in addition to using the first sample set, a video of the tower in the actual environment - the second sample set - is also used. Therefore, the complex uncertain factors in the actual project can be grasped, and accurate tower structure damage detection results can be given, thereby effectively improving the accuracy of tower structure damage detection. On the other hand, an efficient and accurate target detection model is used, so the accuracy of tower structure damage detection can be further improved.
[0093] Optional, such as Figure 3 As shown, a method for determining tower structure damage information proposed in an embodiment of the present application may also include:
[0094] Step 105, obtaining the location information of the drone when shooting the video of the first tower.
[0095] Specifically, the drone may be equipped with a positioning sensor, which, while collecting video of the first tower, simultaneously collects its own position data and transmits the two together with a time stamp back to a device that executes a method for determining tower structure damage information proposed in an embodiment of the present application.
[0096] Step 106: determine a specific structural damage location on the first tower based on the location information.
[0097] It can be understood that the structural damage position output by the target detection model is the position of the structural damage in the input tower image. Here, combined with the drone position information obtained by the positioning sensor carried by the drone, the structural damage on the tower can be located.
[0098] The above introduces a method for determining tower structure damage information proposed in an embodiment of the present application. Corresponding to the above method for determining tower structure damage information, an embodiment of the present application also proposes a device for determining tower structure damage information, which is described below.
[0099] like Figure 4 As shown, a tower structure damage information determination device 400 proposed in an embodiment of the present application includes: a model training module 401.
[0100] The model training module 401 is used to train the target detection model based on the training data set.
[0101] In the embodiment of the present application, the target detection model refers to a general term for a model that can realize the task of detecting damage to a tower structure. As an example, the target detection model may be YOLO v4 (You Only Look Once Version 4).
[0102] In the embodiment of the present application, the iron tower may include but is not limited to a single-tube communication tower.
[0103] In some embodiments, the training data set includes a first sample set, and the first sample set includes tower images with labeled structural damage information.
[0104] In some embodiments, the training data set includes a second sample set, and the second sample set includes a video of a second iron tower taken by a drone. The second iron tower may be any iron tower that has been confirmed to have structural damage, and there may be multiple second iron towers.
[0105] In some other embodiments, the training data set includes a first sample set and a second sample set.
[0106] 1) In the case where the training data set includes the first sample set, before step 101, the method for determining damage information of a tower structure proposed in the embodiment of the present application may further include:
[0107] Acquire images of several towers with structural damage;
[0108] Preprocessing the plurality of iron tower images;
[0109] marking structural damage information of the preprocessed images of the plurality of iron towers;
[0110] The labeled images of the several iron towers are used as the first sample set.
[0111] As an example, the above-mentioned acquisition of several tower images with structural damage may include: several common tower structural damage images collected through relevant channels, wherein the relevant channels may include but are not limited to manual on-site photography, Internet websites, relevant documents, and the like.
[0112] Taking YOLO v4 as an example of the target detection model mentioned in this application, the above-mentioned preprocessing of the several iron tower images may include: adjusting the size of the several iron tower images with structural damage to a size suitable for input to YOLOv4, specifically, adjusting the size of the above-mentioned several iron tower images to 608×608 pixels.
[0113] The structural damage information of the tower may include but is not limited to at least one of the structural damage location and the structural damage type. The structural damage type may be set in advance according to actual needs. For example, the structural damage type may include but is not limited to corrosion, deformation, etc. Furthermore, for different types, the levels may be divided according to the degree of damage.
[0114] As an example, the above-mentioned labeling of structural damage information of the preprocessed images of the several towers may include: setting common structural damage types of towers in Labeling software, and using a label selection box to label the structural damage location and the corresponding structural damage type in the image in response to the user's labeling operation.
[0115] Furthermore, in the case where the training data set includes a first sample set, step 101 may specifically include: using the tower image in the first sample set as input to train the target detection model.
[0116] 2) When the training data set also includes a second sample set, before step 101, the method for determining damage information of a tower structure proposed in the embodiment of the present application may further include:
[0117] Get the video of the second tower taken by the drone;
[0118] The video of the second tower is used as the second sample set.
[0119] It should be noted that the second iron tower may include the first iron tower or may not include it.
[0120] Furthermore, in the case where the training data set also includes a second sample set, step 101 may also include: using the video in the second sample set as input to train the object detection model.
[0121] YOLO v4 is used as an example of the target detection model mentioned in this application. Specifically, each frame of the video of the second tower taken by the drone is used as the input of YOLO v4, and the image with structural damage information such as the location and type of structural damage is used as the output to build, train and optimize the hyperparameters of the YOLO v4 model.
[0122] In the embodiment of the present application, YOLO v4 is used as an example of the target detection model mentioned in the present application. Among them, YOLO v4 as a target detection model includes a backbone network (Backbone), a neck network (Neck) and a prediction network (Head), and the neck network connects the backbone network and the prediction network, wherein:
[0123] The backbone network is used to extract features from the input tower image;
[0124] The neck network is used to further extract features from the features extracted by the backbone network;
[0125] The prediction network is used to predict the target detection result based on the features extracted by the neck network.
[0126] The backbone network, neck network and prediction network are described in detail below.
[0127] 1) Backbone
[0128] As an example, this application uses the CSPDarknet53 network as the backbone network in the YOLO v4 model to extract features from the input tower image. After the tower image with a size of 608×608 pixels is input into the YOLO v4 model, the CSPDarknet53 network is first used to extract the features of the tower damage target.
[0129] In an embodiment of the present application, the backbone network adopts the Mish activation function instead of the original RELU activation function, and adds a Dropblock block to further enhance the generalization ability of the backbone network.
[0130] In an embodiment of the present application, the backbone network includes 5 residual block modules, and the input tower image is downsampled through a convolution kernel of size 3×3 in each residual block module. After the backbone network performs feature extraction on the input tower detection image, the feature map size changes five times, and finally changes from 608×608 to a feature map of size 19×19, thereby achieving rapid dimensionality reduction of the tower image. The feature map output by the residual block module is fused in the neck network of the YOLO v4 model.
[0131] Among them, please refer to the above for the formula of the Mish activation function, which will not be repeated here.
[0132] 2) Neck network
[0133] The neck network (Neck) connects the backbone network (Backbone) and the prediction network (Head) to further improve the diversity and robustness of features.
[0134] The neck network includes a spatial pyramid pooling (SPP) module and a path aggregation network (PAN).
[0135] The SPP module concatenates the max-pooling outputs of low-resolution feature maps to extract the most representative features. The SPP module uses kernels of size 1×1, 5×5, 9×9, and 13×13 for max pooling. The concatenation of feature maps increases the receptive field of the features extracted by the backbone network, thereby improving the accuracy of the entire network in detecting small objects.
[0136] The feature maps spliced by the SPP module are fused with the high-resolution feature maps through PAN. PAN sets up bottom-up and top-down paths through upsampling and downsampling operations to achieve the combination of low-level and high-level features. The PAN module outputs a set of aggregated features for prediction.
[0137] 3) Prediction Network (Head)
[0138] In the embodiment of the present application, YOLO v4 uses the prediction network of YOLO v3 for multi-scale prediction. The prediction network uses three different levels of feature maps for fusion, which are located in the middle layer, the lower middle layer, and the bottom layer. The sizes of the three feature layers are different, which improves the detection performance of targets of different sizes.
[0139] Furthermore, the structural damage information includes the structural damage location and the structural damage type, and the prediction network includes a first detection layer, a second detection layer and a third detection layer, wherein:
[0140] The first detection layer is used to predict a bounding box;
[0141] The second detection layer is used to predict the type of structural damage;
[0142] The third detection layer is used to predict structural damage.
[0143] That is to say, in the embodiment of the present application, the prediction network has three detection layers. Each detection layer is a YOLO v3 that calculates the final prediction. The YOLO v4 model outputs feature maps of size 19×19, 38×38, and 76×76, which are used to predict bounding boxes, structural damage type scores, and structural damage scores, respectively. YOLO v4 uses the complete intersection over union (CIoU) as the loss function. Please refer to the above for the calculation formula of the CIoU loss function, which will not be repeated here.
[0144] In a method for determining tower structure damage information proposed in an embodiment of the present application, on the one hand, in addition to using the first sample set to train the target detection model, a second sample set of tower videos in an actual environment is also used to train the target detection model. Therefore, the complex uncertain factors in the actual project can be grasped, and accurate tower structure damage detection results can be given, thereby effectively improving the accuracy of tower structure damage detection; on the other hand, the efficient and accurate YOLO v4 is used as the target detection model, so the accuracy of tower structure damage detection can be further improved.
[0145] It should be noted that Figure 4 The tower structure damage information determination device 400 can achieve Figure 1 The method of the method embodiment of the present invention can achieve the same technical effect, and the specific reference can be made to the above Figure 1 The introduction of the method for determining the damage information of the tower structure in the illustrated embodiment will not be repeated here.
[0146] like Figure 5 As shown, an apparatus 400 for determining damage information of a tower structure provided in an embodiment of the present application may include: an image acquisition module 402 , an image input module 403 and a result receiving module 404 .
[0147] The image acquisition module 402 is used to acquire the video of the first tower taken by the drone.
[0148] The image input module 403 is used to input the images contained in the video frame by frame into the target detection model.
[0149] The target detection model refers to a general term for models that can realize the task of detecting structural damage of iron towers. The target detection model is used to detect and annotate structural damage information in input iron tower images. The training data set includes a first sample set and a second sample set. The first sample set includes iron tower images annotated with structural damage information, and the second sample set includes videos of the second iron tower taken by a drone.
[0150] It can be understood that if the images contained in the video of the first tower captured by the drone are input into the target detection model frame by frame for damage detection, the structural damage information of the first tower can be output in real time, that is, the structural damage marking and classification of the first tower can be completed in real time, which can improve the efficiency of structural damage detection of the first tower.
[0151] The result receiving module 404 is used to receive the structural damage information of the first tower output by the target detection model.
[0152] As mentioned above, the structural damage information includes but is not limited to the structural damage location and the structural damage type.
[0153] Specifically, the target detection model outputs an image of the first tower annotated with structural damage information, wherein the image includes the location and type of the structural damage.
[0154] The target detection model used in a device for determining tower structure damage information proposed in an embodiment of the present application, on the one hand, in addition to using the first sample set during training, also uses the tower video in the actual environment - the second sample set, so that it can grasp the complex uncertain factors in the actual project and give accurate tower structure damage detection results, thereby effectively improving the accuracy of tower structure damage detection; on the other hand, an efficient and accurate target detection model is used, so that the accuracy of tower structure damage detection can be further improved.
[0155] It should be noted that Figure 5 The tower structure damage information determination device 400 can achieve Figure 2 The method of the method embodiment of the present invention can achieve the same technical effect, and the specific reference can be made to the above Figure 2 The introduction of the method for determining the damage information of the iron tower structure in the illustrated embodiment will not be repeated here.
[0156] Optional, such as Figure 6 As shown, a tower structure damage information determination device 400 proposed in an embodiment of the present application may further include: a position information acquisition module 405 and a damage location module 406.
[0157] The position information acquisition module 405 is used to obtain the position information of the drone when shooting the video of the first tower.
[0158] Specifically, the drone may be equipped with a positioning sensor, which, while collecting video of the first tower, simultaneously collects its own position data and transmits the two together with a time stamp back to a device that executes a method for determining tower structure damage information proposed in an embodiment of the present application.
[0159] The damage location module 406 is used to determine the specific structural damage location on the first tower based on the location information.
[0160] It can be understood that the structural damage position output by the target detection model is the position of the structural damage in the input tower image. Here, combined with the drone position information obtained by the positioning sensor carried by the drone, the structural damage on the tower can be located.
[0161] It should be noted that Figure 6 The tower structure damage information determination device 400 can achieve Figure 3 The method of the method embodiment of the present invention can achieve the same technical effect, and the specific reference can be made to the above Figure 3The introduction of the method for determining the damage information of the tower structure in the illustrated embodiment will not be repeated here.
[0162] Figure 7 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Figure 7 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. The memory may include a memory, such as a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. Of course, the electronic device may also include hardware required for other services.
[0163] The processor, network interface and memory can be interconnected through an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.
[0164] The memory is used to store the program. Specifically, the program may include a program code, and the program code includes a computer operation instruction. The memory may include a memory and a non-volatile memory, and provides instructions and data to the processor.
[0165] The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a tower structure damage information determination device at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations:
[0166] Get the video of the First Tower taken by the drone;
[0167] Input the images contained in the video frame by frame into a target detection model, wherein the target detection model refers to a general term for a model capable of realizing structural damage detection of an iron tower, and the target detection model is used to detect and annotate structural damage information in an input iron tower image, and the target model is obtained by training based on a training data set, and the training data set includes a first sample set and a second sample set, wherein the first sample set includes an iron tower image with annotated structural damage information, and the second sample set includes a video of the second iron tower taken by a drone;
[0168] Receive structural damage information of the first tower output by the target detection model.
[0169] The above application Figure 2 The tower structure damage information determination method disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or instructions in software form. The above processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in one or more embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in combination with one or more embodiments of the present application can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.
[0170] The electronic device may also perform Figure 2 The method for determining the tower structure damage information is not described in detail in this application.
[0171] The embodiment of the present application also provides a computer-readable storage medium, which stores one or more programs, wherein the one or more programs include instructions, which, when executed by a portable electronic device including a plurality of application programs, enable the portable electronic device to execute Figure 2 The method of the embodiment shown is specifically used to perform the following operations:
[0172] Get the video of the First Tower taken by the drone;
[0173] Input the images contained in the video frame by frame into a target detection model, wherein the target detection model is a general term for a model capable of realizing the detection of structural damage of an iron tower, and the target detection model is used to detect and annotate the structural damage information in the input iron tower image. The target model is obtained by training based on a training data set, and the training data set includes a first sample set and a second sample set. The first sample set includes the iron tower images annotated with structural damage information, and the second sample set includes the video of the second iron tower taken by a drone;
[0174] Receive structural damage information of the first tower output by the target detection model.
[0175] The embodiment of the present application also proposes a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method for determining tower structure damage information provided in the embodiment of the present application.
[0176] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0177] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0178] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0179] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0180] It should be noted that the various embodiments in this application are described in a related manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0181] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
[0182] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.
Claims
1. A method for determining damage information of a tower structure, characterized in that: The method comprises: Get the video of the First Tower taken by the drone; Input the images contained in the video frame by frame into a target detection model, wherein the target detection model refers to a general term for a model that can realize the task of detecting structural damage of an iron tower, and the target detection model is used to detect and annotate structural damage information in the input iron tower image. The target model is obtained by training based on a training data set, and the training data set includes a first sample set and a second sample set. The first sample set includes images of the iron tower annotated with structural damage information, and the second sample set includes a video of the second iron tower taken by a drone; Receive structural damage information of the first tower output by the target detection model.
2. The method according to claim 1, characterized in that The method further comprises: Obtaining the location information of the drone when shooting the video of the first tower; The specific structural damage location on the first tower is determined based on the location information.
3. The method according to claim 1, characterized in that Before inputting the images contained in the video frame by frame into the target detection model, the method further includes: The object detection model is trained based on the training data set.
4. The method according to claim 3, characterized in that Before training the target detection model based on the training data set, the method further includes: Acquire images of several towers with structural damage; Preprocessing the plurality of iron tower images; marking structural damage information of the preprocessed images of the plurality of iron towers; taking the labeled plurality of tower images as the first sample set; Wherein, training the target detection model based on the training data set includes: The tower image in the first sample set is used as input to train the target detection model.
5. The method according to claim 4, characterized in that Before training the target detection model based on the training data set, the method further includes: Obtain a video of the second iron tower taken by a drone; Using the video of the second tower as the second sample set; Wherein, the training of the target detection model based on the training data set further includes: The video in the second sample set is used as input to train the object detection model.
6. The method according to any one of claims 1 to 5, characterized in that: The target detection model is YOLO v4, and the target detection model includes a backbone network, a neck network and a prediction network, wherein: The backbone network is used to extract features from the input tower image; The neck network is used to further extract features from the features extracted by the backbone network; The prediction network is used to predict and obtain target detection results based on the features extracted by the neck network.
7. The method according to claim 6, characterized in that The loss function used by the backbone network is the Mish activation function.
8. The method according to claim 6, characterized in that The structural damage information includes the structural damage location and the structural damage type, and the prediction network includes a first detection layer, a second detection layer and a third detection layer, wherein: The first detection layer is used to predict a bounding box; The second detection layer is used to predict the type of structural damage; The third detection layer is used to predict structural damage.
9. A device for determining damage information of a tower structure, characterized in that: The device comprises: An image acquisition module, used to acquire the video of the first tower taken by the drone; An image input module is used to input the images contained in the video into a target detection model frame by frame, wherein the target detection model is a general term for a model that can realize the task of detecting damage to the iron tower structure, and the target detection model is used to detect and annotate the structural damage information in the input iron tower image, and the training data set includes a first sample set and a second sample set, wherein the first sample set includes the iron tower image with annotated structural damage information, and the second sample set includes the video of the second iron tower taken by a drone; A result receiving module is used to receive the structural damage information of the first tower output by the target detection model.
10. An electronic device comprising: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method according to any one of claims 1-8.
11. A computer-readable storage medium, when instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method according to any one of claims 1 to 8.
12. A computer program product, characterized in that The program product is stored in a storage medium, and the program product is executed by at least one processor to implement the method according to any one of claims 1 to 8.
