A method, device and equipment for identifying vibration of a cable and a storage medium

Abstract

The application relates to a cable vibration identification method, device and equipment and a storage medium. The method extracts high semantic features and low semantic features of a to-be-detected image by presetting a target detection model, extracts multi-scale features from the high semantic features according to pyramid pooling in an encoding stage, extracts features from the low semantic features by using an effective channel attention mechanism in a decoding stage, then inputs the features into a convolution layer in parallel, fuses the output features, and finally fuses the features of the two stages to serve as the final features of a target image. Compared with the prior art, the method designs a target detection model, adds an effective channel attention mechanism and two parallel 1x1 convolution layers in the decoding stage, captures deep semantic information of the to-be-detected image by pyramid pooling in the encoding stage, can improve the representation ability of the target detection model for deep features of an image, and improves the segmentation precision and generalization ability of the target detection model for a complex image background.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of computer vision, and particularly relate to a stay vibration identification method, device, equipment and storage medium. BACKGROUND

[0002] In the construction process of cable-supported bridges, it is necessary to accurately measure the cable force to ensure the stability of the structure and the smooth progress of the construction. In the later operation process, due to the influence of external factors such as corrosion, wind load, vehicle load, etc., the cable is prone to diseases such as rust, stress relaxation, and loss of prestress, thereby causing various changes in cable force. The aging and degradation of the cable lead to a decrease in its carrying capacity, causing redistribution of internal forces in the structure. Therefore, it is necessary to evaluate the internal tension of the cable in a timely manner and take appropriate maintenance and reinforcement measures. The traditional contact sensor measurement method has the disadvantages of time-consuming, high cost, and difficulty in installing sensors, which limits its applicability. Compared with the cable vibration measurement relying on contact sensors, the non-contact measurement method based on computer vision has the advantages of low cost, time saving, and convenience, and this method usually relies on the identification of cable movement in the vibration video.

[0003] The frequency method is the most advantageous test method in cable force testing, and has been most widely used in cable force testing of cable-supported system bridges at home and abroad. One of the key problems to be solved in cable force identification based on the frequency method is to accurately identify the natural frequency of the cable structure. In the actual cable measurement process, bad weather such as fog will affect the quality of the cable vibration captured by the video acquisition device, and the visibility of the cable target is poor; moreover, the movement of objects in complex background images will also affect the accuracy of the cable vibration identification method. In the field of computer vision, due to the movement of the background image, there is no static background in the captured video, and the existence of complex environment seriously hinders the application of computer vision technology in cable-supported bridges. When dealing with complex image backgrounds and cable vibration in bad weather, accurate identification of cable vibration is a challenge. SUMMARY

[0004] The following is a summary of the subject matter described in detail herein.

[0005] The main purpose of the embodiments of the present application is to propose a stay vibration identification method, device, equipment and storage medium, which can improve the feature extraction capability of the target detection model, and improve the complex image background segmentation accuracy and generalization capability of the target detection model.

[0006] To achieve the above-mentioned purpose, a first aspect of the embodiments of the present application provides a stay vibration identification method, which comprises:

[0007] obtaining a to-be-detected image;

[0008] extract a target image from the to-be-detected image according to a preset target detection model, the target image being an image of a stay rope detected in the to-be-detected image; wherein the extracting the target image from the to-be-detected image according to the preset target detection model comprises:

[0009] extracting first features and second features of the to-be-detected image according to MobileNetv2, the first features being high semantic features compared to the second features, and the second features being low semantic features compared to the first features;

[0010] extracting first intermediate features from the second features according to an effective channel attention mechanism, inputting the first intermediate features into two 1x1 convolution layers in parallel to obtain second intermediate features output by the two 1x1 convolution layers respectively, and fusing the two second intermediate features to obtain third features;

[0011] extracting fourth features from the first features according to pyramid pooling;

[0012] fusing the third features and the fourth features to obtain fused features, and calculating the target image according to the fused features;

[0013] recognizing the stay rope vibration according to the target image.

[0014] The method provided in the embodiments of the present application extracts high semantic features and low semantic features of a to-be-detected image by MobileNetv2, extracts multi-scale features from the high semantic features according to pyramid pooling in an encoding stage to capture deep semantic information of the to-be-detected image, extracts intermediate features from the low semantic features by using an effective channel attention mechanism in a decoding stage, inputs the intermediate features into convolution layers in parallel, and finally fuses the output features to obtain fused features. The fused features are used as final features for calculating a target image. Compared with the prior art, the method designs a target detection model including an encoding stage and a decoding stage, adds the effective channel attention mechanism and two 1x1 convolution layers in parallel in the decoding stage, and captures deep semantic information of the to-be-detected image by pyramid pooling in the encoding stage. The method can improve the representation ability of the target detection model for deep features of an image, and improve the segmentation accuracy and generalization ability of the target detection model for a complex image background.

[0015] In some embodiments of the present application, the extracting the fourth features from the first features according to pyramid pooling comprises:

[0016] The first feature is input into a 1*1 convolution layer to obtain a fourth intermediate feature, and a fifth intermediate feature is extracted from the fourth intermediate feature according to an effective channel attention mechanism;

[0017] The first feature is input into a 1*1 convolution layer to obtain a fourth intermediate feature, and a fifth intermediate feature is extracted from the fourth intermediate feature according to an effective channel attention mechanism;

[0018] The third intermediate feature and the fifth intermediate feature are fused to obtain a fourth feature.

[0019] In some embodiments of the present application, the third feature and the fourth feature are fused to obtain a fused feature, and a target image is calculated according to the fused feature, comprising:

[0020] The fourth feature is bilinearly interpolated and upsampled to make the size of the fourth feature consistent with the to-be-detected image;

[0021] The fourth feature consistent with the to-be-detected image is fused with the third feature to obtain a fused feature;

[0022] The fused feature is input into a 3*3 convolution layer, and the output feature of the 3*3 convolution layer is bilinearly interpolated and upsampled to obtain the target image with the same size as the to-be-detected image.

[0023] In some embodiments of the present application, the target detection model is trained by transfer learning, and the training process of the transfer learning comprises:

[0024] A first model is constructed, and the first model is a DeepLabV3+ model;

[0025] A second model is constructed according to the first model, and the second model is a model formed after replacing a backbone network Xception model in the first model with MobileNetv2;

[0026] The second model is pre-trained on a PASCAL VOC dataset, so that the second model has prior knowledge of the PASCAL VOC dataset; the PASCAL VOC dataset is formed after all initial images are labeled, and the initial image is a stay rope image collected by a collection device;

[0027] A branch is added in parallel to the empty pyramid pooling in the encoding stage of the second model, and an ECM module is added to the decoding stage of the second model to obtain a third model; the branch comprises a cascaded 1*1 convolution layer and an effective channel attention mechanism, and the ECM module comprises an effective channel attention mechanism and two 1*1 convolution layers in parallel.

[0028] training the third model on the enhanced dataset to obtain a trained target detection model; the enhanced dataset is a dataset formed after feature enhancement on the basis of the PASCAL VOC dataset.

[0029] In some embodiments of the present application, the means of feature enhancement comprises:

[0030] performing at least one operation of horizontal flipping, upside-down flipping, Gaussian noise, Gaussian blur, and occlusion enhancement transformation on the PASCAL VOC dataset.

[0031] In some embodiments of the present application, the identifying the cable vibration according to the target image comprises:

[0032] composing a cable vibration continuous image sequence according to the target image;

[0033] performing motion amplification on the cable vibration in the cable vibration continuous image sequence, creating a time sequence according to the amplified cable vibration, performing Hilbert transform to obtain real time and imaginary time sequences, and performing dimension reduction processing on the real time and imaginary time sequences by principal component analysis;

[0034] performing BSS blind source separation to analyze the original signals that are not observed from multiple mixed signals and visualize the signals, creating modal coordinates and visualizing the modal coordinates, and performing Fourier transform to obtain a power spectral density diagram of the cable vibration.

[0035] In some embodiments of the present application, before the target image is extracted from the to-be-detected image according to the preset target detection model, the method further comprises:

[0036] performing defogging on the input image according to RefineDNet.

[0037] To achieve the above-mentioned purpose, the second aspect of the embodiments of the present application provides a cable vibration identification device, the device comprises:

[0038] an image acquisition unit configured to acquire a to-be-detected image;

[0039] The target detection unit is configured to extract a target image from the to-be-detected image according to a preset target detection model, the target image being an image of the inhaul cable detected in the to-be-detected image; wherein the extraction of the target image from the to-be-detected image according to the preset target detection model comprises: extracting a first feature and a second feature of the to-be-detected image according to MobileNetv2, the first feature being a high semantic feature compared with the second feature, and the second feature being a low semantic feature compared with the first feature; extracting a first intermediate feature from the second feature according to an effective channel attention mechanism, inputting the first intermediate feature into two 1x1 convolution layers arranged in parallel to obtain two second intermediate features output by the two 1x1 convolution layers respectively, and fusing the two second intermediate features to obtain a third feature; extracting a fourth feature from the first feature according to pyramid pooling; fusing the third feature and the fourth feature to obtain a fused feature, and calculating the target image according to the fused feature.

[0040] The vibration identification unit is configured to identify the inhaul cable vibration according to the target image.

[0041] To achieve the above object, a third aspect of the embodiment of the present application provides an electronic device, comprising: at least one control processor and a memory in communication connection with the at least one control processor; the memory stores instructions executable by the at least one control processor, and the instructions are executed by the at least one control processor to enable the at least one control processor to execute the inhaul cable vibration identification method of the first aspect.

[0042] To achieve the above object, a fourth aspect of the embodiment of the present application provides a computer readable storage medium, which stores computer executable instructions for enabling a computer to execute the inhaul cable vibration identification method of the first aspect.

[0043] It can be understood that the beneficial effects of the second aspect to the fourth aspect and the related technology compared with the first aspect and the related technology are the same, and can be referred to the related description in the first aspect, which will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiments or related technical description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0045] Figure 1is a flowchart of a cable vibration identification method provided by an embodiment of the present application;

[0046] Figure 2 is a flowchart of extracting a fourth feature from a first feature according to pyramid pooling provided by an embodiment of the present application;

[0047] Figure 3 is a flowchart of fusing a third feature and a fourth feature to obtain fused features, and calculating a target image according to the fused features provided by an embodiment of the present application;

[0048] Figure 4 is a flowchart of identifying cable vibration according to a target image provided by an embodiment of the present application;

[0049] Figure 5 is a flowchart of a cable vibration identification method provided by another embodiment of the present application;

[0050] Figure 6 is a flowchart of an image defogging model and a target detection model provided by an embodiment of the present application;

[0051] Figure 7 is a flowchart of calculating cable vibration identification provided by an embodiment of the present application;

[0052] Figure 8 is a schematic diagram of a cable model provided by an embodiment of the present application;

[0053] Figure 9 is a schematic diagram of a non-paired training sample set data type provided by an embodiment of the present application;

[0054] Figure 10 is a cable complex image background segmentation data type provided by an embodiment of the present application;

[0055] Figure 11 is a flowchart of image augmentation provided by an embodiment of the present application;

[0056] Figure 12 is a schematic diagram of a cable image defogging model framework provided by an embodiment of the present application;

[0057] Figure 13 is a flowchart of defogging model transfer learning provided by an embodiment of the present application;

[0058] Figure 14 is a result of image defogging provided by an embodiment of the present application;

[0059] Figure 15 is a skeleton network of a target detection model provided by an embodiment of the present application;

[0060] Figure 16 FIG. 18 is a schematic diagram of a flow of target detection model transfer learning provided by an embodiment of the present application;

[0061] Figure 17 FIG. 19 is a schematic diagram of a Loss-Epoch curve of a target detection model provided by an embodiment of the present application;

[0062] Figure 18 FIG. 20 is a schematic diagram of a detection result of a target detection model provided by an embodiment of the present application;

[0063] FIG. 19(a) is a cable vibration frequency spectrum diagram sampled by an accelerometer provided by an embodiment of the present application;

[0064] FIG. 19(b) is a vibration modal coordinate and its power spectral density identification result identified by a cable vibration identification method provided by an embodiment of the present application;

[0065] Figure 20 FIG. 20 is a schematic diagram of a detection result of a target detection model provided by an embodiment of the present application;

[0066] Figure 21 FIG. 21 is a structural diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0067] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not intended to limit the present application.

[0068] It should be noted that although the functional modules are divided in the device schematic diagram, and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a manner different from the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification and claims and the above-described drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0069] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application, and are not intended to limit the present application.

[0070] With reference to Figure 1 , an embodiment of the present application provides a cable vibration identification method, the method comprising:

[0071] Step S100, acquiring a to-be-detected image.

[0072] Step S200, extracting a target image from the to-be-detected image according to a preset target detection model, the target image being an image of the inhaul cable detected from the to-be-detected image; wherein the extracting of the target image from the to-be-detected image according to the preset target detection model comprises:

[0073] According to MobileNetv2, the first feature and the second feature of the to-be-detected image are extracted, the first feature being a high semantic feature compared with the second feature, and the second feature being a low semantic feature compared with the first feature;

[0074] According to the effective channel attention mechanism, the first intermediate feature is extracted from the second feature, the first intermediate feature is input into the two 1x1 convolution layers in parallel to obtain the second intermediate features output by the two 1x1 convolution layers respectively, and the third feature is obtained by fusing the two second intermediate features;

[0075] According to the pyramid pooling, the fourth feature is extracted from the first feature;

[0076] The third feature and the fourth feature are fused to obtain a fused feature, and the target image is calculated according to the fused feature.

[0077] Step S300, recognizing the inhaul cable vibration according to the target image.

[0078] In step S100, the to-be-detected image refers to an image that needs to be subjected to target detection, and the target here is the inhaul cable.

[0079] In step S200, a target detection model is preset to extract the target image from the to-be-detected image through the target detection model, the target image being an image of the inhaul cable detected from the to-be-detected image by the model. The target detection model mainly includes an encoding stage (Encoder) and a decoding stage (Decoder), wherein MobileNetv2 (the main idea of MobileNet-v2 is to introduce a linear bottleneck and an inverse residual to improve the representation ability of the network on the basis of v1, and it is also a lightweight convolutional neural network) is used to extract high semantic features and low semantic features of the to-be-detected image. In the encoding stage, multi-scale features are extracted from the high semantic features according to the pyramid pooling to capture deep semantic information of the to-be-detected image; compared with the prior art, in the decoding stage, the intermediate feature is extracted from the low semantic feature by using the effective channel attention mechanism, and then the intermediate feature is input into the convolution layer in parallel, and finally the output feature is obtained by fusion. Finally, the features output by the encoding stage and the features output by the decoding stage are fused as the final features for calculating the target image.

[0080] Compared with the technical scheme at the present stage, the method is designed to include an encoding stage and a decoding stage. In the decoding stage, an effective channel attention mechanism and two parallel 1×1 convolution layers are added. In the encoding stage, deep semantic information of the image to be detected is captured through pyramid pooling, which can improve the feature extraction capability of the target detection model, and improve the segmentation precision and generalization capability of the target detection model in complex image backgrounds.

[0081] Referring to Figure 2 In an embodiment of the present application, the fourth feature is extracted from the first feature according to the pyramid pooling in step S200, which includes:

[0082] In step S210, the first feature is subjected to dilated pyramid pooling to obtain a multi-scale feature, and the multi-scale feature is input into a 1×1 convolution layer to obtain a third intermediate feature.

[0083] In step S220, the first feature is input into a 1×1 convolution layer to obtain a fourth intermediate feature, and a fifth intermediate feature is extracted from the fourth intermediate feature according to the effective channel attention mechanism.

[0084] In step S230, the third intermediate feature and the fifth intermediate feature are fused to obtain the fourth feature.

[0085] In step S210, the dilated pyramid pooling is used to extract the multi-scale feature from the first feature, and then the 1×1 convolution layer is used for feature compression to make the feature size consistent with the image to be detected. In steps S220 and S230, a parallel feature extraction module is added to the dilated pyramid pooling, which includes a 1×1 convolution layer and an effective channel attention mechanism. Through the feature extraction module, the shortcomings of the target detection model in the image to be detected, such as missed detection and over-detection, can be improved, the feature extraction precision of the model can be improved, and the cable feature in the target image can be more discriminative.

[0086] Referring to Figure 3 In step S200, the third feature and the fourth feature are fused to obtain a fused feature, and the target image is calculated according to the fused feature, which includes:

[0087] In step S240, the fourth feature is subjected to bilinear interpolation up-sampling to make the size of the fourth feature consistent with the image to be detected.

[0088] In step S250, the fourth feature with the same size as the image to be detected is fused with the third feature to obtain a fused feature.

[0089] In step S260, the fused feature is input into a 3×3 convolution layer, and the output feature of the 3×3 convolution layer is subjected to bilinear interpolation up-sampling to obtain a target image with the same size as the image to be detected.

[0090] In step S240, the multi-scale features (fourth features) are bilinearly interpolated and up-sampled to make the size of the fourth features consistent with the image to be detected. In step S250, the fourth features are fused with the third features to obtain fused features. In step S260, the fused features are input into a 3x3 convolutional layer, and the output features of the 3x3 convolutional layer are bilinearly interpolated and up-sampled to obtain a target image of the same size as the image to be detected, so as to capture the detailed information contained in the shallow features and obtain deep features of the cable image.

[0091] In some embodiments of the present application, the target detection model is trained by using transfer learning, and the training process of the transfer learning includes:

[0092] In step S510, a first model is constructed, and the first model is a DeepLabV3+ model.

[0093] In step S520, a second model is constructed according to the first model, and the second model is a model formed after replacing the Xception model in the first model with a MobileNetv2 model.

[0094] In step S530, the second model is pre-trained on a PASCAL VOC dataset, so that the second model has prior knowledge of the PASCAL VOC dataset; the PASCAL VOC dataset is formed after all initial images are labeled, and the initial images are cable images collected by a collection device.

[0095] In step S540, a branch is added to the parallel hollow pyramid pooling in the encoding stage of the second model, and an ECM module is added to the decoding stage of the second model to obtain a third model; the branch includes a cascaded 1x1 convolutional layer and an effective channel attention mechanism, and the ECM module includes an effective channel attention mechanism and two parallel 1x1 convolutional layers.

[0096] In step S550, the third model is trained on an enhanced dataset to obtain a trained target detection model; the enhanced dataset is formed by feature enhancement based on the PASCAL VOC dataset.

[0097] In the training process of the target detection model, the transfer learning method is used to make the target detection model have sufficient generalization ability. The data set collection and enhancement are introduced in subsequent embodiments.

[0098] In some embodiments, the feature enhancement means includes:

[0099] Step S610, at least one of horizontal flip, vertical flip, Gaussian noise, Gaussian blur, and occlusion augmentation is performed on the PASCAL VOC dataset.

[0100] Referring to Figure 4 In some embodiments, the cable vibration is identified according to the target image, including:

[0101] Step S310, a continuous image sequence of the cable vibration is formed according to the target image.

[0102] Step S320, motion amplification is performed on the cable vibration in the continuous image sequence of the cable vibration, and a time sequence is created according to the amplified cable vibration, a Hilbert transform is performed to obtain real time and imaginary time sequences, and principal component analysis is performed to reduce the dimension of the real time and imaginary time sequences.

[0103] Step S330, BSS blind source separation is used to analyze the original signal without observation and visualize the signal from the multiple mixed signals, create modal coordinates and visualize the modal coordinates, and Fourier transform is performed to obtain the power spectral density diagram of the cable vibration.

[0104] Referring to Figure 5 to Figure 1 9, the embodiment provides a cable vibration identification method, and the method includes:

[0105] Step S710, a dataset of cable images is established.

[0106] Considering different appearances, perspectives, illuminations, backgrounds, and objects similar to the cable captured in the same image, a laboratory cable model (as shown in Figure 8 is used to sample the dataset through mobile terminal devices such as drones and GoPro cameras. The collected cable image samples consider the changes of cable appearance, shooting perspective, illumination, distance, and background, as shown in Figure 9 .

[0107] The collected image samples are labeled using the Labelme data labeling tool to create a database that can be used for model training. During the labeling process, the cable in the image sample is labeled and saved according to the format of the Pascal VOC dataset to obtain the dataset of cable images.

[0108] Step S720, feature augmentation is performed on the image samples in the dataset.

[0109] Image samples are usually difficult to collect, and here the Imgau data augmentation toolkit is used to simulate image samples in complex environments such as foggy, rainy, snowy, and motion blur, color change, and occlusion, etc. to expand the original database of cable images. The image expansion process is as shown in Figure 11 .

[0110] From horizontal flip, upside-down flip, Gaussian noise, Gaussian blur, occlusion enhancement transformation, randomly select 1 kind of enhancement transformation, and randomly select 2 kinds of enhancement transformation from multi-fog / snow, rain enhancement transformation, then set appropriate enhancement multiple, enhance the database, and obtain the foggy image dataset and the enhanced dataset after expansion. The number and quality of training samples of the enhanced dataset meet the basic requirements of model training, and the dataset is divided into training set, validation set and test set according to the ratio of 8:1:1; the foggy image dataset is used as the unpaired training sample set containing foggy images.

[0111] Step S730, constructing a dehazing model, training the dehazing model.

[0112] Taking RefineDNet as the framework of the dehazing model, the unpaired training sample set (foggy image dataset) is sent to the dehazing model, and the ambient light A, the preliminary dehazing image J DCP , and the transmission map T DCP are generated by the dark channel prior (DCP).

[0113] The preliminary dehazing image and the transmission map are input into the refinement network R T and R J , respectively, to obtain T ref and j ref .

[0114] By refining and reconstructing the blurred input image T rec into I rec , and then updating the refinement R T by minimizing the difference between I rec and I real .

[0115] By predicting the image and the clear image, the discriminator D of the clean image and the foggy image is generated, and the dehazing model is obtained, as shown in Figure 12 .

[0116] In order to ensure that the dehazing model has sufficient generalization ability, a transfer learning method is used in the model training process, including taking RefineDNet as the basic model framework, taking fefined_DCP as the backbone network of the model and pre-training on the RESIDE dataset. Finally, RefineDNet is pre-trained on the unpaired training sample set, and finally a dehazing model capable of realizing the dehazing of the foggy image of the cable and enhancing the visibility of the cable target is obtained, and the transfer learning process is as shown in Figure 13 .

[0117] In this embodiment, Figure 14 is the dehazing result of the dehazing model of the cable image.

[0118] Step S740, a target detection model is constructed, and the target detection model is trained.

[0119] The image in the enhanced data set is taken as a to-be-detected image.

[0120] The DeepLabV3+ model is taken as a framework of the target detection model, the backbone network Xception of the DeepLabV3+ is replaced by the MobileNetv2, the ASPP module of the encoding stage is replaced by the ASPP-I, and the ECM module is added in the decoding stage. As shown in FIG. 6, the target detection model training process includes the following steps. Figure 15

[0121] Step S741, the to-be-detected image is sent into the MobileNetv2, and high semantic features and low semantic features of the to-be-detected image are obtained.

[0122] Step S742, a feature extraction module is connected in parallel with the ASPP module of the encoding stage of the target detection model, the ASPP-I is constructed, the ASPP-I is taken as a multi-scale feature extraction network of the target detection model, and deep semantic information of the to-be-detected image is captured. An effective channel attention module is added in the decoding stage of the target detection model, the output features of the attention module are respectively input into two Conv1x1 layers, then the Conv results are fused, a new feature module is constructed, and here, the feature extraction module is named as the ECM.

[0123] The high semantic features are input into the ASPP-I module of the encoding stage, multi-scale features of the to-be-detected image are obtained, and the feature channel number is compressed through the 1x1 convolution. The results after 4 times upsample of the bilinear interpolation module are fused, the detail information contained in the shallow features is captured, and the deep features of the to-be-detected image are obtained.

[0124] Step S743, the low semantic features are input into the ECM module of the decoding stage of the target detection model, and are concatenated with the deep features of the to-be-detected image to obtain a fused feature map.

[0125] Step S744, 3x3 convolution layers are used to extract semantic information and detail information of the fused feature map, the to-be-detected image size is gradually restored through 4 times upsample of the bilinear interpolation, an ultimate semantic segmentation prediction result output layer (a target image) is generated, and a target detection model capable of detecting targets in a complex image background is obtained.

[0126] ​In order to ensure that the target detection model has sufficient generalization ability and detection precision, the embodiment proposes to use the transfer learning method in the model training process, including: taking DeepLabV3+ as the basic model framework, replacing the skeleton network Xception of DeepLabV3+ with a lightweight convolutional neural network MobileNetv2 and pre-training on the PASCAL VOC dataset, so that DeepLabV3+ has the prior knowledge of the PASCAL VOC dataset. At the same time, replace the Atrous Spatial Pyramid Pooling ASPP in the encoding stage of DeepLabV3+ with ASPP-I, and add the ECM module in the decoding stage to obtain DeepLabV3+ ECM and pre-train on the enhanced dataset, and finally obtain the target detection model, and the transfer learning process is as shown in Figure 16

[0127] In the embodiment, Figure 17 is the Loss-Epoch curve of the target detection model.

[0128] In the embodiment, Figure 18 is the detection result of the target detection model.

[0129] Step S750, cable vibration identification is performed on the target image.

[0130] According to the target image, a continuous image sequence of cable vibration is constructed, the cable vibration in the continuous image sequence is magnified in motion, a vibration time sequence is created, a real time and a virtual time sequence are obtained by Hilbert transform, and dimension reduction processing is performed on the real time and the virtual time sequence by principal component analysis. The BSS blind source separation is used to analyze the original signals that are not observed from multiple mixed signals and visualize the signals, create modal coordinates and visualize the modal coordinates, and then obtain the power spectrum density diagram of the cable vibration by Fourier transform.

[0131] In the embodiment, FIG. 19 is the recognition result of the cable vibration method in a complex environment, wherein (a) is the cable vibration frequency spectrum diagram sampled by the accelerometer, and (b) is the cable vibration modal coordinate and its power spectrum density obtained by the method provided in the embodiment.

[0132] The following provides a set of control experiment embodiments:

[0133] In the embodiment, when the ASPP-I and the ECM module are increased at the same time, the cable recognition precision is improved, and the results are as follows, and the MIoU is increased by 1.71%. Among them, MIoU, MPA and PA are semantic segmentation evaluation indexes.

[0134]

[0135] Table 1 comparative results on the cable image verification set

[0136] Referring to​Figure 20 , this embodiment provides a cable vibration recognition device, the device comprising: an image acquisition unit 1100, a target detection unit 1200 and a vibration recognition unit 1300;

[0137] The image acquisition unit 1100 is used to acquire an image to be detected.

[0138] The target detection unit 1200 is used to extract a target image from the image to be detected according to a preset target detection model, where the target image is an image of a cable detected in the image to be detected; wherein, extracting the target image from the image to be detected according to the preset target detection model includes: extracting a first feature and a second feature of the image to be detected according to MobileNetv2, where the first feature is a high-semantic feature compared to the second feature, and the second feature is a low-semantic feature compared to the first feature; extracting a first intermediate feature from the second feature according to an effective channel attention mechanism, inputting the first intermediate feature into two parallel 1×1 convolutional layers to obtain second intermediate features output by the two 1×1 convolutional layers respectively, and fusing the two second intermediate features to obtain a third feature; extracting a fourth feature from the first feature according to pyramid pooling; fusing the third feature and the fourth feature to obtain a fused feature, and calculating the target image based on the fused feature.

[0139] The vibration recognition unit 1300 is used to recognize cable vibration according to the target image.

[0140] The device provided in this embodiment and the above-mentioned method are based on the same inventive concept, so the content of the above-mentioned method embodiment is also applicable to the content of the present device embodiment and will not be repeated here.

[0141] like Figure 21 , an embodiment of the present application further provides an electronic device, the electronic device comprising:

[0142] at least one memory;

[0143] at least one processor;

[0144] at least one program;

[0145] The programs are stored in the memory, and the processor executes at least one program to implement the cable vibration identification method described above in the present disclosure.

[0146] The electronic device may be any intelligent terminal including a mobile phone, a tablet computer, a personal digital assistant (PDA), a car computer, etc.

[0147] The electronic device according to the embodiment of the present application is described in detail below.

[0148] The processor 1600 can be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute related programs to implement the technical solutions provided by the embodiments of the present application.

[0149] The memory 1700 can be implemented by a read only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1700 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the program codes are stored in the memory 1700 and are called and executed by the processor 1600 to implement the cable vibration identification method of the embodiments of the present application.

[0150] The input / output interface 1800 is configured to implement information input and output.

[0151] The communication interface 1900 is configured to implement the communication interaction between the device and other devices. The communication can be realized by a wired manner (for example, a USB, a network cable, etc.) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, etc.).

[0152] The bus 2000 is configured to transmit information between various components (for example, the processor 1600, the memory 1700, the input / output interface 1800, and the communication interface 1900) of the device.

[0153] The processor 1600, the memory 1700, the input / output interface 1800, and the communication interface 1900 are connected to each other by the bus 2000 to realize the communication connection between the device.

[0154] The embodiments of the present application further provide a storage medium, which is a computer readable storage medium and stores computer executable instructions. The computer executable instructions are configured to make a computer execute the cable vibration identification method.

[0155] The memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory disposed remotely with respect to the processor, and these remote memories can be connected to the processor through a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0156] The embodiments described in the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0157] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than shown in the figures, or combine certain steps, or different steps.

[0158] The device embodiments described above are only schematic, and the units illustrated as separate components can or can not be physically separate, i.e., can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0159] Those skilled in the art can understand that all or some of the steps in the above disclosed method, the functional modules / units in the system and the device can be implemented as software, firmware, hardware and their appropriate combinations.

[0160] The terms "first", "second", "third", "fourth" and the like used in the specification of the present application and the above-described drawings, if any, are used to distinguish similar objects, and do not necessarily have to describe a particular order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to only those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0161] It should be understood that, in the application, "at least one" refers to one or more, and "multiple" refers to two or more. "And / or" is used to describe the association relationship of the associated objects, which means that there can be three relationships, for example, "A and / or B" can represent three cases of only A, only B and A and B existing at the same time, wherein A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after it. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can represent a, b, c, "a and b", "a and c", "b and c", or "a and b and c", wherein a, b and c can be single or multiple.

[0162] In several embodiments provided in the application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of units is only a logical function division, and actual implementation can have another division manner, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed units can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0163] The units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, that is, they can be located in one place, or they can be distributed on multiple network units. According to actual needs, part or all of the units can be selected to achieve the purpose of the embodiment scheme.

[0164] In addition, the functional units in each embodiment of the application can be integrated into a processing unit, or each unit can be physically present, or two or more units can be integrated into one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0165] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or say the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes multiple instructions for causing an electronic device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program storage media.

[0166] The above is a specific description of the preferred implementation of the embodiments of the present application, but the embodiments of the present application are not limited to the above-mentioned implementation, and those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the embodiments of the present application. These equivalent modifications or replacements are all included in the scope defined by the claims of the embodiments of the present application.

[0167] The above is a specific description of the preferred implementation of the embodiments of the present application, but the embodiments of the present application are not limited to the above-mentioned implementation, and those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the embodiments of the present application. These equivalent modifications or replacements are all included in the scope defined by the claims of the embodiments of the present application.

Claims

1. A cable vibration identification method, characterized in that: The method comprises: Obtain the image to be detected; Extracting a target image from the image to be detected according to a preset target detection model, wherein the target image is an image of a cable detected in the image to be detected; wherein extracting the target image from the image to be detected according to the preset target detection model includes: Extracting a first feature and a second feature of the image to be detected according to MobileNetv2, wherein the first feature is a high-semantic feature compared to the second feature, and the second feature is a low-semantic feature compared to the first feature; Extracting a first intermediate feature from the second feature according to the effective channel attention mechanism, inputting the first intermediate feature into two parallel 1×1 convolutional layers to obtain second intermediate features output by the two 1×1 convolutional layers respectively, and fusing the two second intermediate features to obtain a third feature; Extracting a fourth feature from the first feature according to pyramid pooling; fusing the third feature and the fourth feature to obtain a fused feature, and calculating a target image based on the fused feature; The cable vibration is identified based on the target image.

2. The cable vibration identification method according to claim 1, characterized in that: Extracting the fourth feature from the first feature according to pyramid pooling includes: Performing dilated pyramid pooling on the first feature to obtain a multi-scale feature, and inputting the multi-scale feature into a 1×1 convolutional layer to obtain a third intermediate feature; Input the first feature into a 1×1 convolutional layer to obtain a fourth intermediate feature, and extract a fifth intermediate feature from the fourth intermediate feature according to an effective channel attention mechanism; The third intermediate feature and the fifth intermediate feature are fused to obtain the fourth feature.

3. The cable vibration identification method according to claim 2, characterized in that: The fusing the third feature and the fourth feature to obtain a fused feature, and calculating a target image according to the fused feature, includes: Performing bilinear interpolation upsampling on the fourth feature so that a size of the fourth feature is consistent with the image to be detected; fusing the fourth feature, which has the same size as the image to be detected, with the third feature to obtain a fused feature; The fused features are input into a 3×3 convolutional layer, and the output features of the 3×3 convolutional layer are up-sampled by bilinear interpolation to obtain the target image of the same size as the image to be detected.

4. The cable vibration identification method according to claim 3, characterized in that: The target detection model is trained using transfer learning, and the training process of the transfer learning includes: Constructing a first model, wherein the first model is a DeepLabV3+ model; Construct a second model based on the first model, where the second model is formed by replacing the backbone network Xception model in the first model with MobileNetv2; Pre-training the second model on a PASCAL VOC dataset so that the second model has prior knowledge of the PASCAL VOC dataset; the PASCAL VOC dataset is a dataset formed by annotating all initial images, wherein the initial images are images of the cable captured by an acquisition device; A branch is added in parallel to the dilated pyramid pooling in the encoding stage of the second model, and an ECM module is added to the decoding stage of the second model to obtain a third model; the branch includes a cascaded 1×1 convolutional layer and an effective channel attention mechanism, and the ECM module includes an effective channel attention mechanism and two parallel 1×1 convolutional layers; The third model is trained on an enhanced dataset to obtain a trained object detection model; the enhanced dataset is a dataset formed by feature enhancement based on the PASCAL VOC dataset.

5. The cable vibration identification method according to claim 4, characterized in that: The feature enhancement means include: Perform at least one of horizontal flipping, upside down flipping, Gaussian noise, Gaussian blurring, and occlusion enhancement transformation on the PASCAL VOC dataset.

6. The cable vibration identification method according to claim 1, characterized in that: The identifying the cable vibration according to the target image includes: forming a cable vibration continuous image sequence according to the target image; Performing motion amplification on the cable vibration in the continuous image sequence of the cable vibration, creating a time series based on the amplified cable vibration, performing Hilbert transform to obtain real time and imaginary time series, and performing dimensionality reduction processing on the real time and imaginary time series using principal component analysis; BSS blind source separation is used to analyze the unobserved original signal from multiple mixed signals and visualize the signal, create modal coordinates and visualize the modal coordinates, and obtain the power spectrum density diagram of the cable vibration through Fourier transform.

7. The cable vibration identification method according to claim 1, characterized in that: Before extracting the target image from the image to be detected according to the preset target detection model, the method further includes: The input image is dehazed according to RefineDNet.

8. A cable vibration identification device, characterized in that: The device comprises: An image acquisition unit, used for acquiring an image to be detected; A target detection unit is configured to extract a target image from the image to be detected according to a preset target detection model, where the target image is an image in which a cable in the image to be detected is detected; wherein the extracting the target image from the image to be detected according to the preset target detection model includes: extracting a first feature and a second feature of the image to be detected according to MobileNetv2, where the first feature is a high-semantic feature compared to the second feature, and the second feature is a low-semantic feature compared to the first feature; extracting a first intermediate feature from the second feature according to an effective channel attention mechanism, inputting the first intermediate feature into two parallel 1×1 convolutional layers to obtain second intermediate features outputted by the two 1×1 convolutional layers respectively, and fusing the two second intermediate features to obtain a third feature; extracting a fourth feature from the first feature according to pyramid pooling; fusing the third feature with the fourth feature to obtain a fused feature, and calculating the target image according to the fused feature; A vibration recognition unit is used to recognize the cable vibration according to the target image.

9. An electronic device, characterized in that: include: at least one control processor and a memory for communicatively coupling with the at least one control processor; The memory stores instructions that can be executed by the at least one control processor, and the instructions are executed by the at least one control processor to enable the at least one control processor to perform the cable vibration identification method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the cable vibration identification method according to any one of claims 1 to 7.

Images