Multi-modal fusion power transmission line inspection method and device

Abstract

The invention provides a multi-modal fusion power transmission line inspection method and device, and relates to the technical field of power transmission line inspection application, and the method comprises the steps: collecting a multi-modal image of a to-be-inspected power transmission line at a current moment; inputting a feature pyramid network embedded into a multi-scale context adaptive joint learning network to obtain a multi-modal image fusion feature, and judging whether an abnormal event exists in the image corresponding to the to-be-inspected power transmission line under the action of the multi-modal image fusion feature; if the position information of the unmanned aerial vehicle exists, establishing an incidence relation function between the position information of the unmanned aerial vehicle at the current moment and the multi-modal image fusion features based on the similarity between the multi-modal features representing the position information of the unmanned aerial vehicle; inputting the corresponding image of the to-be-inspected power transmission line with the abnormal event mark into the incidence relation function, and determining longitude and latitude position information of the abnormal event in the to-be-inspected power transmission line; the technical problem that a power transmission line has potential safety hazards due to poor accuracy of an inspection result is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of power transmission line inspection application, and in particular to a multi-modal fusion power transmission line inspection method and device. Background Art

[0002] Transmission lines usually pass through areas with complex geographical environments, including hills and mountains. These environmental factors pose severe challenges to the safe operation of transmission lines. In complex environments, how to quickly identify and eliminate potential safety hazards of transmission lines to ensure their safe, stable and efficient operation is a major problem currently faced by the operation and inspection management of transmission lines.

[0003] In order to solve the above problems and improve the operation and maintenance efficiency of transmission lines, new inspection methods such as drones and helicopters have begun to be used to conduct full coverage inspections of transmission corridors and take photos, and analyze equipment defects through image data. Although this image-based inspection method has improved the inspection efficiency to a certain extent, it also has significant shortcomings such as poor accuracy of inspection results, which makes the transmission lines have safety hazards. First, it is difficult to provide clear inspection data in harsh environments such as fog and nighttime by relying solely on visible light images, which affects the accuracy of inspection results. Secondly, facing massive image data, relying on manual judgment is not only time-consuming and labor-intensive, but also prone to missed inspections or misjudgments due to human subjectivity. In addition, the real-time transmission and processing of inspection images are restricted by the communication conditions of the transmission lines, and timely response cannot be achieved, which in turn affects the safe operation of the transmission lines. Summary of the invention

[0004] The purpose of the present invention is to provide a multi-modal fusion power transmission line inspection method and device to alleviate the technical problem that the power transmission line has safety hazards due to poor accuracy of inspection results.

[0005] In a first aspect, an embodiment of the present invention provides a multi-modal fusion transmission line inspection method, comprising:

[0006] At least one camera installed on the drone is used to collect a multimodal image of the power transmission line to be inspected at the current moment;

[0007] The multimodal image is preprocessed and input into a feature pyramid network embedded in a multi-scale context adaptive joint learning network to obtain a multimodal image fusion feature, and under the action of the multimodal image fusion feature, it is determined whether there is an abnormal event in the image corresponding to the power transmission line to be inspected;

[0008] If so, establishing a correlation function between the current position information of the UAV and the multimodal image fusion feature based on the similarity between the multimodal features representing the position information of the UAV;

[0009] The image corresponding to the power transmission line to be inspected with abnormal event marks is input into the association function to determine the longitude and latitude position information of the abnormal event in the power transmission line to be inspected.

[0010] In combination with the first aspect, an embodiment of the present invention provides a first possible implementation of the first aspect, wherein a feature pyramid network embedded in a multi-scale context adaptive joint learning network is input to obtain a multimodal image fusion feature, and a step of determining whether there is an abnormal event in the image corresponding to the power transmission line to be inspected under the action of the multimodal image fusion feature includes:

[0011] Based on the multi-scale context adaptive joint learning network, a fusion feature of the multimodal image is obtained;

[0012] Outputting an image corresponding to the power transmission line to be inspected at the current moment according to the fusion features of the multimodal image and the detection function of the feature pyramid network;

[0013] Based on whether there is an abnormal event mark in the image corresponding to the power transmission line to be inspected at the current moment, it is determined whether there is an abnormal event in the image corresponding to the power transmission line to be inspected.

[0014] In combination with the first aspect, an embodiment of the present invention provides a second possible implementation of the first aspect, wherein the multi-scale context adaptive joint learning network includes a feature enhancement branch, a feature screening branch and an adaptive multi-scale fusion branch; the feature enhancement branch is composed of a global average pooling layer, a first fully connected layer and a second fully connected layer; the feature screening branch is composed of parallel feature value convolution layers and gated value convolution layers; and the adaptive multi-scale fusion branch is composed of a learnable weighted fusion module.

[0015] In combination with the first aspect, an embodiment of the present invention provides a third possible implementation of the first aspect, wherein the step of obtaining the fusion features of the multimodal image based on the multi-scale context adaptive joint learning network includes:

[0016] Extracting global features of the preprocessed multimodal image through the global average pooling layer;

[0017] Determine a feature weight of each multimodal image based on the global feature, a first weight corresponding to the first fully connected layer, and a second weight corresponding to the second fully connected layer;

[0018] Performing an element-wise product of the feature weight of each multimodal image and the cross-layer feature to obtain an enhanced feature of each multimodal image; wherein the cross-layer feature is a feature obtained by splicing each two adjacent layers in the multi-scale feature; and the multi-scale feature is a multi-layer image feature generated by inputting each pre-processed multimodal image into the feature pyramid network;

[0019] Inputting the enhanced features of each multimodal image into the feature value convolution layer to obtain high-level features of each multimodal image;

[0020] Inputting the enhanced features of each multimodal image into the gated value convolution layer to obtain the gated features of each multimodal image;

[0021] Performing an element-wise product of the high-level features of each multimodal image and the gated features to obtain a screening feature of each multimodal image;

[0022] According to the multi-scale features, weights are configured for the screening features and the enhanced features of each multimodal image to obtain extraction features of each multimodal image at each scale;

[0023] The extracted features of each multimodal image at each scale are fused to obtain multimodal image fusion features.

[0024] In combination with the first aspect, an embodiment of the present invention provides a fourth possible implementation of the first aspect, wherein the step of establishing a correlation relationship function between the location information of the drone at the current moment and the multimodal image fusion feature based on the similarity between the multimodal features characterizing the location information of the drone includes:

[0025] When an abnormal event occurs in the image corresponding to the power transmission line to be inspected, respectively obtaining text features corresponding to the satellite navigation position of the UAV at the current moment and sensor features for representing the position information of the UAV at the current moment;

[0026] Calculating a first similarity between the sensor feature and the text feature;

[0027] Calculating a second similarity between the multimodal image fusion feature and the text feature;

[0028] Based on the first similarity and the second similarity, an association relationship function between the multimodal image fusion feature and the sensor feature used to characterize the position information of the drone is established.

[0029] In combination with the first aspect, an embodiment of the present invention provides a fifth possible implementation of the first aspect, wherein the step of inputting the image corresponding to the transmission line to be inspected with the abnormal event mark into the association function to determine the latitude and longitude position information of the abnormal event in the transmission line to be inspected includes:

[0030] Inputting the image corresponding to the power transmission line to be inspected, which is output by the feature pyramid network and has abnormal event marks, into the association function to determine a joint embedding vector of the multimodal image fusion feature and the sensor feature used to characterize the location information of the drone;

[0031] The joint embedding vector is subjected to nonlinear calculation to generate longitude and latitude position information of abnormal events in the power transmission line to be inspected.

[0032] In combination with the first aspect, an embodiment of the present invention provides a sixth possible implementation of the first aspect, wherein the step of preprocessing the multimodal image includes:

[0033] Adjusting the frame rate according to the sampling rate of each multimodal image to unify the frame rates of the various multimodal images;

[0034] Perform preprocessing operations on the multimodal images after the frame rate is unified;

[0035] The various multimodal images after the preprocessing operation are spatially aligned based on the intrinsic parameters and extrinsic parameters of each camera.

[0036] In a second aspect, an embodiment of the present invention further provides a multi-modal fusion power transmission line inspection device, comprising:

[0037] A collection module, which collects a multimodal image of the power transmission line to be inspected at the current moment through at least one camera installed on the drone;

[0038] A judgment module preprocesses the multimodal image, inputs the preprocessed multimodal image into a feature pyramid network embedded in a multi-scale context adaptive joint learning network to obtain a multimodal image fusion feature, and judges whether there is an abnormal event in the image corresponding to the power transmission line to be inspected under the action of the multimodal image fusion feature;

[0039] An establishment module, if any, establishes a correlation function between the location information of the drone at the current moment and the multimodal image fusion feature based on the similarity between the multimodal features representing the location information of the drone;

[0040] The determination module inputs the image corresponding to the power transmission line to be inspected with the abnormal event mark into the association function to determine the longitude and latitude position information of the abnormal event in the power transmission line to be inspected.

[0041] In a third aspect, an embodiment provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the computer program, the steps of the method described in any of the aforementioned implementation methods are implemented.

[0042] In a fourth aspect, an embodiment provides a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the steps of the method described in any one of the aforementioned implementations.

[0043] The embodiment of the present invention provides a multimodal fusion power transmission line inspection method and device. Through various cameras installed on a drone, a multimodal image of the power transmission line to be inspected at the current moment is obtained, a multiscale context adaptive joint learning network is embedded in a feature pyramid network, and the multimodal image after synchronization alignment preprocessing is input into the feature pyramid network. Various multimodal image feature fusions are realized through the multiscale context adaptive joint learning network. Based on the multimodal image fusion features, the feature pyramid network can realize more accurate detection of abnormal events in the corresponding image of the power transmission line. If there is an abnormal event in the power transmission line at the current moment, a correlation coefficient function of the drone position information and the multimodal image fusion features can be established based on the similarity between the multimodal features that can characterize the current position of the drone. Based on this function, the corresponding image of the power transmission line to be inspected with abnormal event marks output by the feature pyramid network is input, and the longitude and latitude position information of the abnormal event in the power transmission line to be inspected can be further determined, so as to facilitate timely maintenance when abnormal events occur in the transmission line.

[0044] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description and the drawings.

[0045] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0047] Figure 1 A flow chart of a multi-modal fusion power transmission line inspection method provided by an embodiment of the present invention;

[0048] Figure 2 A flow chart of another multi-modal fusion power transmission line inspection method provided by an embodiment of the present invention;

[0049] Figure 3 A schematic diagram of functional modules of a multi-modal fusion power transmission line inspection device provided in an embodiment of the present invention;

[0050] Figure 4 A schematic diagram of the hardware architecture of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] In recent years, image recognition and feature extraction methods based on convolutional neural networks have developed rapidly and have gradually been applied to various visual recognition tasks. In particular, with the continuous improvement of computer hardware, intelligent inspection modes for power transmission lines that combine target detection, sensing technology, and communication technology have gradually emerged. Practical applications have shown that this new inspection mode can be expanded according to different needs and has excellent performance in speed and accuracy. However, in complex actual environments, relying solely on traditional visible light images is still difficult to meet the inspection needs of all scenarios, especially in low visibility conditions such as heavy fog and nighttime, and the performance of existing intelligent inspection systems is still insufficient.

[0053] Based on this, the embodiments of the present invention provide a multimodal fusion transmission line inspection method and device, which can accurately identify abnormal events in transmission lines under harsh environments through the fusion results of multimodal image features, and can determine the location of the abnormal events, facilitating timely maintenance.

[0054] To facilitate understanding of this embodiment, a multimodal fusion transmission line inspection method disclosed in an embodiment of the present invention is first introduced in detail. This method can be applied to controllers, intelligent control modules, host computers, servers and other execution devices. The execution device can be a controller set on a drone, or a host computer or server that communicates with the controller of the drone.

[0055] Figure 1A flow chart of a multi-modal fusion power transmission line inspection method provided in an embodiment of the present invention.

[0056] Reference Figure 1 , the method may include the following steps:

[0057] Step S102: collecting a multimodal image of the power transmission line to be inspected at the current moment by using at least one camera installed on the drone.

[0058] Among them, the multimodal image may include an infrared image captured by an infrared camera, an RGB image captured by an RGB camera, a point cloud image captured by a lidar, and the like; for ease of explanation, the multimodal image in an embodiment of the present invention can be exemplified by infrared images and RGB images.

[0059] In step S104, the multimodal image is preprocessed and input into a feature pyramid network embedded in a multi-scale context adaptive joint learning network to obtain a multimodal image fusion feature. Under the action of the multimodal image fusion feature, it is determined whether there is an abnormal event in the image corresponding to the power transmission line to be inspected.

[0060] Among them, a multi-scale context adaptive joint learning network is embedded in the feature pyramid network. In addition to identifying abnormal events from the corresponding images of the transmission line to be inspected through the structure with detection function in the feature pyramid network, multi-modal image fusion features are generated under the action of the multi-scale context adaptive joint learning network. Based on the multi-modal image fusion features, the abnormal event recognition effect of the feature pyramid network is further enhanced; the abnormal event can be understood as an identification box in the image. If the representation box appears, an abnormal event exists, and vice versa.

[0061] The multi-scale context-adaptive joint learning network includes a feature enhancement branch, a feature screening branch and an adaptive multi-scale fusion branch; the feature enhancement branch is composed of a global average pooling layer, a first fully connected layer and a second fully connected layer; the feature screening branch is composed of parallel eigenvalue convolutional layers and gated value convolutional layers; the adaptive multi-scale fusion branch is composed of a learnable weighted fusion module.

[0062] Step S106: if it exists, based on the similarity between the multimodal features representing the location information of the drone, a correlation function between the location information of the drone at the current moment and the multimodal image fusion features is established.

[0063] Among them, the multimodal features may include text features, sensor features and multimodal image fusion features obtained in the steps of the aforementioned embodiment. Such multimodal features can reflect the position of the drone when an abnormal event occurs; based on the text features, the association between the sensor features and the multimodal image fusion features is constructed.

[0064] It should be noted that if there is an abnormal event, a correlation function between the drone's location information and the multimodal image fusion features can be established based on the similarity association between multiple features that can characterize the drone's location, so that subsequent steps can determine the longitude and latitude position information of the abnormal event in the transmission line based on the correlation function.

[0065] Step S108, inputting the image corresponding to the power transmission line to be inspected with abnormal event marks into the association function to determine the longitude and latitude position information of the abnormal event in the power transmission line to be inspected.

[0066] In a preferred embodiment of practical application, a multimodal image of the power transmission line to be inspected at the current moment is obtained by various cameras installed on the drone, a multi-scale context adaptive joint learning network is embedded in the feature pyramid network, and the multimodal image preprocessed by synchronization alignment is input into the feature pyramid network. Various multimodal image feature fusions are realized through the multi-scale context adaptive joint learning network. Based on this multimodal image fusion feature, the feature pyramid network can realize more accurate detection of abnormal events in the corresponding image of the power transmission line; if there is an abnormal event in the power transmission line at the current moment, a correlation coefficient function of the drone position information and the multimodal image fusion feature can be established based on the similarity between the multimodal features that can characterize the current position of the drone; based on this function, the corresponding image of the power transmission line to be inspected with the abnormal event mark output by the feature pyramid network is input, and the longitude and latitude position information of the abnormal event in the power transmission line to be inspected can be further determined, so as to facilitate timely maintenance when an abnormal event occurs in the transmission line.

[0067] In some embodiments, preprocessing the multimodal images in step S104 may be understood as synchronously acquiring images of each modality, aligning the scale, performing noise reduction processing, and then synchronizing the images to the same space, including:

[0068] Step 1.1), adjusting the frame rate according to the sampling rate of each multimodal image, and unifying the frame rates of various multimodal images.

[0069] Assume that the sampling rate of the infrared camera is 20 frames per second and the sampling rate of the RGB camera is 25 frames per second. Since the sampling frequencies of the infrared image and the RGB image are different, direct time synchronization will result in data misalignment. Therefore, the embodiment of the present invention uses the infrared camera with a lower sampling frequency as a reference to synchronize the infrared image and the RGB image in time.

[0070] Specifically, first, the sampling frequency of the infrared camera is selected as the time reference; second, the RGB image is downsampled to be aligned with the infrared image in time.

[0071] For example, the RGB camera takes one frame every five frames, and the infrared camera takes one frame every four frames; that is, since the sampling rates of different cameras are inconsistent, the frame sampling frequencies are adjusted to make the frame sampling rates of different cameras consistent, thereby achieving temporal synchronization of multimodal images.

[0072] Step 1.2), preprocessing operations are performed on the multimodal images after the frame rate is unified.

[0073] It can be understood that the preprocessing operation may include operations such as filtering, noise reduction, and cropping to reduce errors.

[0074] In step 1.3), various multimodal images after preprocessing are spatially aligned based on the intrinsic and extrinsic parameters of each camera.

[0075] Since the infrared camera and the RGB camera have certain differences in viewing angle and position when they are installed, spatial calibration is required to ensure that the images captured by the two cameras can be aligned in the same spatial reference system. In the embodiment of the present invention, the infrared camera and the RGB camera are first calibrated using a calibration plate to obtain their intrinsic parameters and extrinsic parameters respectively. Then, the infrared image and the RGB image are converted to the same coordinate system using the calibration results.

[0076] Specifically, the infrared image is converted to the world coordinate system using the intrinsic and extrinsic parameters of the infrared camera; the RGB image is converted to the world coordinate system using the intrinsic and extrinsic parameters of the RGB camera; and the coordinate transformation matrix under the same reference system is used to achieve spatial alignment of the infrared image and the RGB image.

[0077] On the basis of the above-mentioned embodiment, the input in step S104 is embedded in the feature pyramid network of the multi-scale context adaptive joint learning network to obtain the multi-modal image fusion feature. Under the action of the multi-modal image fusion feature, it is judged whether there is an abnormal event operation in the image corresponding to the power transmission line to be inspected. By embedding the pyramid network of the multi-scale context adaptive joint learning network, a more accurate abnormal event recognition effect can be achieved under the action of the multi-modal image fusion feature, which may specifically include:

[0078] In step 2.1), based on the multi-scale context adaptive joint learning network, the fusion features of the multimodal image are obtained.

[0079] Here, based on the embedding of the multi-scale context-adaptive joint learning network in the feature pyramid network, the fusion features of images of multiple sizes and multiple modalities can be obtained.

[0080] Step 2.2), based on the fusion features of the multimodal image and the detection function of the feature pyramid network, the corresponding image of the power transmission line to be inspected at the current moment is output.

[0081] It should be noted that the structure of the existing feature pyramid network has the function of detecting the identification box corresponding to abnormal events, and can more accurately identify abnormal events based on the fusion features of multimodal images.

[0082] Step 2.3) is to determine whether there is an abnormal event in the image corresponding to the power transmission line to be inspected based on whether there is an abnormal event mark in the image corresponding to the power transmission line to be inspected at the current moment.

[0083] Here, if the abnormal identification mark (identification box) exists, it is considered that an abnormal event exists in the transmission line to be inspected, and vice versa.

[0084] Based on the above embodiment, step 2.1) can be implemented by the following steps, specifically including:

[0085] Step 2.1.1), extracting the global features of the preprocessed multimodal image through a global average pooling layer;

[0086] In the feature enhancement branch, the global features of the input RGB image and infrared image are first extracted through the global average pooling layer to ensure the capture of the overall information of each modality image; the global feature extraction is implemented by the following formula:

[0087] F global_RGB =GAP(I RGB ),F global_IR =GAP(I IR )

[0088] Among them, I RGB ,I IR Represent the input RGB image and infrared image respectively, F global_RGB , F global_IR They are the global features of RGB images and infrared images respectively. GAP() can be understood as the functional function corresponding to the global average pooling layer.

[0089] Step 2.1.2), based on the global features, the first weight corresponding to the first fully connected layer and the second weight corresponding to the second fully connected layer, determine the feature weight of each multimodal image.

[0090] The feature weight of each multimodal image is obtained by the preset weights corresponding to the two fully connected layers. The feature weight can emphasize the features of the important areas in each multimodal image. The feature weight generation of each multimodal image can be achieved by the following formula:

[0091] W RGB =σ(W2·ReLU(W1·F global_RGB )),

[0092] W IR=σ(W2·ReLU(W1·F global_IR ))

[0093] Where W1 is the first weight of the first fully connected layer, W2 is the second weight of the second fully connected layer, σ represents the Sigmoid activation function, and W RGB is the feature weight of the RGB image, W IR is the feature weight of the IR image, and ReLU is the linear rectification activation function.

[0094] In step 2.1.3), the feature weight of each multimodal image is element-wise multiplied with the cross-layer feature to obtain the enhanced feature of each multimodal image.

[0095] Specifically, the feature weights of the multimodal images are used to strengthen the important areas of the cross-layer features of the RGB images and the cross-layer features of the infrared images, so that the network can extract key information more accurately. Then, the feature weights (attention weights) of these multimodal images are applied to the cross-layer features of the RGB images and the cross-layer features of the infrared images through pixel-level multiplication, further significantly displaying the important content, and finally obtaining the high-level features of the RGB images and the high-level features of the infrared images. This process enables the feature enhancement branch to effectively highlight the details of the key areas in the image in extremely dark scenes. The specific enhanced feature calculation formula is as follows:

[0096] F enhanced_RGB =W RGB ⊙F cross_RGB ,F enhanced_IR =W IR ⊙F cross_IR

[0097] Among them, ⊙ represents the element-by-element multiplication operation, F cross_RGB 、F cross_IR They represent the RGB features and infrared features extracted across layers, that is, the cross-layer features corresponding to the RGB image and the infrared image, respectively. enhanced_RGB is the enhanced feature of RGB image, F enhanced_IR It is the enhanced feature of infrared image.

[0098] Among them, the cross-layer feature is the feature obtained by splicing every two adjacent layers in the multi-scale feature; the multi-scale feature is the multi-layer image feature generated by inputting each preprocessed multimodal image into the feature pyramid network; for example, the multi-layer image feature generated by inputting the feature pyramid network of multimodal image A is five layers, then the cross-layer feature of multimodal image A is the feature obtained by splicing the first and second layers, the second and third layers, the third and fourth layers, and the fourth and fifth layers.

[0099] In step 2.1.4), the enhanced features of each multimodal image are input into the eigenvalue convolution layer to obtain the high-level features of each multimodal image.

[0100] The two parallel convolutional layers in the feature screening branch are used to calculate the eigenvalue and the gate value to enhance the flexibility and robustness of feature screening. First, the eigenvalue convolutional layer is used to extract the high-level features of the RGB image and the infrared image. The specific formula is as follows:

[0101] F value_RGB =Conv value (F enhanced_RGB )

[0102] F value_IR =Conv value (F enhanced_IR )

[0103] Among them, F value_RGB is the high-level feature of the RGB image, F value_IR Conv is the high-level feature of infrared images. value is the corresponding function of the eigenvalue convolution layer.

[0104] Step 2.1.5), input the enhanced features of each multimodal image into the gated value convolution layer to obtain the gated features of each multimodal image. Here, the formula for generating the gated features using the gated value convolution layer is as follows:

[0105] F gate_RGB =σ(Conv gate (F enhanced_RGB ))

[0106] F gate_IR =σ(Conv gate (F enhanced_IR ))

[0107] Among them, F gate_RGB is the gated feature of the RGB image, F gate_IR is the gated feature of the infrared image, Conv gate is the corresponding function of the gated convolution layer.

[0108] Step 2.1.6), perform element-wise multiplication of the high-level features of each multimodal image with the gated features to obtain the screening features of each multimodal image; these two feature outputs are fused together by element-wise multiplication, as shown in the following formula:

[0109] F selected_RGB =F value_RGB ⊙F gate_RGB

[0110] F selected_IR =F value_IR⊙F gate_IR

[0111] Among them, F selected_RGB is the screening feature of RGB image, F selected_IR is the screening feature of infrared image.

[0112] The feature screening branch allows the network to filter out unimportant features more flexibly, thereby improving the accuracy and effectiveness of feature extraction and avoiding information redundancy. In addition, the gating mechanism designed in the embodiment of the present invention can adaptively adjust and learn the strategy of filtering features, so as to better screen the high-level features in RGB images under the visible spectrum and infrared images, that is, the screening features obtained through the above processing can overcome the adverse effects of lighting defects on recognition accuracy in actual environments.

[0113] Step 2.1.7), weight the screening features and enhanced features of each multimodal image according to the multi-scale features, and obtain the extracted features of each multimodal image at each scale, which can be calculated by the following formula:

[0114] F multi-scale_RGB =Concat(F enhanced_RGB ,F selected_RGB )

[0115] F multi-scale_IR =Concat(F enhanced_IR ,F selected_IR )

[0116] Among them, F multi-scale_RGB is the multi-scale feature of the RGB image, that is, the extracted features of the RGB image at each scale, F multi-scale_IR is the multi-scale feature of the infrared image, that is, the extracted features of the infrared image at each scale, and Concat is a function used to connect arrays.

[0117] In the adaptive multi-scale fusion branch, the embodiment of the present invention designs a learnable weighted fusion module to dynamically adjust the weights of the enhanced features from the feature enhancement branch and the filtered features from the feature filtering branch at different scales. This branch uses the multi-scale receptive field to integrate multi-scale information from different branches, thereby enhancing the network's perception of multi-scale objects.

[0118] Step 2.1.8), the extracted features of each multimodal image at each scale are fused to obtain the multimodal image fusion features, which can be calculated by the following formula:

[0119]

[0120] Among them, i is used to represent the specific scale, is the extracted features of the RGB image at scale i, is the extracted features of the infrared image at scale i, F fused It is a multi-modal image fusion feature.

[0121] Through this multi-scale fusion, the network can simultaneously capture the detail information in the RGB image and the thermal information in the infrared image, making the fused features more comprehensive and rich, and thus making the abnormal event recognition results based on the fused features more accurate.

[0122] In some embodiments, if an abnormal event occurs in the power transmission line, step S106 is performed to further obtain the location where the abnormal event occurs, specifically including:

[0123] Step 3.1), when there is an abnormal event in the image corresponding to the power transmission line to be inspected, the text features corresponding to the satellite navigation position of the drone at the current moment and the sensor features used to characterize the position information of the drone at the current moment are obtained respectively.

[0124] In order to obtain the precise location of the abnormal position of the transmission line, an embodiment of the present invention designs a cross-modal positioning module, designs the location information of the drone into a set of learnable text tags, and gives them to the text encoder to form a fuzzy description, which is compared and learned with the features obtained by the multi-scale context adaptive joint learning network to obtain the precise location of the abnormal event of the transmission line.

[0125] Specifically, the IMU signal composed of the accelerometer and gyroscope measurements on the X, Y, and Z axes and the GNSS signal within 1 second are extracted and projected using a 1D convolution with a convolution kernel size of 4 to obtain the sensor feature U used to characterize the position information of the drone at the current moment. m Generate corresponding text tags according to the current GNSS position of the drone, design a set of text tags for each abnormal event, and input them into the text encoder to obtain the text features T corresponding to the satellite navigation position of the drone m .

[0126] Assume that the current GNSS coordinates of the drone are lat d =34.0522°N, lon d =118.2437°W, the relative GNSS position of the target is Δlat=0.0001°, Δlon=0.0002°, and the heading angle of the drone is θ y =45°, the pitch angle is θ p =10°, the roll angle is θ r =5°, the target distance is about 20 meters, and the direction is "30 degrees north by east".

[0127] A text mark designed in an embodiment of the present invention is expressed as: "The target position relative to the drone: the current latitude of the drone: 34.0522N; the current longitude of the drone: 118.2437W; relative latitude: 0.0001 degrees; relative longitude: 0.0002 degrees; the heading angle of the drone: 45 degrees; the pitch angle of the drone: 10 degrees; the roll angle of the drone: 5 degrees; the target distance from the drone is approximately: 20 meters; the target direction: 30 degrees north-east".

[0128] Among them, GNSS is the Global Navigation Satellite System, which is an air-based radio navigation and positioning system that can provide users with all-weather three-dimensional coordinates, speed and time information at any location on the earth's surface or in near-Earth space. The longitude and latitude coordinates of the drone can be obtained based on the GNSS coordinates.

[0129] Step 3.2), calculate the first similarity between the sensor feature and the text feature; use contrastive learning to calculate the similarity between the sensor feature and the text feature used to characterize the drone location information, and the calculation method is as follows:

[0130]

[0131] Among them, τ represents the temperature coefficient used to control the smoothness of similarity, CE(.) represents the cross entropy loss, and M is the number of text features obtained by historical text tagging. represents the matching probability of the mth sensor feature and the text feature T under the current text feature T, until all M text features are traversed, at this time, is a similarity matrix, s(.) represents cosine similarity; E (T,U)～D [] indicates expected calculation, and D has no physical meaning. represents the matching probability of the mth text feature and the sensor feature U under the current sensor feature U, until all M text features are traversed. At this time, is a similarity matrix, y u2t Represents the true label from sensor features to text features; y t2u Represents the true label of text features to sensor features, is the first similarity.

[0132] Step 3.3), calculating the second similarity between the multimodal image fusion feature and the text feature;

[0133] Contrastive learning is used to calculate the similarity of infrared images, RGB image fusion features and text tag features. The calculation method is as follows:

[0134]

[0135] Among them, τ represents the temperature coefficient used to control the smoothness of similarity, CE(.) represents the cross entropy loss, and M is the number of text features obtained by historical text tagging. represents the matching probability of the mth text feature and the fusion feature I under the current fusion feature I, until all M text features are traversed, at this time, is a similarity matrix, s(.) represents cosine similarity; represents the matching probability of the mth fusion feature and the text feature T under the current text feature T, until all M text features are traversed. At this time, is a similarity matrix, E (I,T)～D [] indicates expected calculation, D has no physical meaning, y v2t Represents the true label of the fusion feature to the text feature; y t2v Represents the true label from text features to fusion features; is the second similarity.

[0136] Step 3.4), based on the first similarity and the second similarity, establish a correlation function Fusion between the multimodal image fusion feature and the sensor feature used to characterize the location information of the drone.

[0137] Here, the first similarity model and the second similarity model are added to obtain the association relationship function Fusion model, and training is performed. When the gradient of the added model at the current moment and the previous moment no longer decreases, the association relationship function Fusion training is completed.

[0138] Based on the above-mentioned embodiment, the specific geographical location where the abnormal event occurs may be determined according to step S108, including:

[0139] In step 4.1), the image corresponding to the power transmission line to be inspected output by the feature pyramid network and marked with abnormal events is input into the association function to determine the joint embedding vector of the multimodal image fusion features and the sensor features used to characterize the location information of the drone.

[0140] E joint =Fusion(E GNSS ,F img )

[0141] Among them, E joint represents the joint embedding vector generated by the fused image encoding and the UAV position information encoding, NN represents nonlinear calculation, which is used to calculate the GNSS position information of the abnormal event according to the joint embedding vector, (E GNSS ,F img ) is the corresponding image of the power transmission line to be inspected output by the feature pyramid network and marked with abnormal events.

[0142] Here, when an abnormal event is identified, the text features collected by the satellite navigation system of the drone are used as a bridge to associate the sensor features used to characterize the geographic location of the drone at this time with the multimodal image fusion features to obtain a joint embedding vector.

[0143] In step 4.2), the joint embedding vector is nonlinearly calculated to generate the longitude and latitude location information of the abnormal event in the transmission line to be inspected.

[0144]

[0145] Among them, E joint represents the joint embedding vector generated by the fused image encoding and the UAV position information encoding, NN represents nonlinear calculation, To calculate the GNSS location information (longitude and latitude coordinates) of the abnormal event based on the joint embedding vector.

[0146] Figure 2 A flow chart of another multi-modal fusion power transmission line inspection method provided in an embodiment of the present invention.

[0147] like Figure 2 As shown, an embodiment of the present invention proposes a multimodal fusion transmission line inspection method and device combining infrared and RGB images. First, the RGB image and the infrared image are preprocessed, and the feature fusion of the RGB image and the infrared image is realized through a multi-scale context adaptive joint learning network. The network includes three branches: a feature enhancement branch, a feature screening branch, and an adaptive multi-scale fusion branch. By utilizing the complementarity of infrared images and RGB images, the accuracy and robustness problems of transmission line inspection in low visibility environments such as fog and night are effectively solved. Infrared imaging technology can obtain thermal radiation information of the transmission line and its surrounding environment when visible light conditions are insufficient, thereby providing richer image information for defect detection. Through multimodal fusion technology, the infrared image and the visible light image are effectively combined, which not only retains the rich details of the visible light image, but also combines the temperature distribution characteristics of the infrared image, so that the transmission line inspection can be accurately completed even in complex environments.

[0148] In addition, the embodiment of the present invention also inputs the multimodal fusion features into the cross-modal positioning module, combining GNSS technology and a multi-scale context adaptive joint learning network to achieve accurate positioning of abnormal events in the transmission line. Specifically, the location information of the UAV is designed as a set of learnable text tags, and a fuzzy description is generated by a text encoder. The description is compared and learned with the features extracted by the multi-scale context adaptive joint learning network to accurately locate abnormal events in the transmission line. This method not only supports UAVs to quickly patrol over a large area, discover and locate problems in a timely manner, and improve patrol efficiency, but also realizes accurate positioning of defect locations by associating defect information with GNSS location data. This innovation improves the rapid locking and maintenance efficiency of fault locations, while providing comprehensive patrol data for power grid companies to support subsequent operation and maintenance decisions.

[0149] The embodiments of the present invention provide a multimodal fusion power transmission line inspection method and device. By combining the multimodal fusion technology of infrared and RGB images and integrating the text description of information such as drone GNSS, the accuracy and robustness of the power transmission line inspection system in low visibility environments such as fog and night are improved, while the positioning of GNSS information of the defect position is achieved, thereby improving the rapid locking of the fault position and the maintenance efficiency.

[0150] In some embodiments, Figure 3 As shown, an embodiment of the present invention provides a multi-modal fusion power transmission line inspection device, comprising:

[0151] A collection module, which collects a multimodal image of the power transmission line to be inspected at the current moment through at least one camera installed on the drone;

[0152] A judgment module preprocesses the multimodal image, inputs the preprocessed multimodal image into a feature pyramid network embedded in a multi-scale context adaptive joint learning network to obtain a multimodal image fusion feature, and judges whether there is an abnormal event in the image corresponding to the power transmission line to be inspected under the action of the multimodal image fusion feature;

[0153] An establishment module, if any, establishes a correlation function between the location information of the drone at the current moment and the multimodal image fusion feature based on the similarity between the multimodal features representing the location information of the drone;

[0154] The determination module inputs the image corresponding to the power transmission line to be inspected with the abnormal event mark into the association function to determine the longitude and latitude position information of the abnormal event in the power transmission line to be inspected.

[0155] Furthermore, the judgment module is specifically used to obtain the fusion features of the multimodal image based on the multi-scale context adaptive joint learning network; output the corresponding image of the transmission line to be inspected at the current moment according to the fusion features of the multimodal image and the detection function of the feature pyramid network; and determine whether there is an abnormal event in the corresponding image of the transmission line to be inspected based on whether there is an abnormal event mark in the corresponding image of the transmission line to be inspected at the current moment.

[0156] Furthermore, the multi-scale context adaptive joint learning network includes a feature enhancement branch, a feature screening branch and an adaptive multi-scale fusion branch; the feature enhancement branch is composed of a global average pooling layer, a first fully connected layer and a second fully connected layer; the feature screening branch is composed of a parallel eigenvalue convolution layer and a gated value convolution layer; and the adaptive multi-scale fusion branch is composed of a learnable weighted fusion module.

[0157] Furthermore, the judgment module is specifically used to extract the global features of the preprocessed multimodal image through the global average pooling layer; based on the global features, the first weight corresponding to the first fully connected layer and the second weight corresponding to the second fully connected layer, determine the feature weight of each multimodal image; perform element-wise product of the feature weight of each multimodal image with the cross-layer feature to obtain the enhanced feature of each multimodal image; wherein the cross-layer feature is a feature obtained by splicing each adjacent two layers in the multi-scale feature; the multi-scale feature is a multi-layer image generated by inputting each preprocessed multimodal image into the feature pyramid network Features; inputting the enhanced features of each multimodal image into the feature value convolution layer to obtain the high-level features of each multimodal image; inputting the enhanced features of each multimodal image into the gated value convolution layer to obtain the gated features of each multimodal image; performing element-wise product of the high-level features of each multimodal image with the gated features to obtain the screening features of each multimodal image; configuring weights for the screening features and enhanced features of each multimodal image according to the multi-scale features to obtain the extracted features of each multimodal image at each scale; fusing the extracted features of each multimodal image at each scale to obtain the multimodal image fusion features.

[0158] Furthermore, a module is established, which is specifically used to, when an abnormal event exists in the image corresponding to the transmission line to be inspected, respectively obtain the text features corresponding to the satellite navigation position of the UAV at the current moment and the sensor features used to characterize the position information of the UAV at the current moment; calculate the first similarity between the sensor features and the text features; calculate the second similarity between the multimodal image fusion features and the text features; and establish an association relationship function between the multimodal image fusion features and the sensor features used to characterize the position information of the UAV based on the first similarity and the second similarity.

[0159] Furthermore, the determination module is specifically used to input the corresponding image of the transmission line to be inspected, which is output by the feature pyramid network and marked with abnormal events, into the association function, determine the joint embedding vector of the multimodal image fusion features and the sensor features used to characterize the location information of the drone; perform nonlinear calculation on the joint embedding vector to generate the longitude and latitude position information of the abnormal event in the transmission line to be inspected.

[0160] Furthermore, the judgment module is specifically used to adjust the frame rate according to the sampling rate of each multimodal image, unify the frame rates of various multimodal images; perform preprocessing operations on the multimodal images after the frame rate is unified; and spatially align the various multimodal images after the preprocessing operation based on the internal parameters and external parameters of each camera.

[0161] An embodiment of the present invention provides an electronic device for implementing an electronic device. In this embodiment, the electronic device may be, but is not limited to, a personal computer (PC), a laptop computer, a monitoring device, a server, or other computer device with analysis and processing capabilities.

[0162] As an exemplary embodiment, see Figure 4 The electronic device 110 includes a communication interface 111, a processor 112, a memory 113 and a bus 114. The processor 112, the communication interface 111 and the memory 113 are connected via the bus 114. The memory 113 is used to store a computer program that supports the processor 112 to execute the method. The processor 112 is configured to execute the program stored in the memory 113.

[0163] The machine-readable storage medium mentioned in this article can be any electronic, magnetic, optical or other physical storage device that can contain or store information, such as executable instructions, data, etc. For example, the machine-readable storage medium can be: RAM (Radom Access Memory), volatile memory, non-volatile memory, flash memory, storage drive (such as hard disk drive), any type of storage disk (such as CD, DVD, etc.), or similar storage medium, or a combination thereof.

[0164] The non-volatile medium may be a non-volatile memory, a flash memory, a storage drive (such as a hard drive), any type of storage disk (such as a CD, DVD, etc.), or a similar non-volatile storage medium, or a combination thereof.

[0165] It can be understood that the specific operation methods of each functional module in this embodiment can refer to the detailed description of the corresponding steps in the above method embodiment, and will not be repeated here.

[0166] The computer-readable storage medium provided in the embodiment of the present invention stores a computer program. When the computer program code is executed, the method described in any of the above embodiments can be implemented. For specific implementation, please refer to the method embodiment, which will not be described in detail here.

[0167] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0168] In addition, in the description of the embodiments of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0169] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0170] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present invention, which are used to illustrate the technical solutions of the present invention rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the aforementioned embodiments, those of ordinary skill in the art should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the aforementioned embodiments within the technical scope disclosed by the present invention, or can easily conceive of changes, or make equivalent replacements for some of the technical features therein. Such modifications, changes or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the protection scope of the present invention.

Claims

1. A multi-modal fusion transmission line inspection method, characterized in that: include: At least one camera installed on the drone is used to collect a multimodal image of the power transmission line to be inspected at the current moment; The multimodal image is preprocessed and input into a feature pyramid network embedded in a multi-scale context adaptive joint learning network to obtain a multimodal image fusion feature, and under the action of the multimodal image fusion feature, it is determined whether there is an abnormal event in the image corresponding to the power transmission line to be inspected; If so, establishing a correlation function between the current position information of the UAV and the multimodal image fusion feature based on the similarity between the multimodal features representing the position information of the UAV; The image corresponding to the power transmission line to be inspected with abnormal event marks is input into the association function to determine the longitude and latitude position information of the abnormal event in the power transmission line to be inspected.

2. The method according to claim 1, characterized in that The step of inputting a feature pyramid network embedded in a multi-scale context-adaptive joint learning network to obtain a multimodal image fusion feature, and judging whether there is an abnormal event in the image corresponding to the power transmission line to be inspected under the action of the multimodal image fusion feature includes: Based on the multi-scale context adaptive joint learning network, a fusion feature of the multimodal image is obtained; Outputting an image corresponding to the power transmission line to be inspected at the current moment according to the fusion features of the multimodal image and the detection function of the feature pyramid network; Based on whether there is an abnormal event mark in the image corresponding to the power transmission line to be inspected at the current moment, it is determined whether there is an abnormal event in the image corresponding to the power transmission line to be inspected.

3. The method according to claim 2, characterized in that The multi-scale context adaptive joint learning network includes a feature enhancement branch, a feature screening branch and an adaptive multi-scale fusion branch; The feature enhancement branch is composed of a global average pooling layer, a first fully connected layer and a second fully connected layer; the feature screening branch is composed of a parallel eigenvalue convolution layer and a gated value convolution layer; and the adaptive multi-scale fusion branch is composed of a learnable weighted fusion module.

4. The method according to claim 3, characterized in that The step of obtaining fusion features of multimodal images based on the multi-scale context adaptive joint learning network includes: Extracting global features of the preprocessed multimodal image through the global average pooling layer; Determine a feature weight of each multimodal image based on the global feature, a first weight corresponding to the first fully connected layer, and a second weight corresponding to the second fully connected layer; Performing an element-wise product of the feature weight of each multimodal image and the cross-layer feature to obtain an enhanced feature of each multimodal image; wherein the cross-layer feature is a feature obtained by splicing each two adjacent layers in the multi-scale feature; and the multi-scale feature is a multi-layer image feature generated by inputting each pre-processed multimodal image into the feature pyramid network; Inputting the enhanced features of each multimodal image into the feature value convolution layer to obtain high-level features of each multimodal image; Inputting the enhanced features of each multimodal image into the gated value convolution layer to obtain the gated features of each multimodal image; Performing an element-wise product of the high-level features of each multimodal image and the gated features to obtain a screening feature of each multimodal image; According to the multi-scale features, weights are configured for the screening features and the enhanced features of each multimodal image to obtain extraction features of each multimodal image at each scale; The extracted features of each multimodal image at each scale are fused to obtain multimodal image fusion features.

5. The method according to claim 1, characterized in that The step of establishing a correlation relationship function between the current position information of the drone and the multimodal image fusion feature based on the similarity between the multimodal features representing the position information of the drone comprises: When an abnormal event occurs in the image corresponding to the power transmission line to be inspected, respectively obtaining text features corresponding to the satellite navigation position of the UAV at the current moment and sensor features for representing the position information of the UAV at the current moment; Calculating a first similarity between the sensor feature and the text feature; Calculating a second similarity between the multimodal image fusion feature and the text feature; Based on the first similarity and the second similarity, an association relationship function between the multimodal image fusion feature and the sensor feature used to characterize the position information of the drone is established.

6. The method according to claim 1, characterized in that The step of inputting the image corresponding to the power transmission line to be inspected with abnormal event marks into the association function to determine the longitude and latitude position information of the abnormal event in the power transmission line to be inspected includes: Inputting the image corresponding to the power transmission line to be inspected, which is output by the feature pyramid network and has abnormal event marks, into the association function to determine a joint embedding vector of the multimodal image fusion feature and the sensor feature used to characterize the location information of the drone; The joint embedding vector is subjected to nonlinear calculation to generate longitude and latitude position information of abnormal events in the power transmission line to be inspected.

7. The method according to claim 1, characterized in that The step of preprocessing the multimodal image comprises: Adjusting the frame rate according to the sampling rate of each multimodal image to unify the frame rates of the various multimodal images; Perform preprocessing operations on the multimodal images after the frame rate is unified; The various multimodal images after the preprocessing operation are spatially aligned based on the intrinsic parameters and extrinsic parameters of each camera.

8. A multi-modal fusion power transmission line inspection device, characterized in that: include: A collection module, which collects a multimodal image of the power transmission line to be inspected at the current moment through at least one camera installed on the drone; A judgment module preprocesses the multimodal image, inputs the preprocessed multimodal image into a feature pyramid network embedded in a multi-scale context adaptive joint learning network to obtain a multimodal image fusion feature, and judges whether there is an abnormal event in the image corresponding to the power transmission line to be inspected under the action of the multimodal image fusion feature; An establishment module, if any, establishes a correlation function between the location information of the drone at the current moment and the multimodal image fusion feature based on the similarity between the multimodal features representing the location information of the drone; The determination module inputs the image corresponding to the power transmission line to be inspected with the abnormal event mark into the association function to determine the longitude and latitude position information of the abnormal event in the power transmission line to be inspected.

9. An electronic device, characterized in that: The method comprises a memory, a processor, and a program stored in the memory and capable of being run on the processor, wherein the processor implements the method according to any one of claims 1 to 7 when executing the program.

10. A computer-readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed, the method according to any one of claims 1 to 7 is implemented.

Images