Power line image segmentation method and system based on depth features and semantic features, terminal equipment and storage medium

Abstract

The invention discloses an electric power line image segmentation method and system based on depth features and semantic features, terminal equipment and a storage medium, and belongs to the field of image segmentation. The method comprises the steps of extracting the depth features of an electric power line image by introducing a Transform encoder, capturing a global dependency relationship of the image by using a multi-head attention mechanism, and segmenting the depth features of the electric power line image; the defects of the CNN in complex background and fine target segmentation are overcome; high-frequency feature information of the image is extracted through a high-frequency information feature extraction module, the image is processed through a dimension conversion module, key feature information of a depth feature map is reserved, the semantic feature map can describe feature information of a power line more finely, and the image segmentation precision is improved; and finally, carrying out image segmentation according to the segmentation mask generated by the mask decoder. The problem that in the prior art, when an electric power line image is segmented, the image cannot be segmented more finely, so that the segmented image of the electric power line is inaccurate is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image segmentation, and in particular to a power line image segmentation method and system based on deep features and semantic features, a terminal device and a storage medium. BACKGROUND

[0002] With the development of information technology and unmanned aerial vehicle technology, power line inspection has gradually realized automation and intelligentization. In particular, in the process of monitoring and maintaining power equipment, image data has become a crucial reference. However, due to the influence of complex environment and variable lighting conditions on power line images, manual analysis and processing of these image data becomes increasingly difficult and inefficient.

[0003] Traditional power line image segmentation methods mainly rely on manual inspection and manual image annotation, which is not only inefficient, but also easily affected by subjective factors, making it difficult to ensure high precision. With the continuous progress of deep learning technology, automatic image processing technology based on deep learning methods such as convolutional neural networks (CNN) has begun to be widely applied, but in the application of power line images, there are still many challenges. However, due to the fact that power line images often contain complex backgrounds and power lines themselves are relatively thin and easily occluded, using automatic image processing technology based on convolutional neural networks (CNN) to segment power line images will result in inaccurate power line segmentation images due to the inability to segment images more finely. SUMMARY

[0004] The present application provides a power line image segmentation method and system based on deep features and semantic features, a terminal device and a storage medium, which can solve the problem of inaccurate power line segmentation images caused by the inability to segment images more finely when using automatic image processing technology based on convolutional neural networks (CNN) to segment power line images.

[0005] To solve the above technical problems, an embodiment of the present application provides a power line image segmentation method based on deep features and semantic features, comprising:

[0006] Obtaining a current power line image;

[0007] Inputting the current power line image into a trained deep feature extraction model, and performing deep feature extraction and feature fusion on the power line image through the Transformer encoder and feature fusion module in the trained deep feature image extraction model to obtain a deep feature map.

[0008] The deep feature map is input into a semantic feature extraction model, semantic feature extraction of the deep feature map is performed by an image semantic encoder in the semantic feature extraction model, a deep semantic feature map is generated, and a segmentation mask is generated according to the deep semantic feature map by a mask decoder; wherein the image semantic encoder is composed of a plurality of high-frequency information feature extraction modules and improved ViT blocks connected alternately; the improved ViT block comprises a dimension transformation module; the dimension transformation module comprises a dimension-reduced multilayer perceptron, a GELU activation function and a dimension-restored multilayer perceptron; the dimension transformation module is used to retain key feature information of the deep feature map and reduce redundant information;

[0009] The segmentation mask and the power line image are combined through a bitwise operation to generate a power line segmentation image.

[0010] Further, the model training of the deep feature extraction model comprises:

[0011] A teacher model and a student model of the deep feature extraction model are respectively constructed by using a DINOv2 self-supervised strategy;

[0012] The general image dataset is transformed and input into the teacher model, so that the teacher model performs deep feature extraction on each general image of different versions through a Transformer encoder, obtains deep feature maps of the same image of different versions, calculates a loss value according to the deep feature maps of the same image of different versions, optimizes parameters of the teacher model according to the loss value, until the loss value converges, and a pre-trained teacher model is obtained;

[0013] The collected power line images under different environments are respectively input into the pre-trained teacher model and the student model to be trained, so that the pre-trained teacher model and the student model to be trained perform deep feature extraction on each power line image through a Transformer encoder, and obtain a teacher deep feature map and a student deep feature map of each power line image;

[0014] A loss value is calculated according to the teacher deep feature map and the student deep feature map of each power line image, and parameters of the student model are optimized according to the loss value until the loss value converges, and a trained student model is obtained; wherein the trained student model is a trained deep feature extraction model.

[0015] Further, the deep feature image extraction model further comprises an image processing module; before the Transformer encoder and the feature fusion module of the trained deep feature image extraction model perform deep feature extraction and feature fusion on the power line image, further comprising:

[0016] The input image is divided into a plurality of image blocks by an image processing module, and the plurality of image blocks are processed by a full connection layer to obtain a plurality of low-dimensional image blocks.

[0017] Further, the Transformer encoder comprises a first normalization layer, a multi-head attention mechanism, a residual connection, a second normalization layer and a first multi-layer perception; the deep feature image extraction model is used for deep feature extraction and feature fusion of the power line image, comprising:

[0018] For each low-dimensional image block, a normalized image block is obtained by performing a normalization operation on the low-dimensional image block by the first normalization layer;

[0019] The attention values of different regions in the normalized image block are calculated by the multi-head attention mechanism to obtain attention feature information of the image block;

[0020] The low-dimensional image block and the feature information are connected in a residual connection to obtain an image block with attention feature information, and the image block with attention feature information is sequentially input into the second normalization layer and the first multi-layer perception for normalization operation and deep feature extraction to obtain a deep feature map of the image block;

[0021] The deep feature map of the image block and the image block with attention feature information are connected in a residual connection to obtain a final deep feature map of the image block.

[0022] Further, after the Transformer encoder of the deep feature image extraction model extracts the deep features of the power line image, the method further comprises:

[0023] The feature fusion module is used for fusing the final deep feature maps of the image blocks generated by the Transformer encoder to obtain a deep feature map of a complete image.

[0024] Further, the semantic feature extraction model comprises:

[0025] The high-frequency information feature of the deep feature map is extracted by the fast Fourier transform, high-pass filtering and inverse fast Fourier transform of the high-frequency information feature extraction module to obtain the high-frequency information feature of the deep feature map.

[0026] The semantic feature map is extracted from the deep feature map by the improved ViT block to obtain the semantic feature map.

[0027] The high-frequency information feature and the semantic feature image are combined to generate a deep semantic feature map.

[0028] Further, the improved ViT block further comprises a third normalization layer, a multi-head attention mechanism, a residual connection, a fourth normalization layer and a second multi-layer perceptron; the semantic feature image extraction on the deep feature map through the improved ViT block obtains a semantic feature map, comprising:

[0029] The deep feature map is normalized through the third normalization layer to obtain a normalized deep feature map;

[0030] The attention values of different regions in the normalized deep feature map are calculated through the multi-head attention mechanism to obtain a deep feature map with attention feature information;

[0031] The deep feature map with attention feature information is sequentially subjected to dimension reduction, nonlinear activation and dimension recovery through the dimension-reduced multi-layer perceptron, GELU activation function and dimension-restored multi-layer perceptron in the dimension transformation module to obtain a deep feature map with attention feature information after one-time dimension transformation;

[0032] The deep feature map and the deep feature map with attention feature information after dimension transformation are subjected to residual connection, and are input into the fourth normalization layer for normalization operation to obtain a normalized deep feature map after residual connection;

[0033] The normalized deep feature map after residual connection is subjected to second dimension transformation and semantic feature image extraction through the dimension transformation module and the second multi-layer perceptron respectively, and the deep feature map after second dimension transformation and the semantic feature image are subjected to residual connection and attention enhancement to obtain a final semantic feature map.

[0034] On the basis of the above method embodiment, the present application correspondingly provides a system embodiment;

[0035] An embodiment of the present application provides a power line image segmentation system based on deep features and semantic features, comprising: an image acquisition module, a deep feature map generation module, a segmentation mask generation module and an image segmentation module.

[0036] The image acquisition module is used for acquiring a current power line image.

[0037] The deep feature map generation module is used for inputting the current power line image into a trained deep feature extraction model, performing deep feature extraction and feature fusion on the power line image through a Transformer encoder and a feature fusion module in the trained deep feature image extraction model to obtain a deep feature map.

[0038] The segmentation mask generation module is configured to input the depth feature map into a semantic feature extraction model, perform semantic feature extraction on the depth feature map by an image semantic encoder in the semantic feature extraction model, generate a depth semantic feature map, and generate a segmentation mask according to the depth semantic feature map by a mask decoder; wherein the image semantic encoder is composed of a plurality of high-frequency information feature extraction modules and improved ViT blocks connected alternately; the improved ViT block comprises a dimension transformation module; the dimension transformation module comprises a dimension-reduced multilayer perceptron, a GELU activation function and a dimension-restored multilayer perceptron; the dimension transformation module is configured to retain key feature information of the depth feature map and reduce redundant information.

[0039] The image segmentation module is configured to combine the segmentation mask and the power line image by bit operation to generate a power line segmentation image.

[0040] On the basis of the above-mentioned method embodiment, another embodiment of the present application provides a terminal device, comprising a processor, a memory and a computer program stored in the memory and configured to be executed by the processor, when the processor executes the computer program, a kind of power line image segmentation method based on depth feature and semantic feature as described in the above embodiment is realized.

[0041] On the basis of the above-mentioned method embodiment, another embodiment of the present application provides a computer readable storage medium, the computer readable storage medium includes stored computer program, wherein, when the computer program runs, the device where the computer readable storage medium is located executes the power line image segmentation method based on depth feature and semantic feature described in the above embodiment.

[0042] Compared with the prior art, the embodiment of the present application has the following beneficial effects:

[0043] The present invention introduces a Transformer encoder to extract deep features of power line images, and uses its multi-head attention mechanism and multi-layer perceptron to effectively capture the global dependencies and high-level semantic information of the image, thereby solving the limitations of CNN in the segmentation of complex backgrounds and small targets; further, the high-frequency feature information of the image is extracted through the high-frequency information feature extraction module in the semantic feature extraction model, and the image is processed through the dimensionality transformation module of the improved ViT block, retaining the key feature information of the deep feature map and reducing redundant information, providing a high-precision image basis for subsequent image recognition and extraction, so that the semantic feature map can more finely describe the feature information of the power line, thereby improving the accuracy of image segmentation; finally, a segmentation mask is generated by combining the deep semantic features with the mask decoder, thereby solving the problem that the existing technology uses the automated image processing technology based on the convolutional neural network (CNN) to segment the power line image, which leads to inaccurate power line segmentation images due to the inability to segment the image more finely. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A flowchart of a method for segmenting power line images based on depth features and semantic features provided by an embodiment of the present invention;

[0045] Figure 2 A structural diagram of a deep feature extraction model provided by an embodiment of the present invention;

[0046] Figure 3 A structural diagram of a Transformer encoder provided in an embodiment of the present invention;

[0047] Figure 4 A structural diagram of a semantic feature extraction model provided by an embodiment of the present invention;

[0048] Figure 5 A structural diagram of a high-frequency information feature extraction module provided in an embodiment of the present invention;

[0049] Figure 6 A structural diagram of an improved ViT block provided in an embodiment of the present invention;

[0050] Figure 7 This is a structural diagram of a power line image segmentation system based on depth features and semantic features provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0051] With reference to the accompanying drawings, the technical solutions in the embodiments of the present application will be clearly and completely described below, obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0052] In the description of the present application, it should be understood that the terms "first", "second", "third" and "fourth" are only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features.

[0053] Embodiment 1:

[0054] Referring to Figure 1 In order to solve the problem that the power line image segmentation is inaccurate due to the inability to finely segment the image when the prior art uses the automatic image processing technology based on the convolutional neural network (CNN) to segment the power line image, the method at least includes the following steps:

[0055] Step S1: acquiring a current power line image;

[0056] In this embodiment, the power line image can be collected from a drone or other high-resolution camera equipment.

[0057] In this embodiment, after acquiring the current power line image, it further includes uniformly formatting (such as JPEG, PNG) the collected image, and performing standardization processing (such as adjusting the image size, removing noise, cropping the region of interest, etc.) on the image.

[0058] Step S2: inputting the current power line image into the trained deep feature extraction model, and performing deep feature extraction and feature fusion on the power line image through the Transformer encoder and the feature fusion module in the trained deep feature image extraction model to obtain a deep feature map;

[0059] In this embodiment, referring to Figure 2 The deep feature extraction model structure diagram provided by the embodiment of the present application; the deep feature image extraction model further includes an image processing module; before the deep feature extraction and feature fusion of the power line image through the Transformer encoder and the feature fusion module in the trained deep feature image extraction model, it further includes:

[0060] The image processing module is configured to divide an input image into a plurality of image patches, and perform dimension reduction processing on the plurality of image patches through a fully connected layer to obtain a plurality of low-dimensional image patches.

[0061] In the embodiment, refer to Figure 2 The power line image is first input into an image processing module of a deep feature image extraction model, and the image processing module is configured to divide an input image into a plurality of image patches, and perform dimension reduction processing on the plurality of image patches through a fully connected layer to realize division of the image patches and linear projection of the image patches, so as to obtain a plurality of low-dimensional image patches. Then, the plurality of low-dimensional image patches are input into a Transformer encoder to perform deep feature extraction, so as to obtain a deep feature map of each image patch. Finally, the deep feature map of each image patch is input into a feature fusion module, and the feature fusion module is configured to perform feature fusion on the final deep feature map of each image patch generated by the Transformer encoder, so as to obtain a deep feature map of a complete image.

[0062] In the embodiment, refer to Figure 3 The Transformer encoder provided in the embodiment of the application has the structure shown in the figure, and the deep feature extraction and feature fusion of the power line image are performed by the Transformer encoder and the feature fusion module in the trained deep feature image extraction model, including:

[0063] For each low-dimensional image patch, a first normalization layer is used to perform normalization operation to obtain a normalized image patch.

[0064] The attention value of different regions in the normalized image patch is calculated through a multi-head attention mechanism to obtain attention feature information of the image patch.

[0065] In the embodiment, the calculation formula of the attention value is In the formula, Q represents information to be queried at present, in the processing of the image patch, is a feature vector after linear transformation of the current image patch; K represents a key of the information to be queried, in the processing of the image patch, is a feature vector after linear transformation of all image patches (including the current patch); V represents a value of the information corresponding to the key vector, in the processing of the image patch, is a feature vector after another linear transformation of all image patches; d k The dimension of the key vector K. The multi-head attention mechanism calculates the query, key and value of different regions, each attention head can capture important spatial information in the region it focuses on, so that the model can establish a connection between different image regions.

[0066] Performing a residual connection between the low-dimensional image patch and the feature information to obtain an image patch with attention feature information, and inputting the image patch with attention feature information into the second normalization layer and the first multi-layer perceptron for normalization and deep feature extraction to obtain a deep feature map of the image patch;

[0067] The depth feature map of the image patch is residually connected with the image patch with attention feature information to obtain the final depth feature map of the image patch.

[0068] In this embodiment, the model training of the deep feature extraction model includes:

[0069] Using the DINOv2 self-supervision strategy, we construct the teacher model and student model of the deep feature extraction model respectively;

[0070] The general image dataset is transformed and input into the teacher model, so that the teacher model extracts deep features from each general image of different transformed versions through the Transformer encoder, obtains deep feature maps of different transformed versions of the same image, and calculates the loss value based on the deep feature maps of different transformed versions of the same image. The parameters of the teacher model are optimized according to the loss value until the loss value converges, thereby obtaining a pre-trained teacher model.

[0071] The collected power line images under different environments are input into the pre-trained teacher model and the student model to be trained respectively, so that the pre-trained teacher model and the student model to be trained perform deep feature extraction on each power line image through the Transformer encoder to obtain the teacher depth feature map and the student depth feature map of each power line image;

[0072] According to the teacher depth feature map and the student depth feature map of each power line image, the loss value is calculated, and the parameters of the student model are optimized according to the loss value until the loss value converges to obtain a trained student model; wherein the trained student model is a trained deep feature extraction model.

[0073] Step S3: Refer to Figure 4A structural diagram of the semantic feature extraction model provided by the embodiment of the present application is shown in the figure. In the figure, the Chinese interpretation of Image Encoder is image semantic encoder, the Chinese interpretation of Adapter-Feature is high-frequency information feature extraction module, the Chinese interpretation of Modiffied Transformer Block is improved ViT block, and the Chinese interpretation of Mask Decoder is mask decoder. The depth feature map is input into the semantic feature extraction model, the semantic feature extraction of the depth feature map is performed by the image semantic encoder in the semantic feature extraction model, the depth semantic feature map is generated, and the segmentation mask is generated according to the depth semantic feature map by the mask decoder. The image semantic encoder is composed of a plurality of high-frequency information feature extraction modules and improved ViT blocks which are alternately connected. The improved ViT block includes a dimension transformation module. The dimension transformation module includes a dimension-reduced multilayer perceptron, a GELU activation function and a dimension-restored multilayer perceptron. The dimension transformation module is used to retain the key feature information of the depth feature map and reduce the redundant information.

[0074] In the embodiment, the semantic feature extraction of the depth feature map by the image semantic encoder in the semantic feature extraction model to generate the depth semantic feature map includes:

[0075] The high-frequency information feature of the depth feature map is obtained by performing high-frequency information feature extraction on the depth feature map by the fast Fourier transform, high-pass filtering and inverse fast Fourier transform of the high-frequency information feature extraction module.

[0076] In the embodiment, the high-frequency information feature of the depth feature map is obtained by performing high-frequency information feature extraction on the depth feature map by the fast Fourier transform, high-pass filtering and inverse fast Fourier transform of the high-frequency information feature extraction module. Figure 5 A structural diagram of the high-frequency information feature extraction module provided by the embodiment of the present application is shown in the figure. In the figure, the Chinese interpretation of FFT Transform is fast Fourier transform, the Chinese interpretation of High-pass Filter is high-pass filtering, and the Chinese interpretation of IFFT Transform is inverse fast Fourier transform. After the high-frequency information feature of the depth feature map is obtained by performing high-frequency information feature extraction on the depth feature map by the fast Fourier transform, high-pass filtering and inverse fast Fourier transform of the high-frequency information feature extraction module, the dimension adjustment and feature enhancement operation processing of the high-frequency information feature of the depth feature map are further performed by two layers of MLP (MLP_tune and MLP_up) and GULE. The MLP is multilayer perceptron, and MLP_tune and MLP_up are used for dimension adjustment to retain the key high-frequency information feature.

[0077] The semantic feature map is obtained by performing semantic feature image extraction on the depth feature map by the improved ViT block.

[0078] The depth semantic feature map is generated by combining the high-frequency information feature and the semantic feature image.

[0079] In the embodiment, referring to Figure 6 , a structure diagram of the improved ViT block provided by the embodiment of the application; in the diagram, the Chinese interpretation of Positional Encoding is position encoding, the Chinese interpretation of the first Layer Normalization is the third normalization layer, the Chinese interpretation of Multi-head Attention is the multi-head attention mechanism, the Chinese interpretation of Adapter-Scale is the dimension transformation module, the Chinese interpretation of Dimension Reduction is dimension reduction, the Chinese interpretation of GELU is the GELU activation function, the Chinese interpretation of Dimension Restoration is dimension restoration, the Chinese interpretation of the second Layer Normalization is the fourth normalization layer, and MLP is the second multi-layer perceptron; the improved ViT block further includes a third normalization layer, a multi-head attention mechanism, a residual connection, a fourth normalization layer, and a second multi-layer perceptron, and the Chinese interpretation of Attention-augmented features is attention enhancement; the improved ViT block is used to perform semantic feature image extraction on the deep feature map to obtain a semantic feature map, including:

[0080] performing position encoding on the deep feature map to obtain a position-encoded deep feature map;

[0081] performing normalization operation on the deep feature map through the third normalization layer to obtain a normalized deep feature map;

[0082] calculating attention values of different regions in the normalized deep feature map through the multi-head attention mechanism to obtain a deep feature map with attention feature information;

[0083] performing dimension reduction, nonlinear activation, and dimension restoration on the deep feature map with attention feature information in sequence through the dimension-reduced multi-layer perceptron, the GELU activation function, and the dimension-restored multi-layer perceptron in the dimension transformation module to obtain a deep feature map with attention feature information after first dimension transformation;

[0084] performing residual connection on the deep feature map and the deep feature map with attention feature information after dimension transformation, and inputting the residual connection deep feature map into the fourth normalization layer to perform normalization operation to obtain a normalized residual connection deep feature map;

[0085] performing second dimension transformation and semantic feature image extraction on the normalized residual connection deep feature map through the dimension transformation module and the second multi-layer perceptron respectively, and performing residual connection and attention enhancement on the second dimension transformation deep feature map and the semantic feature image to obtain a final semantic feature map.

[0086] Step S4: combining the segmentation mask and the power line image through bitwise operation to generate a power line segmentation image.

[0087] In this embodiment, a bitwise multiplication operation is performed to retain the portion of the mask marked as the power line, while the background portion is set to 0 or another default color. This method can extract the precise area of ​​the power line from the original image, forming a segmented image containing the power line.

[0088] Example 2:

[0089] Reference Figure 7 , which is a structural diagram of a power line image segmentation system based on depth features and semantic features provided by an embodiment of the present invention; to address the problem that conventional automated image processing technology based on convolutional neural networks (CNNs) cannot segment power line images more finely, resulting in inaccurate power line segmentation images, the system includes at least the following modules: an image acquisition module, a depth feature map generation module, a segmentation mask generation module, and an image segmentation module;

[0090] The image acquisition module is used to acquire the current power line image;

[0091] The deep feature map generation module is used to input the current power line image into the trained deep feature extraction model, and perform deep feature extraction on the power line image through the Transformer encoder in the trained deep feature image extraction model to obtain a deep feature map; wherein the Transformer encoder includes a first normalization layer, a multi-head attention mechanism, a residual connection, a second normalization layer and a first multi-layer perceptron;

[0092] The segmentation mask generation module is used to input the depth feature map into the semantic feature extraction model, perform semantic feature extraction on the depth feature map through the image semantic encoder in the semantic feature extraction model to generate a depth semantic feature map, and generate a segmentation mask based on the depth semantic feature map through the mask decoder; wherein, the image semantic encoder is composed of a plurality of high-frequency information feature extraction modules and an improved ViT block that are alternately connected; the improved ViT block includes a third normalization layer, a multi-head attention mechanism, a dimensionality transformation module, a residual connection, a fourth normalization layer and a second multi-layer perceptron; the dimensionality transformation module includes a multi-layer perceptron for dimensionality reduction, a GELU activation function and a multi-layer perceptron for dimensionality recovery;

[0093] The image segmentation module is used to combine the segmentation mask and the power line image through bitwise operation to generate a power line segmentation image.

[0094] Another embodiment of the present application also provides a terminal device, comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, and the processor implements the power line image segmentation method based on depth features and semantic features as described in the above embodiments when executing the computer program. The terminal device can be a desktop computer, a notebook computer, a palm computer, a cloud server, and other computing devices. The terminal device can include, but is not limited to, a processor and a memory.

[0095] The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor and the like, and the processor is a control center of the terminal device, and connects all parts of the terminal device through various interfaces and lines.

[0096] The memory can be used to store the computer program, and the processor realizes various functions of the terminal device by running or executing the computer program stored in the memory and calling data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required by a function, and the like; and the data storage area can store data created according to the use of the mobile phone and the like. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, for example, a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state memory devices.

[0097] Another embodiment of the present application provides a computer readable storage medium, comprising a stored computer program, wherein the computer readable storage medium controls the device where the computer readable storage medium is located to execute the power line image segmentation method based on depth features and semantic features as described in the above embodiments when the computer program runs.

[0098] The storage medium is a computer readable storage medium, and the computer program is stored in the computer readable storage medium. When the computer program is executed by a processor, steps of each method embodiment described above can be implemented. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate forms, etc. The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc.

[0099] The above specific embodiments further illustrate the purpose, technical solutions and beneficial effects of the present application. It should be understood that the above description is only for specific embodiments of the present application and is not intended to limit the protection scope of the present application. It is particularly pointed out that any modification, equivalent replacement, improvement, etc. made by those skilled in the art within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A power line image segmentation method based on depth features and semantic features, characterized in that: include: Get the current power line image; Input the current power line image into the trained deep feature extraction model, and perform deep feature extraction and feature fusion on the power line image through the Transformer encoder and feature fusion module in the trained deep feature image extraction model to obtain a deep feature map; The depth feature map is input into the semantic feature extraction model, and the image semantic encoder in the semantic feature extraction model performs semantic feature extraction on the depth feature map to generate a depth semantic feature map, and a mask decoder generates a segmentation mask based on the depth semantic feature map; wherein the image semantic encoder is composed of a plurality of high-frequency information feature extraction modules and an improved ViT block that are alternately connected; the improved ViT block includes a dimensionality transformation module; the dimensionality transformation module includes a multi-layer perceptron for dimensionality reduction, a GELU activation function, and a multi-layer perceptron for dimensionality recovery; the dimensionality transformation module is used to retain the key feature information of the depth feature map and reduce redundant information; The segmentation mask and the power line image are combined through bitwise operation to generate the power line segmentation image.

2. The power line image segmentation method based on depth features and semantic features according to claim 1, characterized in that: The model training of the deep feature extraction model includes: Using the DINOv2 self-supervision strategy, we construct the teacher model and student model of the deep feature extraction model respectively; The general image dataset is transformed and input into the teacher model, so that the teacher model extracts deep features from each general image of different transformed versions through the Transformer encoder, obtains deep feature maps of different transformed versions of the same image, and calculates the loss value based on the deep feature maps of different transformed versions of the same image. The parameters of the teacher model are optimized according to the loss value until the loss value converges, thereby obtaining a pre-trained teacher model. The collected power line images under different environments are input into the pre-trained teacher model and the student model to be trained respectively, so that the pre-trained teacher model and the student model to be trained perform deep feature extraction on each power line image through the Transformer encoder to obtain the teacher depth feature map and the student depth feature map of each power line image; According to the teacher depth feature map and the student depth feature map of each power line image, the loss value is calculated, and the parameters of the student model are optimized according to the loss value until the loss value converges to obtain a trained student model; wherein the trained student model is a trained deep feature extraction model.

3. The power line image segmentation method based on depth features and semantic features according to claim 2, characterized in that: The depth feature image extraction model also includes an image processing module; Before performing deep feature extraction and feature fusion on the power line image through the Transformer encoder and feature fusion module in the trained deep feature image extraction model, the method further includes: The input image is divided into several small image blocks through the image processing module, and the several image blocks are subjected to dimensionality reduction processing through the fully connected layer to obtain several low-dimensional image blocks.

4. The method for power line image segmentation based on depth features and semantic features according to claim 3, characterized in that: The Transformer encoder includes a first normalization layer, a multi-head attention mechanism, a residual connection, a second normalization layer and a first multi-layer perceptron; The deep feature extraction and feature fusion of the power line image are performed using the Transformer encoder and feature fusion module in the trained deep feature image extraction model, including: For each low-dimensional image patch, normalization is performed through the first normalization layer to obtain a normalized image patch; The attention values ​​of different regions in the normalized image patch are calculated through the multi-head attention mechanism to obtain the attention feature information of the image patch; Performing a residual connection between the low-dimensional image patch and the feature information to obtain an image patch with attention feature information, and inputting the image patch with attention feature information into the second normalization layer and the first multi-layer perceptron for normalization and deep feature extraction to obtain a deep feature map of the image patch; The depth feature map of the image patch is residually connected with the image patch with attention feature information to obtain the final depth feature map of the image patch.

5. The method for power line image segmentation based on depth features and semantic features according to claim 4, characterized in that: The feature fusion module is used to fuse the final depth feature maps of each image block generated by the Transformer encoder to obtain the depth feature map of the complete image.

6. The method for power line image segmentation based on depth features and semantic features according to claim 5, characterized in that: The method of extracting semantic features from the depth feature map by using the image semantic encoder in the semantic feature extraction model to generate the depth semantic feature map includes: The high-frequency information feature extraction module extracts high-frequency information features from the depth feature map through fast Fourier transform, high-pass filtering and inverse fast Fourier transform to obtain high-frequency information features of the depth feature map; The semantic feature image is extracted from the depth feature map through the improved ViT block to obtain the semantic feature map; Combine high-frequency information features and semantic feature images to generate deep semantic feature maps.

7. The method for power line image segmentation based on depth features and semantic features according to claim 6, characterized in that: The improved ViT block also includes a third normalization layer, a multi-head attention mechanism, a residual connection, a fourth normalization layer and a second multi-layer perceptron; The semantic feature image is extracted from the depth feature map by the improved ViT block to obtain the semantic feature map, including: The depth feature map is normalized by the third normalization layer to obtain a normalized depth feature map; The attention values ​​of different regions in the normalized deep feature map are calculated through the multi-head attention mechanism to obtain a deep feature map with attention feature information; The depth feature map with attention feature information is sequentially reduced in dimension, nonlinearly activated, and restored through the multi-layer perceptron with dimension reduction, GELU activation function, and multi-layer perceptron with dimension recovery in the dimension transformation module to obtain a depth feature map with attention feature information after one dimension transformation. Perform a residual connection between the depth feature map and the depth feature map with attention feature information after dimension transformation, and input it into the fourth normalization layer for normalization to obtain the normalized residual connection depth feature map; The normalized residual-connected depth feature map is subjected to secondary dimensional transformation and semantic feature image extraction through the dimensional transformation module and the second multi-layer perceptron respectively, and the secondary dimensional transformed depth feature map and semantic feature image are subjected to residual connection and attention enhancement to obtain the final semantic feature map.

8. A power line image segmentation system based on depth features and semantic features, characterized in that: include: Image acquisition module, depth feature map generation module, segmentation mask generation module and image segmentation module; The image acquisition module is used to acquire the current power line image; The depth feature map generation module is used to input the current power line image into the trained depth feature extraction model, and perform depth feature extraction and feature fusion on the power line image through the Transformer encoder and feature fusion module in the trained depth feature image extraction model to obtain a depth feature map; The segmentation mask generation module is used to input the depth feature map into the semantic feature extraction model, perform semantic feature extraction on the depth feature map through the image semantic encoder in the semantic feature extraction model to generate a depth semantic feature map, and generate a segmentation mask based on the depth semantic feature map through the mask decoder; wherein, the image semantic encoder is composed of a plurality of high-frequency information feature extraction modules and an improved ViT block that are alternately connected; the improved ViT block includes a dimensionality transformation module; the dimensionality transformation module includes a multi-layer perceptron for dimensionality reduction, a GELU activation function, and a multi-layer perceptron for dimensionality recovery; the dimensionality transformation module is used to retain the key feature information of the depth feature map and reduce redundant information; The image segmentation module is used to combine the segmentation mask and the power line image through bitwise operation to generate a power line segmentation image.

9. A terminal device comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, it implements a power line image segmentation method based on depth features and semantic features as described in any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium includes a stored computer program, wherein when the computer program is running, the device where the storage medium is located is controlled to execute the power line image segmentation method based on depth features and semantic features as described in any one of claims 1 to 7.

Images