Equipment icing image segmentation and model training method, electronic equipment and storage medium

Through feature extraction, optimization and fusion network, the problem of inaccurate ice-covered segmentation in the prior art is solved, the segmentation accuracy of ice-covered areas is improved, and it is suitable for the detection of complex backgrounds and fine ice-covered features.

CN119068198AActive Publication Date: 2024-12-03GUANGDONG POWER GRID CO LTD +1
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202411209248.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-30
Publication Date
2024-12-03
Estimated Expiration
2044-08-30

AI Technical Summary

Technical Problem

When the prior art deals with complex backgrounds and fine ice-covering features, the ice-covering segmentation method based on edge detection is difficult to accurately capture the details of the ice-covering edge, resulting in inaccurate segmentation results, affecting the normal operation and service life of the equipment.

Method used

Feature extraction network, feature expression optimization network and feature fusion network are used to obtain shallow and deep features, optimize feature expression, and obtain segmentation results of ice-covered image areas through feature fusion network to improve segmentation accuracy.

Benefits of technology

It significantly improves the segmentation accuracy of the ice-covered area, can more accurately capture the fine-grained features of the ice-covered image area, enhances the processing ability of complex backgrounds, and improves the detection effect of the ice-covered edge.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068198A_ABST
    Figure CN119068198A_ABST
Patent Text Reader

Abstract

The embodiment of the invention discloses an equipment icing image segmentation and model training method, electronic equipment and a storage medium, and the method comprises the steps: inputting a to-be-segmented equipment image into a pre-trained equipment icing image segmentation model which comprises a feature extraction network, a feature expression optimization network and a feature fusion network; obtaining shallow layer features and deep layer features corresponding to the icing image area in the to-be-segmented equipment image through a feature extraction network; optimizing feature expressions of the shallow-layer features and the deep-layer features through a feature expression optimization network to correspondingly obtain optimized shallow-layer features and optimized deep-layer features; and through a feature fusion network, based on the optimized shallow-layer features and the optimized deep-layer features, obtaining a segmentation result of an iced image region in the to-be-segmented equipment image. According to the embodiment of the invention, the accuracy of detecting and segmenting the icing area image from the to-be-segmented image can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a device ice-covered image segmentation and model training method, electronic equipment and storage medium. Background Art

[0002] In cold and humid climates, ice will form on the surface of outdoor equipment, such as conductors and related facilities on transmission lines. When ice forms on the surface of the equipment, it will affect the normal operation of the equipment, reduce the service life of the equipment, and even cause damage to the equipment. For example, when ice forms on the surface of the transmission line, the resistance of the conductor increases, which will reduce the efficiency of power transmission; the weight of the conductor increases, which increases the sag of the conductor and intensifies the vibration, which will increase the risk of line breakage and even cause damage to the line tower structure. Therefore, it is very important to detect the ice on the surface of the equipment in time and deal with the ice in time. The prior art detects whether there is ice on the surface of the corresponding equipment by attempting to segment and obtain the image area of ​​the iced device from the device image corresponding to the device that may have ice. The prior art can segment the iced image area based on the edge detection ice segmentation method, but the segmentation method based on edge detection is often difficult to achieve ideal results when dealing with complex image backgrounds and small ice features. At the same time, it is difficult to accurately capture the details of the ice edge, resulting in inaccurate segmentation results. Summary of the invention

[0003] The embodiments of the present invention provide a device ice-covered image segmentation and model training method, an electronic device and a storage medium, which can improve the accuracy of detecting and segmenting an ice-covered area image from an image to be segmented.

[0004] In a first aspect, an embodiment of the present invention provides a method for segmenting an ice-covered device image, comprising:

[0005] Inputting the device image to be segmented into a pre-trained device ice-covered image segmentation model, wherein the device ice-covered image segmentation model includes: a feature extraction network, a feature expression optimization network and a feature fusion network;

[0006] Acquire shallow features and deep features corresponding to the ice-covered image area in the device image to be segmented through the feature extraction network;

[0007] The feature expressions of the shallow features and the deep features are optimized by the feature expression optimization network, and the optimized shallow features and the optimized deep features are obtained accordingly; and

[0008] Through the feature fusion network, based on the optimized shallow features and the optimized deep features, a segmentation result of the ice-covered image area in the image of the device to be segmented is obtained.

[0009] In a second aspect, an embodiment of the present invention provides a method for training an equipment ice image segmentation model, comprising:

[0010] Inputting the sample device image into the device ice-covered image segmentation model to be trained, wherein the device ice-covered image segmentation model includes: a feature extraction network, a feature expression optimization network and a feature fusion network;

[0011] Acquire shallow features and deep features corresponding to the ice-covered image area in the sample device image through the feature extraction network;

[0012] The feature expressions of the shallow features and the deep features are optimized by the feature expression optimization network, and the optimized shallow features and the optimized deep features are obtained accordingly;

[0013] Obtaining, through the feature fusion network, a segmentation result of the ice-covered image area in the sample device image based on the optimized shallow features and the optimized deep features; and

[0014] The function value of the training loss function is determined based on the segmentation result of the ice-covered image area, and the network parameters in the device ice-covered image segmentation model are adjusted according to the function value of the training loss function.

[0015] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements a device ice-covered image segmentation method or a device ice-covered image segmentation model training method as described in any one of the embodiments of the present invention.

[0016] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a device ice-covered image segmentation method or a device ice-covered image segmentation model training method as described in any one of the embodiments of the present invention.

[0017] The embodiments of the present invention provide a device ice-covered image segmentation and model training method, electronic device and storage medium. First, shallow features and deep features corresponding to the ice-covered area are extracted through a feature extraction network. Then, the feature expression of the shallow features and deep features is further optimized through a feature expression optimization network, so as to enhance the feature expression capability of the shallow features and deep features for the ice-covered image area and capture finer-grained features of the ice-covered image area. Finally, based on the optimized shallow features and deep features with enhanced expression capability and finer-grained ice-covered image features, a feature fusion network is used to obtain the segmentation result of the ice-covered image area in the device image to be segmented through a fusion operation, so as to improve the accuracy of detecting and segmenting the ice-covered area image from the image to be segmented. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solution of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 It is a flowchart of a method for segmenting an image of ice-covered equipment provided by an embodiment of the present invention;

[0020] Figure 2 It is a structural schematic diagram of a feature extraction network in the device ice-covered image segmentation method provided by an embodiment of the present invention;

[0021] Figure 3 It is a structural schematic diagram of a multi-scale convolutional attention module in the device ice-covered image segmentation method provided by an embodiment of the present invention;

[0022] Figure 4 It is a structural schematic diagram of a deep feature optimization module in the device ice-covered image segmentation method provided by an embodiment of the present invention;

[0023] Figure 5 is another flowchart of the device ice-covered image segmentation method provided by an embodiment of the present invention;

[0024] Figure 6 It is a structural schematic diagram of a feature fusion network in the device ice image segmentation method provided by an embodiment of the present invention;

[0025] Figure 7 It is a flow chart of a device ice image segmentation model training method provided by an embodiment of the present invention;

[0026] Figure 8 It is a structural schematic diagram of a device ice-covered image segmentation apparatus provided in an embodiment of the present invention;

[0027] Fig. 9 It is a structural schematic diagram of a device ice image segmentation model training device provided by an embodiment of the present invention;

[0028] Fig.10 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] Figure 1 A flowchart of a device ice-covered image segmentation method provided in an embodiment of the present invention is provided. The method can be performed by a device ice-covered image segmentation apparatus provided in an embodiment of the present invention, and the apparatus can be implemented in software and / or hardware. In a specific embodiment, the apparatus can be integrated in an electronic device, such as a computer, a server, etc. The following embodiments will be described by taking the apparatus integrated in an electronic device as an example. Reference Figure 1 , the method may specifically include the following steps:

[0032] Step 101, input the device image to be segmented into a pre-trained device ice-covered image segmentation model, which includes: a feature extraction network, a feature expression optimization network and a feature fusion network. This step can help obtain the shallow features and deep features corresponding to the ice-covered image area in the device image to be segmented through the feature extraction network.

[0033] Specifically, the above-mentioned equipment image to be segmented may be an image obtained by photographing a power transmission line equipment that may have surface ice covered by an outdoor device.

[0034] Specifically, the device image to be segmented may also be an image obtained by pre-processing an original device image obtained by photographing a power transmission line device that may have surface ice covered using an external device.

[0035] Optionally, before step 101, an original device image corresponding to the device image to be segmented is obtained, and defogging is performed on the original device image to obtain the device image to be segmented.

[0036] Optionally, the above-mentioned process of dehazing the original device image includes: obtaining the original device image corresponding to the device image to be segmented; converting the original device image into a grayscale image, and performing bilateral filtering on the grayscale image to obtain a bilateral filtered image; determining the transmittance of each pixel in the original device image based on the bilateral filtered image; and obtaining the dehazed image corresponding to the original device image based on the transmittance of each pixel to obtain the above-mentioned device image to be segmented.

[0037] Specifically, the process of determining the transmittance of the original device image based on the bilateral filtered image can be performed based on the following formula:

[0038]

[0039] Where T(x) represents the transmittance of each pixel in the original device image, I(x) represents the pixel value of each pixel in the bilateral filtered image, K represents the global atmospheric light estimation value in the original device image, and ω is a constant used to adjust the estimation accuracy and sensitivity of the transmittance.

[0040] Specifically, the process of obtaining the defogging image corresponding to the original device image based on the transmittance of each pixel to obtain the device image to be segmented can be performed based on the following formula:

[0041]

[0042] Where M(x) is the pixel value of each pixel in the dehazed image, m(x) is the pixel value of each pixel in the original device image, T(x) represents the transmittance of each pixel in the original device image, and T min Indicates the minimum threshold value of transmittance.

[0043] Specifically, dehazing the native device image can avoid the interference of foggy images and improve the segmentation accuracy.

[0044] Step 102, obtain shallow features and deep features corresponding to the ice-covered image area in the device image to be segmented through a feature extraction network. This step can be helpful in optimizing the feature expression of shallow features and deep features, obtaining optimized shallow features and optimized deep features, and then obtaining a high-precision segmentation result of the ice-covered image area based on the optimized shallow features and optimized deep features.

[0045] Specifically, the features in each embodiment of the present invention may be feature maps. For example, the shallow features may be shallow feature maps, and the deep features may be deep feature maps.

[0046] Optionally, the feature extraction network includes: a first convolution normalization module, a second convolution normalization module, a third convolution normalization module, a fourth convolution normalization module, a fifth convolution normalization module, a first downsampling convolution layer, a second downsampling convolution layer, a third downsampling convolution layer and a fourth downsampling convolution layer.

[0047] Optionally, the above-mentioned shallow features and deep features corresponding to the ice-covered image area in the image of the device to be segmented are obtained through the feature extraction network, including: performing convolution normalization and activation function processing on the image of the device to be segmented through a first convolution normalization module to obtain a first convolution normalization output feature; obtaining a first down-sampled convolution input feature based on the first convolution normalization output feature, and performing down-sampled convolution processing on the first down-sampled convolution input feature through a first down-sampled convolution layer to obtain a first down-sampled convolution output feature; performing convolution normalization and activation function processing on the first down-sampled convolution output feature through a second convolution normalization module to obtain a second convolution normalization output feature; obtaining a second down-sampled convolution input feature based on the second convolution normalization output feature, and performing down-sampled convolution processing on the second down-sampled convolution input feature through a second down-sampled convolution layer to obtain a shallow feature; The shallow features are processed by the third convolution normalization module through convolution normalization and activation function to obtain the third convolution normalization output features; based on the third convolution normalization output features, the third down-sampled convolution input features are obtained, and the third down-sampled convolution input features are processed by down-sampled convolution through the third down-sampled convolution layer to obtain the third down-sampled convolution output features; the third down-sampled convolution output features are processed by convolution normalization and activation function to obtain the fourth convolution normalization output features; based on the fourth convolution normalization output features, the fourth down-sampled convolution input features are obtained, and the fourth down-sampled convolution input features are processed by down-sampled convolution through the fourth down-sampled convolution layer to obtain the fourth down-sampled convolution output features; the fourth down-sampled convolution output features are processed by convolution normalization and activation function to obtain the deep features.

[0048] In an optional specific example, the feature extraction network further includes: a convolutional block attention module (CBAM) arranged after the first convolution normalization module and before the first downsampling convolution layer, and after the second convolution normalization module and before the first downsampling convolution layer, which can be specifically as follows: Figure 2 shown.

[0049] Optionally, the aforementioned process of obtaining the first down-sampled convolution input feature based on the first convolution standardized output feature includes: performing attention adjustment on the first convolution standardized output feature through a first CBAM module to obtain the first down-sampled convolution input feature.

[0050] Optionally, the aforementioned process of obtaining a second down-sampled convolution input feature based on the second convolution standardized output feature includes: performing attention adjustment on the second convolution standardized output feature through a second CBAM module to obtain a second down-sampled convolution input feature.

[0051] Specifically, the size of the convolution kernel of each of the above downsampling convolution layers can be set to 3×3, and the step size can be set to 2.

[0052] In an optional specific example, a device image to be segmented with a size of 512×512 pixels and 3 channels is input into the device ice image segmentation model; a shallow feature map F with a size of 128×128 pixels and 36 channels output by the second downsampling convolutional layer is obtained. 1 ; Get the deep feature map F with a size of 32×32 pixels and 128 channels output by the fifth convolution normalization module 2 .

[0053] Specifically, convolution normalization and activation function processing are performed through the convolution normalization module, which can better extract the shallow and deep features of the image to be segmented; downsampling convolution processing is performed through the downsampling convolution layer to avoid the model being too complicated and reduce the amount of calculation; by setting the convolution block attention module to adjust the attention of the shallow features when extracting shallow features, the model's attention to different parts of the image is enhanced, which effectively enhances the model's ability to process image details, so it can handle complex backgrounds of images and capture the details of ice edges.

[0054] Step 103, optimize the feature expression of shallow features and deep features through the feature expression optimization network, and obtain optimized shallow features and optimized deep features accordingly. This step optimizes the feature expression of shallow features and deep features, enhances the feature expression ability of shallow features and deep features for ice-covered image regions, captures more fine-grained features of ice-covered image regions, and can facilitate obtaining a more accurate ice-covered region image segmentation result based on the fusion of optimized shallow features and deep features.

[0055] Optionally, the feature expression optimization network includes: a multi-scale convolutional attention module (Multi-scale Convolutional Attention, MSCA), the multi-scale convolutional attention module includes: a depth convolution unit, a first strip convolution branch, a second strip convolution branch, a third strip convolution branch and a point-by-point convolution unit, which can be specifically as follows: Figure 3 shown.

[0056] Optionally, the process of optimizing the feature expression of shallow features through the feature expression optimization network to obtain optimized shallow features includes: optimizing the shallow features through multiple scale convolutional attention modules to obtain optimized shallow features.

[0057] Optionally, the process of optimizing the shallow features by the scale convolution attention module to obtain the optimized shallow features includes:

[0058] The shallow features are deep convolved by the deep convolution unit to obtain the output features of the deep convolution unit; the output features of the deep convolution unit are deep strip convolved by the first strip convolution branch to obtain the output features of the first strip convolution branch; the output features of the deep convolution unit and the output features of the first strip convolution branch are fused to obtain the input features of the second strip convolution branch, and the input features of the second strip convolution branch are deep strip convolved by the second strip convolution branch to obtain the output features of the second strip convolution branch; the output features of the deep convolution unit and the output features of the second strip convolution branch are fused. The input feature of the third strip convolution branch is obtained, and the third strip convolution branch is used to perform a deep strip convolution on the input feature of the third strip convolution branch to obtain the output feature of the third strip convolution branch; the deep convolution unit output feature, the first strip convolution branch output feature, the second strip convolution branch output feature and the third strip convolution branch output feature are fused to obtain the point-by-point convolution unit input feature, and the point-by-point convolution unit is used to perform a point-by-point convolution on the point-by-point convolution unit input feature to obtain the attention feature; and the shallow feature is adjusted based on the attention feature to obtain the optimized shallow feature.

[0059] Optionally, the above process of fusing the output features of the deep convolution unit and the output features of the first strip convolution branch includes: adding and fusing the output features of the deep convolution unit and the output features of the first strip convolution branch pixel by pixel.

[0060] Optionally, the above process of fusing the output features of the deep convolution unit and the output features of the second strip convolution branch includes: adding and fusing the output features of the second strip convolution branch and the output features of the first strip convolution branch pixel by pixel.

[0061] Optionally, the process of adjusting the attention of the shallow feature based on the attention feature includes: multiplying the attention feature and the shallow feature pixel by pixel.

[0062] Specifically, the above-mentioned depth convolution unit may include a depth convolution layer with a convolution kernel size of 3×3, and may also include multiple depth convolution layers.

[0063] Optionally, the first strip convolution branch includes a first deep strip convolution layer and a second deep strip convolution layer, and the process of performing deep strip convolution on the output features of the deep convolution unit through the first strip convolution branch to obtain the output features of the first strip convolution branch includes: performing strip convolution on the output features of the deep convolution unit through the first deep strip convolution layer and the second deep strip convolution layer in sequence to obtain the output features of the first strip convolution branch.

[0064] Specifically, the convolution kernels of the first depthwise strip convolution layer and the second depthwise strip convolution layer can be set to 1×5 and 5×1 strip convolution kernels respectively.

[0065] Optionally, the second strip convolution branch includes a third deep strip convolution layer and a fourth deep strip convolution layer, and the process of performing deep strip convolution on the second strip convolution branch input features through the second strip convolution branch to obtain the second strip convolution branch output features includes: sequentially performing strip convolution on the second strip convolution branch input features through the third deep strip convolution layer and the fourth deep strip convolution layer to obtain the second strip convolution branch output features.

[0066] Specifically, the convolution kernels of the third depth strip convolution layer and the fourth depth strip convolution layer can be set to 1×9 and 9×1 strip convolution kernels respectively.

[0067] Optionally, the third strip convolution branch includes a fifth deep strip convolution layer and a sixth deep strip convolution layer, and the process of performing deep strip convolution on the third strip convolution branch input features through the third strip convolution branch to obtain the third strip convolution branch output features includes: sequentially performing strip convolution on the third strip convolution branch input features through the fifth deep strip convolution layer and the sixth deep strip convolution layer to obtain the third strip convolution branch output features.

[0068] Specifically, the convolution kernels of the fifth depthwise strip convolution layer and the sixth depthwise strip convolution layer may be set to 1×13 and 13×1 strip convolution kernels respectively.

[0069] In an optional embodiment, if Figure 3 As shown, use and Correspondingly, the depth convolution unit, the first strip convolution branch, the second strip convolution branch, and the third strip convolution branch output are represented by F 3 Indicates the optimization of shallow features, then: and F 3 They can be expressed as:

[0070]

[0071] Optionally, the process of optimizing the feature expression of the deep feature through the feature expression optimization network to obtain the optimized deep feature includes: optimizing the deep feature through the scale convolution attention module to obtain the optimized deep feature.

[0072] Specifically, the process of optimizing deep features through the above-mentioned scale convolutional attention module can be the same as the process of optimizing shallow features through the above-mentioned scale convolutional attention module to obtain optimized shallow features, which will not be repeated here.

[0073] Specifically, the embodiment of the present invention optimizes the shallow features and deep features corresponding to the ice-covered image area through the MSCA module. First, the local feature information of the corresponding features is aggregated through the deep convolution unit of the MSCA module, and then the multi-scale contextual information of the corresponding features is captured through the three strip convolution branches of the MSCA module. The connection between different channel features is further obtained through the point-by-point convolution unit of the MSCA module, thereby enhancing the feature expression ability of the corresponding feature image. In addition, when the embodiment of the present application uses a single strip convolution branch to capture the contextual information of the corresponding feature, the output features of the previous strip convolution branch are integrated into the input features of the next strip convolution branch, which can further capture more contextual information, thereby further enhancing the feature expression ability of the corresponding feature image.

[0074] Step 104, through the feature fusion network, based on the optimization of shallow features and deep features, obtain the segmentation result of the ice-covered image area in the device image to be segmented. This step can obtain the segmentation result of the ice-covered image area in the device image to be segmented through a fusion operation based on the optimized shallow features and deep features of the ice-covered image features with enhanced expression ability and finer granularity obtained through steps 101 to 103, which can improve the accuracy of detecting and segmenting the ice-covered area image from the image to be segmented.

[0075] Specifically, the segmentation result of the ice-covered image area can be an image that distinguishes the ice-covered image area in the image of the device to be segmented when the ice-covered image area exists, or it can be an image without the relevant display of the ice-covered image area when the ice-covered image area does not exist.

[0076] Optionally, the process of obtaining the segmentation result of the ice-covered image area in the image of the device to be segmented by the feature fusion network based on optimizing shallow features and optimizing deep features includes:

[0077] The number of channels of the optimized shallow features and the optimized deep features is adjusted to the preset number of channels, and the sizes of the optimized shallow features and the optimized deep features are adjusted to the preset sizes to obtain the optimized shallow features to be fused and the optimized deep features to be fused; the optimized shallow features to be fused and the optimized deep features to be fused are fused to obtain the optimized fused features; the optimized fused features are subjected to standard convolution processing to obtain the fused features to be segmented; and the number of channels and the size of the fused features to be segmented are adjusted to be the same as the number of channels and the size of the device image to be segmented, to obtain the segmentation result of the ice-covered image area.

[0078] Optionally, the preset number of channels is greater than the number of channels of the device image to be segmented, and the preset size is smaller than the size of the device image to be segmented.

[0079] Specifically, the number of channels for optimizing shallow features and optimizing deep features can be adjusted to a preset number of channels based on a point-by-point convolution operation, or the number of channels can be adjusted based on other methods in the prior art, such as through grouped convolution and fully connected layers.

[0080] Specifically, the sizes of the optimized shallow features and the optimized deep features can be adjusted to the preset sizes based on the transposed convolution operation, and the sizes of the optimized shallow features and the optimized deep features can be adjusted to the preset sizes through the interpolation operation.

[0081] Specifically, the shallow features to be fused and optimized and the deep features to be fused and optimized can be fused based on a pixel-by-pixel addition fusion operation, or can be fused through a channel superposition fusion operation.

[0082] The embodiment of the present invention adjusts the number and size of channels of optimized shallow features and optimized deep features to a transitional number and size of channels for fusion, and then performs standard convolution processing on the fused features. The fused features after the standard convolution processing are further adjusted to the target number of channels and target size. After the fusion of optimized shallow features and optimized deep features, further feature extraction can be performed, and the features of the ice-covered image area can be better extracted, which is beneficial to further improve the accuracy of subsequent segmentation results.

[0083] Optionally, the above-mentioned process of obtaining the segmentation result of the ice-covered image area in the image of the device to be segmented through the feature fusion network based on the optimized shallow features and the optimized deep features includes: adjusting the channel dimensions of the optimized shallow features and the optimized deep features to be the same as the channel dimensions and sizes of the device image to be segmented; and fusing the optimized shallow features and the optimized deep features that have been dimensionally adjusted and size adjusted to obtain the fused features to be segmented.

[0084] The following further introduces a method for segmenting ice-covered images of equipment provided by another embodiment of the present invention.

[0085] Optionally, the feature expression optimization network includes a deep feature optimization module, and the deep feature optimization module includes: a first dilated convolution branch, a second dilated convolution branch, a first strip pooling branch, and a second strip pooling branch, which can be specifically as follows: Figure 4 shown.

[0086] Optional, such as Figure 5 As shown Figure 1 Step 103 in the method may include the following steps:

[0087] Step 1031, through the first dilated convolution branch, perform deep dilated separable convolution on the deep features to obtain the first dilated convolution branch output features.

[0088] Optionally, the first atrous convolution branch includes a first depthwise atrous separable convolution layer and a second depthwise atrous separable convolution layer.

[0089] Optionally, the expansion rate of the first depth-atrous separable convolution layer is 4, and the size of the convolution kernel is 3×3; the expansion rate of the second depth-atrous separable convolution layer is 6, and the size of the convolution kernel is 3×3.

[0090] Optionally, the process of performing a deep atrous separable convolution on the deep features through the first atrous convolution branch to obtain the output features of the first atrous convolution branch includes: performing a deep atrous separable convolution on the deep features through a first deep atrous separable convolution layer and a second deep atrous separable convolution layer connected in series, in sequence.

[0091] Step 1032, through the second dilated convolution branch, perform deep dilated separable convolution on the deep features to obtain the second dilated convolution branch output features.

[0092] Optionally, the second atrous convolution branch includes: a third depth-atrous separable convolution layer and a fourth depth-atrous separable convolution layer.

[0093] Optionally, the expansion rate of the third depth-atrous separable convolution layer is 8, and the size of the convolution kernel is 3×3; the expansion rate of the fourth depth-atrous separable convolution layer is 10, and the size of the convolution kernel is 3×3.

[0094] The process of performing a deep atrous separable convolution on the deep features through the second atrous convolution branch to obtain the output features of the second atrous convolution branch includes: performing a deep atrous separable convolution on the deep features through a third deep atrous separable convolution layer and a fourth deep atrous separable convolution layer connected in series.

[0095] Step 1033, through the first strip pooling branch, average pooling is performed on the deep features based on the horizontal strip pooling kernel, and pixel expansion is performed on the corresponding average pooling result based on the deep features to obtain the first strip pooling branch output.

[0096] Specifically, the size of the horizontal strip pooling kernel can be set according to the size of the deep feature map. For example, for a deep feature map F with a size of 32×32 pixels, 2 , the size of the above horizontal strip pooling kernel can be set to 1×32.

[0097] Optionally, the above process of performing pixel expansion on the corresponding average pooling result based on the deep feature includes:

[0098] Based on the pixel size of the deep features, the average pooling result of the first strip pooling branch is expanded up and down by the same pixels to obtain the above-mentioned first strip pooling branch output.

[0099] Step 1034, through the second strip pooling branch, average pooling is performed on the deep features based on the vertical strip pooling kernel, and pixel expansion is performed on the corresponding average pooling result based on the deep features to obtain the second strip pooling branch output.

[0100] Optionally, the above-mentioned process of pixel expansion of the corresponding average pooling result based on the deep feature includes: based on the pixel size of the deep feature, expanding the average pooling result of the second strip pooling branch by the same pixels left and right to obtain the above-mentioned first strip pooling branch output.

[0101] Specifically, the sizes of the horizontal strip pooling kernel and the vertical strip pooling kernel can be set according to the size of the deep feature map. For example, for a deep feature map F with a size of 32×32 pixels, 2 , the sizes of the above horizontal strip pooling core and vertical strip pooling core can be set to 1×32 and 32×1 respectively.

[0102] Step 1035, the output features of the first dilated convolution branch, the output features of the second dilated convolution branch, the output of the first strip pooling branch, and the output of the second strip pooling branch are fused to obtain a multi-branch output fusion feature.

[0103] Optionally, the above process of fusing the output features of the first atrous convolution branch, the output features of the second atrous convolution branch, the output of the first strip pooling branch, and the output of the second strip pooling branch includes: stacking the output features of the first atrous convolution branch, the output features of the second atrous convolution branch, the output of the first strip pooling branch, and the output of the second strip pooling branch in the channel dimension.

[0104] Specifically, the first dilated convolution branch output feature, the second dilated convolution branch output feature, the first strip pooling branch output and the second strip pooling branch output can also be fused in other ways, such as based on pixel-by-pixel addition.

[0105] Step 1036, performing channel dimension reduction on the multi-branch output fusion features based on point-by-point convolution operations to obtain optimized deep features.

[0106] In an optional specific embodiment, the deep feature F with a size of 32×32 pixels and 128 channels is 2 Input to the deep feature optimization module, F 2 After the deep hole separable convolution processing of the first hole convolution branch, the output feature of the first hole convolution branch is obtained F 2 After the deep hole separable convolution processing of the first hole convolution branch, the output feature of the second hole convolution branch is obtained F 2 After the average pooling and expansion of the first strip pooling branch, the output of the first strip pooling branch is obtained. F 2 After the average pooling and expansion of the first strip pooling branch, the output of the first strip pooling branch is obtained. and The size of is 32×32 pixels and the number of channels is 128. and The channel dimension is stacked, and then the channel dimension is reduced by point-by-point convolution to obtain the optimized deep feature F with a minimum of 32×32 pixels and 48 channels. 4 .

[0107] In an optional specific embodiment of the present invention, Figure 6 As shown, the process of obtaining the segmentation result of the ice-covered image area in the image of the device to be segmented by the feature fusion network based on optimizing shallow features and optimizing deep features includes:

[0108] The optimized shallow feature F with a size of 128×128 pixels and 36 channels is 3 , perform point-by-point convolution to obtain the shallow features to be fused and optimized with a size of 128×128 pixels and 12 channels The optimized deep feature F with a size of 32×32 pixels and 48 channels is 4 , up-sample to obtain the optimized deep features to be fused with a size of 128×128 pixels and 12 channels Will and The optimized fusion features are obtained by adding and fusing them pixel by pixel. The optimized fusion features are subjected to a 3×3 standard convolution to obtain the fusion features to be segmented with a size of 128×128 pixels and 12 channels. Last pair Based on the pixel size and the number of channels of the device image to be segmented, upsampling is performed to obtain the segmentation result of the ice-covered image area with a size of 512×512 pixels and a number of channels of 3.

[0109] The embodiment of the present invention optimizes the feature expression of deep features based on multiple feature optimization branches, and at the same time, by setting a hole convolution branch, it improves the perception of global features, improves the gradient propagation problem of the network, and reduces the risk of gradient disappearance or explosion; according to the physical shape of the transmission line and the distribution characteristics of ice, a strip pooling branch is set to enable more context information to be added, and more ice features to be extracted, thereby improving the segmentation accuracy of the ice-covered area image. In addition, the embodiment of the present invention further sets two hole convolution layers with different expansion rates in series in each hole convolution branch, which can effectively avoid grid artifacts.

[0110] Figure 7 A flowchart of a device ice-covered image segmentation model training method provided in an embodiment of the present invention is provided. The method can be executed by a device ice-covered image segmentation model training device provided in an embodiment of the present invention, and the device can be implemented in software and / or hardware. In a specific embodiment, the device can be integrated in an electronic device, such as a computer, a server, etc. The following embodiments will be described by taking the device integrated in an electronic device as an example. Reference Figure 7 , the method may specifically include the following steps:

[0111] Step 701, inputting a sample device image into a device ice-covered image segmentation model to be trained, wherein the device ice-covered image segmentation model includes: a feature extraction network, a feature expression optimization network and a feature fusion network;

[0112] Step 702: Obtain shallow features and deep features corresponding to the ice-covered image area in the sample device image through a feature extraction network.

[0113] Specifically, the specific steps included in the above step 702 are the same as the specific steps included in step 102 in the above-mentioned device ice-covered image segmentation method.

[0114] Step 703, optimizing the feature expressions of shallow features and deep features through a feature expression optimization network, and obtaining optimized shallow features and optimized deep features accordingly.

[0115] Optionally, the feature extraction network includes: a first convolution normalization module, a second convolution normalization module, a third convolution normalization module, a fourth convolution normalization module, a fifth convolution normalization module, a first downsampling convolution layer, a second downsampling convolution layer, a third downsampling convolution layer and a fourth downsampling convolution layer.

[0116] In an optional specific example, the feature extraction network further includes: a convolutional block attention module (CBAM) arranged after the first convolution normalization module and before the first downsampling convolution layer, and after the second convolution normalization module and before the first downsampling convolution layer, which can be specifically as follows: Figure 2 shown.

[0117] Specifically, the specific steps included in the above step 703 are the same as the specific steps included in step 103 in the above-mentioned device ice-covered image segmentation method.

[0118] Step 704 , obtaining the segmentation result of the ice-covered image area in the sample device image through the feature fusion network based on optimizing shallow features and optimizing deep features.

[0119] Specifically, the specific steps included in the above step 704 are the same as the specific steps included in step 104 in the above-mentioned device ice-covered image segmentation method.

[0120] Optionally, the feature expression optimization network includes: a multi-scale convolutional attention module (Multi-scale Convolutional Attention, MSCA), the multi-scale convolutional attention module includes: a depth convolution unit, a first strip convolution branch, a second strip convolution branch, a third strip convolution branch and a point-by-point convolution unit, which can be specifically as follows: Figure 3 shown.

[0121] Optionally, the feature expression optimization network includes a deep feature optimization module, and the deep feature optimization module includes: a first dilated convolution branch, a second dilated convolution branch, a first strip pooling branch, and a second strip pooling branch, which can be specifically as follows: Figure 4 shown.

[0122] Optionally, the first atrous convolution branch includes a first depthwise atrous separable convolution layer and a second depthwise atrous separable convolution layer.

[0123] Optionally, the second atrous convolution branch includes: a third depth-atrous separable convolution layer and a fourth depth-atrous separable convolution layer.

[0124] Step 705: determine the function value of the training loss function based on the segmentation result of the ice-covered image area, and adjust the network parameters in the device ice-covered image segmentation model according to the function value of the training loss function.

[0125] Optionally, the device ice-covered image segmentation model training method provided in the above-mentioned embodiment of the present invention also includes: before inputting the sample device image into the device ice-covered image segmentation model to be trained, obtaining multiple ice-covered device images, and obtaining sample ice-covered device images corresponding to each ice-covered device image based on a geometric transformation operation.

[0126] Specifically, the plurality of ice-covering equipment images include ice-covered area images.

[0127] Optionally, the plurality of ice-covered equipment images may be obtained by defogging the originally captured ice-covered equipment images.

[0128] Optionally, the process of determining the function value of the training loss function based on the segmentation result of the ice-covered image area includes: determining the function value of the training loss function based on the segmentation result of the ice-covered image area and a labeled ice-covered area image corresponding to the sample device image.

[0129] Specifically, the annotated ice-covered area image is obtained by annotating the ice-covered image area in the corresponding sample ice-covered equipment image.

[0130] Specifically, the dataset can be divided into a training dataset, a validation dataset and a test dataset in a ratio of 8:1:1, and then the training dataset, the validation dataset and the test dataset are used to train the ice image segmentation model of the device to be trained.

[0131] Specifically, the process of adjusting the network parameters in the device ice image segmentation model according to the function value of the training loss function includes: firstly randomly initializing the network weight parameters and setting the network hyperparameters, such as the learning rate size, regularization parameters, etc., using the cross entropy as the loss function, and then selecting the stochastic gradient descent optimization algorithm to update the network model parameters. During the training process, the validation data set is periodically used to evaluate the performance of the network model, so as to timely adjust the network model structure and hyperparameters so that the network can converge quickly. When the function value of the loss function of the device ice image segmentation model tends to converge, the training ends, and then a trained device ice image segmentation model is obtained.

[0132] The device ice-covered image segmentation model trained by the embodiment of the present invention can enhance the feature expression ability of shallow features and deep features for the ice-covered image region when segmenting the ice-covered area in the device ice-covered image, and capture finer-grained ice-covered image region features; then based on the optimized shallow features and deep features with enhanced expression ability and finer-grained ice-covered image features, the segmentation result of the ice-covered image region in the device image to be segmented is obtained through a fusion operation, which can improve the accuracy of detecting and segmenting the ice-covered area image from the image to be segmented.

[0133] Figure 8 is a structural diagram of a device for segmenting ice-covered images provided in an embodiment of the present invention, and the device is suitable for executing a method for segmenting ice-covered images provided in an embodiment of the present invention. Figure 8 As shown, the device may specifically include:

[0134] The input module 801 to be segmented is used to input the device image to be segmented into a pre-trained device ice-covered image segmentation model, which includes: a feature extraction network, a feature expression optimization network and a feature fusion network. It is helpful to obtain the shallow features and deep features corresponding to the ice-covered image area in the device image to be segmented through the feature extraction network.

[0135] The feature extraction module 802 to be segmented is used to obtain the shallow features and deep features corresponding to the ice-covered image area in the device image to be segmented through the feature extraction network. It can be helpful to optimize the feature expression of the shallow features and the deep features to obtain the optimized shallow features and the optimized deep features, and then obtain the high-precision segmentation result of the ice-covered image area based on the optimized shallow features and the optimized deep features.

[0136] Optionally, the feature extraction network includes: a first convolution normalization module, a second convolution normalization module, a third convolution normalization module, a fourth convolution normalization module, a fifth convolution normalization module, a first downsampling convolution layer, a second downsampling convolution layer, a third downsampling convolution layer and a fourth downsampling convolution layer.

[0137] Optionally, the above-mentioned feature extraction module 802 to be segmented can be specifically used to perform convolution normalization and activation function processing on the image of the device to be segmented through a first convolution normalization module to obtain a first convolution normalization output feature; obtain a first down-sampled convolution input feature based on the first convolution normalization output feature, and perform down-sampled convolution processing on the first down-sampled convolution input feature through a first down-sampled convolution layer to obtain a first down-sampled convolution output feature; perform convolution normalization and activation function processing on the first down-sampled convolution output feature through a second convolution normalization module to obtain a second convolution normalization output feature; obtain a second down-sampled convolution input feature based on the second convolution normalization output feature, and perform down-sampled convolution processing on the second down-sampled convolution input feature through a second down-sampled convolution layer to obtain a shallow feature; perform a third convolution normalization module on the shallow The first layer features are convolution-normalized and activated by adding a convolution function to obtain third convolution-normalized output features; based on the third convolution-normalized output features, the third down-sampled convolution input features are obtained, and the third down-sampled convolution input features are down-sampled by the third down-sampled convolution layer to obtain third down-sampled convolution output features; the third down-sampled convolution output features are convolution-normalized and activated by adding a convolution function to obtain fourth convolution-normalized output features; based on the fourth convolution-normalized output features, the fourth down-sampled convolution input features are obtained, and the fourth down-sampled convolution input features are down-sampled by the fourth down-sampled convolution layer to obtain fourth down-sampled convolution output features; and the fourth down-sampled convolution output features are convolution-normalized and activated by adding a convolution function to obtain deep features.

[0138] The feature optimization module 803 to be segmented is used to optimize the feature expression of shallow features and deep features through the feature expression optimization network, and obtain the optimized shallow features and optimized deep features accordingly. It can optimize the feature expression of shallow features and deep features, enhance the feature expression ability of shallow features and deep features for ice-covered image areas, capture more fine-grained features of ice-covered image areas, and facilitate obtaining a more accurate ice-covered area image segmentation result based on the fusion of optimized shallow features and deep features.

[0139] Optionally, the feature expression optimization network includes a multi-scale convolutional attention module, and the multi-scale convolutional attention module includes: a depth convolution unit, a first strip convolution branch, a second strip convolution branch, a third strip convolution branch and a point-by-point convolution unit.

[0140] Optionally, the above-mentioned feature optimization module 803 to be segmented can be specifically used to perform deep convolution on shallow features through a deep convolution unit to obtain deep convolution unit output features; perform deep strip convolution on the deep convolution unit output features through a first strip convolution branch to obtain the first strip convolution branch output features; fuse the deep convolution unit output features and the first strip convolution branch output features to obtain the second strip convolution branch input features, and perform deep strip convolution on the second strip convolution branch input features through the second strip convolution branch to obtain the second strip convolution branch output features; fuse the deep convolution unit output features and the second strip convolution branch output features. The output features of the strip convolution branch are fused to obtain the input features of the third strip convolution branch, and the input features of the third strip convolution branch are subjected to depthwise strip convolution through the third strip convolution branch to obtain the output features of the third strip convolution branch; the output features of the depthwise convolution unit, the output features of the first strip convolution branch, the output features of the second strip convolution branch and the output features of the third strip convolution branch are fused to obtain the input features of the point-by-point convolution unit, and the input features of the point-by-point convolution unit are subjected to point-by-point convolution through the point-by-point convolution unit to obtain the attention features; and the shallow features are adjusted based on the attention features to obtain the optimized shallow features.

[0141] Optionally, the feature expression optimization network includes a deep feature optimization module, and the deep feature optimization module includes: a first dilated convolution branch, a second dilated convolution branch, a first strip pooling branch, and a second strip pooling branch.

[0142] Optionally, the above-mentioned feature optimization module 803 to be segmented can be specifically used to perform deep hole separable convolution on the deep features through the first hole convolution branch to obtain the first hole convolution branch output features; perform deep hole separable convolution on the deep features through the second hole convolution branch to obtain the second hole convolution branch output features; perform average pooling on the deep features based on the horizontal strip pooling kernel through the first strip pooling branch, and perform pixel expansion on the corresponding average pooling results based on the deep features to obtain the first strip pooling branch output; perform average pooling on the deep features based on the vertical strip pooling kernel through the second strip pooling branch, and perform pixel expansion on the corresponding average pooling results based on the deep features to obtain the second strip pooling branch output; fuse the first hole convolution branch output features, the second hole convolution branch output features, the first strip pooling branch output and the second strip pooling branch output to obtain multi-branch output fusion features; and perform channel dimensionality reduction on the multi-branch output fusion features based on the point-by-point convolution operation to obtain optimized deep features.

[0143] Optionally, the first atrous convolution branch includes a first depthwise atrous separable convolution layer and a second depthwise atrous separable convolution layer.

[0144] Optionally, the second atrous convolution branch includes a third depthwise atrous separable convolution layer and a fourth depthwise atrous separable convolution layer.

[0145] Optionally, the to-be-segmented feature optimization module 803 can be specifically used to perform a deep-dilute separable convolution on the deep features in sequence through a first deep-dilute separable convolution layer and a second deep-dilute separable convolution layer connected in series.

[0146] Optionally, the above-mentioned feature optimization module 803 to be segmented can be specifically used to perform deep-hole separable convolution on deep features in sequence through a third deep-hole separable convolution layer and a fourth deep-hole separable convolution layer connected in series.

[0147] The feature fusion module 804 to be segmented is used to obtain the segmentation result of the ice-covered image area in the device image to be segmented based on the optimization of shallow features and deep features through the feature fusion network. It can be combined with modules 801 to 803 to first extract the shallow features and deep features corresponding to the ice-covered area through the feature extraction network, and then further optimize the feature expression of the shallow features and deep features through the feature expression optimization network, enhance the feature expression ability of the shallow features and deep features for the ice-covered image area, and capture finer-grained features of the ice-covered image area; finally, the feature fusion network is based on the optimized shallow features and deep features with enhanced expression ability and finer granularity of the ice-covered image features, and the segmentation result of the ice-covered image area in the device image to be segmented is obtained through fusion operation, which can improve the accuracy of detecting and segmenting the ice-covered area image from the image to be segmented.

[0148] Optionally, the above-mentioned feature fusion module 804 to be segmented can be specifically used to adjust the number of channels of the optimized shallow features and the optimized deep features to a preset number of channels, and adjust the sizes of the optimized shallow features and the optimized deep features to a preset size to obtain the optimized shallow features to be fused and the optimized deep features to be fused; fuse the optimized shallow features to be fused and the optimized deep features to be fused to obtain the optimized fused features; perform standard convolution processing on the optimized fused features to obtain the fused features to be segmented; and adjust the number of channels and the size of the fused features to be segmented to be the same as the number of channels and the size of the device image to be segmented, to obtain the segmentation result of the ice-covered image area.

[0149] Fig. 9 1 is a structural diagram of a device ice image segmentation model training apparatus provided in an embodiment of the present invention, and the apparatus is suitable for executing an ice image segmentation model training method provided in an embodiment of the present invention. Fig. 9 As shown, the device may specifically include:

[0150] The sample input module 901 is used to input the sample device image into the device ice-covered image segmentation model to be trained. The device ice-covered image segmentation model includes: a feature extraction network, a feature expression optimization network and a feature fusion network.

[0151] The sample feature extraction module 902 is used to obtain shallow features and deep features corresponding to the ice-covered image area in the sample device image through a feature extraction network.

[0152] The sample feature optimization module 903 is used to optimize the feature expressions of shallow features and deep features through a feature expression optimization network, and obtain optimized shallow features and optimized deep features accordingly;

[0153] The sample feature fusion module 904 is used to obtain the segmentation result of the ice-covered image area in the sample device image based on the optimization of shallow features and the optimization of deep features through a feature fusion network.

[0154] The parameter adjustment module 905 is used to determine the function value of the training loss function based on the segmentation result of the ice-covered image area, and adjust the network parameters in the device ice-covered image segmentation model according to the function value of the training loss function.

[0155] The device ice-covered image segmentation model trained by the embodiment of the present invention can enhance the feature expression ability of shallow features and deep features for the ice-covered image region when segmenting the ice-covered area in the device ice-covered image, and capture finer-grained ice-covered image region features; then based on the optimized shallow features and deep features with enhanced expression ability and finer-grained ice-covered image features, the segmentation result of the ice-covered image region in the device image to be segmented is obtained through a fusion operation, which can improve the accuracy of detecting and segmenting the ice-covered area image from the image to be segmented.

[0156] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example for illustration. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. The specific working process of the functional modules described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0157] An embodiment of the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the device ice-covered image segmentation method or the device ice-covered image segmentation model training method provided in any of the above embodiments is implemented.

[0158] An embodiment of the present invention further provides a computer-readable medium having a computer program stored thereon, and when the program is executed by a processor, the device ice-covered image segmentation method or the device ice-covered image segmentation model training method provided in any of the above embodiments is implemented.

[0159] The embodiment of the present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the device ice-covered image segmentation method or device ice-covered image segmentation model training method as described in any one of the embodiments of the present invention.

[0160] Reference below Fig.10 , which shows a schematic diagram of the structure of a computer system 1000 of an electronic device suitable for implementing an embodiment of the present invention. Fig.10 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0161] like Fig.10 As shown, the computer system 1000 includes a central processing unit (CPU) 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage part 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the system 1000 are also stored. The CPU 1001, the ROM 1002, and the RAM 1003 are connected to each other through a bus 1004. An input feature / output (I / O) interface 1005 is also connected to the bus 1004.

[0162] The following components are connected to the I / O interface 1005: an input feature section 1006 including a keyboard, a mouse, etc.; an output section 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN card, a modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1010 as needed, so that a computer program read therefrom is installed into the storage section 1008 as needed.

[0163] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1009, and / or installed from the removable medium 1011. When the computer program is executed by the central processing unit (CPU) 1001, the above-mentioned functions defined in the system of the present invention are executed.

[0164] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0165] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0166] The modules and / or units involved in the embodiments of the present invention may be implemented by software or hardware. The modules and / or units described may also be arranged in a processor, for example, may be described as: a processor including a to-be-segmented input module, a to-be-segmented feature extraction module, a to-be-segmented feature optimization module, and a to-be-segmented feature fusion module; or, may be described as: a processor including a sample input module, a sample feature extraction module, a sample feature optimization module, and a parameter adjustment module. The names of these modules do not, in some cases, constitute limitations on the modules themselves.

[0167] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently and not be assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device implements: inputting the device image to be segmented into a pre-trained device ice-covered image segmentation model, the device ice-covered image segmentation model includes: a feature extraction network, a feature expression optimization network and a feature fusion network; obtaining shallow features and deep features corresponding to the ice-covered image area in the device image to be segmented through the feature extraction network; optimizing the feature expression of shallow features and deep features through the feature expression optimization network, and correspondingly obtaining optimized shallow features and optimized deep features; and obtaining the segmentation result of the ice-covered image area in the device image to be segmented based on the optimized shallow features and optimized deep features through the feature fusion network.

[0168] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for segmenting equipment ice-covered images, characterized in that: include: Inputting the device image to be segmented into a pre-trained device ice-covered image segmentation model, wherein the device ice-covered image segmentation model includes: a feature extraction network, a feature expression optimization network and a feature fusion network; Acquire shallow features and deep features corresponding to the ice-covered image area in the device image to be segmented through the feature extraction network; The feature expressions of the shallow features and the deep features are optimized by the feature expression optimization network, and the optimized shallow features and the optimized deep features are obtained accordingly; and Through the feature fusion network, based on the optimized shallow features and the optimized deep features, a segmentation result of the ice-covered image area in the image of the device to be segmented is obtained.

2. The device ice image segmentation method according to claim 1, characterized in that: The feature expression optimization network includes a multi-scale convolution attention module, and the multi-scale convolution attention module includes: a depth convolution unit, a first strip convolution branch, a second strip convolution branch, a third strip convolution branch and a point-by-point convolution unit; The step of optimizing the feature expression of the shallow feature by the feature expression optimization network to obtain the optimized shallow feature includes: Performing deep convolution on the shallow features through the deep convolution unit to obtain deep convolution unit output features; Performing a deep strip convolution on the output feature of the deep convolution unit through the first strip convolution branch to obtain a first strip convolution branch output feature; The output feature of the deep convolution unit and the output feature of the first strip convolution branch are merged to obtain a second strip convolution branch input feature, and the second strip convolution branch is used to perform a deep strip convolution on the input feature of the second strip convolution branch to obtain a second strip convolution branch output feature; The output feature of the depth convolution unit and the output feature of the second strip convolution branch are merged to obtain a third strip convolution branch input feature, and the third strip convolution branch performs a depth strip convolution on the input feature of the third strip convolution branch through the third strip convolution branch to obtain a third strip convolution branch output feature; The output feature of the depth convolution unit, the output feature of the first strip convolution branch, the output feature of the second strip convolution branch and the output feature of the third strip convolution branch are merged to obtain a point-by-point convolution unit input feature, and the point-by-point convolution unit is used to perform point-by-point convolution on the input feature of the point-by-point convolution unit to obtain an attention feature; and Attention adjustment is performed on the shallow feature based on the attention feature to obtain the optimized shallow feature.

3. The device ice image segmentation method according to claim 1, characterized in that: The feature expression optimization network includes a deep feature optimization module, and the deep feature optimization module includes: a first dilated convolution branch, a second dilated convolution branch, a first strip pooling branch, and a second strip pooling branch; The step of optimizing the feature expression of the deep feature by the feature expression optimization network to obtain the optimized deep feature includes: Through the first atrous convolution branch, the deep feature is subjected to a deep atrous separable convolution to obtain a first atrous convolution branch output feature; Through the second atrous convolution branch, a deep atrous separable convolution is performed on the deep feature to obtain a second atrous convolution branch output feature; Through the first strip pooling branch, average pooling is performed on the deep features based on the horizontal strip pooling kernel, and pixel expansion is performed on the corresponding average pooling result based on the deep features to obtain the first strip pooling branch output; Through the second strip pooling branch, average pooling is performed on the deep features based on the vertical strip pooling kernel, and pixel expansion is performed on the corresponding average pooling result based on the deep features to obtain the second strip pooling branch output; Fusing the first atrous convolution branch output feature, the second atrous convolution branch output feature, the first strip pooling branch output, and the second strip pooling branch output to obtain a multi-branch output fusion feature; and The optimized deep features are obtained by performing channel dimension reduction on the multi-branch output fusion features based on a point-by-point convolution operation.

4. The device ice image segmentation method according to claim 3, characterized in that: The first atrous convolution branch includes a first depth-atrous separable convolution layer and a second depth-atrous separable convolution layer; the second atrous convolution branch includes a third depth-atrous separable convolution layer and a fourth depth-atrous separable convolution layer; The step of performing a deep hole separable convolution on the deep feature through the first hole convolution branch to obtain a first hole convolution branch output feature includes: Performing a deep hole separable convolution on the deep features through a first deep hole separable convolution layer and a second deep hole separable convolution layer connected in series in sequence; The step of performing a deep hole separable convolution on the deep feature through the second hole convolution branch to obtain a second hole convolution branch output feature includes: The deep features are sequentially subjected to a deep atrous separable convolution through a third deep atrous separable convolution layer and a fourth deep atrous separable convolution layer connected in series.

5. The device ice image segmentation method according to claim 1, characterized in that: The feature extraction network includes: a first convolution normalization module, a second convolution normalization module, a third convolution normalization module, a fourth convolution normalization module, a fifth convolution normalization module, a first down-sampling convolution layer, a second down-sampling convolution layer, a third down-sampling convolution layer and a fourth down-sampling convolution layer; The step of obtaining shallow features and deep features corresponding to the ice-covered image area in the device image to be segmented through the feature extraction network includes: Performing convolution normalization and activation function processing on the device image to be segmented through the first convolution normalization module to obtain a first convolution normalization output feature; Acquire a first down-sampled convolution input feature based on the first convolution normalized output feature, and perform down-sampled convolution processing on the first down-sampled convolution input feature through the first down-sampled convolution layer to obtain a first down-sampled convolution output feature; Performing convolution normalization and activation function processing on the first down-sampled convolution output feature through the second convolution normalization module to obtain a second convolution normalization output feature; Acquire a second down-sampled convolution input feature based on the second convolution normalized output feature, and perform down-sampled convolution processing on the second down-sampled convolution input feature through the second down-sampled convolution layer to obtain the shallow feature; The shallow features are subjected to convolution normalization and activation function processing by the third convolution normalization module to obtain third convolution normalization output features; Acquire a third down-sampled convolution input feature based on the third convolution normalized output feature, and perform down-sampled convolution processing on the third down-sampled convolution input feature through the third down-sampled convolution layer to obtain a third down-sampled convolution output feature; Performing convolution normalization and activation function processing on the third down-sampled convolution output feature through the fourth convolution normalization module to obtain a fourth convolution normalization output feature; Acquire a fourth down-sampled convolution input feature based on the fourth convolution normalized output feature, and perform down-sampled convolution processing on the fourth down-sampled convolution input feature through the fourth down-sampled convolution layer to obtain a fourth down-sampled convolution output feature; and The fourth down-sampled convolution output feature is subjected to convolution normalization and activation function processing by the fifth convolution normalization module to obtain the deep feature.

6. The device ice image segmentation method according to claim 1, characterized in that: The step of obtaining a segmentation result of the ice-covered image area in the image of the device to be segmented by the feature fusion network based on the optimized shallow features and the optimized deep features includes: The number of channels of the optimized shallow features and the optimized deep features is adjusted to a preset number of channels, and the sizes of the optimized shallow features and the optimized deep features are adjusted to a preset size, to obtain the optimized shallow features to be fused and the optimized deep features to be fused; Fusing the shallow features to be fused and optimized and the deep features to be fused and optimized to obtain optimized fused features; Performing standard convolution processing on the optimized fusion features to obtain fusion features to be segmented; and The number of channels and the size of the fusion features to be segmented are adjusted to be the same as the number of channels and the size of the device image to be segmented, so as to obtain the segmentation result of the ice-covered image area.

7. A method for training an equipment ice image segmentation model, characterized in that: include: Inputting the sample device image into the device ice-covered image segmentation model to be trained, wherein the device ice-covered image segmentation model includes: a feature extraction network, a feature expression optimization network and a feature fusion network; Acquire shallow features and deep features corresponding to the ice-covered image area in the sample device image through the feature extraction network; The feature expressions of the shallow features and the deep features are optimized by the feature expression optimization network, and the optimized shallow features and the optimized deep features are obtained accordingly; Obtaining, through the feature fusion network, a segmentation result of the ice-covered image area in the sample device image based on the optimized shallow features and the optimized deep features; and The function value of the training loss function is determined based on the segmentation result of the ice-covered image area, and the network parameters in the device ice-covered image segmentation model are adjusted according to the function value of the training loss function.

8. The equipment ice image segmentation model training method according to claim 7 is characterized in that: Also includes: Before inputting the sample device image into the device ice image segmentation model to be trained, A plurality of ice-covered equipment images are acquired, and the sample ice-covered equipment images corresponding to the respective ice-covered equipment images are acquired based on a geometric transformation operation.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the device ice-covered image segmentation method as described in any one of claims 1 to 6 is implemented, or the device ice-covered image segmentation method as described in any one of claims 7 or 8 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the device ice-covered image segmentation method as described in any one of claims 1 to 7 is implemented, or the device ice-covered image segmentation method as described in any one of claims 7 or 8 is implemented.

Citation Information

Patent Citations

  • Remote sensing image cloud and cloud shadow fine segmentation detection method based on deep learning

    CN116052016A

  • Real-time semantic segmentation method and device for multidirectional feature refinement network, and medium

    CN117197446A

  • Optical cable icing detection method and device, electronic equipment and medium

    CN118196008A

  • Overhead transmission line icing form segmentation method

    CN118411724A

  • Icing detection model training method and device, electronic equipment and storage medium

    CN118504650A