SegMama-based ultra-lightweight photovoltaic module image segmentation method

Through SegMamba technology, the problems of low segmentation accuracy, high computational complexity and insufficient robustness in the traditional photovoltaic module image segmentation method are solved, and efficient and fine photovoltaic module image segmentation is achieved, suitable for variable environments and high-resolution images Processing reduces hardware costs and computing resource requirements.

CN119006498AActive Publication Date: 2024-11-22NORTHWEST NORMAL UNIVERSITY

Patent Information

Application Number
CN202411438524.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-15
Publication Date
2024-11-22
Estimated Expiration
2044-10-15

AI Technical Summary

Technical Problem

Traditional photovoltaic module image segmentation methods have problems such as limited segmentation accuracy, high computational complexity, insufficient robustness, insufficient model generalization capabilities, and poor detail and edge processing, especially in variable environments and high-resolution image processing.

Method used

Using the ultra-lightweight photovoltaic module image segmentation method based on SegMamba, through data preprocessing, feature extraction, feature fusion and image postprocessing, PatchEmbed2D, PMM Layer block and multi-scale cross feature fusion decoder, combining bilateral filtering, Gaussian filtering and Morphological processing to achieve the generation of high-resolution segmented images.

Benefits of technology

It significantly improves image segmentation accuracy, reduces computing resource requirements, improves processing speed and robustness, adapts to complex and changeable practical application scenarios, reduces hardware deployment costs, and enhances the deployability and flexibility of the system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119006498A_ABST
    Figure CN119006498A_ABST
Patent Text Reader

Abstract

The invention discloses an ultra-lightweight photovoltaic module image segmentation method based on SegMama. The method comprises the following steps: S1, data preprocessing and optimization; s2, feature extraction: image features are extracted through PatchEmbedded 2D and PMM Layer layer depth in sequence, a first-stage feature map is generated after normalization operation, and so on, the feature map generated in the previous stage is subjected to the operation to generate a feature map of the next stage, and finally, four images with different resolutions are generated; s3, feature fusion; the method has the advantages that the image segmentation precision is remarkably improved, and the model is ultra-lightweight; PatchEmbedded 2D blocking and PMM Layer channel number equally-dividing operation are introduced into an encoder, so that the parameter quantity and the calculation quantity of the model are greatly reduced while the diversity and deep mining of features are realized; a recursive cross-level feature fusion strategy is adopted in the decoder, so that comprehensive interaction of high-dimensional semantic information and low-dimensional detail information is ensured; the parallel complementary mechanism accurately captures fine features and differences of the photovoltaic module image, the segmentation precision is improved, and a solid guarantee is provided for subsequent photovoltaic module defect detection and other applications.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to an ultra-lightweight photovoltaic module image segmentation method based on SegMamba. Background Art

[0002] Traditional photovoltaic module image segmentation methods are mostly based on manual annotation segmentation and some traditional deep learning methods. Manual methods use the quadrilateral features of photovoltaic arrays to segment photovoltaic arrays, but the segmentation accuracy is limited and time-consuming. In recent years, the Transformer model based on the self-attention mechanism has been applied to the field of computer vision. It realizes the interaction of global information through the self-attention mechanism, which can reveal the dependencies between different positions and scales of input data. It has certain advantages over traditional manual methods and traditional models (such as FCN, UNet, Segformer, etc.) in IRT image segmentation, but it still has high computational complexity and low efficiency. Dependence on specific features and lack of robustness: Traditional segmentation methods based on quadrilateral features are highly dependent on specific geometric features and require frequent parameter adjustments in the face of changing environments, which limits the segmentation accuracy and robustness.

[0003] High computational cost: Although the self-attention mechanism Transformer model (such as Segformer) can capture global information and dependencies between different scales, it has high computational complexity when processing high-resolution images and is not suitable for scenarios with high real-time requirements.

[0004] Insufficient model generalization ability: Although existing deep learning methods perform well, they are greatly affected by the diversity and quality of training data. When faced with new scenarios, they lack generalization ability, which affects segmentation accuracy.

[0005] Details and edges are not processed precisely: Although models such as Segformer can capture global information and dependencies between different scales, they may not be precise enough when processing image details and edges, especially in key areas such as the boundaries of photovoltaic arrays and shadow junctions, where segmentation discontinuities or blurred edges may occur. For this reason, the present invention proposes an ultra-lightweight photovoltaic module image segmentation method based on SegMamba. Summary of the invention

[0006] The object of the present invention is to provide an ultra-lightweight photovoltaic module image segmentation method based on SegMamba to solve the problems raised in the above background technology.

[0007] To achieve the above-mentioned purpose, the present invention provides the following technical solutions: an ultra-lightweight photovoltaic module image segmentation method based on SegMamba, the specific steps comprising: S1: data preprocessing and optimization: S11: using a drone equipped with a dual-spectrum camera to perform low-altitude high-resolution photography, and simultaneously capturing infrared and visible light images of the photovoltaic array; after multiple shots, a total of several infrared and visible light images are obtained, each image having a resolution of 640×512 pixels; using infrared thermal imaging for analysis; after screening, finally retaining high-quality infrared thermal images; S12: Use the Labelme annotation tool to manually annotate the infrared image to obtain accurate image label information, divide the image into two categories: background and photovoltaic array Pv, and annotate it in PASCAL VOC format; S13: To eliminate the noise in the image, a two-step filtering strategy is adopted. First, bilateral filtering technology is applied to reduce the edge blur and detail loss in the image. Then, Gaussian filtering is used to further smooth the image and remove the remaining random noise. S2: Feature extraction: This paper proposes an ultra-lightweight encoder to solve the problem of application overhead caused by large segmentation model calculation and parameter quantity. The image is extracted through PatchEmbed2D, and then the number of channels of the image is changed through a block composed of two PMM Layer blocks in series, and then a normalization operation is performed to generate the first stage feature map. Similarly, the feature map generated in the previous stage is subjected to the above operation to generate the feature map of the next stage, and finally four images with different resolutions are generated; S3: Feature Fusion: The decoder uses multi-scale cross-feature fusion to efficiently restore the resolution of the image from multiple scale levels through horizontal cross connections. First, the features of different resolutions generated by the encoder in four different stages and the post-processed features are used as input. A recursive and cross-level strategy is adopted. Through multiple iterations of upsampling and skip connection mechanisms, the rich semantic information of the high level is directly passed to the lower level, deeply and carefully fused with the features of the subsequent stages and converted into the final segmentation mask. A high-resolution segmentation map is generated through the classification and segmentation module cls_seg. S4: Image post-processing: Morphological processing is used to eliminate tiny noises and fill defects in the target area; edge smoothing technology is used to smooth segmentation boundaries and reduce jagged edges; regional optimization strategies are used to merge adjacent or subdivide overly large areas to accurately depict object contours; and threshold fine-tuning is used to further improve segmentation accuracy.

[0008] Preferably, in each PMM Layer block, the feature X with the number of channels C is first normalized and then divided into Four features, each with C / 4 channels; each feature is then simultaneously input into a PM Block consisting of two serial Mamba modules for convolution; in each channel, 1 / 4 of the original input feature is added to the output feature after passing through a Sigmoid function, which ensures that the total number of channels remains unchanged and maintains high accuracy while minimizing the number of parameters to make the model lightweight Preferably, the process formula of step S2 is expressed as: i=1,2,3,4 i = 1,2,3,4 i=1,2,3,4 in, is the input feature with C channels; The number of channels output after being evenly divided is C / 4 feature; represent The result after passing the sigmoid activation function; for The output characteristics after passing through the PMM Layer module; For the general and The result after addition; It is the result of connecting the four parallel output results, and the number of channels is C; is the final mapped feature; LN is LayerNorm, which represents the layer normalization function; Sp is the Split operation used to evenly divide the input features; Sigmoid is the activation function; PM is the PMMLayer operation, and Cat is the Concat operation used to fuse the extracted features output by four independent channels to generate features with C channels; Pro is the linear projection operation, which helps to integrate information from different sub-channels and Mamba modules to ensure the consistency and integrity of the features.

[0009] Preferably, the encoder is composed of PatchEmbed2D, PMM Layer blocks and Groupnorm normalization blocks which are alternately connected in series.

[0010] Preferably, the step S4 can also effectively distinguish the foreground and background of the image by adaptive threshold processing Otsu method to generate a clear binary image.

[0011] Compared with the prior art, the present invention has the following beneficial effects: the image segmentation accuracy is significantly improved: By introducing the Parallel Multi-Mamba Layer (PMM Layer), feature diversity and deep mining are achieved in the feature extraction stage. This parallel complementary mechanism accurately captures the subtle features and key differences of photovoltaic module images, greatly improving the segmentation accuracy and providing a solid guarantee for applications such as photovoltaic module defect detection and crack identification.

[0012] Efficient processing and real-time performance: While pursuing high precision, the ultra-lightweight design greatly reduces the number of model parameters, achieving efficient processing and real-time performance. The number of parameters in the data set used by the present invention is as low as 0.85M, which is about 78% lower than the traditional segmentation method (such as segformer). This feature enables the invention to operate efficiently in an environment with limited computing resources, shorten processing time, and improve processing speed. This is especially important for photovoltaic monitoring systems that require real-time feedback, which can ensure timely detection and response to potential problems and improve overall operation and maintenance efficiency.

[0013] Deployment costs are significantly reduced: The lightweight design of the model not only reduces the demand for computing resources, but also significantly reduces the cost of hardware deployment. Photovoltaic power stations and distributed photovoltaic systems can flexibly select hardware equipment according to actual needs, without relying on high-cost high-performance computing equipment, thereby reducing the overall investment threshold and enhancing the deployability and flexibility of the system.

[0014] Robustness and generalization ability: Mamba's structure may have better noise resistance than convolution and Transformer for noise and abnormal data in photovoltaic array infrared images. Since Mamba's state space model contains memory capabilities and can process continuous input data more smoothly, it can provide more robust predictions when there are abnormal points or lighting changes in the image, and better adapt to complex and changeable practical application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0016] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0017] See also Figure 1 , which is the first embodiment of the present invention, provides a technical solution: a method for image segmentation of ultra-lightweight photovoltaic modules based on SegMamba, the specific steps of which include: S1: data preprocessing and optimization: S11: using a drone equipped with a dual-spectrum camera to perform low-altitude high-resolution photography, and simultaneously capture infrared and visible light images of the photovoltaic array; after multiple shots, a total of several infrared and visible light images are obtained, each with a resolution of 640×512 pixels; infrared thermal imaging is used for analysis; after screening, high-quality infrared thermal images are finally retained; S12: Then, the infrared image is manually labeled using the Labelme labeling tool to obtain accurate image label information, and the image is divided into two categories: background and photovoltaic array Pv, and labeled using the PASCAL VOC format; S13: In view of the fact that infrared thermal imaging technology is susceptible to the dual influence of external factors such as reflection interference, medium absorption and reflection, background radiation, and internal factors of the instrument, the image is often accompanied by significant noise; a two-step filtering strategy is adopted, firstly, bilateral filtering technology is applied to reduce edge blur and detail loss in the image, and then Gaussian filtering is used to further smooth the image and remove the remaining random noise; S2: Feature extraction: First input The image is extracted through PatchEmbed2D, and then the number of channels of the image is changed through a block composed of two PMM Layer blocks in series, and then a normalization operation is performed to generate the first stage feature map. Similarly, the feature map generated in the previous stage is subjected to the above operation to generate the feature map of the next stage, and finally four images with different resolutions are generated; S3: Feature fusion: The decoder adopts multi-scale cross-feature fusion to efficiently restore the resolution of the image from multiple scale levels through horizontal cross-connections. First, the features of different resolutions generated by the encoder in four different stages and the post-processed features are taken as input. Taking the features generated in the fourth stage as an example, the resolution is first adjusted to the same as stage3 by upsampling, that is, Then, the PMM Layer or ConvModule (1x1 convolution) is used to halve the number of feature channels of the adjusted stage4 and the original stage3 to C / 2 for fusion; a recursive and cross-level strategy is adopted, through multiple iterations of upsampling and skip connection mechanism, the rich semantic information of the high level is directly transmitted to the lower level, deeply and carefully fused with the features of the subsequent stages and converted into the final segmentation mask; a high-resolution segmentation map is generated through the classification and segmentation module cls_seg; S4: Image post-processing: Morphological processing is used to eliminate tiny noises and fill defects in the target area; edge smoothing technology is used to smooth segmentation boundaries and reduce jagged edges; regional optimization strategies are used to merge adjacent or subdivide overly large areas to accurately depict object contours; and threshold fine-tuning is used to further improve segmentation accuracy.

[0018] In this embodiment, preferably, in each PMM Layer block, the feature X with the number of channels C is first normalized and then divided into Four features, each with C / 4 channels; each feature is then simultaneously and parallelly input into a PM Block consisting of two serial Mamba modules for convolution; in each channel, 1 / 4 of the input original features are added to the output features after passing through a Sigmoid function, which ensures that the total number of channels remains unchanged and maintains high accuracy while minimizing the number of parameters to make the model lightweight.

[0019] In this embodiment, preferably, the process formula of step S2 is expressed as: i=1,2,3,4 i = 1,2,3,4 i=1,2,3,4 in, is the input feature with C channels; The number of channels output after being evenly divided is C / 4 feature; represent The result after passing the sigmoid activation function; for The output characteristics after passing through the PMM Layer module; For the general and The result after addition; It is the result of connecting the four parallel output results, and the number of channels is C; is the final mapped feature; LN is LayerNorm, which represents the layer normalization function; Sp is the Split operation used to evenly divide the input features; Sigmoid is the activation function; PM is the PMMLayer operation, and Cat is the Concat operation used to fuse the extracted features output by four independent channels to generate features with C channels; Pro is the linear projection operation, which helps to integrate information from different sub-channels and Mamba modules to ensure the consistency and integrity of the features.

[0020] In this embodiment, preferably, the encoder is composed of PatchEmbed2D, PMM Layer blocks and Groupnorm normalization blocks which are alternately connected in series.

[0021] In this embodiment, preferably, the step S4 can also effectively distinguish the foreground and background of the image through the adaptive threshold processing Otsu method to generate a clear binary image.

[0022] Although embodiments of the present invention have been shown and described (see the above detailed description for details), it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for ultra-lightweight photovoltaic module image segmentation based on SegMamba, characterized by: The specific steps include: S1: data preprocessing and optimization: S11: using a drone equipped with a dual-spectrum camera to take low-altitude high-resolution photos, and simultaneously capture infrared and visible light images of the photovoltaic array; after multiple shots, a total of several infrared and visible light images were obtained, each with a resolution of 640×512 pixels; infrared thermal imaging was used for analysis; after screening, high-quality infrared thermal images were finally retained; S12: Use the Labelme annotation tool to manually annotate the infrared image to obtain accurate image label information, divide the image into two categories: background and photovoltaic array Pv, and annotate it in PASCAL VOC format; S13: To eliminate the noise in the image, a two-step filtering strategy is adopted. First, bilateral filtering technology is applied to reduce the edge blur and detail loss in the image. Then, Gaussian filtering is used to further smooth the image and remove the remaining random noise. S2: Feature extraction: To address the application overhead caused by the large number of segmentation model calculations and parameters, an ultra-lightweight encoder is proposed. The image is extracted through PatchEmbed2D, and then the number of channels of the image is changed through a block composed of two PMM Layer blocks in series, and then a normalization operation is performed to generate the first stage feature map. Similarly, the feature map generated in the previous stage is subjected to the above operation to generate the feature map of the next stage, and finally four images with different resolutions are generated; S3: Feature Fusion: The decoder uses multi-scale cross-feature fusion to efficiently restore the resolution of the image from multiple scale levels through horizontal cross connections. First, the features of different resolutions generated by the encoder in four different stages and the post-processed features are used as input. A recursive and cross-level strategy is adopted. Through multiple iterations of upsampling and skip connection mechanisms, the rich semantic information of the high level is directly passed to the lower level, deeply and carefully fused with the features of the subsequent stages and converted into the final segmentation mask. A high-resolution segmentation map is generated through the classification and segmentation module cls_seg. S4: Image post-processing: Morphological processing is used to eliminate tiny noises and fill defects in the target area; edge smoothing technology is used to smooth segmentation boundaries and reduce jagged edges; regional optimization strategies are used to merge adjacent or subdivide overly large areas to accurately depict object contours; and threshold fine-tuning is used to further improve segmentation accuracy.

2. The ultra-lightweight photovoltaic module image segmentation method based on SegMamba according to claim 1, characterized in that: In each PMM Layer block, the feature X with the number of channels C is first normalized and then divided into Four features, each with C / 4 channels; then each feature is simultaneously and parallelly input into the PMM Block composed of two serial Mamba modules for convolution operation; in each channel, 1 / 4 of the original input features are added to the output features after passing through the Sigmoid function, and finally the four features are added. This parallel convolution strategy can extract the spatial dimension features of the image at different scales at the same time, and realize the comprehensive and accurate capture of local details and global context information, ensuring the diversity and depth of feature extraction, and minimizing the number of parameters while ensuring that the total number of channels remains unchanged and maintains high precision, making the model lightweight.

3. The ultra-lightweight photovoltaic module image segmentation method based on SegMamba according to claim 2, characterized in that: The process formula of step S2 is expressed as: i=1,2,3,4 i = 1,2,3,4 i=1,2,3,4 in, is the input feature with C channels; The number of channels output after being evenly divided is C / 4 feature; represent The result after passing the sigmoid activation function; for The result features after passing through the PMMLayer module; For the general and The result after addition; It is the result of connecting the four parallel output results, and the number of channels is C; is the final mapped feature; LN is LayerNorm, which represents the layer normalization function; Sp is the Split operation used to evenly divide the input features; Sigmoid is the activation function; PM is the PMM Layer operation, and Cat is the Concat operation used to fuse the extracted features output by four independent channels to generate features with a channel number of C; Pro is the linear projection operation, which helps to integrate information from different sub-channels and Mamba modules to ensure the consistency and integrity of the features.

4. The ultra-lightweight photovoltaic module image segmentation method based on SegMamba according to claim 1, characterized in that: The encoder is composed of PatchEmbed2D, PMM Layer blocks and Groupnorm normalization blocks which are alternately connected in series.

5. The ultra-lightweight photovoltaic module image segmentation method based on SegMamba according to claim 1, characterized in that: The step S4 can also effectively distinguish the foreground and background of the image by using the adaptive threshold processing Otsu method to generate a clear binary image.

Citation Information

Patent Citations

  • Remote sensing image semantic segmentation method based on double-branch feature fusion

    CN115797931A

  • Medical image segmentation method based on feature interaction

    CN118134952A

  • Battery pack SOC and SOE estimation method driven by Mama architecture

    CN118444159A

  • Photovoltaic panel recognition method, ground station, control apparatus, and unmanned aerial vehicle

    US20200134318A1

  • Multivalent binding compositions with reactive groups

    US20240117428A1

Cited By

  • Cabin air conditioner prediction method based on bidirectional interaction Mama with independent channels

    CN119719679A