An improved global self-attention mechanism-based visible light photovoltaic support panoramic image automatic identification method

Abstract

The application provides an automatic identification method for visible light photovoltaic support panoramic pictures based on an improved global self-attention mechanism, relates to the technical field of image processing, and is used for automatic identification and labeling of photovoltaic panoramic support pictures, reduction of manual labeling cost, reduction of picture processing calculation amount, improvement of precision, and comprises the following steps: step 1, collecting a plurality of panoramic pictures containing visible light photovoltaic supports, and automatically cutting the panoramic pictures; step 2, calculating the cutting picture coordinates of the support corner points according to the panoramic picture coordinates of the support corner points; step 3, obtaining the connected domain of the support according to the support corner point coordinates, and converting the connected domain into a binary graph containing coordinate labeling information, training through a support automatic identification network, and learning the conversion relationship between the cutting picture and the binary graph; and step 4, after cutting any panoramic picture, performing support automatic identification, generating a binary graph containing coordinate labeling information, and restoring the panoramic picture to obtain the support region and the coordinates of the panoramic picture.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to an automatic recognition method for panoramic images of visible light photovoltaic supports based on an improved global self-attention mechanism. Background Technology

[0002] The rapid growth in renewable energy production and consumption has strongly promoted the construction of a clean, low-carbon, and green energy system, and facilitated the transformation of the energy economy towards an ecologically centered development model.

[0003] Existing panoramic photovoltaic images suffer from high resolution and significant background interference. Manual segmentation of the panoramic support structure and modules is costly and requires substantial annotation work. UAV remote sensing images typically have resolutions in the hundreds of millions, making direct automatic segmentation computationally intensive and unacceptable. Direct downsampling for recognition would also result in unacceptable accuracy loss. Summary of the Invention

[0004] This invention designs an automatic recognition method for panoramic images of visible light photovoltaic supports based on an improved global self-attention mechanism. This method is used for the automatic recognition and annotation of panoramic photovoltaic support images, reducing the cost of manual annotation, reducing the computational load of directly downsampling high-resolution images in conventional processing methods, and improving accuracy.

[0005] A lining structure connection method and construction method for underground docking of tunnel boring machines includes the following steps:

[0006] Step 1: Collect several panoramic images containing visible light photovoltaic brackets. The panoramic images are all the same size and have their coordinates marked in advance. Then, automatically cut the panoramic images according to the same rules.

[0007] Step 2: Calculate the coordinates of the cutaway diagram of the bracket corner points based on the panoramic view coordinates of the bracket corner points;

[0008] Step 3: Obtain the connected components of the support based on the corner coordinates of the support, and convert them into a binary image containing coordinate annotation information. Train the support automatic recognition network on the cutting image and the binary image to learn the transformation relationship between the cutting image and the binary image.

[0009] Step 4: After cutting any panoramic image containing a visible light photovoltaic bracket according to the same rules as in Step 1, automatically identify the bracket through the trained bracket automatic recognition network, generate a binary image containing coordinate annotation information, and restore the panoramic image to obtain the bracket area and coordinates of the panoramic image.

[0010] Furthermore, by iteratively slicing and cropping the panoramic image, with the cropped image size being scale, the rows and columns of the panoramic image to which the cropped image belongs are as follows:

[0011]

[0012]

[0013] Where i represents the row of the panoramic image to which the cut image belongs, j represents the column of the panoramic image to which the cut image belongs, w represents the horizontal coordinate of the panoramic image to which the cut image belongs, and h represents the vertical coordinate of the panoramic image to which the cut image belongs.

[0014] Preferably, in order to avoid the problem of segmentation and breakage at the boundary during subsequent merging operations, an overlap rate gap is set during cutting.

[0015] Furthermore, the relative coordinates of the bracket corner points in the cut image to the panoramic image are determined using the following formula:

[0016]

[0017] Where, x split y split The x and y coordinates of the bracket corner points in the cutting diagram are respectively represented by x. ori ,y ori The x and y coordinates of the corner point of the bracket in the panoramic image are represented by w0 and h0, respectively, and the width and height of a single bracket in the image are represented by w0 and h0, respectively.

[0018] Furthermore, the automatic stent recognition network includes an Encoder feature extraction layer and an RTformer head layer. The head layer uses the global autonomous force mechanism in RTformer to obtain the segmentation results.

[0019] Furthermore, after the cut image is input into the scaffold automatic recognition network, features are extracted through multi-layer 3*3 and 5*5 DepthSepConv depth-separable convolutions. Then, the SE module simulates residual operations to perform multi-feature fusion, and the SE module is placed in the last two layers of DepthSepConv to improve model accuracy.

[0020] Furthermore, the automatic bracket recognition network adds an SE module and a large 5*5 convolutional kernel at a deeper level, which, based on the acquisition of deep semantic information, allows for the acquisition of more edge information of the photovoltaic panel.

[0021] Furthermore, SPPF is used to fuse features from different layers to improve the model's generalization ability.

[0022] Preferably, the panoramic images are merged according to the segmentation rules, and the broken parts of the brackets at the junctions are connected.

[0023] Furthermore, the coordinates of the support frame in the panoramic view are given by the following formula:

[0024]

[0025] Where x′ ori ,y′ ori These represent the x and y coordinates of the support frame on the panoramic image after recalculation, respectively. split y′ split These represent the x and y coordinates of the cut image calculated by the scaffold automatic recognition network, respectively.

[0026] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0027] This invention designs an automatic recognition method for panoramic images of visible light photovoltaic supports based on an improved global self-attention mechanism. This method is used for the automatic recognition and annotation of panoramic photovoltaic support images, reducing the cost of manual annotation and minimizing the accuracy loss caused by directly downsampling high-resolution images in conventional processing methods.

[0028] 1. This invention uses an improved automatic support identification network to identify and label supports, effectively reducing background interference, improving edge recognition, and obtaining more accurate support corner information.

[0029] 2. In order to avoid the problem of segmentation breakage at the boundary during subsequent merging operations, the present invention sets the overlap rate gap to 30% of the width of a single bracket. In this way, during inference, post-processing is performed based on the segmentation results of the merged area to avoid abnormal segmentation positions; that is, step 4 can better segment the position of the bracket when merging the final panoramic image.

[0030] 3. In this invention, the SE module is placed in the last two layers of the SE module, DepthSepConv, because the closer the SE module is to the end of the network, the greater the improvement in model accuracy. This approach can effectively improve the accuracy of the model.

[0031] 4. The tail of the Encoder layer in this invention uses multiple 5*5DepthSepConv modules for feature extraction. This is because larger convolutional kernels have a more significant effect in the middle and later parts of the network. In this way, in addition to obtaining deep semantic information, more edge information of the photovoltaic panel can also be obtained.

[0032] 5. In the final step of feature extraction, this invention uses SPPF to fuse features from different layers, which greatly improves the generalization ability of the model. Attached Figure Description

[0033] Figure 1 This is the overall process framework of the present invention;

[0034] Figure 2 This invention relates to image segmentation;

[0035] Figure 3 This is the bracket of the present invention and its binary image;

[0036] Figure 4 This is a schematic diagram of the main architecture of the RTFormer head of this invention;

[0037] Figure 5 This invention is a cutting method. Figure 2 Schematic diagram of automatic value stitching;

[0038] Figure 6 This invention relates to a lightweight encoder layer architecture;

[0039] Figure 7 This is a schematic diagram of the Encoder feature extraction of the present invention;

[0040] Figure 8 This is the operation mode of the SE module in this invention;

[0041] Figure 9 This is a schematic diagram of the overlapping position setting in this invention. Detailed Implementation

[0042] The following describes in detail, with reference to the accompanying drawings and specific implementation methods, an automatic recognition method for panoramic images of visible light photovoltaic brackets based on an improved global self-attention mechanism.

[0043] This invention proposes an automatic panoramic image recognition method for visible light photovoltaic (PV) support structures based on an improved global self-attention mechanism. The main process includes: automatic segmentation of the panoramic image; calculating the coordinates of the segmented images based on prior information about the PV support structure; automatic annotation of the segmented images to obtain labeled images for training; automatic recognition using an improved Rtformer global self-attention architecture, which effectively reduces background interference, improves edge recognition, and obtains more accurate corner information; and finally, automatic PV recognition and panoramic image reconstruction using the trained network. After obtaining the segmented panoramic image, the corner information of the PV support structure is obtained by extracting the contour of the binary image and calculating the minimum bounding rectangle of the contour. Finally, the pixel-to-GPS correspondence is used to convert the information into the actual latitude and longitude coordinates of the panoramic image.

[0044] Step 1: Automatic Cutting

[0045] UAV remote sensing images typically have resolutions in the hundreds of millions, making direct automatic segmentation computationally intensive. Direct downsampling for recognition would also result in unacceptable accuracy loss. Therefore, after obtaining the scale of the remote sensing image, we segment it according to a specific scale, referring to the formula below.

[0046]

[0047]

[0048] `scale` represents the size of the cropped image; here, a fixed size of 640 is chosen to balance inference speed and neural network accuracy. The overlap ratio `gap` is set to 30%. The fixed scale size `w` represents the x-coordinate of the panorama to which the cropped image belongs, `h` represents the y-coordinate, `i` represents the row, and `j` represents the column. The original remote sensing image is cyclically sliced ​​based on the values ​​of `i` and `j`. The results are shown above. Figure 2 Here, the background of the completely black image is filtered based on the prior information of the scaffold.

[0049] The innovative points are as follows:

[0050] To avoid segmentation breakage issues at the boundaries during subsequent merging operations, the overlap gap is set to 30% equal to the width of a single support. This allows for post-processing based on the segmentation results of the merged area during inference, avoiding locations with abnormal segmentation. In other words, step 4 can better segment the positions of the supports during the final panoramic image merging.

[0051] As you can see in the left image, without gap filling and post-processing, the middle section of the scaffold, when cut into two parts, is prone to breakage and misidentification as two separate scaffolds. The right image, after post-processing, effectively avoids this problem. The 30% size is a hyperparameter that considers both recognition performance and time settings.

[0052] Step 2: Automatic Labeling

[0053] After cutting, the relative coordinates of the support structure need to be recalculated. Directly generating an image by annotating the support structure coordinates from the original large image would be time-consuming due to the high resolution of the original image. Therefore, we recalculate the support structure coordinates of the automatically cut support structure. The specific process is as follows:

[0054] 1. Convert the GPS coordinates of the support structures saved in the panoramic image to coordinates in the panoramic image coordinate system. Only the GPS coordinates of the four corner points of each support structure are needed here.

[0055] Based on the panoramic image scale, slices were created using a fixed size of 640 pixels and an overlap rate of 30%. (Based on the original scaffold image...) With slice image x split The coordinates of the scaffold on the slice image are recalculated based on the corresponding relationship. The specific formula is as follows:

[0056]

[0057] Where x split y splitThe x and y coordinates of the bracket corner points in the cutting diagram are respectively represented by x. ori ,y ori These represent the x and y coordinates of the bracket corner point in the panoramic image, respectively. w0 and h0 represent the width and height of a single bracket in the image, respectively. i represents the row and j of the panoramic image to which the cutout belongs. The x-coordinates of each bracket in the small image are obtained. split y split Next, the connected components of the support structure corresponding to the coordinates of the small image are obtained, and the data is converted into the type required for image segmentation training. Binary image training samples are obtained based on the support structure contour information. See the specific generated results for reference. Figure 3 This eliminates the need for manual labeling.

[0058] Step 3: Training scaffold automatically identifies the network

[0059] After obtaining the automatically segmented small images and their corresponding binary images from step 2, the images are fed into the scaffold automatic recognition network for training. Based on the training results, the network can automatically learn the conversion relationship between the segmented images and the binary images. Following this conversion, the original large image is segmented to obtain small images, which are then fed into the trained network. This automatically calculates the segmented binary images, which are then merged according to the segmentation rules to obtain the segmented large image corresponding to the panoramic image.

[0060] The network we referenced here is the RTFormer global self-attention network. It mainly consists of an encoder feature extraction layer and an RTFormer head layer. The head layer uses the global self-attention mechanism in RTFormer to obtain the segmentation result. Here, we mainly optimized the encoder feature extraction side. The specific optimization steps will be explained below, and will not be elaborated on here.

[0061] Step 3.1 Encoder Feature Extraction Module

[0062] The encoder part uses an optimized network, mainly referencing the depthwise separable convolution and SE module in MobileNet v3 for computational optimization and multi-layer perceptual information fusion. For the specific architecture, refer to [link to relevant documentation]. Figure 6 .

[0063] DW and PW represent the two processes of depthwise separable convolution, namely DepthSepConv. Input is the input slice image.

[0064] Specific details are as follows Figure 7First, after receiving the segmented small image as input, the model extracts features through multiple layers of 3*3 and 5*5 DepthSepConv depthwise separable convolutions. Then, the SE module simulates residual operations to perform multi-feature fusion. For the SE module operation method, please refer to [link / reference]. Figure 8 :

[0065] Assuming the input is a W*H*C feature vector, where W, H, and C represent the width, height, and number of channels of the input feature vector, respectively (i.e., the feature vector after convolution of the cropped images), the following steps are performed: First, compression is applied to a size of 1*1*C. Then, excitation is performed, which linearly activates the vector. Finally, the 1*1*C scaled excitation vector is multiplied by the original input to obtain the multi-feature fusion result.

[0066] The feature fusion results from the SE modules are fed into subsequent SPPF network layers. Multi-scale fusion is achieved by applying SPPF spatial pyramid operations to each of the 5*5DepthSepConv modules. After pooling different scales of each module to the same size, feature fusion is performed, thus obtaining deep, high-resolution semantic information. This further improves the model's performance, ultimately significantly surpassing existing MobileNetv3 models. Figure 8 In this diagram, Stem conv represents the convolutional operation of the backbone network, 3*3 / 5*5 represents the DepthSepConv (the kernel size of the depthwise separable convolution), and SPPF represents the multi-scale fusion operation. Here we obtain the final feature vector, which is then fed into the rtformer head layer for automatic segmentation.

[0067] We selected blocks with the simplest possible structure and the highest possible speed to construct our encoder. Based on this, we summarized the following two methods for the encoder's feature extraction layer, which can better simplify the network and reduce the model's runtime.

[0068] The first point is the use of the SE channel attention mechanism, commonly used in current small networks. This effectively improves the model's accuracy; the specific process has been explained above and will not be repeated here. This module also introduces significant latency on the ARM side, so we need to adjust both accuracy and speed. Our research shows that the closer the SE module is to the end of the network, the greater the improvement in model accuracy. As shown in the figure above, we placed the SE module in the last two layers of DepthSepConv. The second point is that the size of the convolutional kernel at different positions affects the model's performance. Here, we use different convolutional kernels for feature extraction, which are the different convolutional kernels in DepthSepConv mentioned above. Our research shows that larger convolutional kernels have a more significant effect in the middle and later parts of the network. Therefore, in the figure above, multiple 5*5 DepthSepConv modules are used at the end of the Encoder layer for feature extraction.

[0069] Here, we primarily employ the controlled variable method, introducing SE modules and large 5x5 convolutional kernels at different layers. Through comparative experiments, we trained the model and found that under the same input conditions, the closer the SE modules and large convolutional kernels were to the end of the network, the higher the accuracy. This is theoretically based; we know that deeper networks can capture more semantic information about the photovoltaic panels. Adding SE modules and large 5x5 convolutional kernels to deeper layers allows us to capture not only deeper semantic information but also more edge information about the photovoltaic panels. Therefore, the improved model accuracy is to be expected.

[0070] Thirdly, because photovoltaic support scenarios are complex and panels vary in size, we used SPPF to fuse features from different layers at the end of feature extraction, which greatly improved the model's generalization ability. Ultimately, our results optimized the inference time on the ARM side without affecting accuracy.

[0071] Step 3.2 Construct the main architecture of RTFormer head

[0072] The head module uses the tarnsformer network from the latest RTFormer. (See reference...) Figure 4 .

[0073] The attention mechanism uses the External Attention (EA) module, and the formula for EA can be expressed as:

[0074] EA(X,K,V)=DN(X*K T )*V

[0075] Where X∈N×d represents the current input feature, N is the number of elements (or pixels in the image), d is the feature dimension, K,V∈RM×d are the parameters that the network needs to learn, which are the attention parameters in the attention mechanism, and their size is the same as the size of the input feature. DN is the double normalization operation. The multi-head version of EA, Multi-Head EA, can be interpreted as:

[0076] MHEA(X) = Concat(h1,h2,...,h) H ),h i =EA(X) i (,K',V'), i∈[1,H]

[0077] Where K′, V′∈RM×d′, d′=d / H, H is the number of heads, and d is the scale of the input X. And X i It is the i-th head of X. For example... Figure 3 As shown in the upper part, the multi-head mechanism generates H attention maps to increase the capacity of the EA, which allows matrix multiplication to be divided into several groups, similar to grouped convolution. Although the EA uses shared K′ and V′ for different heads, this can significantly speed up computation. That is... Figure 3 The process in the first part.

[0078] The purpose of the attention mechanism is to enable the neural network to learn better scaffold features, facilitating scaffold information recognition. The K and V matrices represent the parameters learned by the network. The attention calculation formula is as follows:

[0079]

[0080] Where K g V g ∈RM g ×d,M g =M×H, GDN represents grouped double normalization, which decomposes the second normalization of the original double normalization into H groups, such as Figure 3 As shown in the lower part, the attention mechanism aggregates features from different outputs to further extract multi-scale features, at which point the output is 1 / 8 of the original size. In the last three stages, the stride of the high-resolution features remains unchanged at 8, while the strides of the low-resolution features are 16 and 32 respectively. This means the current network output's relationship to the original scaffold coordinates is as follows:

[0081]

[0082] x here split_ y split_ This is the result obtained from the current network output, which incorporates the scaffold features relative to the original cut. Figure 1 Results at scales of / 8, 1 / 16, and 1 / 32.

[0083] Then for each pixel x split_ y split_ The training process involves classification. It determines whether the current output is a real scaffold and trains to establish a correspondence between each cut image and the segmentation result. This way, after obtaining smaller images from the original large image through cutting, these smaller images are fed into the trained network to automatically calculate the segmented binary image.

[0084] Step 4. Merge panoramic images

[0085] After the neural network obtains all the small image segmentation results, it merges the panoramic images according to the segmentation rules, and uses some traditional image processing or computational operations to connect the broken parts of the support at the junction.

[0086] In steps 1 and 2, we sliced ​​the image using a fixed scale of 640 and an overlap ratio of 30%. Therefore, the coordinates of the bracket in the panorama can be recalculated using the following formula:

[0087]

[0088] Where x′ ori ,y′ ori These represent the x and y coordinates of the support frame on the panoramic image after recalculation, respectively. split y′ split These represent the x and y coordinates of the cut image obtained through neural network calculation, respectively.

[0089] The overlap ratio gap of 30% is mainly used to facilitate post-processing of overlapping positions during merging. This mainly involves performing an OR operation on the corresponding overlapping bits. (See reference...) Figure 9 :

[0090] As you can see in the left image, without post-processing, if the same support is cut into two parts, the cut point can easily break and be mistaken for two separate supports. The right image, after post-processing, effectively avoids this problem.

[0091] Final merged image reference Figure 5 After obtaining the support area of ​​the entire large image, the coordinates of the four corner points of each support are x... ori ,y ori The drone's internal parameters are converted into actual latitude, longitude, and altitude.

[0092] The above embodiments are only for illustrating the technical concept and features of the present invention, and are intended to enable those skilled in the art to understand the content of the present invention and implement it accordingly. They should not be construed as limiting the scope of protection of the present invention. All equivalent changes or modifications made in accordance with the spirit and essence of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A method for automatic recognition of panoramic images of visible light photovoltaic supports based on an improved global self-attention mechanism, characterized in that, Includes the following steps: Step 1: Collect several panoramic images containing visible light photovoltaic brackets. The panoramic images are all the same size and have their coordinates marked in advance. Then, automatically cut the panoramic images according to the same rules. Step 2: Calculate the coordinates of the cutaway diagram of the bracket corner points based on the panoramic view coordinates of the bracket corner points; Step 3: Obtain the connected components of the support based on the corner coordinates of the support, and convert them into a binary image containing coordinate annotation information. Train the support automatic recognition network on the cutting image and the binary image to learn the transformation relationship between the cutting image and the binary image. Step 4: After cutting any panoramic image containing a visible light photovoltaic bracket according to the same rules as in Step 1, automatically identify the bracket through the trained bracket automatic recognition network, generate a binary image containing coordinate annotation information, and restore the panoramic image to obtain the bracket area and coordinates of the panoramic image. The automatic stent recognition network consists of an Encoder feature extraction layer and an RTformer head layer. The head layer uses the global autonomous force mechanism in RTformer to obtain the segmentation results. After the segmentation image is input into the automatic stent recognition network, features are extracted through multiple layers of 3*3 and 5*5 DepthSepConv depthwise separable convolutions. Then, the SE module simulates residual operations to perform multi-feature fusion. The SE module is placed in the last two DepthSepConv layers to improve model accuracy. The overlap gap is 30% of the width of a single stent.

2. The method for automatic recognition of panoramic images of visible light photovoltaic supports based on an improved global self-attention mechanism according to claim 1, characterized in that: If a panoramic image is repeatedly sliced ​​and the sliced ​​image size is 'scale', then the rows and columns of the panoramic image to which the sliced ​​image belongs are as follows: ； ； Where i represents the row of the panoramic image to which the cut image belongs, and j represents the column of the panoramic image to which the cut image belongs.

3. The method for automatic recognition of panoramic images of visible light photovoltaic supports based on an improved global self-attention mechanism according to claim 2, characterized in that: To avoid the problem of segmentation and breakage at the boundary during subsequent merging operations, an overlap rate gap is set during the cutting process.

4. The method for automatic recognition of panoramic images of visible light photovoltaic supports based on an improved global self-attention mechanism according to claim 3, characterized in that: The relative coordinates of the bracket corner points in the cut view to the panoramic view are based on the following formula: ； in, , These represent the x and y coordinates of the corner points of the support in the cutting diagram, respectively. , These represent the x and y coordinates of the corner point of the bracket in the panoramic image, respectively. and These represent the width and height of a single support in the image, respectively.

5. The method for automatic recognition of panoramic images of visible light photovoltaic supports based on an improved global self-attention mechanism according to claim 4, characterized in that: The automatic bracket recognition network adds an SE module and a large 5*5 convolutional kernel to the deeper layers, thereby obtaining more edge information of the photovoltaic panel based on the deep semantic information.

6. The method for automatic recognition of panoramic images of visible light photovoltaic supports based on an improved global self-attention mechanism according to claim 5, characterized in that: SPPF is used to fuse features from different layers to improve the model's generalization ability.

7. The method for automatic recognition of panoramic images of visible light photovoltaic supports based on an improved global self-attention mechanism according to claim 6, characterized in that: The panoramic images were merged according to the segmentation rules, and the broken parts of the brackets at the junctions were connected.

8. The method for automatic recognition of panoramic images of visible light photovoltaic supports based on an improved global self-attention mechanism according to claim 1, characterized in that: The coordinates of the support structure in the panoramic view are shown in the following formula: ； in , These represent the x and y coordinates of the support frame on the panoramic image after recalculation. , These represent the x and y coordinates of the cut image calculated by the scaffold automatic recognition network, respectively.

Images