Rooftop photovoltaic recognition and evaluation method based on key point detection and enhanced semantic segmentation

Abstract

The present invention discloses a rooftop photovoltaic recognition and evaluation method based on key point detection enhanced semantic segmentation and its system and storage medium. The method comprises: extracting significant feature points of the roof in satellite images as key points; generating a descriptor vector for each detected key point; combining the generated key point information with a pre-trained semantic segmentation network, and optimizing the initial segmentation result of the semantic segmentation network using the key point information; using the descriptor vector to match and compare similarities of key points to identify the outline and important structural features of the roof, and classifying and identifying the roof according to the matching results to obtain the final roof segmentation result. The present invention realizes high-precision recognition and description of roofs in complex environments through the combination of key point detection and semantic segmentation, and has good robustness and adaptability.

Description

Technical Field

[0001] The present invention belongs to the technical field of image recognition, and relates to rooftop photovoltaic recognition, and in particular to a rooftop photovoltaic recognition and evaluation method based on key point detection enhanced semantic segmentation. Background Art

[0002] As the global demand for renewable energy continues to grow, rooftop photovoltaic power generation has become an important part of urban energy transformation. Using roof space for photovoltaic power generation not only helps reduce urban carbon emissions, but also effectively utilizes existing building structures. However, accurately assessing the photovoltaic potential of building roofs is a complex task that requires detailed analysis of multiple factors, including the roof area, inclination, orientation and surrounding environment.

[0003] Existing methods for assessing rooftop photovoltaic potential usually rely on manual surveys and on-site measurements, which are not only time-consuming and labor-intensive, but also difficult to cover buildings in large areas. With the development of satellite remote sensing technology, high-resolution satellite images have become an effective alternative, enabling automated analysis of building roofs over a wide area. However, current image processing techniques still face many challenges in extracting and identifying rooftop features from satellite images.

[0004] In the field of satellite image processing, semantic segmentation technology is widely used for object recognition and region segmentation in images, but it is still difficult to accurately identify roofs and evaluate photovoltaic potential. Existing semantic segmentation methods may not be able to accurately handle roof details under complex backgrounds and multi-scale conditions, resulting in limited accuracy of segmentation results. This is because traditional methods often fail to provide enough detailed information when dealing with different lighting, weather conditions, and differences in roof materials and colors, thus affecting the overall segmentation effect. Summary of the invention

[0005] Purpose of the invention: In order to overcome the shortcomings of the prior art, a rooftop photovoltaic identification and evaluation method based on key point detection enhanced semantic segmentation is provided. By combining the advanced technologies of key point detection and semantic segmentation, the key point features of the roof can be accurately extracted to enhance the effect of semantic segmentation, thereby achieving more accurate rooftop identification and photovoltaic potential evaluation in complex environments.

[0006] Technical solution: To achieve the above purpose, the present invention provides a rooftop photovoltaic recognition and evaluation method based on key point detection and enhanced semantic segmentation, comprising the following steps:

[0007] S1: Extract the significant feature points of the roof in the satellite image as key points;

[0008] S2: Generate a descriptor vector for each detected key point;

[0009] S3: Combine the generated key point information with the pre-trained semantic segmentation network, accurately divide the boundaries of the roof area through the location information of the key points, enhance the detail features, and use the key point information to optimize the initial segmentation results of the semantic segmentation network;

[0010] S4: Use the descriptor vector to match and compare the key points to identify the outline and important structural features of the roof, classify and identify the roof based on the matching results, and obtain the final roof segmentation result.

[0011] Furthermore, the key point extraction in step S1 utilizes a key point detection module that combines deep learning and traditional image processing algorithms to extract significant feature points of the roof in the satellite image. The feature points include the edges and corners of the roof, which can accurately indicate the outline and structural features of the roof. The specific method includes:

[0012] A1: Preprocess the input satellite images to optimize image quality and improve the accuracy of subsequent processing;

[0013] A2: Perform feature detection on the preprocessed image: Use a convolutional neural network to extract deep features in the image and generate a multi-scale feature map; perform edge detection on the feature map and use the Sobel operator to enhance the image edge information;

[0014] A3: Calculate the response strength of each pixel through the feature map and generate the key point candidate area;

[0015] A4: Use image pyramid technology to perform multi-scale feature fusion processing to improve the stability of key points;

[0016] A5: Perform fine processing on the detected key points to improve positioning accuracy.

[0017] Furthermore, the image preprocessing in step A1 includes noise removal, color correction and scale normalization. Noise removal uses Gaussian filtering and bilateral filtering techniques to reduce noise and artifacts in the image. The purpose of color correction and brightness adjustment is to ensure the contrast and color uniformity of the image and adapt to image processing under different lighting conditions. Scale normalization ensures the consistency of images at different resolutions, thereby improving the stability of key point detection;

[0018] The specific process of image preprocessing is as follows:

[0019] A1-1: Use Gaussian filtering to smooth the image. The formula of Gaussian filter is:

[0020]

[0021] in, is the weight of the Gaussian filter, and are image coordinates, is the standard deviation of the Gaussian filter; the noise is smoothed by the convolution operation;

[0022] A1-2: Adjust the brightness and contrast of the image through linear transformation. The correction formula is:

[0023]

[0024] in, is the corrected pixel value, is the original pixel value, is the contrast adjustment factor, is the brightness adjustment factor;

[0025] A1-3: Use resampling technology to adjust the image to a uniform resolution; for image scaling, use the bilinear interpolation formula:

[0026]

[0027] in, and is the interpolation factor, is the pixel value after scaling;

[0028] The multi-scale feature map generated in step A2 is expressed as:

[0029]

[0030] in, is the feature map, is the convolution kernel weight, is the displacement of the convolution kernel, is the input image;

[0031] The convolution operation formula of the Sobel operator is:

[0032]

[0033]

[0034] in, and The Sobel operator is and The convolution kernel of the direction is used; the edge intensity map is obtained by calculating the gradient amplitude, specifically:

[0035]

[0036] In the generation of the above multi-scale feature map, in order to enhance the edge information in the feature map, the Sobel operator and the Canny edge detection method in the image processing method are combined. This combination can effectively extract the significant features in the image, especially the key areas such as the roof edge and corner points.

[0037] The step A3 is used to generate candidate key point regions, and the non-maximum suppression (NMS) algorithm is used to process the candidate regions to remove redundant key points with low response strength and retain the feature points with local maximum response. Specifically, Harris corner detection and Shi-Tomasi algorithm are used to identify candidate key point regions and ensure the accuracy of the detected feature points under different conditions.

[0038] The calculation formula for the response intensity of key points is:

[0039]

[0040] in, is the response strength, and The images are and The second derivative of direction;

[0041] Non-maximum suppression (NMS): The NMS algorithm is used to remove redundant key points with low response intensity. The formula of the NMS algorithm is:

[0042]

[0043] in, is the response intensity after NMS processing;

[0044] The multi-scale feature fusion processing in step A4 is to independently extract features and detect key points at different scales. The scale space theoretical formula is:

[0045]

[0046] in, is in scale The image below, is a Gaussian filter, is the input image; key point fusion, which fuses the key points at different scales to obtain a complete and accurate set of key points;

[0047] In the above multi-scale feature fusion process, multi-scale image pyramid technology is used. Features are extracted and key points are detected independently at different scales, and then the results at these scales are fused to obtain a complete and accurate set of key points. The application of scale space theory ensures the scale invariance of key point detection, so that the key points remain consistent under different image conditions.

[0048] In step A5, the refinement process is performed by performing sub-pixel level adjustment through local gradient calculation, and the adjustment formula is:

[0049]

[0050]

[0051] in, and are the adjusted coordinates, and is the gradient of the key point, and It is the second-order derivative. Geometric consistency verification eliminates false positive points that do not match the roof structure and ensures the accuracy of key points. Geometric consistency verification calculates the geometric relationship of key points to ensure that they are consistent with the actual structural characteristics of the roof.

[0052] In the above-mentioned refinement processing, the positioning accuracy of key points is improved through sub-pixel level adjustment. This process combines local gradients and image details to accurately locate key points. At the same time, geometric consistency verification is performed to eliminate false detection points that do not conform to the roof structure, ensuring that the retained key points accurately represent the outline and structural characteristics of the roof.

[0053] Furthermore, generating the descriptor vector in step S2 includes the following steps:

[0054] B1: Define local region of interest (ROI)

[0055] For each detected key point, a local region of interest ROI is defined to extract local features around it; the size and shape of the ROI are adjusted according to the scale and response strength of the key point to ensure that the descriptor can capture enough local information around the key point; the ROI is defined as assuming that the coordinates of the key point are , and the scale of the key point is s, then the range of ROI is defined as:

[0056]

[0057] in, is the radius of the ROI, usually proportional to the scale s of the keypoint;

[0058] B2: Extracting local features

[0059] In the ROI area, a convolutional neural network (CNN) is used to extract local features and generate feature maps. :

[0060]

[0061] in, It is the feature map in the ROI area. is the weight of the convolution kernel, is the displacement of the convolution kernel, is the image pixel value within the ROI area;

[0062] The feature map contains the color, texture and structure information around the key points, which is crucial for the generation of descriptors. Convolutional neural networks effectively extract and encode local features through multi-layer convolution operations.

[0063] B3: Feature Encoding

[0064] The extracted local features are encoded to generate a descriptor vector. The descriptor vector is a fixed-length vector constructed by aggregating and reducing the dimension of local features; the principal component analysis (PCA) technique is used to reduce the feature dimension and retain the most important feature information;

[0065] Specifically, principal component analysis is used to reduce the dimension and transform the feature map The local features in are encoded as descriptor vectors , specifically:

[0066]

[0067] in, is the generated descriptor vector, PCA is a dimensionality reduction technique used to map the high-dimensional features of the feature map to a low-dimensional space;

[0068] B4: Aggregate all local features within the ROI to generate a descriptor vector. The formula is as follows:

[0069]

[0070] in, is the number of feature points in the ROI area, is the aggregated descriptor vector.

[0071] Furthermore, in step B4, the aggregated descriptor vector is normalized and feature fused, as follows:

[0072] Normalization: Normalize the descriptor vector to unit length using L2 normalization:

[0073]

[0074] in, is the normalized descriptor vector, is the L2 norm (i.e., Euclidean distance) of the descriptor vector;

[0075] Feature fusion: fuse multiple descriptor information, concatenate or weightedly fuse descriptor vectors from different scales or different feature extraction models:

[0076]

[0077] in, is the fused descriptor vector, and are descriptors from different sources, and is the weight factor.

[0078] Feature normalization techniques are used to normalize the descriptor vector to ensure its stability under different conditions. In addition, feature fusion methods are used to combine multiple descriptor information to enhance the expressiveness and robustness of the descriptor.

[0079] After normalization and feature fusion processing, descriptor storage and indexing are required. The generated descriptor vectors will be stored and indexed for subsequent key point matching and similarity comparison.

[0080] Furthermore, the step S3 is specifically as follows:

[0081] C1: Input feature preparation

[0082] The key point information and descriptors extracted by the key point detection module are fused with the original satellite image. The key point information usually includes the location and feature descriptors of the key points. The descriptor vector captures the local image features. The key point information is fused with the original image features through feature cascading or weighted fusion. The formula is as follows:

[0083]

[0084] in, is the fused feature map, is the feature map of the original image, It is the feature map obtained by key point extraction. is the fusion weight factor;

[0085] C2: Semantic Segmentation Network Enhancement

[0086] The fused feature map is input into the semantic segmentation network to enhance the segmentation capability of the roof area;

[0087] Network input: The input feature map of the semantic segmentation network is , the output of the network is the class prediction probability map for each pixel ;

[0088]

[0089] in, A function representing a semantic segmentation network, is the predicted probability distribution;

[0090] Loss function: The cross entropy loss function is used to optimize the semantic segmentation network, calculating the difference between the predicted probability and the true label:

[0091]

[0092] in, is the semantic segmentation loss, is the actual category label, is the probability of predicting category c, is the total number of categories;

[0093] C3: Keypoint Enhancement

[0094] The ability of the segmentation network to resolve detailed features is further enhanced through key point information. This is achieved by integrating the key point location information into the attention mechanism in the network. An attention module is introduced into the network to enhance the response of the feature map in the area around the key point;

[0095] Key point enhancement is achieved by integrating key point position information into the attention mechanism in the network. Assume that the attention map is , and its calculation formula is:

[0096]

[0097] in, is the activation function, is the attention weight matrix, is the bias term;

[0098] By applying the attention map to the feature map:

[0099]

[0100] in, is the enhanced feature map;

[0101] C4: Boundary refinement

[0102] Refine the boundaries in the segmentation results to improve the accuracy of boundary recognition in the roof area;

[0103] C5: Regional Accuracy Verification

[0104] The enhanced segmentation results are checked using a geometric consistency verification method to ensure that the segmentation results are consistent with the actual roof geometry.

[0105] Furthermore, in step C4, the conditional random field CRF is used to optimize the boundary to achieve boundary refinement, and CRF is applied to the segmentation probability map to further adjust the segmentation result so that the label consistency between adjacent pixels is higher. The energy function formula of CRF is:

[0106]

[0107] in, is the data item energy, is the similarity term for adjacent pixel pairs, and It is a pixel tag.

[0108] Furthermore, in step C5, the result is verified by calculating the geometric overlap between the segmented area and the actual roof structure:

[0109]

[0110] in, is the Jaccard index, and are the predicted segmentation and true label regions, respectively.

[0111] Furthermore, the step S4 is specifically as follows:

[0112] D1: Establishing feature database

[0113] In order to effectively match and identify key points in images, it is necessary to establish a feature database that contains descriptor vectors of all known targets. These descriptors are extracted from training images and represent the characteristic information of various targets. The purpose of constructing a feature database is to provide a comprehensive reference set for efficient feature matching in practical applications. The descriptor vectors in the database provide benchmark data for subsequent matching and recognition;

[0114] Assume that the database contains known targets, each The descriptor vector is , feature database It is expressed as:

[0115]

[0116] Among them, each is the descriptor vector of the known target;

[0117] D2: Descriptor Matching

[0118] Each key point descriptor in the image to be processed is matched to find the descriptor that is most similar to the descriptor in the feature database; the matching process uses a distance metric, and the present invention uses Euclidean distance and cosine similarity to evaluate the similarity between descriptors. In this way, the most similar descriptor in the database can be found, and the possible category or position of the key point to be processed can be determined.

[0119] Specifically, assuming that the key point descriptor in the image to be processed is , with each descriptor in the database The similarity is calculated by Euclidean distance and cosine similarity; the formulas for Euclidean distance and cosine similarity are as follows:

[0120]

[0121]

[0122] in, and is the L2 norm of the descriptor;

[0123] Matching results, select the descriptor with the smallest distance or the highest similarity as the matching result, the matching method is as follows:

[0124]

[0125]

[0126] D3: Apply matching strategy

[0127] In order to improve the accuracy of matching, the present invention introduces matching strategies to further improve the matching accuracy, including nearest neighbor matching (NN) and nearest neighbor ratio test. The nearest neighbor matching identifies key points by finding the best match of the descriptor in the feature database. If the similarity of the found match exceeds a predetermined threshold, the match is considered successful. The nearest neighbor ratio test further filters out false matches by calculating the ratio between the nearest neighbor match and the next neighbor match, thereby ensuring the accuracy of recognition;

[0128] Nearest neighbor matching: find the descriptor in the database that is most similar to the query descriptor; if the similarity of the match exceeds a predetermined threshold, the match is considered successful:

[0129]

[0130] Nearest neighbor ratio test: Calculate the ratio between the nearest neighbor match and the next neighbor match to filter out false matches; assume that the descriptor of the nearest neighbor is and the next neighbor descriptor is , then the ratio test formula is:

[0131]

[0132] If the ratio If the ratio is less than a predetermined threshold, the match is considered valid;

[0133] D4: Identification of matching results

[0134] By comparing the matched descriptors with known target categories, the categories of key points in the image to be processed can be determined. This category information helps to infer regional features in the entire image, such as the specific structure of the roof or the type of area. The matching results provide specific information about different areas in the image.

[0135] Specifically, the category of the key points in the image to be processed is determined by comparing the matched descriptors with the target categories in the database:

[0136]

[0137] in, is the predicted category, Is the descriptor in the database The category label of the image is used to infer the regional features in the entire image. If the key point is identified as a certain type of roof, the structural type of the entire roof area is inferred.

[0138] D5: Local geometry verification and global consistency check

[0139] In order to improve the robustness and recognition accuracy of the system, the present invention also combines local geometry verification and global consistency check. Local geometry verification verifies the accuracy of matching by comparing the geometric positions of key points in the image, ensuring that the key point matching conforms to the actual geometric structure. The global consistency check ensures that the matching result is consistent with the overall structural features of the image, thereby further improving the reliability and accuracy of recognition.

[0140] Verify geometric consistency; verify the accuracy of matching by comparing the geometric positions of key points in the image; the formula for calculating the geometric distance of matching points is as follows:

[0141]

[0142] if If it is less than a certain threshold, the match is considered accurate;

[0143] Global consistency check: Check whether the matching result is consistent with the overall structural features of the image, for example, by calculating the overlap between the segmented region and the overall structure:

[0144]

[0145] in, and are the predicted segmentation and true label regions, respectively.

[0146] The present invention also provides a rooftop photovoltaic recognition and evaluation system based on key point detection and enhanced semantic segmentation, the system comprising a memory and a processor; wherein,

[0147] The memory is used to store computer program instructions that can be executed on the processor;

[0148] The processor is used to execute the steps of the rooftop photovoltaic identification and evaluation method based on key point detection enhanced semantic segmentation when running the computer program instructions.

[0149] The present invention also provides a computer storage medium, which stores a program of a rooftop photovoltaic identification and evaluation method based on key point detection enhanced semantic segmentation. When the program of the rooftop photovoltaic identification and evaluation method based on key point detection enhanced semantic segmentation is executed by at least one processor, the steps of the rooftop photovoltaic identification and evaluation method based on key point detection enhanced semantic segmentation are implemented.

[0150] The method of the present invention combines advanced key point detection technology with semantic segmentation technology to form a comprehensive solution. Specifically, the present invention mainly includes the following parts:

[0151] Key point extraction: The key point detection module extracts the significant feature points of the roof from the satellite image. These feature points include the edges and corners of the roof, which can accurately indicate the outline and structural features of the roof, thus providing accurate information for subsequent image processing.

[0152] Descriptor generation: Generate a descriptor vector for each detected key point. The descriptor vector includes local image features around the key point, such as color, texture, and structural information, which are used for key point matching and similarity comparison to enhance the recognition ability of feature points. These descriptor vectors provide strong support for feature matching and region division.

[0153] Semantic segmentation enhancement: Combine key point information with a pre-trained semantic segmentation network. Through the location information of key points, the boundaries of the roof area are accurately divided and the detailed features are enhanced. This step not only improves the accuracy of semantic segmentation, but also enhances the ability to recognize roof features in complex environments.

[0154] Feature matching and identification: Descriptor vectors are used to match key points and compare similarities to identify the outline and important structural features of the roof. Roof classification and identification are achieved based on the matching results to obtain the final roof segmentation results. This process helps to effectively evaluate the potential of distributed rooftop photovoltaic systems.

[0155] Photovoltaic potential assessment: Based on the accurate identification of the roof area, the photovoltaic potential of the roof is further analyzed. By integrating the geometric information of the roof and the installation requirements of the photovoltaic system, the installation plan and design recommendations of the photovoltaic system are generated.

[0156] The present invention optimizes the model from two directions. On the one hand, by combining deep learning with traditional image processing technology, the key point features of the roof are accurately extracted to improve the recognition ability of the model in different environments; on the other hand, by enhancing the detail feature parsing ability and descriptor matching accuracy of the semantic segmentation network, the model's description accuracy of complex roof areas is improved.

[0157] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0158] 1. High-precision key point detection, by combining deep learning with traditional image processing technology, accurately detects the significant feature points of the roof, improving the detection accuracy in different environments and scales.

[0159] 2. Refined descriptor generation, using convolutional neural network (CNN) and dimensionality reduction technology to generate detailed and stable descriptor vectors, effectively capturing local features around key points and improving the accuracy of feature matching.

[0160] 3. Enhanced semantic segmentation accuracy: Combining key point information with an improved semantic segmentation network, using attention mechanism and conditional random field (CRF), significantly improves the segmentation details and boundary recognition capabilities of the roof area.

[0161] 4. Accurate feature matching and recognition, by building a feature database and applying advanced matching strategies (such as nearest neighbor matching and ratio test), combined with local geometry verification and global consistency check, to ensure efficient and reliable key point identification and region classification.

[0162] 5. Strong robustness and adaptability. The multi-scale image pyramid technology is used to enhance the processing ability of satellite images in complex backgrounds and overcome the recognition challenges of traditional methods in complex environments.

[0163] 6. Broad application prospects. The high accuracy and robustness of the method of the present invention make it not only suitable for rooftop photovoltaic potential assessment, but also have application potential in many fields such as urban planning and building monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0164] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0165] The present invention is further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, various equivalent forms of modifications to the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0166] like Figure 1 As shown, the present invention provides a rooftop photovoltaic recognition and evaluation method based on key point detection enhanced semantic segmentation, comprising the following steps:

[0167] S1: Extract the significant feature points of the roof as key points in the satellite image through the key point detection module;

[0168] S2: For each detected key point, a descriptor vector is generated through the descriptor generation module;

[0169] S3: Perform semantic segmentation enhancement through the semantic segmentation enhancement module, combine the generated key point information with the pre-trained semantic segmentation network, accurately divide the boundary of the roof area through the position information of the key points, enhance the detail features, and use the key point information to optimize the initial segmentation result of the semantic segmentation network;

[0170] S4: Perform feature matching and recognition through the feature matching and recognition module, use the descriptor vector to match and compare the key points to identify the outline and important structural features of the roof, classify and recognize the roof according to the matching results, and obtain the final roof segmentation result;

[0171] S5: Photovoltaic potential assessment: Based on the accurate identification of the roof area, further analyze the photovoltaic potential of the roof; by integrating the geometric information of the roof and the installation requirements of the photovoltaic system, generate installation planning and design recommendations for the photovoltaic system.

[0172] In step S1, the key point extraction utilizes a key point detection module that combines deep learning and traditional image processing algorithms to extract significant feature points of the roof in the satellite image. The feature points include the edges and corners of the roof, which can accurately indicate the contour and structural features of the roof. The specific method includes:

[0173] A1: Preprocess the input satellite images to optimize image quality and improve the accuracy of subsequent processing;

[0174] Image preprocessing includes noise removal, color correction and scale normalization. Noise removal uses Gaussian filtering and bilateral filtering techniques to reduce noise and artifacts in the image. The purpose of color correction and brightness adjustment is to ensure the contrast and color uniformity of the image and adapt to image processing under different lighting conditions. Scale normalization ensures the consistency of images at different resolutions, thereby improving the stability of key point detection;

[0175] The specific process of image preprocessing is as follows:

[0176] A1-1: Use Gaussian filtering to smooth the image. The formula of Gaussian filter is:

[0177]

[0178] in, is the weight of the Gaussian filter, and are image coordinates, is the standard deviation of the Gaussian filter; the noise is smoothed by the convolution operation;

[0179] A1-2: Adjust the brightness and contrast of the image through linear transformation. The correction formula is:

[0180]

[0181] in, is the corrected pixel value, is the original pixel value, is the contrast adjustment factor, is the brightness adjustment factor;

[0182] A1-3: Use resampling technology to adjust the image to a uniform resolution; for image scaling, use the bilinear interpolation formula:

[0183]

[0184] in, and is the interpolation factor, is the pixel value after scaling.

[0185] A2: Perform feature detection on the preprocessed image: Use a convolutional neural network to extract deep features in the image and generate a multi-scale feature map; perform edge detection on the feature map and use the Sobel operator to enhance the image edge information;

[0186] The generated multi-scale feature map is represented as:

[0187]

[0188] in, is the feature map, is the convolution kernel weight, is the displacement of the convolution kernel, is the input image;

[0189] The convolution operation formula of the Sobel operator is:

[0190]

[0191]

[0192] in, and The Sobel operator is and The convolution kernel of the direction is used; the edge intensity map is obtained by calculating the gradient amplitude, specifically:

[0193]

[0194] In the generation of the above multi-scale feature map, in order to enhance the edge information in the feature map, the Sobel operator and the Canny edge detection method in the image processing method are combined. This combination can effectively extract the significant features in the image, especially the key areas such as the roof edge and corner points.

[0195] A3: Calculate the response strength of each pixel through the feature map and generate the key point candidate area;

[0196] Key point candidate region generation, using the non-maximum suppression (NMS) algorithm to process the candidate region to remove redundant key points with low response strength and retain the feature points with local maximum response. Specifically, Harris corner detection and Shi-Tomasi algorithm are used to identify candidate key point regions and ensure the accuracy of the detected feature points under different conditions;

[0197] The response intensity calculation formula of the key point is:

[0198]

[0199] in, is the response strength, and The images are and The second derivative of direction;

[0200] Non-maximum suppression (NMS): The NMS algorithm is used to remove redundant key points with low response intensity. The formula of the NMS algorithm is:

[0201]

[0202] in, is the response intensity after NMS processing;

[0203] A4: Use image pyramid technology to perform multi-scale feature fusion processing to improve the stability of key points;

[0204] Multi-scale feature fusion processing extracts features and detects key points independently at different scales. The scale space theoretical formula is:

[0205]

[0206] in, is in scale The image below, is a Gaussian filter, is the input image; key point fusion, which fuses the key points at different scales to obtain a complete and accurate set of key points;

[0207] In the above multi-scale feature fusion process, multi-scale image pyramid technology is used. Features are extracted and key points are detected independently at different scales, and then the results at these scales are fused to obtain a complete and accurate set of key points. The application of scale space theory ensures the scale invariance of key point detection, so that the key points remain consistent under different image conditions.

[0208] A5: Perform fine processing on the detected key points to improve positioning accuracy.

[0209] The refinement process is performed by calculating the local gradient to adjust the sub-pixel level. The adjustment formula is:

[0210]

[0211]

[0212] in, and are the adjusted coordinates, and is the gradient of the key point, and It is the second-order derivative. Geometric consistency verification eliminates false detection points that do not conform to the roof structure and ensures the accuracy of key points. Geometric consistency verification calculates the geometric relationship of key points to ensure that they are consistent with the actual structural characteristics of the roof;

[0213] In the above-mentioned refinement processing, the positioning accuracy of key points is improved through sub-pixel level adjustment. This process combines local gradients and image details to accurately locate key points. At the same time, geometric consistency verification is performed to eliminate false detection points that do not conform to the roof structure, ensuring that the retained key points accurately represent the outline and structural characteristics of the roof.

[0214] Generating a descriptor vector in step S2 includes the following steps:

[0215] B1: Define local region of interest (ROI)

[0216] For each detected key point, a local region of interest ROI is defined to extract local features around it; the size and shape of the ROI are adjusted according to the scale and response strength of the key point to ensure that the descriptor can capture enough local information around the key point; the ROI is defined as assuming that the coordinates of the key point are , and the scale of the key point is s, then the range of ROI is defined as:

[0217]

[0218] in, is the radius of the ROI, usually proportional to the scale s of the keypoint;

[0219] B2: Extracting local features

[0220] In the ROI area, a convolutional neural network (CNN) is used to extract local features and generate feature maps. :

[0221]

[0222] in, It is the feature map in the ROI area. is the weight of the convolution kernel, is the displacement of the convolution kernel, is the image pixel value within the ROI area;

[0223] The feature map contains the color, texture and structure information around the key points, which is crucial for the generation of descriptors. Convolutional neural networks effectively extract and encode local features through multi-layer convolution operations.

[0224] B3: Feature Encoding

[0225] The extracted local features are encoded to generate a descriptor vector. The descriptor vector is a fixed-length vector constructed by aggregating and reducing the dimension of local features; the principal component analysis (PCA) technique is used to reduce the feature dimension and retain the most important feature information;

[0226] Specifically, principal component analysis is used to reduce the dimension and transform the feature map The local features in are encoded as descriptor vectors , specifically:

[0227]

[0228] in, is the generated descriptor vector, PCA is a dimensionality reduction technique used to map the high-dimensional features of the feature map to a low-dimensional space;

[0229] B4: Aggregate all local features within the ROI to generate a descriptor vector. The formula is as follows:

[0230]

[0231] in, is the number of feature points in the ROI area, is the aggregated descriptor vector;

[0232] B5: Standardization and feature fusion processing, as follows:

[0233] Normalization: Normalize the descriptor vector to unit length using L2 normalization:

[0234]

[0235] in, is the normalized descriptor vector, is the L2 norm (i.e., Euclidean distance) of the descriptor vector;

[0236] Feature fusion: fuse multiple descriptor information, concatenate or weightedly fuse descriptor vectors from different scales or different feature extraction models:

[0237]

[0238] in, is the fused descriptor vector, and are descriptors from different sources, and is the weight factor.

[0239] Feature normalization techniques are used to normalize the descriptor vector to ensure its stability under different conditions. In addition, feature fusion methods are used to combine multiple descriptor information to enhance the expressiveness and robustness of the descriptor.

[0240] B6: Descriptor storage and indexing, the generated descriptor vectors will be stored and indexed for subsequent key point matching and similarity comparison.

[0241] Step S3 is specifically as follows:

[0242] C1: Input feature preparation

[0243] The key point information and descriptors extracted by the key point detection module are fused with the original satellite image. The key point information usually includes the location and feature descriptors of the key points. The descriptor vector captures the local image features. The key point information is fused with the original image features through feature cascading or weighted fusion. The formula is as follows:

[0244]

[0245] in, is the fused feature map, is the feature map of the original image, It is the feature map obtained by key point extraction. is the fusion weight factor;

[0246] C2: Semantic Segmentation Network Enhancement

[0247] The fused feature map is input into the semantic segmentation network to enhance the segmentation capability of the roof area;

[0248] Network input: The input feature map of the semantic segmentation network is , the output of the network is the class prediction probability map for each pixel ;

[0249]

[0250] in, A function representing a semantic segmentation network, is the predicted probability distribution;

[0251] Loss function: The cross entropy loss function is used to optimize the semantic segmentation network, calculating the difference between the predicted probability and the true label:

[0252]

[0253] in, is the semantic segmentation loss, is the actual category label, is the probability of predicting category c, is the total number of categories;

[0254] C3: Keypoint Enhancement

[0255] The ability of the segmentation network to resolve detailed features is further enhanced through key point information. This is achieved by integrating the key point location information into the attention mechanism in the network. An attention module is introduced into the network to enhance the response of the feature map in the area around the key point;

[0256] Key point enhancement is achieved by integrating key point position information into the attention mechanism in the network. Assume that the attention map is , and its calculation formula is:

[0257]

[0258] in, is the activation function, is the attention weight matrix, is the bias term;

[0259] By applying the attention map to the feature map:

[0260]

[0261] in, is the enhanced feature map;

[0262] C4: Boundary refinement

[0263] Refine the boundaries in the segmentation results to improve the accuracy of boundary recognition in the roof area;

[0264] Conditional random field (CRF) is used to optimize the boundary to achieve boundary refinement, and CRF is applied to the segmentation probability map to further adjust the segmentation result so that the label consistency between adjacent pixels is higher; the energy function formula of CRF is:

[0265]

[0266] in, is the data item energy, is the similarity term for adjacent pixel pairs, and It is a pixel tag.

[0267] C5: Regional Accuracy Verification

[0268] The enhanced segmentation results are checked using the geometric consistency verification method to ensure that the segmentation results are consistent with the actual roof geometry;

[0269] Verify the result by calculating the geometric overlap between the segmented area and the actual roof structure:

[0270]

[0271] in, is the Jaccard index, and are the predicted segmentation and true label regions, respectively.

[0272] Step S4 is specifically as follows:

[0273] D1: Establishing feature database

[0274] In order to effectively match and identify key points in images, it is necessary to establish a feature database that contains descriptor vectors of all known targets. These descriptors are extracted from training images and represent the characteristic information of various targets. The purpose of constructing a feature database is to provide a comprehensive reference set for efficient feature matching in practical applications. The descriptor vectors in the database provide benchmark data for subsequent matching and recognition;

[0275] Assume that the database contains known targets, each The descriptor vector is , feature database It is expressed as:

[0276]

[0277] Among them, each is the descriptor vector of the known target;

[0278] D2: Descriptor Matching

[0279] Match each key point descriptor in the image to be processed to find the most similar descriptor to the descriptor in the feature database; the matching process uses a distance metric, and the present invention uses Euclidean distance and cosine similarity to evaluate the similarity between descriptors. In this way, the most similar descriptor in the database can be found, and the possible category or position of the key point to be processed can be determined;

[0280] Specifically, assuming that the key point descriptor in the image to be processed is , with each descriptor in the database The similarity is calculated by Euclidean distance and cosine similarity; the formulas for Euclidean distance and cosine similarity are as follows:

[0281]

[0282]

[0283] in, and is the L2 norm of the descriptor;

[0284] Matching results, select the descriptor with the smallest distance or the highest similarity as the matching result, the matching method is as follows:

[0285]

[0286]

[0287] D3: Apply matching strategy

[0288] In order to improve the accuracy of matching, the present invention introduces matching strategies to further improve the matching accuracy, including nearest neighbor matching (NN) and nearest neighbor ratio test. The nearest neighbor matching identifies key points by finding the best match of the descriptor in the feature database. If the similarity of the found match exceeds a predetermined threshold, the match is considered successful. The nearest neighbor ratio test further filters out false matches by calculating the ratio between the nearest neighbor match and the next neighbor match, thereby ensuring the accuracy of recognition;

[0289] Nearest neighbor matching: find the descriptor in the database that is most similar to the query descriptor; if the similarity of the match exceeds a predetermined threshold, the match is considered successful:

[0290]

[0291] Nearest neighbor ratio test: Calculate the ratio between the nearest neighbor match and the next neighbor match to filter out false matches; assume that the descriptor of the nearest neighbor is and the next neighbor descriptor is , then the ratio test formula is:

[0292]

[0293] If the ratio If the ratio is less than a predetermined threshold, the match is considered valid;

[0294] D4: Identification of matching results

[0295] By comparing the matched descriptors with known target categories, the categories of key points in the image to be processed can be determined. This category information helps to infer regional features in the entire image, such as the specific structure of the roof or the type of area. The matching results provide specific information about different areas in the image;

[0296] Specifically, the category of the key points in the image to be processed is determined by comparing the matched descriptors with the target categories in the database:

[0297]

[0298] in, is the predicted category, Is the descriptor in the database The category label of the image is used to infer the regional features in the entire image. If the key point is identified as a certain type of roof, the structural type of the entire roof area is inferred.

[0299] D5: Local geometry verification and global consistency check

[0300] In order to improve the robustness and recognition accuracy of the system, the present invention also combines local geometry verification and global consistency check. Local geometry verification verifies the accuracy of matching by comparing the geometric positions of key points in the image, ensuring that the key point matching conforms to the actual geometric structure. The global consistency check ensures that the matching result is consistent with the overall structural features of the image, thereby further improving the reliability and accuracy of recognition.

[0301] Verify geometric consistency; verify the accuracy of matching by comparing the geometric positions of key points in the image; the formula for calculating the geometric distance of matching points is as follows:

[0302]

[0303] if If it is less than a certain threshold, the match is considered accurate;

[0304] Global consistency check: Check whether the matching result is consistent with the overall structural features of the image, for example, by calculating the overlap between the segmented region and the overall structure:

[0305]

[0306] in, and are the predicted segmentation and true label regions, respectively.

[0307] This embodiment also provides a rooftop photovoltaic identification and evaluation system based on key point detection enhanced semantic segmentation, the system includes a network interface, a memory and a processor; wherein the network interface is used to realize the reception and transmission of signals in the process of sending and receiving information between other external network elements; the memory is used to store computer program instructions that can be run on the processor; the processor is used to execute the steps of the above-mentioned consensus method when running the computer program instructions.

[0308] The present embodiment also provides a computer storage medium, which stores a computer program, and the method described above can be implemented when the processor executes the computer program. The computer readable medium can be considered to be tangible and non-temporary. Non-limiting examples of non-temporary tangible computer-readable media include non-volatile memory circuits (such as flash memory circuits, erasable programmable read-only memory circuits or mask read-only memory circuits), volatile memory circuits (such as static random access memory circuits or dynamic random access memory circuits), magnetic storage media (such as analog or digital tapes or hard drives) and optical storage media (such as CDs, DVDs or Blu-ray discs), etc. The computer program includes processor executable instructions stored on at least one non-temporary tangible computer-readable medium. The computer program may also include or rely on stored data. The computer program may include a basic input / output system (BIOS) that interacts with the hardware of a special-purpose computer, a device driver that interacts with a specific device of a special-purpose computer, one or more operating systems, user applications, background services, background applications, etc.

[0309] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0310] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0311] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1A function specified in one or more boxes.

[0312] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

Claims

1. Rooftop photovoltaic recognition and evaluation method based on key point detection and enhanced semantic segmentation, characterized by: The steps include: S1: Extract the significant feature points of the roof in the satellite image as key points; S2: Generate a descriptor vector for each detected key point; S3: Combine the generated key point information with the pre-trained semantic segmentation network, and use the key point information to optimize the initial segmentation results of the semantic segmentation network; S4: Use the descriptor vector to match and compare the key points to identify the outline and important structural features of the roof, classify and identify the roof based on the matching results, and obtain the final roof segmentation result; Step S3 is specifically as follows: C1: Fusion of key point information with original image features, achieved through feature cascading or weighted fusion, the formula is as follows: F combined (x,y)=γ·I(x,y)+(1-γ)·F keypoints (x,y) Among them, F combined (x, y) is the fused feature map, I(x, y) is the feature map of the original image, and F keypoints (x, y) is the feature map obtained by key point extraction, and γ is the fusion weight factor; C2: Input the fused feature map into the semantic segmentation network to enhance the segmentation capability of the roof area; Network input: The input feature map of the semantic segmentation network is F combined , the output of the network is the class prediction probability map P for each pixel seg (x,y); P seg (x,y)=f(F combined (x,y)) Among them, f represents the function of the semantic segmentation network, P seg (x,y) is the predicted probability distribution; Loss function: The cross entropy loss function is used to optimize the semantic segmentation network, calculating the difference between the predicted probability and the true label: Among them, L seg is the semantic segmentation loss, y c is the actual category label, P seg,c is the probability of predicting category c, C is the total number of categories; C3: Key point enhancement is achieved by integrating the key point position information into the attention mechanism in the network. Assuming the attention map is A(x,y), its calculation formula is: A(x,y)=σ(W att ·F combined (x,y)+b att ) Among them, σ is the activation function, W att is the attention weight matrix, b att is the bias term; By applying the attention map to the feature map: F enhanced (x,y)=A(x,y)·F combined (x,y) Among them, F enhanced (x,y) is the enhanced feature map; C4: refine the boundary in the segmentation result to improve the boundary recognition accuracy of the roof area; C5: Use the geometric consistency verification method to check the enhanced segmentation results to ensure that the segmentation results are consistent with the geometric structure of the actual roof.

2. The rooftop photovoltaic recognition and evaluation method based on key point detection and enhanced semantic segmentation according to claim 1 is characterized in that: The method for extracting key points in step S1 includes: A1: Preprocess the input satellite images to optimize image quality and improve the accuracy of subsequent processing; A2: Perform feature detection on the preprocessed image: Use a convolutional neural network to extract deep features in the image and generate a multi-scale feature map; perform edge detection on the feature map and use the Sobel operator to enhance the image edge information; A3: Calculate the response strength of each pixel through the feature map and generate the key point candidate area; A4: Use image pyramid technology to perform multi-scale feature fusion processing to improve the stability of key points; A5: Perform fine processing on the detected key points to improve positioning accuracy.

3. The rooftop photovoltaic recognition and evaluation method based on key point detection and enhanced semantic segmentation according to claim 2 is characterized in that: The multi-scale feature map generated in step A2 is represented as: Among them, F(x,y) is the feature map, W i is the convolution kernel weight, d i is the displacement of the convolution kernel, I(x,y) is the input image; The convolution operation formula of the Sobel operator is: Among them, G x and G y are the convolution kernels of the Sobel operator in the x and y directions respectively; the edge intensity map is obtained by calculating the gradient amplitude.

4. The rooftop photovoltaic identification and evaluation method based on key point detection and enhanced semantic segmentation according to claim 2 is characterized in that: The calculation formula for the response intensity of the key points in step A3 is: Where R(x,y) is the response intensity, and are the second-order derivatives of the image in the x and y directions respectively; The NMS algorithm is applied to remove redundant key points with low response intensity. The formula of the NMS algorithm is: Among them, R nms (x,y) is the response intensity after NMS processing.

5. The rooftop photovoltaic recognition and evaluation method based on key point detection and enhanced semantic segmentation according to claim 2 is characterized in that: In step A5, the refinement process is performed by calculating the local gradient to adjust the sub-pixel level. The adjustment formula is: Among them, x adjusted and adjusted are the adjusted coordinates, and is the gradient of the key point, and is the second-order derivative.

6. The rooftop photovoltaic identification and evaluation method based on key point detection and enhanced semantic segmentation according to claim 1 is characterized in that: Generating a descriptor vector in step S2 includes the following steps: B1: For each detected key point, a local region of interest ROI is defined to extract local features around it; the size and shape of the ROI are adjusted according to the scale and response strength of the key point; the definition of the ROI is that assuming the coordinates of the key point are (x0, y0) and the scale of the key point is s, then the range of the ROI is defined as: ROI(x,y)={(x0-δ,y0-δ)≤(x,y)≤(x0+δ,y0+δ)} Where, δ is the radius of the ROI; B2: In the ROI area, obtain local features and generate feature map F ROI (x,y): Among them, F ROI (x, y) is the feature map in the ROI area, W i is the weight of the convolution kernel, d i is the displacement of the convolution kernel, I ROI (x, y) is the image pixel value within the ROI area; B3: Use principal component analysis to reduce the dimension and transform the feature map F ROI The local features in (x, y) are encoded as a descriptor vector d, specifically: d=PCA(F ROI (x,y)) Among them, d is the generated descriptor vector, PCA is a dimensionality reduction technique used to map the high-dimensional features of the feature map to a low-dimensional space; B4: Aggregate all local features within the ROI to generate a descriptor vector. The formula is as follows: Where N is the number of feature points in the ROI area, and d is the aggregated descriptor vector.

7. The rooftop photovoltaic recognition and evaluation method based on key point detection and enhanced semantic segmentation according to claim 6 is characterized in that: In step B4, the aggregated descriptor vector is normalized and feature fused as follows: Normalization: Normalize the descriptor vector to unit length using L2 normalization: Among them, d normalized is the normalized descriptor vector, ||d||2 is the L2 norm of the descriptor vector; Feature fusion: fuse multiple descriptor information, concatenate or weightedly fuse descriptor vectors from different scales or different feature extraction models: d fused =αd1+βd2 Among them, d fused is the fused descriptor vector, d1 and d2 are descriptors from different sources, and α and β are weight factors.

8. The rooftop photovoltaic recognition and evaluation method based on key point detection and enhanced semantic segmentation according to claim 1 is characterized in that: In step C4, conditional random field CRF is used to optimize the boundary to achieve boundary refinement. The energy function formula of CRF is: Among them, φ data (x i ) is the data item energy, φ pair (x i ,x j ) is the similarity term for adjacent pixel pairs, x i and x j It is a pixel tag.

9. The rooftop photovoltaic identification and evaluation method based on key point detection and enhanced semantic segmentation according to claim 1 is characterized in that: In step C5, the result is verified by calculating the geometric overlap between the segmented area and the actual roof structure: Where J(x,y) is the Jaccard index, S pred and S true are the predicted segmentation and true label regions, respectively.

10. The rooftop photovoltaic recognition and evaluation method based on key point detection and enhanced semantic segmentation according to claim 1 is characterized in that: Step S4 is specifically as follows: D1: Establish a feature database: Assume that the database contains N known targets, and the descriptor vector of each target i is d i , the feature database D is expressed as: D={d1,d2,…,d N } Among them, each d i is the descriptor vector of the known target; D2: Descriptor matching: Match each key point descriptor in the processed image to find the descriptor that is most similar to the descriptor in the feature database; Assume that the key point descriptor in the image to be processed is d query , with each descriptor d in the database i The similarity is calculated by Euclidean distance and cosine similarity; the formulas for Euclidean distance and cosine similarity are as follows: d euclidean (d query ,d i )=||d query -d i ||2 Among them, ||d query ||2 and ||d i ||2 is the L2 norm of the descriptor; The descriptor with the smallest distance or the highest similarity is selected as the matching result. The matching method is as follows: D3: Use nearest neighbor matching and nearest neighbor ratio test to further improve matching accuracy; Nearest neighbor matching: find the descriptor in the database that is most similar to the query descriptor; if the similarity of the match exceeds a predetermined threshold, the match is considered successful: Match=True if s cosine (d query ,d match )>Threshold Nearest neighbor ratio test: Calculate the ratio between the nearest neighbor match and the next neighbor match to filter out false matches; assuming the nearest neighbor descriptor is d1 and the next neighbor descriptor is d2, the ratio test formula is: If the ratio r nn If the ratio is less than a predetermined threshold, the match is considered valid; D4: Use the matching results to identify and classify image regions; determine the category of key points in the image to be processed by comparing the matched descriptors with the target categories in the database: C pred =C i for d match =d i Among them, C pred is the predicted category, C i is the descriptor d in the database i The category label of D5: Verify geometric consistency; verify the accuracy of matching by comparing the geometric positions of key points in the image; the formula for calculating the geometric distance of matching points is as follows: If D geom If it is less than a certain threshold, the match is considered accurate; Global consistency check: Check whether the matching results are consistent with the overall structural characteristics of the image.

Images