Window frame structure extraction method and system based on image segmentation and multi-image fusion

Abstract

The invention belongs to the technical field of computer vision, and discloses a window frame structure extraction method and system based on image segmentation and multi-image fusion, and the method comprises the steps: obtaining an orthographic window image of a building elevation, and carrying out the first preprocessing of the orthographic window image; wherein the first preprocessing comprises window clustering grouping and super-resolution processing; performing window frame segmentation on the orthographic window image after the first preprocessing based on an image segmentation algorithm to obtain a segmented window frame mask; fusing the window frame masks of the same cluster based on a multi-graph fusion algorithm to generate a window frame fusion mask; extracting geometric parameters and color parameters of the window frame fusion mask; obtaining a target window frame structure according to the extracted geometric parameters and color parameters; according to the method, the window frame structure corresponding to the orthographic window image can be accurately extracted through the steps of an image segmentation algorithm, a multi-image fusion algorithm, geometric parameter and color parameter extraction and the like.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method and system for extracting window frame structures based on image segmentation and multi-image fusion. Background Technology

[0002] Window detection is a key step in building facade construction analysis and semantic segmentation, and it is of great significance for subsequent steps such as building modeling, energy consumption assessment and style reconstruction.

[0003] Among these, Level of Detail (LOD) modeling is the core framework supporting multi-scale applications of 3D building models, and the fine extraction of window structures is an important step from LOD2 to LOD3. Existing technologies mainly use computer vision methods to process building facade images or point cloud data, employing algorithms such as edge detection, region segmentation, or deep learning to identify window positions and basic outlines, and then redrawing the windows based on the identified structural and geometric features.

[0004] In practical applications, windows on building facades exhibit a high degree of diversity in shape, size, and color. They are also subject to interference from variable lighting conditions, external obstructions (such as vegetation or decorations), and interior scenes (such as curtains or furniture) reflected through the glass panes. This often leads to problems such as insufficient detail, geometric distortion, and detection errors in window frame structure extraction, which severely restricts the application value of window models in LOD3 architectural 3D models.

[0005] Existing technologies still suffer from insufficient precision, geometric distortion, and detection errors in window frame structure extraction. Therefore, existing technologies need further improvement. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a method and system for extracting window frame structure based on image segmentation and multi-image fusion, in order to solve the problems of insufficient precision, geometric distortion and detection errors in existing window frame structure extraction.

[0007] The technical solution adopted by this invention to solve the technical problem is as follows: In a first aspect, the present invention provides a method for extracting window frame structures based on image segmentation and multi-image fusion, comprising: Acquire orthophotos of the building facade windows and perform a first preprocessing on the orthophotos; wherein the first preprocessing includes window clustering and super-resolution processing; The window frame is segmented based on the image segmentation algorithm to obtain the segmented window frame mask. The window frame masks of the same cluster are fused based on the multi-graph fusion algorithm to generate a window frame fusion mask; Extract the geometric and color parameters of the window frame fusion mask; The target window frame structure is obtained based on the extracted geometric and color parameters.

[0008] In one implementation, the step of segmenting the window frame of the first preprocessed orthophoto window image using an image segmentation algorithm to obtain a segmented window frame mask includes: The orthophoto image after the first preprocessing is then subjected to a second preprocessing; wherein, the second preprocessing includes color space conversion, Channel enhancement, image correction, and bilateral filtering denoising; The second preprocessed orthophoto window image is sampled to obtain a dynamic cue point set; Based on the dynamic cue point set, the orthophoto window image is segmented into a window frame to obtain a segmented window frame mask.

[0009] In one implementation, sampling the second preprocessed orthophoto window image to obtain a dynamic cue point set includes: Adaptive sampling of grid points is performed on the second preprocessed orthophoto window image to generate a grid point set; Extract the edge structure of the second preprocessed orthophoto window image to generate the first edge point set; Perform line detection and bilateral brightness comparison on the first edge point set to generate a second edge point set; Enhanced sampling points of the image edge are generated within the strip-shaped region of the edge structure to obtain a third edge point set; A dynamic cue point set is obtained, which includes the grid point set, the second edge point set, and the third edge point set.

[0010] In one implementation, the step of segmenting the orthophoto window image based on the dynamic cue point set to obtain a segmented window frame mask includes: Based on the dynamic cue point set, the orthophoto window image is segmented into a window frame to obtain a first window frame mask set. Based on color similarity and spatial proximity, similar window frame masks in the first window frame mask set are merged to obtain the second window frame mask set; The window frame masks in the second window frame mask set are denoised to obtain the third window frame mask set. The window frame mask is scored based on area threshold, boundary fit, and void ratio, and the window frame mask with the highest score in the third set of window frame masks is output.

[0011] In one implementation, the step of fusing window frame masks from the same cluster based on a multi-graph fusion algorithm to generate a window frame fusion mask includes: Obtain the window frame mask of the same cluster; Adaptive affine alignment is performed on window frame masks of the same cluster to obtain an aligned group of window frame masks. Consensus voting is performed on the aligned window frame image group to generate a consensus mask; Image fusion of the aligned window frame mask group based on the consensus mask is used to obtain the window frame fusion mask.

[0012] In one implementation, extracting the geometric and color parameters of the window frame fusion mask includes: A third preprocessing is performed on the window frame fusion mask; wherein, the third preprocessing includes image scaling, adding black borders, morphological operations, and maximum connected component preservation; Linearity detection and correction are performed on the window frame fusion mask after the third preprocessing step. The geometric parameters of the window frame fusion mask are obtained by performing intersection cluster structured analysis on the window frame fusion mask after line detection and correction. Color attributes are extracted from the window frame fusion mask after line detection and correction to obtain the color parameters of the window frame fusion mask.

[0013] In one implementation, the step of performing intersection cluster structured analysis on the window frame fusion mask after line detection and correction to obtain the geometric parameters of the window frame fusion mask includes: Obtain the effective intersection points of the horizontal and vertical line segments of the window frame fusion mask after line detection and correction; Two-dimensional spatial clustering is performed on the effective intersection points of the horizontal and vertical line segments to form intersection point clusters; Generate the bounding rectangle of each intersection cluster, and perform row and column grouping clustering on the rectangles; Based on the geometric and positional features of the rectangular boxes after row and column grouping, the geometric parameters of the window frame fusion mask are calculated.

[0014] Secondly, the present invention provides a window frame structure extraction system based on image segmentation and multi-image fusion, comprising: An image processing module is used to acquire orthophotos of windows on a building facade and perform a first preprocessing on the orthophotos; wherein the first preprocessing includes window clustering and super-resolution processing. The image segmentation module is used to segment the window frame of the first preprocessed orthophoto window image based on the image segmentation algorithm to obtain the segmented window frame mask. The image fusion module is used to fuse window frame masks of the same cluster based on a multi-image fusion algorithm to generate a window frame fusion mask. The parameter extraction module is used to extract the geometric and color parameters of the window frame fusion mask; The image generation module is used to obtain the target window frame structure based on the extracted geometric and color parameters.

[0015] Thirdly, the present invention provides a terminal, comprising: a processor and a memory, wherein the memory stores a window frame structure extraction program based on image segmentation and multi-image fusion, wherein when the window frame structure extraction program based on image segmentation and multi-image fusion is executed by the processor, it is used to implement the operation of the window frame structure extraction method based on image segmentation and multi-image fusion as described in the first aspect.

[0016] Fourthly, the present invention also provides a computer-readable storage medium storing a window frame structure extraction program based on image segmentation and multi-image fusion, wherein the window frame structure extraction program based on image segmentation and multi-image fusion, when executed by a processor, is used to implement the operation of the window frame structure extraction method based on image segmentation and multi-image fusion as described in the first aspect.

[0017] The present invention, by employing the above technical solution, has the following effects: This invention acquires orthophotos of building facades and windows, and performs a first preprocessing step on these images. This first preprocessing includes window clustering and super-resolution processing. The preprocessed orthophotos are then segmented into window frames using an image segmentation algorithm to obtain segmented window frame masks. These masks from the same cluster are then fused using a multi-image fusion algorithm to generate a fused window frame mask. Geometric and color parameters of the fused window frame mask are extracted. Based on these extracted parameters, the target window frame structure is obtained. This invention, through image segmentation, multi-image fusion, and geometric and color parameter extraction, can accurately extract the window frame structure corresponding to an orthophoto. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0019] Figure 1 This is a flowchart of the window frame structure extraction method based on image segmentation and multi-image fusion in this invention.

[0020] Figure 2 This is a flowchart of the window frame segmentation method of the present invention.

[0021] Figure 3 This is a schematic diagram of a grid point set generated in one implementation of the present invention.

[0022] Figure 4 This is a schematic diagram of the straight line offset prompt point generated in one implementation of the present invention.

[0023] Figure 5 This is a schematic diagram illustrating the generation of edge prompt points in one implementation of the present invention.

[0024] Figure 6 This is a schematic diagram comparing the first preprocessed orthophoto window image with the output window frame mask in one implementation of the present invention.

[0025] Figure 7 This is a flowchart of a window frame mask fusion method in one implementation of the present invention.

[0026] Figure 8 This is a schematic diagram of image fusion of a window frame mask group in one implementation of the present invention.

[0027] Figure 9 This is a schematic diagram of the sampling area for the geometric and color parameters of the window frame in one implementation of the present invention.

[0028] Figure 10 This is a flowchart of a method for extracting geometric and color parameters of a window frame fusion mask in one implementation of the present invention.

[0029] Figure 11 This is a schematic diagram comparing the window frame fusion mask before and after the third preprocessing in one implementation of the present invention.

[0030] Figure 12 This is a schematic diagram of the structured analysis of the intersection cluster of the window frame fusion mask in one implementation of the present invention.

[0031] Figure 13 This is a schematic diagram illustrating the process of generating the intersection cluster rectangle of the window frame fusion mask in one implementation of the present invention.

[0032] Figure 14 This is a schematic diagram of the target window frame image generated in one implementation of the present invention.

[0033] Figure 15This is a functional schematic diagram of the terminal in one implementation of the present invention.

[0034] The objectives, features, and advantages of this invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0036] Exemplary methods In existing technologies, windows on building facades exhibit a high degree of diversity in shape, size, and color. They are also subject to interference from variable lighting conditions, external obstructions (such as vegetation or decorations), and interior scenes (such as curtains or furniture) reflected through the glass panes. This often leads to problems such as insufficient detail, geometric distortion, and detection errors in window frame structure extraction, which severely restricts the application value of window models in LOD3 architectural 3D models.

[0037] To address the above technical problems, this invention provides a method and system for extracting window frame structures based on image segmentation and multi-image fusion. The method includes: acquiring orthophotos of window frames on a building facade; performing a first preprocessing step on the orthophotos; wherein the first preprocessing step includes window clustering and super-resolution processing; segmenting the window frames of the preprocessed orthophotos using an image segmentation algorithm to obtain segmented window frame masks; fusing window frame masks from the same cluster using a multi-image fusion algorithm to generate a fused window frame mask; extracting the geometric and color parameters of the fused window frame mask; and obtaining the target window frame structure based on the extracted geometric and color parameters. This invention, through image segmentation, multi-image fusion, and geometric and color parameter extraction steps, can accurately extract the window frame structure corresponding to an orthophoto.

[0038] like Figure 1 As shown, this embodiment of the invention provides a method for extracting window frame structures based on image segmentation and multi-image fusion, including the following steps: Step S100: Obtain orthophoto images of the building facade and perform a first preprocessing on the orthophoto images; wherein the first preprocessing includes window clustering and super-resolution processing.

[0039] In this embodiment, orthophotos of the building facade windows are acquired, and the orthophotos of the building facade windows are subjected to a first preprocessing, specifically, window clustering and super-resolution processing are performed on the orthophotos of the building facade windows.

[0040] In this embodiment, the clustering method is as follows: First, a pre-trained ResNet50 model (Residual Network 50, a 50-layer deep residual neural network model) is used to extract 2048-dimensional high-level semantic features of each window image. These features include multi-level visual information such as the shape structure, material texture, and color style of the window. Then, the optimal number of clusters is automatically determined using the contour coefficient. Next, the K-means clustering algorithm is used to group visually similar window images in the feature space to obtain window image clusters.

[0041] In this embodiment, based on window image clustering, orthophoto windows are processed in batches using the HYPIR model (image restoration and magnification model) to obtain corresponding HYPIR super-resolution RGB images; thus completing the preprocessing operation of the orthophoto windows.

[0042] like Figure 1 As shown, this embodiment of the invention provides a method for extracting window frame structures based on image segmentation and multi-image fusion, including the following steps: Step S200: Based on the image segmentation algorithm, the window frame of the first preprocessed orthophoto window image is segmented to obtain the segmented window frame mask.

[0043] In this embodiment, SAM (Segment Anything Model) is combined with adaptive preprocessing, dynamic cue point set generation, and a window frame-specific scoring and filtering mechanism. Through illumination robustness enhancement, line-inside offset sampling, and boundary reinforcement strategies, the window frame structure in complex scenes is accurately segmented. Figure 2 The diagram shown is a flowchart of the window frame segmentation method in this embodiment.

[0044] Specifically, in one implementation of this embodiment, step S200 includes the following steps: Step S201: Perform a second preprocessing on the orthophoto window image after the first preprocessing; wherein, the second preprocessing includes color space conversion, Channel enhancement, image correction, and bilateral filtering for noise reduction.

[0045] In this embodiment, the input image is the orthophoto window image after the first preprocessing, i.e., a HYPIR super-resolution RGB image. The input image undergoes a second preprocessing, which is an adaptive preprocessing. Specifically, the second preprocessing includes color space conversion, ... Channel enhancement, image correction, and bilateral filtering for noise reduction.

[0046] like Figure 2 As shown, the second preprocessing includes the following steps: Step S201a, color space conversion: perform color space conversion on the first preprocessed orthophoto window image.

[0047] In this embodiment, the input RGB format orthophoto window image is converted to the YCrCb color space to separate luminance and chromaticity information. The corresponding formula is as follows: ; ; ; The RGB coordinates include: R (red), G (green), and B (blue) are integer values ​​from 0 to 255, representing the three primary color components; The YCrCb coordinates include: Y (luminance, 16~235), Cr (red-cyan hue, 128 is a neutral value), and Cb (blue-yellow hue, 128 is a neutral value). Subsequently, contrast enhancement operations were performed based on the Y channel.

[0048] Step S201b, Channel enhancement for orthophoto windows after color space conversion Channel enhancement.

[0049] In this embodiment, for The channel applies the CLAHE algorithm (Contrast-Limited Adaptive Histogram Equalization) to segment the entire image into... Each region is divided into non-overlapping rectangular regions, limiting the maximum frequency of each gray level to prevent noise amplification. Histogram equalization is then applied to each region, as shown in the following formula: ; ; in, For equalization Channel grayscale values ​​(0~255) stretch local contrast; The threshold for clipping histogram frequencies (by...) Control intensity), suppress noise amplification; yes The cumulative distribution function of gray levels in a channel. It is the minimum value of the cumulative distribution function; To mitigate the cropping constraint, excess pixels are evenly distributed across all gray levels; that is, when the frequency of any gray level exceeds a certain threshold... When the total number of pixels exceeds the limit, the excess pixels will be evenly added to the frequency of 256 gray levels (0~255) to maintain the total number of pixels.

[0050] Furthermore, global histogram balancing is applied as a supplement, with the corresponding formula as follows: ; in, This represents the grayscale value (0~255) output after histogram equalization. Represents the total number of gray levels (in an 8-bit image). ), Indicates the first Each input gray level, and , It is grayscale. The probability density function.

[0051] The probability density function for gray levels is calculated as follows: ; in, Represents grayscale level The number of pixels appearing in an image. This represents the total number of pixels.

[0052] Step S201c, image correction, for Image correction is performed on the orthophoto window image after channel enhancement.

[0053] In this embodiment, the image correction method is gamma correction, which is used to enhance the dark details of the image. The corresponding formula is as follows: ; in, The input pixel value, ranging from 0 to 255; The output pixel values, This is the gamma value.

[0054] Step S201d: Bilateral filtering denoising, performing bilateral filtering denoising on the orthophoto window image after image correction.

[0055] In this embodiment, bilateral filtering is applied to the orthorectified orthophoto window image to denoise it while maintaining the sharp edges of the window frame, resulting in the orthophoto window image after the second preprocessing. The corresponding formula is as follows: ; in, Indicates the position of the filtered image. Pixel value at; It is a pixel in the neighborhood. The original pixel values; Representative with A local neighborhood centered on the element, such as a sliding window; It is a spatial domain weighting function that decays based on pixel location distance; the closer the pixel, the greater the weight. It is a range weighting function that decreases based on the difference in pixel values; the closer the values ​​are, the greater the weight. It is the normalization factor, which is equal to the sum of the weights of all neighboring pixels.

[0056] Step S202: Sample the second preprocessed orthophoto window image to obtain a dynamic cue point set.

[0057] In this embodiment, the orthophoto window image after the second preprocessing is sampled to obtain a dynamic cue point set. This dynamic cue point set includes a grid point set, a generated point set based on line detection and bilateral brightness comparison, and an enhanced sampling point set for image edges, such as... Figure 2 As shown, the three point sets above correspond to three sampling methods: grid point adaptive sampling, straight line offset sampling, and edge region sampling, respectively. The point set obtained by merging the prompt points obtained by the three sampling methods is the dynamic prompt point set.

[0058] Step S202 specifically includes the following steps: Step S202a, adaptive sampling, performing grid point adaptive sampling on the second preprocessed orthophoto window image to generate a grid point set.

[0059] In this embodiment, grid point adaptive sampling is performed on the second preprocessed orthophoto window image to obtain grid cue points, and the image diagonal length is calculated. Based on a preset image diagonal length threshold and and sampling interval threshold and Dynamically determine sampling interval The corresponding formula is as follows:

[0060] Furthermore, based on the sampling interval The number of grid points generated is According to the spacing along the width and height of the image Generate a uniform grid point set ,like Figure 3 The diagram shown is a schematic of the grid point set generated in this embodiment.

[0061] Step S202b: Extract the edge structure of the second preprocessed orthophoto window image to generate the first edge point set.

[0062] In this embodiment, straight-line inner offset sampling is performed on the second preprocessed orthophoto window image. First, it is necessary to extract the edge structure of the second preprocessed orthophoto window image. In this embodiment, a multi-scale edge detection algorithm is used to extract the significant edge structure in the image and generate a first edge point set. The corresponding formula is as follows: ; ; ; ; in, and They are direction and Sobel convolution kernels (3×3) in the directions are defined as follows: and ; and yes and Image gradient in the direction; It is the input grayscale image; It is the image gradient vector; It is the gradient direction angle.

[0063] In this embodiment, a precise single-pixel width edge structure is obtained through non-maximum suppression and double threshold hysteresis processing. The edge structure is a set of edge pixels with a single-pixel width, represented by a binary image.

[0064] Step S202c: Linear offset sampling, performing line detection and bilateral brightness comparison on the first edge point set to generate a second edge point set.

[0065] In this embodiment, based on the first edge point set, which is represented in the form of a binary image, a Hough transform is performed on the binary image to detect line segments: ; The detection results yield a set of line segments. Each straight line segment is defined by the coordinates of its endpoints. definition.

[0066] Furthermore, based on the geometric characteristics of the building's window frames, straight lines that are close to horizontal or vertical are selected: ; ; Retention conditions: ; in, Is a straight line and The original angle of the axis; It is the angle normalized to the interval [0, 180); the final angle is retained. foot or Candidate straight lines.

[0067] For each filtered line Calculate its normal vector: ; ; ; in, The direction vector of the line. It is the normal vector.

[0068] According to density on a straight line Generate basic sampling points: ; ; ; in, This represents the total number of sampling points; The normalized linear parameters vary uniformly from 0 to 1. On a straight line The basic sampling points at the location.

[0069] Furthermore, based on each basic sampling point Generate a candidate offset point pair: ; ; in, This is the offset.

[0070] Calculate the brightness statistics of the 3×3 neighborhood of each of the two candidate points. and The corresponding formula is as follows: ; Finally, based on the prior knowledge that the window frame color is usually darker than the glass pane color, the sampling points on the darker side were selected to be retained. As a clue point for straight line offset, it forms the second edge point set. ,like Figure 4 The image shown is a schematic diagram of the straight line offset prompt point generated in this embodiment.

[0071] In this embodiment, the line offset prompt point satisfy: .

[0072] Step S202d: Edge region sampling. Enhanced sampling points of the image edge are generated in the strip-shaped region at a distance from the edge structure to obtain the third edge point set.

[0073] In this embodiment, enhanced sampling points for the image edge are generated within a strip-shaped region distant from the edge structure. These are the edge cue points; window edges are extracted and enhanced to obtain the third edge point set. The corresponding formula is as follows: ; ; ; in, Generate additional points for the edges. and These are the lower and upper limits for the number of additional points generated; The distance from the image boundary to the sampling band. It is the minimum edge distance. It is the edge distance ratio coefficient; It is the sampling strip width. It is the offset of the sampling strip width. It is the sampling strip width ratio coefficient.

[0074] like Figure 5 The diagram shown is a schematic of the generation of edge tooltips in this embodiment. The distribution range is as follows: Top and bottom edges ; Left and right edges: .

[0075] Step S202e yields a dynamic cue point set containing the grid point set, the second edge point set, and the third edge point set.

[0076] In this embodiment, all prompt points in the obtained grid point set, second edge point set, and third edge point set are merged. Specifically, grid prompt points, straight line offset prompt points, and edge prompt points are merged to obtain a dynamic prompt point set containing all the above prompt points.

[0077] Step S203: Based on the dynamic cue point set, the orthophoto window image is segmented into a window frame to obtain a segmented window frame mask.

[0078] In this embodiment, the orthophoto window image is segmented based on the dynamic cue point set to obtain a segmented window frame mask, as shown below. Figure 2As shown, the window frame segmentation method in this embodiment specifically includes the following steps: Step S203a, SAM model segmentation: Based on the dynamic cue point set, the orthophoto window image is segmented into window frames to obtain a first window frame mask set.

[0079] In this embodiment, the ViT-H architecture SAM model is adopted, which combines the input image boundary cue box and the dynamic cue point set. The corresponding formula is as follows: ; ; Among them, the prompt box Shrinking the image edges A rectangular frame of pixels. This is a set of dynamic prompt points.

[0080] After segmentation using the SAM model, the first window frame mask set is obtained. ;in, .

[0081] Step S203b: Masking process, merging similar window frame masks in the first window frame mask set based on color similarity and spatial proximity to obtain the second window frame mask set.

[0082] In this embodiment, before merging similar window frame masks in the first set of window frame masks based on color similarity and spatial proximity, the method further includes: Sort the mask areas in descending order to obtain the first set of window frame masks after sorting: ; Pixels are assigned to the sorted first window frame mask set to ensure that each pixel is assigned to the mask with the largest covering area, forming a set of non-overlapping first window frame masks: ; in, yes The final mask index assigned to a pixel is related to the mask area. Perform the operation corresponding to the index of the maximum value, and cover the first window frame mask set of the current pixel.

[0083] In this embodiment, adjacent similar masks are merged based on color similarity and spatial proximity, specifically including: Calculate the color distance between any two masks: The threshold is ; Calculate the spatial distance threshold of the mask: ; Wherein, constant 5 is the minimum spatial distance threshold, used to ensure a reasonable merging distance even in small images; It is the maximum clearance ratio coefficient; Perform morphological dilation on the mask, with dilation radius... Expand the boundaries of the original mask outwards. A number of pixels, when the extended parts overlap, satisfy the proximity condition; If and only if When the proximity condition is met, the masks are merged to obtain the second window frame mask set.

[0084] Step S203c, denoising optimization: Denoising is performed on the window frame masks in the second window frame mask set to obtain the third window frame mask set.

[0085] In this embodiment, the window frame masks in the second window frame mask set are denoised by removing excessively small irrelevant areas to reduce noise interference, resulting in a third window frame mask set. The corresponding formula is as follows: ; If and only if The mask is retained at the time.

[0086] Step S203d: Scoring and filtering. The window frame mask is scored based on area threshold, boundary fit and hollowness, and the window frame mask with the highest score in the third set of window frame masks is output.

[0087] In this embodiment, based on the area threshold Boundary fit and hollow rate These three metrics are used to weight and score the window frame mask, and the highest-scoring window frame mask in the third set of window frame masks is output. The formulas for each metric are as follows: ; ; ; in, and The coordinates are the top-left corner of the circumscribed rectangle of the mask. and These are the width and height of the bounding rectangle of the mask; It is the pixel area of ​​the mask. It is the pixel area of ​​the bounding rectangle of the mask. It is the pixel area of ​​the image.

[0088] In this embodiment, based on the area threshold Boundary fit and hollow rate These three indicators are used to weight and score the window frame mask, and the corresponding formulas are as follows: ; in, .

[0089] In this embodiment, the window frame mask with the highest score in the third set of window frame masks is finally output, that is, the optimal window frame mask is output, which is: .

[0090] like Figure 6 The image shown is a schematic diagram comparing the first preprocessed orthophoto window image with the output window frame mask in this embodiment. Figure 6 The left side shows the orthophoto window image after the first preprocessing, and the right side shows the output window frame mask.

[0091] like Figure 1 As shown, this embodiment of the invention provides a method for extracting window frame structures based on image segmentation and multi-image fusion, including the following steps: Step S300: Based on the multi-graph fusion algorithm, window frame masks of the same cluster are fused to generate a window frame fusion mask.

[0092] In this embodiment, occlusion interference and noise in a single image are eliminated through affine alignment, majority voting decision, and color median optimization, generating a geometrically continuous and highly reliable window frame fusion mask, such as... Figure 7 The diagram shown is a flowchart of the window frame mask fusion method in this embodiment.

[0093] Specifically, in one implementation of this embodiment, step S300 includes the following steps: Step S301: Obtain the window frame mask of the same cluster.

[0094] In this embodiment, the input image is a window frame mask corresponding to the orthophoto window image of the same cluster.

[0095] Step S302: Perform adaptive affine alignment on the window frame masks of the same cluster to obtain an aligned window frame mask group.

[0096] like Figure 7 As shown, this embodiment performs adaptive affine alignment on window frame masks of the same cluster, including the following steps: Step S302a: Select a reference datum for adaptive affine alignment of window frame masks of the same cluster, and extract the boundaries.

[0097] In this embodiment, the effective boundary area of ​​the first window image mask in the group is automatically calculated using the first window image mask in the group as a reference; the boundaries of each other image mask in the group are also detected.

[0098] Step S302b: Perform adaptive affine alignment on the window frame masks of the same cluster to obtain the aligned window frame mask group.

[0099] In this embodiment, an affine transformation relationship is established through three pairs of key points (top left, top right, and bottom left) to accurately map all images to the coordinate system of the reference image; the area outside the boundary is uniformly filled with black to provide standardized input for subsequent fusion steps. The steps are as follows: Step a1, represent the affine transformation using a 2×3 transformation matrix: ; Step a2, for each pixel in the image Transformed coordinates The calculation is as follows: ; Step a3, extract the three points from the source image. Three points relative to the reference image Establish corresponding relationships: ; ; ; ; ; ; ; Step a4, solve the above system of linear equations to get: ; Step a5: Apply a transformation matrix to align the images. Using the first image in each group as a reference, align the other images. Use a reverse mapping strategy to avoid holes in the output image. ; Step a6: Calculate pixel values ​​using bilinear interpolation. .

[0100] Step S303: Consensus voting is performed on the aligned window frame image group to generate a consensus mask.

[0101] In this embodiment, in the aligned image group, the frequency of non-black pixels is counted for each pixel position. A threshold is set for the proportion of the frequency to the number of images in the image group. If the frequency exceeds the threshold, the pixel is retained. Pixel positions that do not reach the threshold are assigned black, thus completing consensus voting and generating a consensus mask. The steps are as follows: Step b1, for each group of aligned images First, a binary mask is generated for each image: ; Step b2, calculate the frequency of non-black pixels at each pixel: ; Step b3: Based on the threshold r, determine whether to retain the pixel value at that location, and generate a consensus mask. .

[0102] Step S304: Based on the consensus mask, perform image fusion of the aligned window frame mask group to obtain the window frame fusion mask.

[0103] In this embodiment, the image of the aligned window frame mask group is fused based on the consensus mask to obtain a window frame fusion mask; specifically, a color median optimization method is used after image alignment and consensus mask generation. Then, align the images. Pixel-level fusion is performed, and each group of window frame masks outputs a single window frame fusion mask. For example... Figure 8 The image shown is a schematic diagram of image fusion using a mask group for a window frame. Figure 8 The image on the left shows the aligned window frame mask assembly. Figure 8 The rightmost image is the window frame fusion mask. The image fusion method for the window frame mask group includes the following steps: Step c1, for each pixel position Select the median value from the pixel values ​​at that location in all aligned images: ; Step c2: Multiply the fusion result with the consensus mask to ensure that the fused pixel values ​​are retained only within the consensus region, thus obtaining the final window frame fusion mask after multi-image fusion. .

[0104] like Figure 1 As shown, this embodiment of the invention provides a method for extracting window frame structures based on image segmentation and multi-image fusion, including the following steps: Step S400: Extract the geometric parameters and color parameters of the window frame fusion mask.

[0105] In this embodiment, a geometric parameter extraction method based on line detection and two-dimensional intersection clustering is used to perform edge enhancement and structured point cluster analysis on the output window frame fusion mask, accurately quantifying the window frame thickness, separator distribution, and color attribute extraction, such as... Figure 9 As shown, this is a schematic diagram of the sampling area for the window frame geometric parameters and color parameters in this embodiment. Figure 10 The diagram shown is a flowchart of the method for extracting the geometric and color parameters of the window frame fusion mask in this embodiment.

[0106] Specifically, in one implementation of this embodiment, step S300 includes the following steps: Step S401: Perform a third preprocessing on the window frame fusion mask; wherein the third preprocessing includes image scaling, adding black borders, morphological operations, and maximum connected component preservation.

[0107] In this embodiment, before extracting geometric and color parameters from the window frame fusion mask, a third preprocessing step is performed on the window frame fusion mask. This third preprocessing includes image scaling, adding black borders, morphological operations, and maximum connected component preservation. Figure 10 As shown, the third preprocessing in this embodiment includes the following steps: Step S401a: Image scaling, reading the window frame blending mask, and scaling proportionally. Scale to the target diagonal length to ensure consistent processing: ; in, The target diagonal length, and The original image dimensions.

[0108] Step S401b: Add a black border around the image. Add a black border of fixed width around the image to prevent edge features from being truncated. ; ; in, and These are the width and height of the scaled image. It refers to the width of the added black border.

[0109] Step S401c, morphological operation: connect tiny breakpoints through morphological closing operation and eliminate tiny noise through opening operation to ensure the topological continuity of the window structure.

[0110] Step S401d: Preserve the largest connected component. The largest connected component is preserved as the basis for subsequent structural feature extraction, such as... Figure 11The image shown is a schematic diagram comparing the window frame fusion mask before and after the third preprocessing step in this embodiment. Figure 11 The left side shows the window frame fusion mask before the third preprocessing step. Figure 11 The right side shows the third pre-processed window frame fusion mask.

[0111] Step S402: Perform straight line detection and correction on the third pre-processed window frame fusion mask.

[0112] like Figure 10 As shown, the line detection and correction in this embodiment includes the following steps: Step S402a, Canny edge extraction: The Canny operator is used to extract image edges based on gradient calculation. ; in, It is a Gaussian filter kernel used to perform convolution operations on the image.

[0113] Step S402b, Hough line detection, detecting line segments in the image based on Hough transform, using polar coordinate parameterization: .

[0114] Step S402c, Angle and Direction Filtering: Based on prior information about the window frame, filter the straight angle and direction, with tolerance... : or (Horizontal line); , (vertical line).

[0115] Step S402d: Clustering of lines in the same direction. The DBSCAN algorithm is used to cluster lines in the same direction, and the neighborhood radius is calculated as follows: ; .

[0116] in, for The neighborhood radius of the neighborhood. for The neighborhood radius of the neighborhood.

[0117] Step S403: Perform intersection cluster structured analysis on the window frame fusion mask after line detection and correction to obtain the geometric parameters of the window frame fusion mask.

[0118] like Figure 10 As shown, the method for extracting geometric parameters in this embodiment includes the following steps: Step S403a: Solve for the intersection of horizontal / vertical lines, and obtain the effective intersection points of the horizontal and vertical line segments of the window frame fusion mask after line detection and correction.

[0119] Step S403b: Two-dimensional spatial clustering of intersection points. Two-dimensional spatial clustering is performed on the effective intersection points of the horizontal and vertical line segments to form intersection point clusters.

[0120] In this embodiment, the effective intersection points of the horizontal and vertical line segments are clustered in two-dimensional space to form intersection point clusters, and the clustering radius is... for: .

[0121] Step S403c: Generate the bounding rectangle of the point cluster, generating the bounding rectangle of each intersection point cluster.

[0122] In this embodiment, a bounding rectangle is generated for each cluster of intersection points: ; Step S403d: Rectangular box row and column grouping clustering. These rectangles are grouped by row and column respectively, with the corresponding cluster radii being: ; ; in, For the rectangle in The cluster radius in the direction, i.e., the cluster radius when grouping by column. For the rectangle in The cluster radius in the direction, that is, the cluster radius when grouped by row.

[0123] like Figure 12 The diagram shown is a schematic representation of the structured analysis of the intersection clusters of the window frame fusion mask. Figure 12 The left side shows the intersection of the window frame fusion mask, the straight line of the window frame edge, and the point formed by them. Figure 12 The right side of the image shows the cluster of intersections of the extracted window frame fusion mask.

[0124] Step S403e, geometric parameter calculation: Based on the geometric and positional features of the rectangular boxes after row and column grouping and clustering, calculate the geometric parameters of the window frame fusion mask.

[0125] In this embodiment, based on the geometric and positional features of the rectangular boxes after row and column grouping and clustering, the geometric parameters of the window frame fusion mask are calculated. The specific steps are as follows: Step d1: For the left and right borders, take the arithmetic mean of the widths of the first and last column cluster rectangles; for the top and bottom borders, take the arithmetic mean of the widths of the first and last row cluster rectangles. First, let the columns be grouped as follows: ; Let the rows be grouped as ; Then we have: ; ; ; ; in, , .

[0126] Step d2: For horizontal dividers, take the arithmetic mean of the heights of the rectangles in the middle rows; for vertical dividers, take the arithmetic mean of the widths of the rectangles in the middle columns. ; .

[0127] Step d3: For the position of the separator bar, perform normalized calculation based on the position of the point cluster center coordinates within the pane area: First, let the coordinates of the column center be... ; Let the center coordinates of the row be ; Then we have: ; ; in, and It is the normalized positional ratio of horizontal and vertical dividers in the horizontal and vertical directions.

[0128] like Figure 13 The diagram shown illustrates the process of generating the rectangular frame of the intersection cluster of the window frame fusion mask in this embodiment. Figure 13 Based on the same window frame fusion mask, from left to right, the following diagrams are presented: image edge extraction results based on the Canny operator, line detection and correction results based on the Hough transform, line angle and direction filtering results, effective intersection point generation, and intersection point cluster rectangle generation.

[0129] Step S404: Extract color attributes from the window frame fusion mask after line detection and correction to obtain the color parameters of the window frame fusion mask.

[0130] In this embodiment, color attribute extraction is performed on the window frame fusion mask after line detection and correction to obtain the color parameters of the window frame fusion mask, including the following steps: Step S404a, structural region decomposition: Based on the point cluster rectangle, the window frame fusion mask is decomposed into independent sampling regions: For the border, sample using the following formula: ; ; ; ; ; Among them, sampling area , , , Defined by the Cartesian product of the horizontal and vertical pixel coordinate intervals.

[0131] For the separator bars, the sampling method is as follows: For each vertical dividing position The sampling area is: ; For each horizontal dividing position The sampling area is: .

[0132] Step S404b, Background threshold filtering and color mean calculation, setting the background color threshold. If every RGB band is less than the threshold, it is considered background. Pixels in the sampling area that exceed the threshold are sampled and their mean is calculated.

[0133] ; ; .

[0134] like Figure 1 As shown, this embodiment of the invention provides a method for extracting window frame structures based on image segmentation and multi-image fusion, including the following steps: Step S500: Generate the target window frame image based on the extracted geometric and color parameters.

[0135] In this embodiment, a target window frame image is generated based on the extracted geometric and color parameters. The method for generating the target window frame image can utilize the extracted geometric parameters (window width and height, border width, separation ratio and its width) and color parameters to perform a simple stylized drawing of the window, forming a front and side image of the window, such as... Figure 14 The image shown is a schematic diagram of the generated target window frame. Figure 14The left side of the image shows the front view of the generated target window frame, and the right side shows the side view of the generated target window frame.

[0136] In this embodiment, the generated target window frame image can be used to generate a parametric model corresponding to the orthophoto window image of the building facade, providing a high-fidelity data foundation for building digital twins.

[0137] This embodiment achieves the following technical effects through the above technical solution: (1) The window frame segmentation method based on physical features and intelligent sampling is mainly based on the synergy of intelligent sampling guidance and physical feature screening. First, three types of cue points are used, including image-adaptive grid cue points, line offset cue points based on line detection and bilateral brightness comparison, and edge cue points at the image edge. The area threshold, boundary fit, and hollowness rate are quantified into scoring functions to screen the real window frames. Finally, noise reduction is performed based on the main color of the window frame and connectivity analysis to improve the segmentation accuracy and robustness.

[0138] (2) The window frame enhancement method using multi-image alignment and fusion is key to using multi-image collaborative voting to enhance the window frame structure and suppress the background. After aligning multiple window frame images in the group through affine transformation, a consensus voting mechanism is used to determine stable window frame pixel areas. Then, median color fusion is used to suppress outliers, and finally a clear image with less background interference and strong window frame features is synthesized.

[0139] (3) Based on the analysis of window frame structure features by two-dimensional intersection clustering, the window grid structure is directly analyzed by the row and column grouping clustering of intersection points. After performing two-dimensional clustering on the intersection points of straight lines, the intersection point clusters are clustered horizontally and vertically respectively to achieve row and column grouping. Thus, complete parameters such as border width, separator size and position ratio can be directly inferred from the geometric information statistics of the grouped point clusters.

[0140] Exemplary device Based on the above embodiments, the present invention also provides a window frame structure extraction system based on image segmentation and multi-image fusion, comprising: An image processing module is used to acquire orthophotos of windows on a building facade and perform a first preprocessing on the orthophotos; wherein the first preprocessing includes window clustering and super-resolution processing. The image segmentation module is used to segment the window frame of the first preprocessed orthophoto window image based on the image segmentation algorithm to obtain the segmented window frame mask. The image fusion module is used to fuse window frame masks of the same cluster based on a multi-image fusion algorithm to generate a window frame fusion mask. The parameter extraction module is used to extract the geometric and color parameters of the window frame fusion mask; The image generation module is used to generate a target window frame image based on the extracted geometric and color parameters.

[0141] Based on the above embodiments, the present invention also provides a terminal, the principle block diagram of which can be as follows: Figure 15 As shown.

[0142] The terminal includes: a processor, a memory, an interface, a display screen, and a communication module connected via a system bus; wherein, the processor of the terminal provides computing and control capabilities; the memory of the terminal includes a computer-readable storage medium and internal memory; the computer-readable storage medium stores an operating system and computer programs; the internal memory provides an environment for the operation of the operating system and computer programs in the computer-readable storage medium; the interface is used to connect to external devices; the display screen is used to display relevant information; and the communication module is used to communicate with a cloud server or other devices.

[0143] When executed by the processor, this computer program is used to implement a window frame structure extraction method based on image segmentation and multi-image fusion.

[0144] It will be understood by those skilled in the art that Figure 15 The schematic diagram shown is merely a partial structural diagram related to the present invention and does not constitute a limitation on the terminal to which the present invention is applied. A specific terminal may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0145] In one embodiment, a terminal is provided, comprising: a processor and a memory, the memory storing a window frame structure extraction program based on image segmentation and multi-image fusion, the window frame structure extraction program based on image segmentation and multi-image fusion being executed by the processor to implement the operation of the window frame structure extraction method based on image segmentation and multi-image fusion as described above.

[0146] In one embodiment, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a window frame structure extraction program based on image segmentation and multi-image fusion, the window frame structure extraction program based on image segmentation and multi-image fusion being executed by a processor to implement the operations of the window frame structure extraction method based on image segmentation and multi-image fusion as described above.

[0147] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, database, or other media used in the embodiments provided by this invention can include both non-volatile and volatile memory.

[0148] In summary, this invention provides a method and system for extracting window frame structures based on image segmentation and multi-image fusion, comprising: acquiring orthophotos of windows on a building facade; performing a first preprocessing on the orthophotos; wherein the first preprocessing includes window clustering and super-resolution processing; segmenting the window frames of the preprocessed orthophotos based on an image segmentation algorithm to obtain segmented window frame masks; fusing window frame masks of the same cluster based on a multi-image fusion algorithm to generate a window frame fusion mask; extracting the geometric and color parameters of the window frame fusion mask; and obtaining the target window frame structure based on the extracted geometric and color parameters. This invention, through steps such as image segmentation, multi-image fusion, and extraction of geometric and color parameters, can accurately extract the window frame structure corresponding to an orthophoto.

[0149] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A method for extracting window frame structures based on image segmentation and multi-image fusion, characterized in that, include: Acquire orthophotos of the building facade windows and perform a first preprocessing on the orthophotos; wherein the first preprocessing includes window clustering and super-resolution processing; The window frame is segmented based on the image segmentation algorithm to obtain the segmented window frame mask. The window frame masks of the same cluster are fused based on the multi-graph fusion algorithm to generate a window frame fusion mask; Extract the geometric and color parameters of the window frame fusion mask; The target window frame structure is obtained based on the extracted geometric and color parameters.

2. The window frame structure extraction method based on image segmentation and multi-image fusion according to claim 1, characterized in that, The step of segmenting the window frame of the first preprocessed orthophoto window image using an image segmentation algorithm to obtain a segmented window frame mask includes: The orthophoto image after the first preprocessing is then subjected to a second preprocessing; wherein, the second preprocessing includes color space conversion, Channel enhancement, image correction, and bilateral filtering denoising; The second preprocessed orthophoto window image is sampled to obtain a dynamic cue point set; Based on the dynamic cue point set, the orthophoto window image is segmented into a window frame to obtain a segmented window frame mask.

3. The window frame structure extraction method based on image segmentation and multi-image fusion according to claim 2, characterized in that, The sampling of the second preprocessed orthophoto window image to obtain a dynamic cue point set includes: Adaptive sampling of grid points is performed on the second preprocessed orthophoto window image to generate a grid point set; Extract the edge structure of the second preprocessed orthophoto window image to generate the first edge point set; Perform line detection and bilateral brightness comparison on the first edge point set to generate a second edge point set; Enhanced sampling points of the image edge are generated within the strip-shaped region of the edge structure to obtain a third edge point set; A dynamic cue point set is obtained, which includes the grid point set, the second edge point set, and the third edge point set.

4. The window frame structure extraction method based on image segmentation and multi-image fusion according to claim 2, characterized in that, The step of segmenting the orthophoto window image based on the dynamic cue point set to obtain a segmented window frame mask includes: Based on the dynamic cue point set, the orthophoto window image is segmented into a window frame to obtain a first window frame mask set. Based on color similarity and spatial proximity, similar window frame masks in the first window frame mask set are merged to obtain the second window frame mask set; The window frame masks in the second window frame mask set are denoised to obtain the third window frame mask set. The window frame mask is scored based on area threshold, boundary fit, and void ratio, and the window frame mask with the highest score in the third set of window frame masks is output.

5. The window frame structure extraction method based on image segmentation and multi-image fusion according to claim 1, characterized in that, The method of fusing window frame masks of the same cluster based on a multi-graph fusion algorithm to generate a window frame fused mask includes: Obtain the window frame mask of the same cluster; Adaptive affine alignment is performed on window frame masks of the same cluster to obtain an aligned group of window frame masks. Consensus voting is performed on the aligned window frame image group to generate a consensus mask; Image fusion of the aligned window frame mask group based on the consensus mask is used to obtain the window frame fusion mask.

6. The window frame structure extraction method based on image segmentation and multi-image fusion according to claim 1, characterized in that, The extraction of the geometric and color parameters of the window frame fusion mask includes: A third preprocessing is performed on the window frame fusion mask; wherein, the third preprocessing includes image scaling, adding black borders, morphological operations, and maximum connected component preservation; Linearity detection and correction are performed on the window frame fusion mask after the third preprocessing step. The geometric parameters of the window frame fusion mask are obtained by performing intersection cluster structured analysis on the window frame fusion mask after line detection and correction. Color attributes are extracted from the window frame fusion mask after line detection and correction to obtain the color parameters of the window frame fusion mask.

7. The window frame structure extraction method based on image segmentation and multi-image fusion according to claim 6, characterized in that, The intersection cluster structured analysis of the window frame fusion mask after line detection and correction yields the geometric parameters of the window frame fusion mask, including: Obtain the effective intersection points of the horizontal and vertical line segments of the window frame fusion mask after line detection and correction; Two-dimensional spatial clustering is performed on the effective intersection points of the horizontal and vertical line segments to form intersection point clusters; Generate the bounding rectangle of each intersection cluster, and perform row and column grouping clustering on the rectangles; Based on the geometric and positional features of the rectangular boxes after row and column grouping, the geometric parameters of the window frame fusion mask are calculated.

8. A window frame structure extraction system based on image segmentation and multi-image fusion, characterized in that, include: An image processing module is used to acquire orthophotos of windows on a building facade and perform a first preprocessing on the orthophotos; wherein the first preprocessing includes window clustering and super-resolution processing. The image segmentation module is used to segment the window frame of the first preprocessed orthophoto window image based on the image segmentation algorithm to obtain the segmented window frame mask. The image fusion module is used to fuse window frame masks of the same cluster based on a multi-image fusion algorithm to generate a window frame fusion mask. The parameter extraction module is used to extract the geometric and color parameters of the window frame fusion mask; The image generation module is used to obtain the target window frame structure based on the extracted geometric and color parameters.

9. A terminal, characterized in that, include: The processor and memory, wherein the memory stores a window frame structure extraction program based on image segmentation and multi-image fusion, the window frame structure extraction program based on image segmentation and multi-image fusion being executed by the processor to implement the window frame structure extraction method based on image segmentation and multi-image fusion as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a window frame structure extraction program based on image segmentation and multi-image fusion, which, when executed by a processor, is used to implement the window frame structure extraction method based on image segmentation and multi-image fusion as described in any one of claims 1-7.

Images