Fine-grained semantic segmentation method, device, equipment and medium for window images

Abstract

The present application relates to the field of image recognition technology. The present application discloses a method, device, equipment and medium for fine-grained semantic segmentation of window images, which can realize accurate segmentation of building facades at a fine-grained level. The method for fine-grained semantic segmentation of window images includes acquiring window images; preprocessing the window images to obtain binary images; generating a unified expression paradigm based on the binary images; denoising the binary images using morphological opening operations combined with the unified expression paradigm to obtain denoised images; extracting the denoised images using connected component analysis to obtain fine-grained segmentation results of window glass.

Description

Technical Field

[0001] The present application relates to the field of image recognition technology. More specifically, the present application relates to a method, device, equipment and medium for fine-grained semantic segmentation of window images. Background Art

[0002] Traditional image segmentation methods use deep learning models trained with pre-labeled training data sets to identify the complex structure of building facades, and perform semantic segmentation on each category of building facades to obtain segmentation results. However, this segmentation method only performs segmentation on the coarse-grained level of building facade semantics (such as doors, windows, or eaves of a building), and cannot perform fine-grained semantic segmentation on windows or eaves (such as segmenting window frame areas or glass areas from window images). In addition, existing public training data sets also lack annotations for window images, resulting in the inability of traditional image segmentation methods to implement segmentation operations on building facades at a fine-grained level. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a method, device, equipment and medium for fine-grained semantic segmentation of window images, which can achieve accurate segmentation of building facades at a fine-grained level. The embodiments of the present application are mainly implemented through the following technical solutions:

[0004] A first aspect of an embodiment of the present application provides a fine-grained semantic segmentation method for a window image, comprising:

[0005] Get window images;

[0006] Preprocessing the window image to obtain a binary image;

[0007] Generate a unified expression paradigm based on the binary image;

[0008] The binary image is subjected to denoising processing by using a morphological opening operation combined with the unified expression paradigm to obtain a denoised image;

[0009] Connected component analysis is used to extract and process the denoised image to obtain a fine-grained segmentation result of the window glass.

[0010] According to one embodiment of the present application, the step of preprocessing the window image to obtain a binary image includes:

[0011] Gray-scale the window image to obtain a gray-scale image;

[0012] The grayscale image is binarized using the Otsu threshold method to obtain the binarized image.

[0013] According to one embodiment of the present application, the step of generating a unified expression paradigm based on the binary image includes:

[0014] Calculating the average value of each row of pixel values ​​in the binary image to obtain multiple row means;

[0015] Calculating the average value of each column of pixel values ​​in the binary image to obtain multiple column means;

[0016] Setting a first threshold;

[0017] Counting the number of all row means greater than the first threshold value to obtain a first statistical value;

[0018] Counting the number of all column means greater than the first threshold value to obtain a second statistical value;

[0019] When the first statistical value satisfies a first preset condition and the second statistical value satisfies a second preset condition, taking the binarized image as the unified expression paradigm;

[0020] When the first statistical value does not satisfy a first preset condition and the second statistical value does not satisfy a second preset condition, each pixel in the binary image is inverted and used as the unified expression paradigm.

[0021] According to an embodiment of the present application, the step of performing denoising on the binary image by using the morphological opening operation in combination with the unified expression paradigm to obtain the denoised image includes:

[0022] The binary image is corroded by using the corrosion operation in the morphological opening operation combined with the unified expression paradigm to obtain a corroded image;

[0023] The eroded image is expanded by using the expansion operation in the morphological opening operation in combination with the unified expression paradigm to obtain the denoised image.

[0024] According to an embodiment of the present application, the step of extracting and processing the denoised image using connected component analysis to obtain a fine-grained segmentation result of the window glass includes:

[0025] Creating a first label map, initializing all labels in the first label map to zero, and setting a label map variable to point to the first label map, wherein each of the labels has a one-to-one correspondence with one of the pixels in the denoised image;

[0026] Calculating the number of all pixels in the denoised image to obtain a first total number of pixels;

[0027] Set a first preset value, and initialize the first preset value to 1;

[0028] In the case where the first preset value is less than or equal to the first total number of pixels, the following steps are executed in a loop: a plurality of neighbor pixels of a third target pixel are obtained in a preset connectivity manner, and a plurality of neighbor labels corresponding to the third target pixel are obtained from the label map variable based on the plurality of neighbor pixels, wherein the third target pixel is any pixel in the denoised image; a reset label of the third target pixel is obtained based on the plurality of neighbor labels; in the label map pointed to by the label map variable, the label corresponding to the third target pixel is modified to the reset label to obtain a second label map, and the label map pointed to by the label map variable is modified to the second label map; in the case where at least one neighbor label among the plurality of neighbor labels is different from other neighbor labels, the plurality of neighbor labels are recorded and processed to obtain labels to be merged, and added to the label set to be merged;

[0029] Modify the second label map based on the label set to be merged to obtain a third label map;

[0030] The fine-grained segmentation result is obtained based on the third label map.

[0031] According to an embodiment of the present application, after the step of obtaining the fine-grained segmentation result based on the third label map, the step of extracting the denoised image using connected component analysis to obtain the fine-grained segmentation result of the window glass further includes:

[0032] The centroid corresponding to the first target label is calculated based on the third label map, wherein the first target label is any label in the third label map.

[0033] According to an embodiment of the present application, after the step of obtaining the fine-grained segmentation result based on the third label map, the step of extracting the denoised image using connected component analysis to obtain the fine-grained segmentation result of the window glass further includes:

[0034] A bounding box corresponding to a second target label is calculated based on the third label map, wherein the second target label is any label in the third label map.

[0035] A second aspect of an embodiment of the present application provides a fine-grained semantic segmentation device for window images, comprising:

[0036] A window image acquisition module, used for acquiring window images;

[0037] A binary image acquisition module, used for preprocessing the window image to obtain a binary image;

[0038] A unified expression paradigm generation module, used to generate a unified expression paradigm based on the binarized image;

[0039] A denoised image acquisition module, used for performing denoising processing on the binary image by using a morphological opening operation combined with the unified expression paradigm to obtain a denoised image;

[0040] The fine-grained segmentation result obtaining module is used to extract and process the denoised image using connected component analysis to obtain a fine-grained segmentation result of the window glass.

[0041] The third aspect of the embodiments of the present application provides a terminal device, including: a processor and a memory, the memory is used to store a computer program, the processor is used to call and run the computer program stored in the memory, and execute the steps of the fine-grained semantic segmentation method of window images provided in the first aspect of the embodiments of the present application.

[0042] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium is used to store a computer program, wherein the computer program enables a computer to execute the steps of the fine-grained semantic segmentation method for window images provided in the first aspect of the embodiment of the present application.

[0043] The beneficial effects of the embodiments of the present application include:

[0044] The embodiment of the present application uses a unified expression paradigm, morphological open operations and connected component analysis in combination to achieve the purpose of fine-grained segmentation of window images, so that the fine-grained semantic segmentation method of window images can achieve the segmentation of building facades without relying on deep learning models. Specifically, the embodiment of the present application first pre-processes the window image to obtain a binary image; generates a unified expression paradigm based on the binary image; uses the morphological open operation combined with the unified expression paradigm to denoise the binary image to obtain a denoised image; uses connected component analysis to extract the denoised image to obtain a fine-grained segmentation result of the window glass in the window image. Compared with the prior art, the embodiment of the present application can achieve accurate segmentation of building facades at a fine-grained level. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the conventional technology, the drawings required for use in the embodiments or the conventional technology descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0046] Figure 1A flowchart of a fine-grained semantic segmentation method for window images of the present application in some embodiments;

[0047] Figure 2 A flowchart of the fine-grained semantic segmentation method of window images in some embodiments of the present application;

[0048] Figure 3 A flowchart of a fine-grained semantic segmentation method for window images of the present application in some other embodiments;

[0049] Figure 4 A flowchart of a fine-grained semantic segmentation method for window images of the present application in some further embodiments;

[0050] Figure 5 A flowchart of a fine-grained semantic segmentation method for window images of the present application in some further embodiments;

[0051] Figure 6 A flowchart of a fine-grained semantic segmentation method for window images of the present application in some further embodiments;

[0052] Figure 7 A reference image of a third target pixel and a plurality of neighboring pixels of the present application;

[0053] Figure 8 A principle block diagram of a fine-grained semantic segmentation device for window images of the present application in some embodiments;

[0054] Fig. 9 This is a functional block diagram of the terminal device of the present application in some embodiments. DETAILED DESCRIPTION

[0055] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present application, so the present application is not limited by the specific embodiments disclosed below.

[0056] It should be noted that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0057] The terms "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0058] The terms "comprises," "includes," or any other variations thereof are intended to cover a non-exclusive inclusion, for example, a process, method, system, product, or apparatus that comprises a series of steps or elements is not necessarily limited to those steps or elements explicitly listed but may include other steps or elements not explicitly listed or inherent to such process, method, product, or apparatus.

[0059] Unless otherwise defined, all technical and scientific terms used in the specification of this application have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The term "and / or" used in the specification of this application includes any and all combinations of one or more related listed items.

[0060] The term "building facade" refers to the interface between a building and its exterior space, as well as its appearance and composition, or the components at the interface between the interior and exterior spaces of a building and their combination. For example, the building facade includes doors, windows or eaves of a building.

[0061] The term “fine-grained level” refers to a more refined and specific classification or recognition of an image.

[0062] The specific implementation of the present application is further described below in conjunction with the accompanying drawings.

[0063] like Figure 1 and Figure 2 , which are all flow charts of a fine-grained semantic segmentation method for window images provided in the first aspect of the embodiment of the present application. Figure 1 In the method, the fine-grained semantic segmentation method of the window image includes:

[0064] S1. Acquire window images.

[0065] The window image includes at least one window. Each window includes window glass and a window frame. The window image is an RGB image.

[0066] S2. Preprocess the window image to obtain a binary image.

[0067] Further, refer to Figure 3The flowchart of the present invention comprises the following steps:

[0068] S21, gray-scale the window image to obtain a gray-scale image.

[0069] The specific process of the step S21 is to perform weighted summation processing on the R channel, G channel and B channel of each pixel in the window image to obtain the grayscale value corresponding to each pixel, thereby obtaining the grayscale image. Specifically, the calculation formula of the grayscale value corresponding to each pixel is as follows: Y=0.299*R+0.587*G+0.114*B, where Y is the grayscale value, R is the R channel, G is the G channel, B is the B channel, 0.299 is the weight of the R channel, 0.587 is the weight of the G channel, and 0.114 is the weight of the B channel.

[0070] The process of step S21 can also be understood as converting the color information of each point in the window image into a single grayscale information, that is, extracting the brightness information of the window image.

[0071] S22, using the Otsu threshold method to perform binarization processing on the grayscale image to obtain the binarized image.

[0072] The step S22 includes:

[0073] S221. Calculate the frequency of occurrence of each gray level in the gray image, and add each of the frequencies to a frequency set.

[0074] It should be understood that each element in the frequency set corresponds to the frequency of occurrence of a gray level.

[0075] The step S221 may be understood as calculating the histogram of the grayscale image.

[0076] S222. Calculate the number of pixels in the grayscale image to obtain a second total number of pixels.

[0077] S223. Set a second preset value and initialize it to zero.

[0078] S224. When the second preset value is smaller than the length of the frequency set, loop through the following steps S2241 to S2246.

[0079] S2241. Calculate the foreground probability of a target frequency according to a first preset algorithm, wherein the target frequency is any element in the frequency set.

[0080] Furthermore, the calculation formula of the first preset algorithm is:

[0081] ;

[0082] in, The target frequency The corresponding foreground probability, is the second total number of pixels, is the target frequency, is the frequency set (that is, the frequency of occurrence of each gray level or the histogram), is the subscript of the frequency set.

[0083] S2242. Calculate the foreground mean of the target frequency according to a second preset algorithm.

[0084] Furthermore, the calculation formula of the second preset algorithm is:

[0085] ;

[0086] in, The target frequency The corresponding foreground mean, The target frequency The corresponding foreground probability, is the target frequency, is the frequency set, is the subscript of the frequency set.

[0087] S2243. Calculate the background probability of the target frequency according to a third preset algorithm.

[0088] Furthermore, the calculation formula of the third preset algorithm is:

[0089] ;

[0090] in, The target frequency The corresponding background probability is is the second total number of pixels, is the target frequency, is the frequency set, is the subscript of the frequency set.

[0091] S2244. Calculate the background mean of the target frequency according to a fourth preset algorithm.

[0092] Furthermore, the calculation formula of the fourth preset algorithm is:

[0093] ;

[0094] in, The target frequency The corresponding background mean, The target frequency The corresponding background probability is is the target frequency, is the frequency set, is the subscript of the frequency set.

[0095] S2245. Calculate the inter-class variance corresponding to the target frequency based on the foreground probability, the foreground mean, the background probability and the background mean, and add the inter-class variance to a variance set.

[0096] Furthermore, the calculation formula of the inter-class variance is:

[0097] ;

[0098] in, The target frequency The corresponding between-class variance is The target frequency The corresponding foreground probability, The target frequency The corresponding background probability is The target frequency The corresponding foreground mean, The target frequency The corresponding background mean.

[0099] S2246: Perform a self-increment operation on the second preset value.

[0100] S225 , selecting the inter-class variance with the largest value in the variance set as the target inter-class variance, and taking the frequency corresponding to the target inter-class variance as the target threshold.

[0101] The target threshold is the optimal threshold. Figure 3 The "Determine the optimal threshold" step in .

[0102] Furthermore, the calculation formula of S225 is:

[0103] ;

[0104] in, is the target threshold, is the target frequency, The target frequency The corresponding between-class variance.

[0105] S226, using the target threshold to calculate the pixel value of each pixel in the grayscale image to obtain the binary image. This step can refer to Figure 3 The "Apply Optimal Thresholding to Binarize Image" step in .

[0106] Furthermore, the calculation formula of step S226 is:

[0107] ;

[0108] in, For the grayscale image In Location The pixel value of the pixel at is the target threshold, For the binary image In Location The pixel value of the pixel at .

[0109] It should be understood that the execution of the above step S2 can improve the accuracy of segmenting the image area in the present application.

[0110] S3. Generate a unified expression paradigm based on the binary image.

[0111] The step S3 can be understood as "judging the window type based on the binary pixel distribution of the binary image and converting it into the unified expression paradigm". In the embodiment of the present application, the unified expression paradigm follows the unified expression form in which the pixel value of the window frame area in the binary image is 0 and the pixel value of the glass area in the binary image is 255.

[0112] Further, refer to Figure 2 and Figure 4 As shown, the S3 step includes:

[0113] S31, calculating the average value of pixel values ​​in each row of the binary image to obtain multiple row means.

[0114] Furthermore, the calculation formula of step S31 is:

[0115] ;

[0116] in, is the first The average pixel value of the row, is the total number of columns of pixels in the binary image, For the Column pixel value, is the total number of rows of pixels in the binary image.

[0117] S32, calculating the average value of each column of pixel values ​​in the binary image to obtain multiple column means.

[0118] Furthermore, the calculation formula of step S32 is:

[0119] ;

[0120] in, is the first The average value of the pixel values ​​in the column, is the total number of columns of pixels in the binary image, For the row pixel value, is the total number of rows of pixels in the binary image.

[0121] S33. Set a first threshold.

[0122] In the embodiment of the present application, the pixel values ​​in the binary image are only 0 and / or 255, so the first threshold is set to 127. In other implementations, the specific value of the first threshold can be set by those skilled in the art according to actual needs.

[0123] S34. Count the number of all row means greater than the first threshold to obtain a first statistical value.

[0124] Furthermore, the calculation formula of step S34 is:

[0125] ;

[0126] in, is the first statistical value, is the total number of rows of pixels in the binary image, For the The row mean (i.e. The average value of the pixel values ​​in the row), is the first threshold.

[0127] S35. Count the number of all column means greater than the first threshold to obtain a second statistical value.

[0128] Furthermore, the calculation formula of step S35 is:

[0129] ;

[0130] in, is the second statistical value, is the total number of columns of pixels in the binary image, For the The mean of the columns (i.e. The average value of the pixel values ​​in the column), is the first threshold.

[0131] S36: When the first statistical value satisfies a first preset condition and the second statistical value satisfies a second preset condition, the binary image is used as the unified expression paradigm. The step of "when the first statistical value satisfies the first preset condition and the second statistical value satisfies the second preset condition" can be understood as Figure 4 The step of "whether the row mean is greater than half of the image height and the column mean is greater than half of the image width" in the binary image is used as the unified expression paradigm, that is, Figure 4 The "binarization image conforms to the unified expression paradigm" step in .

[0132] Furthermore, the first preset condition is:

[0133] ;

[0134] in, is the first statistical value, is the total number of rows of pixels in the binary image.

[0135] The second preset condition is:

[0136] ;

[0137] in, is the second statistical value, is the total number of columns of pixels in the binary image.

[0138] When the first statistical value satisfies a first preset condition and the second statistical value satisfies a second preset condition, the calculation formula of the unified expression paradigm is expressed as: ;in, For the binary image In Location The corresponding unified expression paradigm, For the binary image In Location The pixel value of the pixel at .

[0139] S37, when the first statistical value does not satisfy the first preset condition and the second statistical value does not satisfy the second preset condition, performing a negation operation on each pixel in the binary image as the unified expression paradigm. The “negation operation” can be understood as Figure 2 The "image adjustment" step in can also be understood as Figure 4 The step of "inverting image pixels" and the step of "the first statistical value does not satisfy the first preset condition and the second statistical value does not satisfy the second preset condition" can be understood as Figure 4The "Is the row mean less than half the image height and the column mean less than half the image width" step in .

[0140] It should be understood that the first statistical value does not satisfy the first preset condition means The second statistical value does not satisfy the second preset condition. .

[0141] The calculation formula for the unified expression paradigm step after performing the inversion operation on each pixel in the binary image is: ;in, For the binary image In Location The corresponding unified expression paradigm, For the binary image In Location The pixel value of the pixel at .

[0142] In other embodiments, when the first statistical value satisfies the first preset condition and the second statistical value does not satisfy the second preset condition, or when the first statistical value does not satisfy the first preset condition and the second statistical value satisfies the second preset condition, it is considered that the window image quality is poor (refer to Figure 4 ), which is difficult to separate.

[0143] It should be understood that the above process of determining whether the first statistical value satisfies the first preset condition and whether the second statistical value satisfies the second preset condition can be understood as Figure 2 The "Window type determination" step in .

[0144] S4, using the morphological opening operation combined with the unified expression paradigm to perform denoising on the binary image to obtain a denoised image. This step can be referred to Figure 2 The "Morphological Denoising" step in .

[0145] The morphological opening operation is an operation in image processing, which removes small noise points in the image (that is, eliminates small noise and unimportant objects) through the process of first corrosion and then expansion, while retaining the main structure of the image.

[0146] Furthermore, the S4 step includes:

[0147] S41, using the erosion operation in the morphological opening operation combined with the unified expression paradigm to erode the binary image to obtain an eroded image.

[0148] refer to Figure 5 As shown, the step S41 specifically includes:

[0149] S411. Set the first structural element.

[0150] The first structure element is a basic structure with a certain shape. In the embodiment of the present application, the first structure element is set to 3, that is, the 3*3 first structure element is used to perform the morphological opening operation, wherein the "*" symbol can be understood as a multiplication symbol in the field of mathematics.

[0151] The first structure element operates within a specific area by defining a local neighborhood.

[0152] S412: Use the first structure element to cover the first target pixel and its area pixels, wherein the first target pixel is any pixel in the binary image. This step can be understood as Figure 5 The "First Structuring Element Traversal of the Image" step in .

[0153] The first target pixel can be understood as the currently traversed pixel point.

[0154] The domain pixels refer to pixels around the first target pixel. The first target pixel and its domain pixels form a nine-square rectangle. More specifically, the first target pixel is located in the middle of the nine-square rectangle, and the rest of the rectangle is the domain pixels of the first target pixel.

[0155] S413: When the pixel values ​​corresponding to the first target pixel and its domain pixels are both 255, the pixel values ​​corresponding to the first target pixel and its domain pixels are maintained at 255, and when the pixel value of the first target pixel is 0 and / or the pixel value of the domain pixels of the first target pixel is 0, the pixel values ​​corresponding to the first target pixel and its domain pixels are modified to 0 to form the eroded image. S413 can be understood as Figure 5 In the steps of "whether the first target pixel point structure element area is all 255, if so, keep the pixel value of the first target pixel, if not, set the pixel value of the first target pixel to 0". The first target pixel is the first target pixel point.

[0156] It should be understood that when the pixel values ​​corresponding to the first target pixel and its domain pixel are both 255, it indicates that the first target pixel and its domain pixel (ie, the first structural element) are both located in the glass area in the binary image.

[0157] When the pixel value of the first target pixel is 0 and / or the pixel value of the community pixel is 0, it indicates that the first target pixel and its community pixel (ie, the first structure element) are partially located in the window frame area of ​​the binary image.

[0158] Furthermore, the step S413 can be implemented by the following calculation formula:

[0159] ;

[0160] in, For the eroded image In Location The pixel value of the pixel at is the first structural element, The first structural element In Location, For the binary image In Location The corresponding unified expression paradigm.

[0161] It should be understood that the eroded image (ie, the eroded image) will have a smaller foreground area, and the edges of the objects will shrink inwards to remove small noises in the image.

[0162] S42, using the dilation operation in the morphological opening operation combined with the unified expression paradigm to dilate the eroded image to obtain the denoised image.

[0163] refer to Figure 5 As shown, the step S42 specifically includes:

[0164] S421. Set the second structural element.

[0165] The second structure element is a basic structure with a certain shape. In the embodiment of the present application, the second structure element is set to 3, that is, a 3*3 second structure element is used to perform the morphological opening operation, wherein the "*" symbol can be understood as a multiplication symbol in the field of mathematics.

[0166] S422. Use the second structure element to cover a second target pixel and its area pixels, wherein the second target pixel is any pixel in the eroded image.

[0167] The second target pixel is the current pixel in the second traversal process.

[0168] S423: When the pixel value of the second target pixel is 255 and / or the pixel value of the domain pixel of the second target pixel is 255, the pixel values ​​corresponding to the second target pixel and its domain pixel are both modified to 255, and when the pixel values ​​corresponding to the second target pixel and its domain pixel are both 0, the pixel values ​​corresponding to the second target pixel and its domain pixel are maintained to be 0, thereby forming the denoised image. The step S423 can be understood as Figure 5 In the steps of "whether the second target pixel point structure element area is all 0, if so, keep the pixel value of the second target pixel, if not, set the pixel value of the second target pixel to 255". The second target pixel is the second target pixel point.

[0169] It should be understood that when the pixel value of the second target pixel is 255 and / or the pixel value of the domain pixel of the second target pixel is 255, it indicates that the second target pixel and its domain pixel (that is, the second structural element) are partially located in the glass area in the corrosion image.

[0170] When the pixel values ​​corresponding to the second target pixel and its domain pixel are both 0, it indicates that the second target pixel and its domain pixel (ie, the second structure element) are both located in the window frame area of ​​the eroded image.

[0171] Furthermore, the step S423 can be implemented by the following calculation formula:

[0172] ;

[0173] in, For the denoised image In Location The pixel value of the pixel at is the second structural element, The second structural element In Location, For the eroded image In Location The pixel value of the pixel at .

[0174] It should be understood that the dilation operation is the image after the erosion operation. With further processing, it is able to fill in the tiny gaps left by the corrosion process, thus restoring the connectivity and shape of the object.

[0175] S5. Use connected component analysis to extract and process the denoised image to obtain a fine-grained segmentation result of the window glass. This step can be referred to Figure 2 The "connected component analysis" step and the "window glass instantiation extraction" step, more specifically, the "window glass instantiation extraction" step is to obtain the fine-grained segmentation result.

[0176] In the embodiment of the present application, a two-pass labeling method is used to obtain the fine-grained segmentation result (ie, extract the glass area of ​​the window image).

[0177] Furthermore, the step S5 includes:

[0178] S51. Create a first label map, initialize all labels in the first label map to zero, and set a label map variable to point to the first label map, wherein each of the labels has a one-to-one correspondence with one of the pixels in the denoised image.

[0179] This initialization operation means that all pixels in the denoised image have not been processed yet.

[0180] S52: Calculate the number of all pixels in the denoised image to obtain a first total number of pixels.

[0181] S53: Set a first preset value, and initialize the first preset value to 1.

[0182] S54: When the first preset value is less than or equal to the first total number of pixels, the following steps S541 and S542 are executed cyclically. The step S54 can be understood as the first traversal process in the two-pass labeling method.

[0183] S541. Obtain multiple neighbor pixels of a third target pixel using a preset connectivity method, and obtain multiple neighbor labels corresponding to the third target pixel from the label map variable based on the multiple neighbor pixels, wherein the third target pixel is any pixel in the denoised image.

[0184] Since the shape of the glass area of ​​the window is mostly rectangular, the preset connection mode adopts the eight-connection mode. Figure 7 As shown, the eight-connected mode is to use the pixel P2 located at the upper left of the third target pixel P1, the pixel P3 located above the third target pixel P1, the pixel P4 located at the upper right of the third target pixel P1, the pixel P5 located to the left of the third target pixel P1, the pixel P6 located to the right of the third target pixel P1, the pixel P7 located at the lower left of the third target pixel P1, the pixel P8 located below the third target pixel P1 and the pixel P9 located at the lower right of the third target pixel P1 as the neighbor pixels. That is, in Figure 7 In the 3*3 area, all pixels except the central pixel are neighbor pixels of the central pixel. The neighbor pixels include the neighbor pixels in the horizontal direction, vertical direction and diagonal direction.

[0185] In other embodiments, the preset connectivity mode may also adopt a four-connectivity mode. The four-connectivity mode is to use the pixel located above the third target pixel, the pixel located below the third target pixel, the pixel located to the left of the third target pixel, and the pixel located to the right of the third target pixel as the neighboring pixels. The plurality of neighboring pixels determined by the four-connectivity mode are cross-shaped in overall shape, that is, the adjacent pixels in the horizontal direction and the vertical direction are considered.

[0186] The step of "obtaining a plurality of neighbor pixels of the third target pixel by adopting a preset connectivity method" can be understood as a neighbor definition step.

[0187] refer to Figure 6 As shown, the step S541 can be understood as traversing each pixel in the denoised image in row and column order, and the traversal order can be from left to right and from top to bottom.

[0188] Furthermore, the multiple neighbor labels may be expressed in the following form:

[0189] ;

[0190] in, are multiple neighbor labels corresponding to the third target pixel, is the neighbor label corresponding to the pixel located to the left of the third target pixel, is a pixel located below the third target pixel, is a pixel located at the lower left of the third target pixel, is a pixel located at the lower right of the third target pixel.

[0191] S542: Obtain a reset label of the third target pixel based on the multiple neighbor labels.

[0192] Furthermore, the calculation formula of step S542 is:

[0193] ;

[0194] in, is the reset label of the third target pixel (that is, the denoised image In Location The reset label corresponding to the pixel at For the denoised image In Location The pixel value of the pixel at are multiple neighbor labels corresponding to the third target pixel, A non-zero label.

[0195] The reset label can be understood as a newly allocated label, and its allocation follows the calculation formula of step S542.

[0196] It should be understood that when hour, It can be understood as Figure 6 In the steps of "whether the third target pixel is 0, if so, the label value is 0", the label value refers to the value corresponding to the reset label of the third target pixel. hour, It can be understood as Figure 6 In the process of “Is the third target pixel 0? If not, does the third target pixel have neighbors? If not, set a new label”. hour, It can be understood as Figure 6 The steps of "whether the third target pixel is 0, if not, whether the third target pixel has neighbors, and if so, setting the same label value of the connected area" are performed.

[0197] S543. In the label map pointed to by the label map variable, modify the label corresponding to the third target pixel to the reset label, obtain a second label map, and modify the label map pointed to by the label map variable to the second label map.

[0198] S544: If at least one neighbor tag among the plurality of neighbor tags is different from other neighbor tags, the plurality of neighbor tags are recorded and processed to obtain a tag to be merged, and the tag is added to the tag set to be merged. S544 can be understood as Figure 6 In the steps of "whether they belong to the same label, if not, label record".

[0199] In other implementations, when each of the plurality of neighbor labels belongs to the same label, the original label is retained. Figure 6 In the steps of "Do they belong to the same label? If so, keep the original label".

[0200] S55: modify the second label map based on the label set to be merged to obtain a third label map. The step S55 can be understood as: Figure 6 The steps of "secondary traversal of denoised image and label merging" are as follows.

[0201] Furthermore, the step S55 includes:

[0202] S551: Merge all tags in the tag set to be merged to obtain a merged tag.

[0203] S552 modifies the second label map based on the merged label to obtain the third label map. The third label map is the final label map. In the third label map, pixels in the same connected area are marked with the same label.

[0204] The step S55 can be understood as the second traversal process in the two-pass marking method.

[0205] The step S55 can ensure that all connected areas use the same label.

[0206] S56: Obtain the fine-grained segmentation result based on the third label map.

[0207] Specifically, the step S56 may be to perform a segmentation operation on one of the labels in the third label map to obtain the fine-grained segmentation result.

[0208] The fine-grained segmentation result extracts the glass area in the window image.

[0209] The embodiment of the present application adopts a unified expression paradigm, morphological opening operations and connected component analysis to achieve the purpose of fine-grained segmentation of window images, so that the fine-grained semantic segmentation method of window images can achieve the segmentation of building facades without relying on deep learning models. Compared with the prior art, the embodiment of the present application can achieve accurate segmentation of building facades at a fine-grained level.

[0210] In some embodiments, after the step S56, the step S5 further includes:

[0211] S57. Calculate the centroid corresponding to the first target label based on the third label map, wherein the first target label is any label in the third label map.

[0212] The step S57 includes:

[0213] S571. Calculate the number of pixels of the first target label in the third label image to obtain a third total number of pixels.

[0214] S572. Calculate the centroid corresponding to the first target tag based on the third total number of pixels and a fifth preset algorithm.

[0215] Furthermore, the step S572 can be implemented by the following calculation formula:

[0216] ;

[0217] in, is the first target label, is the centroid corresponding to the first target label, is the third total number of pixels, For all pixels of the first target label The sum of the coordinates, For all pixels of the first target label The sum of coordinates.

[0218] In some embodiments, after the step S56, the step S5 further includes:

[0219] S58. Calculate a bounding box corresponding to a second target label based on the third label map, wherein the second target label is any label in the third label map.

[0220] The bounding box is a rectangular area that encloses a connected area. The connected area is the area in the third label map labeled with all the second target labels. The fine-grained segmentation result is fitted using the rectangular area to obtain a fitting result. The fitting is a more refined fine-grained segmentation result.

[0221] The S58 can be realized by the following calculation formula:

[0222] ;

[0223] in, is the bounding box, is the left edge of the bounding box, is the upper boundary of the bounding box, is the right edge of the bounding box, is the lower boundary of the bounding box, is the connected area, is the second target label.

[0224] It should be understood that the present application aims to perform fine-grained segmentation of the glass area and window frame area of ​​the window based on a single window image, and realize the semantic division of glass and window frame and the instance segmentation of glass. More specifically, the fine-grained semantic segmentation method of window images proposed in the present application can effectively classify window images with different color distributions, and realize the conversion of different window categories into a unified expression paradigm; considering the problem of small noise in the unified expression paradigm, the image is denoised using morphological operations to obtain a binary image with semantic distinction between window frame and glass (that is, the denoised image), and then, through connected component analysis, the glass of the binary image (that is, the denoised image) is instantiated and extracted using a two-pass labeling method, so as to finally obtain a fine-grained segmentation result of the window glass.

[0225] like Figure 8, which is a principle block diagram of a fine-grained semantic segmentation device for window images provided in the second aspect of the embodiment of the present application. Figure 8 In the embodiment, the fine-grained semantic segmentation device 100 of the window image comprises:

[0226] The window image acquisition module 101 is used to acquire window images.

[0227] The binary image acquisition module 102 is used to pre-process the window image to obtain a binary image.

[0228] The unified expression paradigm generating module 103 is used to generate a unified expression paradigm based on the binarized image.

[0229] The denoised image acquisition module 104 is used to perform denoising on the binary image by using a morphological opening operation combined with the unified expression paradigm to obtain a denoised image.

[0230] The fine-grained segmentation result obtaining module 105 is used to extract and process the denoised image using connected component analysis to obtain a fine-grained segmentation result of the window glass.

[0231] A third aspect of the embodiments of the present application provides a terminal device. The principle block diagram of the terminal device can be as follows: Fig. 9 As shown. The terminal device includes a processor, a memory, a network interface, a display screen and a temperature sensor connected via a system bus. Among them, the processor is used to provide computing and control capabilities. The memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the terminal device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a fine-grained semantic segmentation method for window images is implemented. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the temperature sensor is pre-set inside the terminal device to detect the operating temperature of the internal device.

[0232] Those skilled in the art will understand that Fig. 9 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the scheme of the present invention, and does not constitute a limitation on the terminal device to which the scheme of the present invention is applied. The specific terminal device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0233] In some embodiments, an embodiment of the present application provides a terminal device, which includes a processor and a memory, the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute the steps of the fine-grained semantic segmentation method of window images provided in the first aspect of the above-mentioned embodiment of the present application.

[0234] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium is used to store a computer program, wherein the computer program enables a computer to execute the steps of the fine-grained semantic segmentation method for window images provided in the first aspect of the embodiment of the present application.

[0235] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0236] The technical features of the above embodiments can be combined without changing the basic principles of the present application. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0237] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the patent application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the scope of patent protection of the present application shall be subject to the attached claims.

Claims

1. A fine-grained semantic segmentation method for window images, characterized in that: include: Get window images; Preprocessing the window image to obtain a binary image; Generate a unified expression paradigm based on the binary image; The binary image is subjected to denoising processing by using a morphological opening operation combined with the unified expression paradigm to obtain a denoised image; Connected component analysis is used to extract and process the denoised image to obtain a fine-grained segmentation result of the window glass; The step of preprocessing the window image to obtain a binary image includes: graying the window image to obtain a gray image; binarizing the gray image using an Otsu threshold method to obtain the binary image; The step of using the Otsu threshold method to binarize the grayscale image to obtain the binarized image comprises: Calculating the frequency of occurrence of each gray level in the grayscale image, and adding each of the frequencies to a frequency set; Calculating the number of pixels in the grayscale image to obtain a second total number of pixels; Set a second preset value and initialize it to zero; In the case where the second preset value is less than the length of the frequency set, the following steps are executed in a loop: calculating the foreground probability of the target frequency according to the first preset algorithm, wherein the target frequency is any element in the frequency set; calculating the foreground mean of the target frequency according to the second preset algorithm; calculating the background probability of the target frequency according to the third preset algorithm; calculating the background mean of the target frequency according to the fourth preset algorithm; calculating the inter-class variance corresponding to the target frequency based on the foreground probability, the foreground mean, the background probability and the background mean, and adding the inter-class variance to the variance set; performing a self-increment operation on the second preset value; Selecting the largest inter-class variance in the variance set as the target inter-class variance, and using the frequency corresponding to the target inter-class variance as the target threshold; Calculating the pixel value of each pixel in the grayscale image using the target threshold to obtain the binary image; The calculation formula of the first preset algorithm is: ; The calculation formula of the second preset algorithm is ; The calculation formula of the third preset algorithm is ; The calculation formula of the fourth preset algorithm is ; The calculation formula of the inter-class variance is ; in, The target frequency The corresponding foreground probability, is the second total number of pixels, is the target frequency, is the frequency set, is the subscript of the frequency set, The target frequency The corresponding foreground mean, The target frequency The corresponding background probability is The target frequency The corresponding background mean, The target frequency The corresponding between-class variance.

2. The fine-grained semantic segmentation method of window images according to claim 1, characterized in that: The step of generating a unified expression paradigm based on the binary image comprises: Calculating the average value of each row of pixel values ​​in the binary image to obtain multiple row means; Calculating the average value of each column of pixel values ​​in the binary image to obtain multiple column means; Setting a first threshold; Counting the number of all row means greater than the first threshold value to obtain a first statistical value; Counting the number of all column means greater than the first threshold value to obtain a second statistical value; When the first statistical value satisfies a first preset condition and the second statistical value satisfies a second preset condition, taking the binarized image as the unified expression paradigm; When the first statistical value does not satisfy a first preset condition and the second statistical value does not satisfy a second preset condition, each pixel in the binary image is inverted and used as the unified expression paradigm.

3. The fine-grained semantic segmentation method of window images according to claim 1, characterized in that: The step of performing denoising on the binary image by using the morphological opening operation combined with the unified expression paradigm to obtain the denoised image comprises: The binary image is corroded by using the corrosion operation in the morphological opening operation combined with the unified expression paradigm to obtain a corroded image; The eroded image is expanded by using the expansion operation in the morphological opening operation in combination with the unified expression paradigm to obtain the denoised image.

4. The fine-grained semantic segmentation method of window images according to claim 1, characterized in that: The step of extracting and processing the denoised image using connected component analysis to obtain a fine-grained segmentation result of the window glass comprises: Creating a first label map, initializing all labels in the first label map to zero, and setting a label map variable to point to the first label map, wherein each of the labels has a one-to-one correspondence with one of the pixels in the denoised image; Calculating the number of all pixels in the denoised image to obtain a first total number of pixels; Set a first preset value, and initialize the first preset value to 1; In the case where the first preset value is less than or equal to the first total number of pixels, the following steps are executed in a loop: a plurality of neighbor pixels of a third target pixel are obtained in a preset connectivity manner, and a plurality of neighbor labels corresponding to the third target pixel are obtained from the label map variable based on the plurality of neighbor pixels, wherein the third target pixel is any pixel in the denoised image; a reset label of the third target pixel is obtained based on the plurality of neighbor labels; in the label map pointed to by the label map variable, the label corresponding to the third target pixel is modified to the reset label to obtain a second label map, and the label map pointed to by the label map variable is modified to the second label map; in the case where at least one neighbor label among the plurality of neighbor labels is different from other neighbor labels, the plurality of neighbor labels are recorded and processed to obtain labels to be merged, and added to the label set to be merged; Modify the second label map based on the label set to be merged to obtain a third label map; The fine-grained segmentation result is obtained based on the third label map.

5. The fine-grained semantic segmentation method of window images according to claim 4, characterized in that: After the step of obtaining the fine-grained segmentation result based on the third label map, the step of extracting the denoised image using connected component analysis to obtain the fine-grained segmentation result of the window glass further includes: The centroid corresponding to the first target label is calculated based on the third label map, wherein the first target label is any label in the third label map.

6. The fine-grained semantic segmentation method of window images according to claim 4, characterized in that: After the step of obtaining the fine-grained segmentation result based on the third label map, the step of extracting the denoised image using connected component analysis to obtain the fine-grained segmentation result of the window glass further includes: A bounding box corresponding to a second target label is calculated based on the third label map, wherein the second target label is any label in the third label map.

7. A fine-grained semantic segmentation device for window images, characterized in that: include: A window image acquisition module, used for acquiring window images; A binary image acquisition module, used for preprocessing the window image to obtain a binary image; A unified expression paradigm generation module, used to generate a unified expression paradigm based on the binarized image; A denoised image acquisition module, used for performing denoising processing on the binary image by using a morphological opening operation combined with the unified expression paradigm to obtain a denoised image; A fine-grained segmentation result obtaining module, used for extracting and processing the denoised image using connected component analysis to obtain a fine-grained segmentation result of the window glass; The fine-grained semantic segmentation device of the window image is also used to grayscale the window image to obtain a grayscale image; and binarize the grayscale image using the Otsu threshold method to obtain the binarized image; The fine-grained semantic segmentation device of the window image is also used to calculate the frequency of occurrence of each gray level in the gray image, and add each of the frequencies to the frequency set; Calculating the number of pixels in the grayscale image to obtain a second total number of pixels; Set a second preset value and initialize it to zero; In the case where the second preset value is less than the length of the frequency set, the following steps are executed in a loop: calculating the foreground probability of the target frequency according to the first preset algorithm, wherein the target frequency is any element in the frequency set; calculating the foreground mean of the target frequency according to the second preset algorithm; calculating the background probability of the target frequency according to the third preset algorithm; calculating the background mean of the target frequency according to the fourth preset algorithm; calculating the inter-class variance corresponding to the target frequency based on the foreground probability, the foreground mean, the background probability and the background mean, and adding the inter-class variance to the variance set; performing a self-increment operation on the second preset value; Selecting the largest inter-class variance in the variance set as the target inter-class variance, and using the frequency corresponding to the target inter-class variance as the target threshold; Calculating the pixel value of each pixel in the grayscale image using the target threshold to obtain the binary image; The calculation formula of the first preset algorithm is: ; The calculation formula of the second preset algorithm is ; The calculation formula of the third preset algorithm is ; The calculation formula of the fourth preset algorithm is ; The calculation formula of the inter-class variance is ; in, The target frequency The corresponding foreground probability, is the second total number of pixels, is the target frequency, is the frequency set, is the subscript of the frequency set, The target frequency The corresponding foreground mean, The target frequency The corresponding background probability is The target frequency The corresponding background mean, The target frequency The corresponding between-class variance.

8. A terminal device, characterized in that: include: A processor and a memory, the memory being used to store a computer program, the processor being used to call and run the computer program stored in the memory to execute the steps of the fine-grained semantic segmentation method for window images as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: Used to store a computer program, wherein the computer program enables a computer to execute the steps of the fine-grained semantic segmentation method of window images as described in any one of claims 1 to 6 above.

Images