A multi-modal channel residual correction method for building facade defect segmentation

Abstract

The application discloses a building facade defect segmentation method based on multi-modal channel residual correction, and belongs to the technical field of image processing and computer vision. In order to solve the problems that the existing unified segmentation or unified correction is difficult to adapt to different error sources of cracks and hollows, and is easy to introduce defect residuals into a background channel, the application obtains an RGB image and an infrared image corresponding to a same facade area of a building to be detected, extracts and fuses multi-scale image features to obtain basic fusion features, generates cross-modal residual features, fuses the cross-modal residual features with the basic fusion features to obtain enhanced fusion features, generates a basic segmentation output based on the enhanced fusion features, and then respectively performs residual correction on a crack channel and a hollow channel, so that no defect residual term is superimposed on the background channel. As a result, the disturbance of the defect residual term on the background channel is reduced, the segmentation deviation of the cracks and the hollows is improved, and the application is suitable for the field of pixel-level defect identification of building facade cracks and hollows.

Description

Technical Field

[0001] This invention belongs to the field of image processing and computer vision technology, and specifically relates to a method for segmenting defects on building facades using multimodal channel residual correction. Background Technology

[0002] Building facades are constantly exposed to the outdoor environment and are prone to defects such as cracks and hollow areas due to factors such as material aging, temperature and humidity changes, construction quality, and external loads. With the development of image acquisition equipment and computer vision technology, using RGB and infrared images for defect identification of building facades has become a common technical approach in facade inspection. RGB images can reflect the texture, color, edges, and surface morphology of the facade, while infrared images can reflect the differences in infrared imaging response across different areas of the facade. The two technologies are complementary in defect identification.

[0003] Existing image processing methods for building facade defects typically include defect classification, target detection, and region segmentation based on RGB or infrared images. Some methods fuse RGB and infrared image features before pixel-level segmentation. Other methods extract visible light and infrared features separately using dual-branch networks, then generate segmentation results using channel weighting, attention mechanisms, feature stitching, and feature decoding. Still other methods target single defects such as hollowness or cracks in exterior walls, employing infrared thresholding, region growing, thermal response analysis, or deep learning models for identification.

[0004] However, although both cracks and hollow areas are defects in building facades, their appearance and sources of error in images differ. Cracks are typically long, discontinuous, and narrow, easily affected by wall textures, shadows, stains, and background edges, requiring more attention to edge, texture, and local detail information for identification. Hollow areas, on the other hand, may exhibit blurred boundaries, weak response, and indistinct transition with the background, relying more on the differences and complementary information between RGB and infrared images for identification. Existing fusion segmentation methods, if they only generate a uniform segmentation result after feature fusion or apply a uniform correction method to multiple category channels, struggle to separately adapt to the errors caused by the long, thin structure of cracks and the weak response errors of hollow areas.

[0005] Furthermore, in the three segmentation tasks of background, cracks, and hollow areas, the background region typically accounts for a higher proportion. If the defect correction is applied indiscriminately to the background, crack, and hollow channels, the correction information for cracks or hollow areas can easily be introduced into the background channel, causing background response disturbances and affecting the final pixel-level classification results. Therefore, it is necessary to perform separate corrections for the crack and hollow channels based on the fusion segmentation of RGB and infrared images, and to limit the scope of the defect residual term to accommodate the different sources of segmentation error for cracks and hollow areas on building facades. Summary of the Invention

[0006] To address the problem that existing technologies using uniform segmentation or uniform correction are ill-suited to the different error sources of cracks and voids, and that defect residuals are easily introduced into the background channel, this invention proposes the following solution: A method for segmenting building facade defects using multimodal channel residual correction includes the following steps: S1. Obtain the RGB and infrared images corresponding to the same exterior facade area of ​​the building to be detected; S2. Extract RGB image features and infrared image features respectively; S3. Fuse RGB image features and infrared image features of the same scale at multiple scale levels to obtain basic fusion features at multiple scale levels. The basic fusion features include at least one shallow fusion feature. S4. Generate cross-modal residual features based on RGB image features, infrared image features, and basic fusion features; S5. Fuse the basic fusion features with the cross-modal residual features to obtain the enhanced fusion features; S6. Input the enhanced fusion features into the segmentation decoder, decode the enhanced fusion features to obtain the decoded features, and generate the basic segmentation outputs for the background, crack and hollow channels based on the decoded features. The basic segmentation outputs include the basic output of the background channel, the basic output of the crack channel and the basic output of the hollow channel. S7. Generate crack channel residuals based on decoding features and at least one shallow fusion feature; S8. Generate the hollow channel residual based on decoding features and cross-modal residual features; S9. Before performing pixel-by-pixel classification on the category channel score output, the crack channel residual is superimposed on the crack channel base output to obtain the crack channel correction output, the hollow channel residual is superimposed on the hollow channel base output to obtain the hollow channel correction output, and the background channel uses the background channel base output and does not superimpose the defect residual term. S10. Perform pixel-by-pixel classification based on the category channel score output consisting of the background channel base output, crack channel correction output, and hollow channel correction output to obtain the three-category segmentation results of the exterior facade of the building to be detected: background, cracks, and hollows.

[0007] Furthermore, S1, obtaining the RGB image and infrared image corresponding to the same facade area of ​​the building to be detected, includes establishing the correspondence between the RGB image and the infrared image according to the sample identifier, and performing spatial registration or coordinate alignment processing on the RGB image and the infrared image so that the RGB image and the infrared image have a correspondence at the pixel scale or feature scale.

[0008] Furthermore, S2 describes extracting RGB image features and infrared image features respectively, including setting RGB feature extraction branches and infrared feature extraction branches, wherein the RGB feature extraction branches and infrared feature extraction branches do not share parameters, and outputting RGB image features and infrared image features at multiple scale levels respectively.

[0009] Furthermore, S3 describes fusing RGB image features and infrared image features of the same scale at multiple scale levels. This includes concatenating RGB image features and infrared image features of the same scale in the channel dimension at each scale level, and performing channel compression and fusion through 1×1 convolution, normalization, and nonlinear activation to obtain the basic fused features of the corresponding scale level.

[0010] Furthermore, the generation of cross-modal residual features in S4 includes generating candidate cross-modal residual features based on RGB image features, infrared image features and the difference between the two, applying gate weights and residual scaling coefficients to the candidate cross-modal residual features to obtain gated residual features, and using the gated residual features or features transformed from the gated residual features as cross-modal residual features.

[0011] Furthermore, S6 describes obtaining decoded features by decoding the enhanced fusion features through a segmentation decoder, which includes projecting and upsampling the enhanced fusion features at multiple scale levels respectively, and fusing the upsampled enhanced fusion features at multiple scale levels to obtain decoded features used to generate the basic segmentation output.

[0012] Furthermore, the at least one shallow fusion feature mentioned in S7 includes a first-scale fusion feature and a second-scale fusion feature. The first-scale fusion feature and the second-scale fusion feature correspond to the fusion features of the RGB image features and infrared image features of the first two shallow-scale levels of the encoder, respectively. The crack channel residual is generated based on the decoding feature, the first-scale fusion feature and the second-scale fusion feature.

[0013] Furthermore, in step S7, when generating the crack channel residual, a crack residual effect weight is also generated; in step S8, when generating the hollow channel residual, a hollow residual effect weight is also generated; in step S9, the crack channel residual is weighted according to the crack residual effect weight to obtain the crack correction amount used to be superimposed on the crack channel base output; and the hollow channel residual is weighted according to the hollow residual effect weight to obtain the hollow correction amount used to be superimposed on the hollow channel base output.

[0014] Furthermore, the pixel-by-pixel classification described in S10 includes taking the background channel base output, crack channel correction output, and hollow channel correction output as category channel score outputs for the same pixel location, and determining whether the pixel location belongs to the background, crack, or hollow based on the comparison result of the category channel score outputs for the same pixel location.

[0015] Based on the same inventive concept, the present invention also proposes a computer storage medium storing a computer program, which, when executed by a processor, implements the method described in any of the above-mentioned embodiments.

[0016] Compared with the prior art, the present invention has the following beneficial effects: By acquiring RGB and infrared images of the same facade area of ​​the building to be detected, and extracting RGB and infrared image features respectively, the texture, edge, and local structural information of the facade and the infrared imaging response difference information can be used together in subsequent segmentation processing, thereby reducing the impact of insufficient single-modal information on crack and hollow recognition.

[0017] By fusing RGB image features and infrared image features at multiple scale levels, basic fusion features, including shallow fusion features, are obtained. Cross-modal residual features are generated based on RGB image features, infrared image features, and basic fusion features. The basic fusion features are then fused with the cross-modal residual features to obtain enhanced fusion features. The enhanced fusion features are then input into the segmentation decoder to generate basic segmentation outputs for three channels: background, cracks, and voids. This allows the basic segmentation outputs to simultaneously carry multi-scale fusion information and cross-modal residual information, and provides three types of basic outputs for subsequent crack and void channel correction, thereby enhancing the connection between the basic segmentation outputs and the defect channel residual correction.

[0018] By generating crack channel residuals based on decoding features and at least one shallow fusion feature, edge, texture, and local structural information in the shallow fusion feature are used for crack channel correction, thereby improving the problems of missed detection and discontinuous connection caused by cracks being thin, discontinuous, and having insufficient local response.

[0019] By generating hollow channel residuals based on decoding features and cross-modal residual features, the difference between RGB image features and infrared image features is used to participate in hollow channel correction, thereby improving the problem of incomplete segmentation caused by weak response, blurred boundaries and local missing parts in hollow areas.

[0020] By superimposing the crack channel residual and the hollow channel residual onto the corresponding defect channel before pixel-by-pixel classification, and by making the background channel use the background channel base output without superimposing the defect residual term, the scope of the defect residual term can be limited, the disturbance of the defect correction amount to the background channel can be reduced, thereby improving the stability of the three-class segmentation results of background, crack and hollow.

[0021] This invention has the ability to perform multimodal fusion segmentation of RGB and infrared images of building facades, and to perform residual correction on crack channels and hollow channels respectively. It can reduce the disturbance to the background channel caused by the uniform correction method and improve the segmentation deviation caused by different error sources of cracks and hollows. It is applicable to the field of pixel-level defect identification of cracks and hollows on building facades. Attached Figure Description

[0022] Figure 1 This is a flowchart of the building facade defect segmentation method for multimodal channel residual correction as described in the implementation method; Figure 2 These are comparison images before and after crack channel correction as described in the implementation method; Figure 3 These are comparison images before and after the hollow channel correction described in the implementation method; Figure 4 This is a schematic diagram of the network structure described in the implementation method. Detailed Implementation

[0023] The technical solutions in the embodiments of the present invention will now be clearly and completely described in conjunction with the accompanying drawings.

[0024] Implementation Method 1 A method for segmenting building facade defects using multimodal channel residual correction includes the following steps: S1. Obtain the RGB and infrared images corresponding to the same exterior facade area of ​​the building to be detected; Preferably, the facade is the external enclosure surface of the building; S2. Extract RGB image features and infrared image features respectively; Preferably, the infrared image is converted into a single-channel tensor or a multi-channel tensor and then used as a corresponding input to the RGB image; S3. Fuse RGB image features and infrared image features of the same scale at multiple scale levels to obtain basic fusion features at multiple scale levels. The basic fusion features include at least one shallow fusion feature. S4. Generate cross-modal residual features based on RGB image features, infrared image features, and basic fusion features; S5. Fuse the basic fusion features with the cross-modal residual features to obtain the enhanced fusion features; S6. Input the enhanced fusion features into the segmentation decoder, decode the enhanced fusion features to obtain the decoded features, and generate the basic segmentation outputs for the background, crack and hollow channels based on the decoded features. The basic segmentation outputs include the basic output of the background channel, the basic output of the crack channel and the basic output of the hollow channel. S7. Generate crack channel residuals based on decoding features and at least one shallow fusion feature; S8. Generate the hollow channel residual based on decoding features and cross-modal residual features; S9. Before performing pixel-by-pixel classification on the category channel score output, the crack channel residual is superimposed on the crack channel base output to obtain the crack channel correction output, the hollow channel residual is superimposed on the hollow channel base output to obtain the hollow channel correction output, and the background channel uses the background channel base output and does not superimpose the defect residual term. S10. Perform pixel-by-pixel classification based on the category channel score output consisting of the background channel base output, crack channel correction output, and hollow channel correction output to obtain the three-category segmentation results of the exterior facade of the building to be detected: background, cracks, and hollows.

[0025] Through the above steps, the multi-scale features corresponding to RGB and infrared images can first form a basic segmentation output, and then residual correction is performed on the crack channel and the hollow channel respectively, so that the background, crack and hollow outputs can form a clear category channel scoring relationship before pixel-by-pixel classification.

[0026] Furthermore, S1, obtaining the RGB image and infrared image corresponding to the same facade area of ​​the building to be detected, includes establishing the correspondence between the RGB image and the infrared image according to the sample identifier, and performing spatial registration or coordinate alignment processing on the RGB image and the infrared image so that the RGB image and the infrared image have a correspondence at the pixel scale or feature scale.

[0027] By establishing a correspondence based on sample identifiers and performing spatial registration or coordinate alignment, RGB image features and infrared image features can have a consistent spatial correspondence basis during subsequent fusion, reducing the impact of misalignment of different modal features on pixel-level segmentation results.

[0028] Furthermore, S2 describes extracting RGB image features and infrared image features respectively, including setting RGB feature extraction branches and infrared feature extraction branches, wherein the RGB feature extraction branches and infrared feature extraction branches do not share parameters, and outputting RGB image features and infrared image features at multiple scale levels respectively.

[0029] By setting up RGB feature extraction branches and infrared feature extraction branches with non-shared parameters, modal features in the two images can be extracted separately, allowing the texture and edge information in the RGB image and the response difference information in the infrared image to enter the subsequent multi-scale fusion process respectively.

[0030] Furthermore, S3 describes fusing RGB image features and infrared image features of the same scale at multiple scale levels. This includes concatenating RGB image features and infrared image features of the same scale in the channel dimension at each scale level, and performing channel compression and fusion through 1×1 convolution, normalization, and nonlinear activation to obtain the basic fused features of the corresponding scale level.

[0031] By performing channel-dimensional stitching, channel compression, and fusion of RGB image features and infrared image features at each scale level, basic fusion features at the corresponding scale level can be formed, enabling multimodal information at different scales to enter the segmentation decoder and residual correction path.

[0032] Furthermore, the generation of cross-modal residual features in S4 includes generating candidate cross-modal residual features based on RGB image features, infrared image features and the difference between the two, applying gate weights and residual scaling coefficients to the candidate cross-modal residual features to obtain gated residual features, and using the gated residual features or features transformed from the gated residual features as cross-modal residual features.

[0033] By generating candidate cross-modal residual features based on RGB image features, infrared image features, and the difference between the two, and applying gating weights and residual scaling factors to them, the scope and intensity of the cross-modal residual features can be controlled, making them usable for subsequent hollow channel residual generation.

[0034] Furthermore, S6 describes obtaining decoded features by decoding the enhanced fusion features through a segmentation decoder, which includes projecting and upsampling the enhanced fusion features at multiple scale levels respectively, and fusing the upsampled enhanced fusion features at multiple scale levels to obtain decoded features used to generate the basic segmentation output.

[0035] By projecting, upsampling, and fusing enhanced fusion features at multiple scale levels, decoding features for generating basic segmentation outputs can be formed, enabling the background channel basic output, crack channel basic output, and hollow channel basic output to share a common source of decoding features.

[0036] Furthermore, the at least one shallow fusion feature mentioned in S7 includes a first-scale fusion feature and a second-scale fusion feature. The first-scale fusion feature and the second-scale fusion feature correspond to the fusion features of the RGB image features and infrared image features of the first two shallow-scale levels of the encoder, respectively. The crack channel residual is generated based on the decoding feature, the first-scale fusion feature and the second-scale fusion feature.

[0037] By using first-scale fusion features and second-scale fusion features for crack channel residual generation, crack channel correction can utilize edge, texture, and local structural information preserved at the shallow scale, thereby improving local missing and connection discontinuities in the crack channel base output.

[0038] Furthermore, in step S7, when generating the crack channel residual, a crack residual effect weight is also generated; in step S8, when generating the hollow channel residual, a hollow residual effect weight is also generated; in step S9, the crack channel residual is weighted according to the crack residual effect weight to obtain the crack correction amount used to be superimposed on the crack channel base output; and the hollow channel residual is weighted according to the hollow residual effect weight to obtain the hollow correction amount used to be superimposed on the hollow channel base output.

[0039] By generating separate weights for crack residuals and hollow residuals, and then weighting the crack channel residuals and hollow channel residuals separately, the influence of crack correction and hollow correction on the corresponding defect channels can be controlled separately, thus avoiding the mixing of correction methods for different defect channels.

[0040] Furthermore, the pixel-by-pixel classification described in S10 includes taking the background channel base output, crack channel correction output, and hollow channel correction output as category channel score outputs for the same pixel location, and determining whether the pixel location belongs to the background, crack, or hollow based on the comparison result of the category channel score outputs for the same pixel location.

[0041] By comparing the background channel base output, crack channel correction output, and hollow channel correction output as category channel score outputs at the same pixel location, the background, crack, or hollow category can be determined at the pixel level, and the corrected category channel score output is transformed into a three-class segmentation result.

[0042] The method described in this embodiment can be executed by a processor calling a computer program, which can be stored in a computer storage medium. When the computer program is executed by the processor, the above-mentioned method for segmenting building facade defects using multimodal channel residual correction can be implemented.

[0043] Implementation Method 2 This embodiment uses RGB and infrared images of the same exterior facade area of ​​the building to be inspected as a specific example, combined with... Figure 1 This paper further explains the method for segmenting building facade defects using multimodal channel residual correction. In this embodiment, the facade refers to the external enclosure surface of the building. RGB images are used to provide information on the texture, edges, and local structure of the facade, while infrared images are used to provide information related to differences in the infrared imaging response of the facade.

[0044] In this embodiment, the cross-modal residual features are used to fuse with the basic fusion features to obtain enhanced fusion features, and also to participate in the generation of residuals for the hollow channel. The enhanced fusion features are input into the segmentation decoder to obtain decoding features, and the basic segmentation outputs for the background, crack, and hollow channels are generated based on the decoding features.

[0045] During execution S1, RGB and infrared images corresponding to the same facade area of ​​the building to be detected are acquired. A correspondence is established between the RGB and infrared images based on sample identifiers, and spatial registration or coordinate alignment is performed on the RGB and infrared images to ensure a correspondence at the pixel or feature scale. The infrared image is converted into a single-channel or multi-channel tensor and then used as input to correspond with the RGB image.

[0046] During the execution of S2, RGB feature extraction branches and infrared feature extraction branches are set up, and these branches do not share parameters. The RGB feature extraction branch is used to extract RGB image features at multiple scale levels, and the infrared feature extraction branch is used to extract infrared image features at multiple scale levels. Thus, the RGB image features and infrared image features retain the image representation in their respective modalities and serve as inputs for subsequent basic fusion and cross-modal residual feature generation.

[0047] During S3 execution, RGB image features and infrared image features of the same scale at multiple scale levels are fused to obtain basic fused features at multiple scale levels. These basic fused features include at least one shallow fused feature. Specifically, at each scale level, RGB image features and infrared image features of the same scale are concatenated along the channel dimension, and channel compression and fusion are performed through 1×1 convolution, normalization, and nonlinear activation to obtain the basic fused features for the corresponding scale level. The shallow fused features are derived from the shallow scale levels of the basic fused features and are used for subsequent crack channel residual generation.

[0048] During execution S4, cross-modal residual features are generated based on RGB image features, infrared image features, and basic fusion features. The cross-modal residual features are feature-level residual representations formed based on RGB image features, infrared image features, and their differences. In one specific implementation, candidate cross-modal residual features are generated based on RGB image features, infrared image features, and their differences. Gating weights and residual scaling factors are applied to these candidate cross-modal residual features to obtain gated residual features. These gated residual features, or features transformed from gated residual features, are then used as the cross-modal residual features.

[0049] The generation process of cross-modal residual features can be represented as follows:

[0050] in, Indicates basic fusion characteristics; Represents candidate cross-modal residual features; This represents the gating weight, used to control the scope of candidate cross-modal residual features; Indicates the residual scaling factor; This represents the gated residual term superimposed on the base fusion feature; This represents the enhanced fusion feature. The enhanced fusion feature is used as input to the segmentation decoder. Features obtained by further transformation of the gated residual term or the gated residual term are also used in the generation of the hollow channel residual.

[0051] During the execution of S5, the basic fusion features are fused with the cross-modal residual features to obtain enhanced fusion features. These enhanced fusion features are then used as input to the segmentation decoder, enabling both the basic fusion features and the cross-modal residual features to participate in the subsequent decoding process.

[0052] During S6 execution, the enhanced fusion features are input into the segmentation decoder. The segmentation decoder decodes the enhanced fusion features to obtain decoded features, and generates basic segmentation outputs for three channels: background, cracks, and voids. Specifically, the enhanced fusion features at multiple scale levels are projected and upsampled, and the upsampled enhanced fusion features at multiple scale levels are fused to obtain the decoded features used to generate the basic segmentation outputs. The basic segmentation outputs include basic outputs for the background channel, crack channel, and void channel.

[0053] The basic segmentation output can be represented as:

[0054] in, Indicates the basic output of the background channel; Indicates the basic output of the crack channel; This represents the base output of the hollow channel. The base segmentation output is the initial category channel score output generated by the segmentation decoder before crack channel correction and hollow channel correction.

[0055] During execution of S7, a crack channel residual is generated based on the decoded features and at least one shallow fusion feature. Further, the at least one shallow fusion feature includes a first-scale fusion feature and a second-scale fusion feature. The first-scale fusion feature and the second-scale fusion feature correspond to the fusion features of the RGB image features and infrared image features from the first two shallow-scale levels of the encoder, respectively. The crack channel residual is generated based on the decoded features, the first-scale fusion feature, and the second-scale fusion feature. The decoded features provide semantic localization information, while the shallow fusion features provide edge, texture, and local structure information; both are used together to generate the crack channel residual.

[0056] When generating the crack channel residual, a crack residual effect weight is also generated. The crack channel correction output can be expressed as:

[0057] in, Indicates the basic output of the crack channel; Indicates the residual of the crack channel; This represents the weight of the crack residual effect, used to control the intensity of the influence of the crack channel residual on the crack channel output; This indicates the output of the crack channel correction.

[0058] During the execution of S8, the hollow channel residual is generated based on the decoding features and cross-modal residual features. The cross-modal residual features reflect the differences and complementary relationships between RGB image features and infrared image features, while the decoding features provide semantic localization information. Both are used together to generate the hollow channel residual.

[0059] When generating the void channel residual, a weight for the void residual effect is also generated. The void channel correction output can be expressed as:

[0060] in, Indicates the basic output of the hollow channel; Indicates the residual value of the hollow channel; This represents the weight of the hollow channel residual effect, which is used to control the intensity of the influence of the hollow channel residual on the hollow channel output. This indicates the output of the hollow drum channel correction.

[0061] When executing S9, before performing pixel-by-pixel classification on the category channel scoring output, the crack channel residual is superimposed on the crack channel base output to obtain the crack channel correction output, and the hollow channel residual is superimposed on the hollow channel base output to obtain the hollow channel correction output. The background channel uses the background channel base output and does not superimpose defect residual terms. Specifically, the crack channel residual is weighted according to the crack residual's effect weight to obtain the crack correction amount superimposed on the crack channel base output, and the hollow channel residual is weighted according to the hollow residual's effect weight to obtain the hollow correction amount superimposed on the hollow channel base output. The background channel does not receive crack correction amounts or hollow correction amounts.

[0062] The final category channel score output used for pixel-by-pixel classification can be represented as:

[0063] in,

[0064]

[0065]

[0066] in, This represents the category channel score output consisting of the background channel, crack channel, and hollow channel; Indicates background channel output; This indicates the output of the crack channel correction. Indicates the output of the hollow channel correction; This represents the basic output of the background channel. From the above relationships, it can be seen that the crack channel residual is only superimposed on the crack channel basic output, the hollow channel residual is only superimposed on the hollow channel basic output, and the background channel uses the background channel basic output without superimposing defect residual terms.

[0067] During execution S10, pixel-by-pixel classification is performed based on the category channel score output, which consists of the background channel base output, crack channel correction output, and hollow channel correction output, resulting in three segmentation results for the exterior facade of the building to be detected: background, cracks, and hollow areas. Further, the background channel base output, crack channel correction output, and hollow channel correction output are used as category channel score outputs for the same pixel location, and the pixel location is determined to belong to the background, cracks, or hollow areas based on the comparison results of the category channel score outputs at the same pixel location.

[0068] like Figure 2 As shown, Figure 2 This is a comparison image before and after crack channel correction. The marked areas in the image represent crack areas, the background areas represent defect areas not currently displayed, and the boxed areas in the RGB image represent magnified local areas. Figure 2 The columns, from left to right, are: RGB image, actual crack annotation, segmentation result before crack channel residual correction, segmentation result after crack channel residual correction, local result before correction, and local result after correction. Figure 2 Used to represent the pixel-level changes in crack segmentation results before and after crack channel residual correction.

[0069] like Figure 3 As shown, Figure 3 This is a comparison image before and after correction of the hollow channel. The marked areas in the image represent hollow areas, the background areas represent defect areas not currently being displayed, and the selected areas in the RGB and infrared images represent magnified local areas. Figure 3 The columns, from left to right, are: RGB image, infrared image corresponding to the RGB image, true annotation of the hollow area, segmentation result before hollow channel residual correction, segmentation result after hollow channel residual correction, local result before correction, and local result after correction. Figure 3 Used to represent the pixel-level changes in the hollow segmentation results before and after the hollow channel residual correction.

[0070] like Figure 4As shown, in this embodiment, the RGB image is input to the RGB feature extraction module, and the infrared image is input to the infrared feature extraction module. The RGB feature extraction module outputs RGB image features, and the infrared feature extraction module outputs infrared image features. The RGB image features and infrared image features are input to the basic fusion feature generation module to obtain basic fusion features. The basic fusion features are input to the cross-modal residual feature generation module, which combines the RGB image features, infrared image features, and their differences to generate cross-modal residual features. The basic fusion features and cross-modal residual features are input to the enhanced fusion feature generation module to obtain enhanced fusion features. The enhanced fusion features are input to the feature decoding module to obtain decoded features. The decoded features are then input to the basic segmentation output generation module, the crack channel residual correction module, and the hollow channel residual correction module, respectively. The basic segmentation output generation module generates the background channel basic output and provides the corresponding defect channel basic output for the crack channel residual correction module and the hollow channel residual correction module. The crack channel residual correction module generates the crack channel correction output, and the hollow channel residual correction module generates the hollow channel correction output. The background channel basic output, crack channel correction output, and hollow channel correction output together form the category channel score output, which is then processed by the pixel-by-pixel classification module to obtain the facade defect segmentation result.

[0071] The embodiments of the present invention have been described above. These embodiments are used to help understand the technical solutions of the present invention and should not be construed as limiting the scope of the technical solutions of the present invention. Equivalent substitutions or conventional modifications made by those skilled in the art to the embodiments without departing from the technical concept of the present invention should all fall within the scope of the technical solutions of the present invention.

Claims

1. A method for segmenting building facade defects using multimodal channel residual correction, characterized in that, Includes the following steps: S1. Obtain the RGB and infrared images corresponding to the same exterior facade area of ​​the building to be detected; S2. Extract RGB image features and infrared image features respectively; S3. Fuse RGB image features and infrared image features of the same scale at multiple scale levels to obtain basic fusion features at multiple scale levels. The basic fusion features include at least one shallow fusion feature. S4. Generate cross-modal residual features based on RGB image features, infrared image features, and basic fusion features; S5. Fuse the basic fusion features with the cross-modal residual features to obtain the enhanced fusion features; S6. Input the enhanced fusion features into the segmentation decoder, decode the enhanced fusion features to obtain the decoded features, and generate the basic segmentation outputs for the background, crack and hollow channels based on the decoded features. The basic segmentation outputs include the basic output of the background channel, the basic output of the crack channel and the basic output of the hollow channel. S7. Generate crack channel residuals based on decoding features and at least one shallow fusion feature; S8. Generate the hollow channel residual based on decoding features and cross-modal residual features; S9. Before performing pixel-by-pixel classification on the category channel score output, the crack channel residual is superimposed on the crack channel base output to obtain the crack channel correction output, the hollow channel residual is superimposed on the hollow channel base output to obtain the hollow channel correction output, and the background channel uses the background channel base output and does not superimpose the defect residual term. S10. Perform pixel-by-pixel classification based on the category channel score output consisting of the background channel base output, crack channel correction output, and hollow channel correction output to obtain the three-category segmentation results of the exterior facade of the building to be detected: background, cracks, and hollows.

2. The method according to claim 1, characterized in that, S1 describes acquiring RGB and infrared images corresponding to the same facade area of ​​the building to be detected, including establishing a correspondence between RGB and infrared images according to sample identifiers, and performing spatial registration or coordinate alignment processing on the RGB and infrared images so that the RGB and infrared images have a correspondence at the pixel scale or feature scale.

3. The method according to claim 2, characterized in that, S2 describes the extraction of RGB image features and infrared image features respectively, including setting RGB feature extraction branches and infrared feature extraction branches, wherein the RGB feature extraction branches and infrared feature extraction branches do not share parameters, and output RGB image features and infrared image features at multiple scale levels respectively.

4. The method according to claim 3, characterized in that, S3 describes fusing RGB image features and infrared image features of the same scale at multiple scale levels. This includes concatenating RGB image features and infrared image features of the same scale in the channel dimension at each scale level, and performing channel compression and fusion through 1×1 convolution, normalization and nonlinear activation to obtain the basic fused features of the corresponding scale level.

5. The method according to claim 4, characterized in that, The generation of cross-modal residual features described in S4 includes generating candidate cross-modal residual features based on RGB image features, infrared image features and the difference between the two, applying gate weights and residual scaling factors to the candidate cross-modal residual features to obtain gated residual features, and using the gated residual features or features transformed from the gated residual features as cross-modal residual features.

6. The method according to claim 5, characterized in that, S6 describes obtaining decoded features by decoding the enhanced fusion features through a segmentation decoder, which includes projecting and upsampling the enhanced fusion features at multiple scale levels respectively, and fusing the upsampled enhanced fusion features at multiple scale levels to obtain decoded features used to generate the basic segmentation output.

7. The method according to claim 6, characterized in that, The at least one shallow fusion feature mentioned in S7 includes a first-scale fusion feature and a second-scale fusion feature. The first-scale fusion feature and the second-scale fusion feature correspond to the fusion features of the RGB image features and infrared image features of the first two shallow-scale levels of the encoder, respectively. The crack channel residual is generated based on the decoding feature, the first-scale fusion feature and the second-scale fusion feature.

8. The method according to claim 7, characterized in that, When generating the crack channel residual in S7, a crack residual effect weight is also generated. When generating the hollow channel residual in S8, a hollow residual effect weight is also generated. In S9, the crack channel residual is weighted according to the crack residual effect weight to obtain the crack correction amount used to be superimposed on the crack channel base output. The hollow channel residual is weighted according to the hollow residual effect weight to obtain the hollow correction amount used to be superimposed on the hollow channel base output.

9. The method according to claim 8, characterized in that, S10 describes pixel-by-pixel classification, which includes taking the background channel base output, crack channel correction output, and hollow channel correction output as category channel score outputs for the same pixel location, and determining whether the pixel location belongs to the background, crack, or hollow based on the comparison results of the category channel score outputs for the same pixel location.

10. A computer storage medium, characterized in that, The computer storage medium stores a computer program, which, when executed by a processor, implements the method described in any one of claims 1 to 9.

Images