Semantic guidance modulation-based building crack image segmentation method

Abstract

The invention provides a building crack image segmentation method based on semantic guidance modulation, and the method comprises the steps: inputting a visible light image of a building structure into a neural network of an encoder-decoder structure, and obtaining a deep semantic feature through aggregating the high-level features at the tail end of an encoder; an edge auxiliary branch is additionally arranged on an early layer of the decoder to extract an initial edge feature map; after up-sampling the deep semantic features, outputting a probability graph of each pixel belonging to a background interference area through a lightweight segmentation head; taking the probability graph as a soft mask, carrying out negation and then carrying out element-by-element multiplication on the probability graph and the initial edge feature graph to obtain an enhanced edge feature for inhibiting non-crack edge response; and performing channel splicing on the enhanced edge features and features of a corresponding layer of a decoder, fusing the enhanced edge features and the features through a convolutional layer, and outputting a crack pixel-level segmentation result by the decoder. By adopting the scheme of the invention, the false alarm rate caused by the similarity of background interferents and crack visual features in building crack image segmentation can be reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of crack image segmentation, and more particularly, to a building crack image segmentation method based on semantic guidance modulation. BACKGROUND

[0002] The accurate detection of surface cracks of building structures is a key link of engineering safety evaluation. With the development of computer vision technology, crack image segmentation methods based on deep learning have gradually replaced methods relying on artificial or traditional image processing and become the mainstream of application. These methods usually adopt an encoder-decoder network structure, can automatically learn and locate crack regions from visible light images, significantly improve the automation level and efficiency of detection, and provide strong technical support for structural health monitoring.

[0003] However, in actual building scenes, the collected visible light images often contain rich background information, such as shadows, stains, water marks, material joints, and regular surface textures. These background interference and real cracks have high similarity in low-level visual features, for example, they can all present as continuous dark lines, similar edge gradient changes, or approximate local texture patterns. Existing general segmentation networks mainly rely on end-to-end learning of the mapping from images to segmentation masks. Although the internal feature representation of the network has strong semantic abstraction ability, it often lacks explicit modeling and targeted suppression mechanisms for interference. In the decoding process, the network is prone to confuse these deceptive interference features coexisting with cracks with real crack features, resulting in a large number of background interference pixels being misjudged as cracks in the segmentation results, i.e., a high false positive rate, which seriously affects the reliability and practical value of automatic detection and makes the generalization ability of the algorithm in complex real scenes face severe challenges. Therefore, how to solve the problem of high false positive rate caused by the similarity of background interference and crack visual features in building crack image segmentation has become a difficult problem in the industry. SUMMARY

[0004] The present application provides a building crack image segmentation method and device based on semantic guidance modulation, which can reduce the false positive rate caused by the similarity of background interference and crack visual features in building crack image segmentation.

[0005] The present application provides a building crack image segmentation method based on semantic guidance modulation, comprising: obtaining a visible light image of a building structure surface containing cracks; inputting the visible light image into a neural network with an encoder-decoder structure, the encoder extracts multi-layer features, and obtains deep semantic features by aggregating high-level features at the end of the encoder; An edge auxiliary branch is added to an early layer of the decoder, and the branch includes a multi-branch convolution module which enhances the gradient response to the crack edge through parallel convolution paths with different receptive fields to extract an initial edge feature map; After upsampling the deep semantic feature, a segmentation head composed of a 1x1 convolution layer and a Sigmoid activation function is used to output a probability map of each pixel belonging to the background interference region; The probability map is taken as a soft mask, and after being inverted, it is multiplied element by element with the initial edge feature map to obtain an enhanced edge feature that suppresses the non-crack edge response. The enhanced edge feature is concatenated with the features of the corresponding layer of the decoder, and a convolution layer is used for fusion, and finally the crack pixel-level segmentation result is output by the decoder.

[0006] Preferably, the visible light image is input into an encoder-decoder neural network, and the encoder extracts multiple layers of features, specifically including: The input visible light image is normalized; The normalized image is input into the backbone network of the encoder; In the backbone network, at least three intermediate layer outputs with different receptive fields in the spatial dimension are selected as the multiple layer features; The at least three multiple layer features are transmitted to the corresponding layer of the decoder through a skip connection.

[0007] Preferably, the deep semantic feature is obtained by aggregating the high-level features at the end of the encoder, specifically including: Obtain the high-level feature maps output by the last two stages of the encoder; Respectively, the two high-level feature maps are processed by channel attention to calibrate the weights of each feature channel; The two feature maps processed by channel attention are respectively adjusted to the same number of channels through a 1x1 convolution layer; The two feature maps with adjusted channel numbers are added element by element to obtain an aggregated feature map; The aggregated feature map is input into a convolution block containing a residual connection for feature refinement, and the deep semantic feature is output.

[0008] Preferably, the edge auxiliary branch is added to the early layer of the decoder, specifically including: Select the feature before the first upsampling layer in the decoder as the input of the edge auxiliary branch; The input feature is sent to a multi-branch convolution module; The multi-branch convolution module includes three parallel convolution paths, wherein the first path uses a standard 3x3 convolution, the second path uses a 3x3 hollow convolution with a hollow rate of 3, and the third path uses a 3x3 hollow convolution with a hollow rate of 5; The output features of the three paths are spliced in the channel dimension; A 1x1 convolution operation is performed on the spliced features to generate an initial edge feature map.

[0009] Preferably, after upsampling the deep semantic features, a segmentation head composed of a 1x1 convolution layer and a Sigmoid activation function is used to output a probability map of each pixel belonging to the background interference region, which specifically includes: The deep semantic features are bilinearly interpolated and upsampled to have a height and width size same as the initial edge feature Figure One ; The upsampled feature map is input into the 1x1 convolution layer to compress the number of channels to 1; The obtained single-channel feature map is input into the Sigmoid activation function to generate an initial probability map; The initial probability map is subjected to Gaussian filtering and smoothing processing to obtain a probability map for modulation.

[0010] Preferably, before inputting the upsampled feature map into the 1x1 convolution layer, the upsampled feature map is passed through a lightweight channel attention module to enhance the response to discriminative channels.

[0011] Preferably, the probability map is taken as a soft mask, and after being inverted, it is multiplied element by element with the initial edge feature map to obtain an enhanced edge feature that suppresses non-crack edge response, which specifically includes: An inversion operation is performed on each pixel value in the probability map to generate an inverted mask map; The inverted mask map is copied and expanded in the channel dimension to match the number of channels of the initial edge feature map; The copied and expanded mask map is multiplied element by element with the initial edge feature map to obtain a preliminary modulation feature; The preliminary modulation feature is subjected to batch normalization processing to obtain an enhanced edge feature that suppresses non-crack edge response.

[0012] Preferably, before the batch normalization processing of the preliminary modulation feature, the preliminary modulation feature is element-wise weighted summed with the initial edge feature map, and the weighted summed feature is subjected to batch normalization processing.

[0013] Preferably, the enhanced edge feature is channel-spliced with the feature of the corresponding layer of the decoder, and fused through a convolution layer, which specifically includes: concatenate the enhanced edge feature and the main path feature map of the corresponding scale in the decoder in channel dimension; fuse the concatenated feature map through a fusion module, the fusion module sequentially comprising a channel recalibration submodule and a depth separable convolution submodule; input the fused feature map into the next up-sampling layer of the decoder.

[0014] Preferably, the crack pixel-level segmentation result output by the decoder specifically comprises: repeat the decoding process until the spatial size of the input image is restored; at the end of the decoder, use a 1x1 convolution to map the number of feature channels to 2, respectively corresponding to the background pixels and the crack pixels; apply a Softmax function to the mapped two-channel feature, and take the output of the crack pixel channel as the final pixel-level segmentation probability map.

[0015] The technical scheme provided by the embodiments disclosed in the application has the following beneficial effects: In the embodiments of the application, first, the visible light image is input into the neural network of the encoder-decoder structure, the encoder extracts multiple layers of features, and the deep layer semantic feature is obtained by aggregating the high layer features at the end of the encoder, which gives the network strong global context perception ability, enabling it to distinguish structural background and abnormal cracks from the semantic level, laying a foundation for subsequent accurate identification and positioning of the interference area; second, an edge auxiliary branch is added to the early layers of the decoder, which enables the network to focus on and strengthen the gradient response to the fine edges of the cracks, extracts high-resolution and high-fidelity initial edge feature maps, effectively prevents the details of the real cracks from being lost in the deep network, and provides high-quality original edge information for subsequent true from false; then, after up-sampling the deep layer semantic feature, a segmentation head composed of a 1x1 convolution layer and a Sigmoid activation function is used to output a probability map of each pixel belonging to the background interference area, and the inverse of the probability map is taken as a soft mask and multiplied by the initial edge feature map, which realizes the active purification of the bottom edge feature using high-level semantic knowledge, directly suppresses the false edge response generated by the interference area such as shadow and stain at the feature level, thereby significantly reducing the source of false positives; finally, the enhanced edge feature after purification is concatenated and fused with the features of the corresponding layer of the decoder, which realizes the organic unification and complementary enhancement of the verified detailed edge information and the mainstream semantic context information, ensuring that the final segmentation result has accurate boundary positioning and reliable semantic consistency; in summary, by constituting an end-to-end lightweight integrated network, the application effectively reduces the false positive rate caused by the similarity between the background interference and the crack visual features in the building crack image segmentation. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 is an application scenario diagram of a building crack image segmentation method based on semantic guidance modulation according to some embodiments of the present application; Figure 2 is an exemplary flowchart of a building crack image segmentation method based on semantic guidance modulation according to some embodiments of the present application; Figure 3 is a flowchart of determining enhanced edge features according to some embodiments of the present application; Figure 4 is a structural diagram of a building crack image segmentation system according to some embodiments of the present application; Figure 5 is a structural diagram of a computer device for implementing a building crack image segmentation method based on semantic guidance modulation according to some embodiments of the present application. DETAILED DESCRIPTION

[0017] In order to better understand the technical solutions of the present application, the technical solutions of the present application will be described in detail below in combination with the drawings in the specification and specific embodiments.

[0018] Reference Figure 1 The figure is an application scenario diagram of a building crack image segmentation method based on semantic guidance modulation according to some embodiments of the present application, which includes an image acquisition device, an edge computing module / cloud server, a communication network and a terminal, wherein the image acquisition device communicates with the edge computing module or the cloud server through the network, and the terminal is connected with the server system through the communication network; the server system obtains the visible light image of the building structure surface provided by the image acquisition device, inputs the image to the pre-trained encoder-decoder neural network after normalizing preprocessing, sequentially executes multi-layer feature extraction, deep semantic feature aggregation, edge auxiliary branch feature extraction, semantic guidance suppression modulation and feature fusion, and finally outputs the crack pixel-level segmentation result image; when receiving the crack detection request for a specific building component or area sent through the terminal, the server system feeds back the segmentation result image, crack quantization parameters (such as length, width) and confidence evaluation to the terminal for the structure detection personnel or operation and maintenance personnel to view and make decisions.

[0019] Among them, the image acquisition device can be a drone equipped with a high-definition camera, an automatic inspection robot, a handheld intelligent detection device or a fixed monitoring camera; the terminal can be but is not limited to an engineering tablet computer, a field portable computer, a monitoring center workstation or a mobile intelligent terminal; the edge computing module / cloud server can be an embedded computing unit deployed on the detection device, a field edge server, or a cloud server cluster providing large-scale computing and model services.

[0020] Reference Figure 2 The figure is an exemplary flow chart of a building crack image segmentation method based on semantic guidance modulation according to some embodiments of the present application, which mainly includes the following steps: In step 101, a visible light image of a building structure surface containing cracks is obtained.

[0021] It should be noted that obtaining a visible light image of a building structure surface containing cracks means that a digital camera, an industrial camera or a camera integrated in a mobile device is used to collect a red, green and blue (Red Green Blue, RGB) color image of a building concrete, masonry or asphalt surface under natural light or artificial uniform lighting conditions. The obtained image is a conventional visible spectrum image, which does not include infrared, ultraviolet or depth special modal images. The resolution is usually not less than 1024x768 pixels to ensure that the fine crack texture and complex background details can be clearly presented. The present application is particularly suitable for crack analysis of images collected in outdoor or indoor scenes such as bridge piers, tunnel lining, building exterior walls and road surfaces. These scenes often contain rich background interference such as shadows, water stains, oil stains, surface peeling, regular joints, etc. These are typical application scenarios that the present application aims to process and reduce false positives.

[0022] In step 102, the visible light image is input into a neural network with an encoder-decoder structure. The encoder extracts multi-layer features, and the deep semantic features are obtained by aggregating the high-level features at the end of the encoder.

[0023] In some embodiments, the visible light image is input into a neural network with an encoder-decoder structure, and the encoder extracts multi-layer features, which can be achieved by the following steps: The input visible light image is normalized; The normalized image is input into the backbone network of the encoder; In the backbone network, at least three intermediate layer outputs with different receptive fields in the spatial dimension are selected as the multi-layer features; The at least three multi-layer features are transmitted to the corresponding layers in the decoder through a skip connection.

[0024] It should be noted that the multi-layer features in the present application are used to describe the image content at different abstraction levels and spatial scales, which are the feature map sets output by different intermediate layers of the backbone network. The skip connection is a feature transmission path that directly transmits and fuses the multi-layer features extracted by the encoder to the corresponding layers of the decoder across network levels.

[0025] In a specific implementation, first, the input visible light image is normalized. This operation first converts the original integer values of the red, green and blue channels of each pixel of the image into floating-point numbers, and then performs a linear transformation of subtracting the mean value and dividing by the standard deviation for each channel based on the channel mean value and standard deviation of a large-scale image data set (such as the ImageNet data set) statistically obtained in advance. This step normalizes the distribution of pixel values in each channel to a range centered at zero and with a standard deviation of one, thereby accelerating model training convergence and improving the stability of performance. In a specific implementation, the image tensor is input into a normalization layer, which performs calculation according to the preset mean value vector and standard deviation vector, and finally outputs the normalized image tensor; second, the normalized image is input into the backbone network of the encoder.The backbone network usually adopts a residual network (such as a ResNet model) pre-trained on a large classification dataset as a basic architecture, which is composed of a plurality of sequentially connected residual stages, each stage containing a plurality of convolutional layers with residual connections for layer-by-layer extraction and abstraction of image features. In specific implementations, the pre-trained weights are loaded to initialize the network parameters, and the normalized image tensor is input into the backbone network to start the forward propagation process, and finally a series of intermediate and final feature maps are output by the backbone network. Then, in the backbone network, at least three intermediate layers with different spatial receptive fields are selected to output features as the multi-layer features. Specifically, the feature output layers at the end of different depth convolution stages in the backbone network are selected, for example, in the ResNet-34 model, the output after the first residual stage is selected as the first layer feature, which has a spatial downsampling factor of four and a small receptive field, the output after the second residual stage is selected as the second layer feature, which has a spatial downsampling factor of eight and a medium receptive field, and the output after the third residual stage is selected as the third layer feature, which has a spatial downsampling factor of sixteen and a large receptive field. These feature maps output at different stages, with different spatial sizes and semantic abstraction levels, are collectively used as the multi-layer features for subsequent decoding and fusion. Finally, the at least three multi-layer features are transmitted to the corresponding layers in the decoder through a skip connection, which is completed when building a complete encoder-decoder network structure. In the decoder part, the spatial size of the feature map is gradually restored through upsampling operations (such as bilinear interpolation or transposed convolution). When the spatial size of the feature map in a certain layer of the decoder is consistent with the spatial size of the multi-layer feature saved by the encoder, the feature of the encoder at that layer is fused with the current feature of the decoder through the skip connection. The fusion operation usually adopts channel dimension splicing or element-wise addition, for example, when the decoder restores the feature map size to one-eighth of the original image size through upsampling, it is spliced with the second layer feature with the same size saved by the encoder. This process is repeated until the multi-layer features with rich details and semantics extracted by the encoder are used as key supplementary information for fine segmentation at each corresponding level of the decoder.

[0026] In some embodiments, obtaining deep semantic features by aggregating high-level features at the end of the encoder can be achieved by the following steps: Obtaining high-level feature maps output by the last two stages of the encoder; Respectively performing channel attention processing on the two high-level feature maps to calibrate the weights of each feature channel; Respectively adjusting the two feature maps after channel attention processing to the same number of channels through a 1x1 convolutional layer; The two feature maps after adjusting the number of channels are added element by element to obtain an aggregated feature map; The aggregated feature map is input into a convolution block containing a residual connection for feature refinement, and the deep semantic feature is output.

[0027] It should be noted that the channel attention processing in the present application refers to a feature enhancement operation based on global context information for adaptively recalibrating the importance weights of each channel of the feature map; the deep semantic feature is a context feature representation rich in global semantic information for guiding subsequent edge feature modulation.

[0028] In a specific implementation, first, in the backbone network of the encoder, the outputs of the second-to-last stage and the last stage are extracted as a first high-level feature map and a second high-level feature map, respectively, where the first high-level feature map has a relatively high spatial resolution but a slightly lower semantic abstraction degree, and the second high-level feature map has the strongest semantic abstraction ability but the lowest spatial resolution. The two feature maps together constitute the source information for aggregation, which is obtained by recording the output tensor of the corresponding network layer during the forward propagation process. Second, channel attention processing is performed on the two high-level feature maps to calibrate the weights of each feature channel. For each high-level feature map, the channel attention processing is independently implemented. The processing first performs global average pooling on the input feature map in the spatial dimension (height and width), compressing the two-dimensional features of each channel into a scalar, thereby generating a feature vector representing the global response of each channel. The feature vector is processed through a subnetwork composed of a fully connected layer with a dimension reduction coefficient, a nonlinear activation function (such as ReLU), and a fully connected layer that restores the original channel dimension. Finally, a Sigmoid activation function is used to generate a channel weight vector between 0 and 1. The weight vector is multiplied with the original input feature map channel by channel, which completes the adaptive reweighting of each channel of the original feature and outputs the feature map after channel attention calibration. Then, two independent 1×1 convolution layers are used to perform convolution operations on the calibrated first high-level feature map and second high-level feature map, respectively. The kernel number of the 1×1 convolution is uniformly set to a preset value, which serves to map the original channel number of each input feature map to the same target channel number without changing the spatial size of the feature map. This operation ensures that the two feature maps are fully aligned in the channel dimension in the subsequent steps, preparing for element-wise addition fusion. Further, the second high-level feature map is upsampled to the same spatial height and width as the first high-level feature map through bilinear interpolation, and point-by-point addition operation is performed on the two tensors. This operation fuses complementary semantic information from different depths, generating an aggregated feature map with richer information.Finally, the aggregated feature map is input into a convolution block with a residual connection for feature refinement, and the deep semantic feature is output. The convolution block is composed of two 3x3 standard convolution layers connected in sequence, and a batch normalization layer and a ReLU activation function are connected in sequence after each convolution layer. A mapping short circuit connection is established between the input of the convolution block and the output of the second convolution layer to form a residual structure. The obtained aggregated feature map is input into the convolution block, and after the transformation, normalization and activation of the first convolution layer, the processing result is further input into the original aggregated feature map (i.e. the other end of the short circuit connection) for element-by-element addition. The residual convolution block performs nonlinear transformation and fusion on the aggregated feature, further refines the representation ability of the feature, and finally outputs the deep semantic feature as the guide for the entire semantic modulation process.

[0029] In step 103, an edge auxiliary branch is added to the early layers of the decoder. The branch includes a multi-branch convolution module that enhances the gradient response to crack edges through parallel convolution paths with different receptive fields to extract an initial edge feature map.

[0030] In some embodiments, the edge auxiliary branch added to the early layers of the decoder can be implemented using the following steps: Selecting features before the first upsampling layer in the decoder as the input of the edge auxiliary branch; Sending the input features into a multi-branch convolution module; The multi-branch convolution module includes three parallel convolution paths, where the first path uses standard 3x3 convolution, the second path uses 3x3 dilated convolution with a dilated rate of 3, and the third path uses 3x3 dilated convolution with a dilated rate of 5; Concatenating the output features of the three paths in the channel dimension; Performing a 1x1 convolution operation on the concatenated features to generate an initial edge feature map.

[0031] It should be noted that the multi-branch convolution module in this application refers to a module for parallel extraction of multi-scale edge responses, composed of several convolution paths with different receptive fields; the initial edge feature map refers to a feature map containing enhanced multi-scale crack edge responses for semantic guided modulation.

[0032] In specific implementation, firstly, the features before the first upsampling layer in the decoder are selected as the input to the edge auxiliary branch. During the forward propagation of the decoder, when the features are passed to the network layer about to undergo the first spatial upsampling, the feature map before this layer is extracted. The reason for selecting features at this location is that the feature map at this time contains both the primary semantic information refined by the encoder and maintains a relatively high spatial resolution, which can provide basic features for subsequent edge detection that combine semantic guidance and detail preservation. Secondly, the input features are fed into a multi-branch convolutional module, which contains three parallel convolutional paths. The first path uses a standard 3×3 convolution to capture local, fine edge gradient changes. The second path uses a 3×3 dilated convolution with a dilation rate of 3 to capture local, fine edge gradient changes. The third path uses a 3×3 dilated convolution with a dilation rate of 5, which has the largest receptive field and is used to capture the large-scale continuity that cracks may have. The three paths operate independently in a parameter-non-shared manner. It should be further noted that different dilation rates are set... The proposed approach aims to efficiently cover the multi-scale spatial range that crack edges may span without significantly increasing the number of parameters. Each approach performs convolution calculations on the input feature map and outputs its own feature map. Then, the output features of the three approaches are concatenated along the channel dimension. After obtaining the output feature maps of the three approaches, they are sequentially connected along the channel dimension to form a new feature map tensor. This is done to collect and preserve the captured local, meso- and macro-level edge information, providing a complete set of multi-scale edge features for the next step of information fusion. Finally, a 1×1 convolution kernel is used to perform convolution operations on the concatenated multi-channel feature map. The role of 1×1 convolution is to weight and integrate the features that have fused multi-scale information along the channel dimension, compressing and refining them into a more representative edge feature expression. The purpose of this is to aggregate multi-path information, eliminate redundancy, and generate a feature map with a fixed number of channels that is more suitable for subsequent modulation operations. This step ultimately uses the integrated and refined feature map as the initial edge feature map for semantically guided modulation.

[0033] In step 104, after upsampling the deep semantic features, a segmentation head consisting of a 1×1 convolutional layer and a Sigmoid activation function is used to output a probability map of each pixel belonging to the background interference region.

[0034] In some embodiments, after upsampling the deep semantic features, the probability map of each pixel belonging to the background interference region can be output through a segmentation head consisting of a 1×1 convolutional layer and a sigmoid activation function by the following steps: The deep semantic features are upsampled using bilinear interpolation so that their height and width dimensions are similar to the initial edge features. Figure OneConclusions; The up-sampled feature map is input into the 1x1 convolution layer to compress the channel number to 1; The obtained single-channel feature map is input into the Sigmoid activation function to generate an initial probability map; The initial probability map is subjected to Gaussian filtering smoothing processing to obtain a probability map for modulation.

[0035] It should be noted that the bilinear interpolation up-sampling in the present application refers to a spatial transformation method based on linear weighted average of adjacent pixels for enlarging the spatial size of the feature map to the target size; and the probability map for modulation refers to a stable probability feature map subjected to smoothing processing for guiding modulation of edge features.

[0036] In a specific implementation, firstly, since deep semantic features usually have a low spatial resolution, while edge feature maps need to maintain a high resolution to preserve details, the spatial size of deep semantic features must be enlarged, and bilinear interpolation up-sampling calculates the value of each new pixel by linearly weighting and averaging the nearest four known pixel points in its original feature map, which is simple to calculate and can maintain the smooth transition of features. In a specific operation, the target height and width are determined according to the size of the initial edge feature map, and then the bilinear interpolation algorithm is performed on the deep semantic feature map to enlarge its spatial size to be consistent with the target size; secondly, the up-sampled feature map still has multiple channels, while the probability map needs to be single-channel to represent a unique probability value at each position. A 1x1 convolutional layer can aggregate and compress the information of multiple channels into one channel without changing the spatial size by linear combination with a convolutional kernel of the same number of input channels. In a specific implementation, the output channel number of the 1x1 convolutional layer is set to 1, and random initialized weight parameters and bias parameters are used. The reason for this is to fuse and simplify the rich multi-channel semantic information into a single response map representing the possibility of background interference. This step finally converts the up-sampled multi-channel feature map into a single-channel preliminary response map; then, the obtained single-channel feature map is input into the Sigmoid activation function to generate an initial probability map. The feature value range output by the 1x1 convolutional layer is arbitrary, which does not meet the requirement of the [0, 1] interval of probability. The Sigmoid activation function can map any real number to the (0, 1) interval, and its function form is 1 / (1+exp(-x)), which exactly meets the requirement of probability representation. The Sigmoid function is independently applied to each pixel value of the single-channel feature map for calculation, and the response value at each position is converted into a probability value. The reason for this is to obtain an initial probability estimate that meets the definition of probability and can be directly used for subsequent calculations, and to convert the single-channel response map into an initial probability map; finally, the initial probability map is subjected to Gaussian filtering smoothing processing to obtain a probability map for modulation. The initial probability map may produce irregular spots or sharp edges due to local uncertainty or noise in network prediction, and direct use for modulation may lead to discontinuous or unreasonable suppression of edge features. Gaussian filtering is a linear smoothing filter that uses a convolution kernel generated according to a Gaussian function (such as 3x3 or 5x5 size, and the standard deviation σ is usually set to 1.0) to convolve the image, which can effectively suppress high-frequency noise and make the image smooth. For each pixel of the initial probability map, a weighted average calculation is performed using this Gaussian kernel. The reason for this is to eliminate local noise and outliers in the probability map and obtain a probability map that changes more continuously and stably in space, thereby ensuring that the subsequent modulation operation is more robust and reliable. This step finally converts the smoothed probability map into the final probability map for guiding modulation.

[0037] It should be noted that before inputting the upsampled feature map into the 1×1 convolutional layer, the process further includes: passing the upsampled feature map through a lightweight channel attention module to enhance the response to the discriminative channels. This lightweight channel attention module is a simplified design based on a channel attention mechanism. Its core structure typically includes a global average pooling layer, a fully connected layer with dimensionality reduction coefficients, a non-linear activation function, and a fully connected layer that restores the original channel dimensions. Finally, a sigmoid function is used to generate the weight vectors for each channel. The role of this module is to adaptively recalibrate the importance weights of each channel in the feature map. It first captures the full importance weights of each channel through global average pooling. The system takes local context information and then learns the nonlinear dependencies between channels through a small fully connected layer. Finally, it outputs a weight vector to scale each channel of the input feature map, thereby enhancing the response of channels with high discriminative power for the current segmentation task (i.e., identifying background interference) while suppressing the response of irrelevant or redundant channels. The benefit of introducing this module before generating the background interference probability map is that it enables the network to focus more on feature channels that are semantically highly related to interferences such as shadows and stains when generating the probability map, thereby improving the accuracy and robustness of interference region identification. This helps to generate more accurate probability maps for subsequent modulation, improving the system's ability to reduce the false alarm rate of cracks from the source.

[0038] In step 105, the probability map is used as a soft mask, inverted, and then multiplied element-wise with the initial edge feature map to obtain enhanced edge features that suppress non-crack edge responses.

[0039] In some embodiments, reference Figure 3 As shown in the figure, this is a flowchart illustrating the process of determining enhanced edge features in some embodiments of this application. In this embodiment, the probability map is used as a soft mask, which is inverted and then multiplied element-wise with the initial edge feature map to obtain enhanced edge features that suppress non-crack edge responses. This can be achieved through the following steps: In step 1051, an inverted operation is performed on each pixel value in the probability map to generate an inverted mask map; In step 1052, the inverted mask image is copied and expanded in the channel dimension so that its number of channels matches the initial edge feature image; In step 1053, the copied and expanded mask image is multiplied element-wise with the initial edge feature image to obtain preliminary modulation features; In step 1054, the preliminary modulation features are batch normalized to obtain enhanced edge features that suppress non-crack edge responses.

[0040] It should be noted that the soft mask in the present application refers to a continuous weight map with a value domain in the interval [0, 1], which is used for non-binary and flexible weight modulation of the feature map; the replication and expansion in the channel dimension is used for element-by-element operation of the single-channel mask graph and the multi-channel feature map, and is achieved by repeating and stacking the single-channel mask graph in the channel dimension; the enhanced edge feature refers to the feature map that has been stabilized in data distribution after batch normalization processing and has been suppressed in non-crack edge response for input into the subsequent fusion module.

[0041] In a specific implementation, first, an inversion operation is performed on each pixel value in the probability map to generate an inverted mask map, the probability map representing the possibility of each pixel belonging to a background interference region, with a value range of [0, 1], and the inversion operation is used to realize a logical conversion from interference suppression to non-interference preservation, and the specific method is to perform a calculation on each pixel value in the probability map: new pixel value = 1 - original pixel value, so that in the original probability map, the area with a high value (close to 1, indicating a high interference probability) becomes a low value (close to 0) in the inverted mask map, meaning that the edge response of this area should be strongly suppressed subsequently, and vice versa, in the original probability map, the area with a low value (close to 0, indicating a low interference probability) becomes a high value (close to 1) in the inverted mask map, meaning that the edge response of this area should be preserved or enhanced subsequently; second, since the inverted mask map is single-channel, while the initial edge feature map is usually multi-channel, in order to perform element-by-element multiplication, both must have the same number of channels, and a copy expansion operation is performed along the channel dimension to repeat the single-channel inverted mask map multiple times until its number of channels is the same as that of the initial edge feature map, thereby generating a multi-channel mask map tensor, in which the numerical value of each channel is exactly the same, and the reason for this is to provide dimensionally matched operands for the next element-by-element multiplication, ensuring that the modulation weight in the spatial position can be equally applied to each channel of the edge feature map, and finally the single-channel inverted mask map is expanded to a multi-channel mask map that is completely consistent in shape with the initial edge feature map; then, the obtained multi-channel mask map and the initial edge feature map are subjected to numerical multiplication operation at the corresponding spatial position and corresponding channel, and this operation is the core of semantic-guided modulation, and its physical meaning is that in the area where the inverted mask map value is 1 (corresponding to the original interference probability of 0), the edge feature is completely preserved, in the area where the inverted mask map value is 0 (corresponding to the original interference probability of 1), the edge feature is completely suppressed, and in the intermediate value area, the edge feature is partially suppressed, and the reason for this is that through the interference region positioning map provided by the high-level semantic information, the false edge response in the initial edge feature map that may come from background interference is directly and actively suppressed, realizing purification at the feature level, and finally the semantic information and the edge feature are combined to generate a preliminary modulation feature;Finally, the preliminary modulation feature is batch normalized to obtain an enhanced edge feature that suppresses non-crack edge response. In the training process, the batch normalization calculates the mean and variance of each channel of the preliminary modulation feature map in the current batch data, normalizes the features of the channel using these statistics, and then performs affine transformation through a learnable scaling parameter and an offset parameter. In the inference stage, the moving average mean and variance calculated in the training stage are used to perform batch normalization on the preliminary modulation feature. The reason for batch normalization is that after the modulation operation, the numerical distribution of the feature map may change. Batch normalization can stabilize it near the distribution of zero mean and unit variance, which helps to accelerate the training convergence of the subsequent network and improve the generalization ability and stability of the model. The feature map after distribution stabilization is output as the enhanced edge feature.

[0042] It should be noted that before batch normalizing the preliminary modulation feature, the preliminary modulation feature and the initial edge feature map are element-wise weighted and summed, and then the weighted and summed feature is batch normalized. Specifically, one or more learnable weight parameters can be set. In the simplest implementation, a learnable scalar parameter is set, the preliminary modulation feature and the initial edge feature map are element-wise weighted and summed, and the calculation formula is: enhanced feature = A * preliminary modulation feature + (1-A) * initial edge feature map, where A is the learnable weight coefficient, and its initial value can be set to 0.5 and optimized through back propagation during network training. Finally, the weighted and summed result is batch normalized. The purpose of this is to prevent excessive suppression and avoid losing real and weak crack edges due to excessive suppression in the case of incomplete semantic information.

[0043] In step 106, the enhanced edge feature is channel spliced with the feature of the corresponding layer of the decoder, and fused through a convolution layer, and finally the crack pixel-level segmentation result is output by the decoder.

[0044] In some embodiments, the enhanced edge feature and the feature of the corresponding layer of the decoder are channel spliced and fused through a convolution layer, which can be implemented by the following steps: The enhanced edge feature and the main path feature map of the corresponding scale in the decoder are spliced in the channel dimension. The spliced feature map is fused through a fusion module, and the fusion module includes a channel recalibration sub-module and a depth separable convolution sub-module in sequence. The fused feature map is input into the next upsampling layer of the decoder.

[0045] It should be noted that the channel recalibration submodule in the present application is used to adaptively redistribute the channel weights of the spliced features, and the depth separable convolution submodule is used to maintain the calculation efficiency while fusing information.

[0046] In a specific implementation, first, the enhanced edge features are spliced with the feature maps of the main path at the corresponding scale in the decoder in the channel dimension, that is, the enhanced edge features from the edge modulation branch are obtained, and at the same time, the feature maps of the same spatial scale in the decoder main path are obtained, the two feature maps are connected along the channel dimension to generate a new feature map with a channel number being the sum of the two; second, the spliced feature map is fused through a fusion module, the fusion module sequentially includes a channel recalibration submodule and a depth separable convolution submodule, the spliced feature map obtained in the first step is input into the fusion module, the channel recalibration submodule (its structure and function are similar to the aforementioned lightweight channel attention module) first calculates an importance weight vector for each channel of the spliced feature, and then performs weighting through channel-by-channel multiplication, the purpose is to adaptively emphasize important feature channels from the two paths and suppress secondary or redundant channels, further, the depth separable convolution submodule first performs two-dimensional spatial convolution on the recalibrated feature map to fuse spatial information, and then performs a 1x1 point-by-point convolution to fuse channel information, thereby completing efficient feature extraction, the reason for using this two-step fusion is that intelligent selection in the channel dimension can improve the relevance of fusion, and then using efficient depth separable convolution for spatial and channel mixing can significantly reduce the calculation parameter amount and time consumption while ensuring the fusion effect, finally, the spliced composite feature map is taken as the fused feature map; then, the fused feature map is input into the next upsampling layer of the decoder, that is, the feature map output by the fusion module is taken as input and transmitted to the next preset upsampling layer in the decoder network structure, the reason for this is that the core task of the decoder is to gradually restore the spatial resolution of the feature map through a series of upsampling operations to perform pixel-level prediction, this step completes all information fusion and processing at the current scale, and sends the result to the standard decoding process to continue the subsequent size restoration and further feature decoding work.

[0047] In some embodiments, the crack pixel-level segmentation result output by the decoder can be realized by the following steps: Repeat the decoding process until the spatial size of the input image is restored; At the end of the decoder, use a 1x1 convolution to map the number of feature channels to 2, corresponding to the background pixels and crack pixels respectively; Apply the Softmax function to the mapped two-channel feature, and take the output of the crack pixel channel as the final pixel-level segmentation probability map.

[0048] It should be noted that the background pixels and crack pixels in the present application refer to two semantic class labels corresponding to non-crack regions and crack regions in the image respectively for pixel-level binary classification; the pixel-level segmentation probability map refers to a two-dimensional probability matrix with a value range of [0, 1] for indicating the possibility of each pixel being predicted as a crack.

[0049] In a specific implementation, firstly, the decoding process is repeated until the spatial size of the input image is restored, the decoding process refers to the iterative operation steps defined in the claims, including feature fusion and upsampling, after completing the feature fusion of the current scale, the fused feature map is input into the upsampling layer to expand its spatial size, and then a new round of fusion is performed with the corresponding scale features transmitted by the encoder through the skip connection, this process is executed in a loop, and each time the upsampling usually enlarges the height and width of the feature map to twice the original size, the iterative process is repeated until the pixel number of the finally output feature map in height and width is completely consistent with the original image input into the neural network, the purpose of this step is to generate a high-resolution feature map corresponding to the input image pixel by pixel, providing a spatial basis for the final pixel-level classification, and finally the spatial size of the feature map is restored to the spatial size of the input image; secondly, at the end of the decoder, a 1x1 convolution is used to map the number of feature channels to 2, respectively corresponding to the background pixels and the crack pixels, after the last layer of the decoder network structure, i.e., the feature map whose spatial size has been restored to the input size, a 1x1 convolution layer is connected, the convolution layer is configured to have two convolution kernels, and the number of channels of each convolution kernel is the same as that of the input feature map, during forward propagation, the convolution layer performs dot product operation on each spatial position of the input feature map, and outputs two corresponding response values, the two response values constitute a two-dimensional feature vector at each pixel position, the first channel is defined as the non-normalized score corresponding to the background pixel class, and the second channel is defined as the non-normalized score corresponding to the crack pixel class, the purpose of this step is to convert the rich multi-channel feature semantics into direct class scores for binary classification, and finally map the high-dimensional features to a double-channel feature score map respectively corresponding to the background and the crack at each pixel position; finally, the Softmax function is applied to the mapped double-channel features, and the output of the crack pixel channel is taken as the final pixel-level segmentation probability map, the Softmax function is independently applied to each pixel position of the double-channel feature score map obtained in the above step for calculation, the Softmax function performs exponential normalization on the scores of the two channels, so that the sum of the two output values at each position is 1, and each value thus represents the predicted probability that the pixel belongs to the corresponding class, specifically, the probability value of the second channel (i.e., the crack pixel channel) after the Softmax calculation is taken to constitute a new single-channel two-dimensional map, and the value of each pixel in the map represents the probability that the network predicts the pixel to belong to the crack, which ranges from 0 to 1, the purpose of this step is to obtain a normalized output that meets the probability interpretation and can be directly used for evaluation or binaryzation by setting a threshold (such as 0.5) to obtain the final segmentation mask, and finally the probability output of the crack channel is taken as the final pixel-level segmentation probability map.

[0050] On the other hand, in some embodiments, the present application provides a building crack image segmentation system, referring to Figure 4The figure is a schematic diagram of the structure of a building crack image segmentation system according to some embodiments of this application. The building crack image segmentation system 400 includes: an acquisition module 401, a processing module 402, and an execution module 403, which are described below: Acquisition module 401 is used to acquire visible light images of the surface of a building structure containing cracks; Processing module 402 is used to input visible light images into a neural network with an encoder-decoder structure. The encoder extracts multi-layer features and obtains deep semantic features by aggregating the high-level features at the end of the encoder. In this application, the processing module 402 is also used to add an edge auxiliary branch in the early layer of the decoder. The branch includes a multi-branch convolution module, which enhances the gradient response to the crack edge through parallel convolutional paths with different receptive fields to extract the initial edge feature map. In this application, the processing module 402 is also used to upsample the deep semantic features and then output a probability map of each pixel belonging to the background interference region through a segmentation head composed of a 1×1 convolutional layer and a Sigmoid activation function. In this application, the processing module 402 is further configured to use the probability map as a soft mask, invert it, and multiply it element-wise with the initial edge feature map to obtain enhanced edge features that suppress non-crack edge responses. The execution module 403 is used to concatenate the enhanced edge features with the features of the corresponding layer of the decoder through a channel, and then fuse them through a convolutional layer. Finally, the decoder outputs the crack pixel-level segmentation result.

[0051] In addition, this application also provides a computer device, the computer device including a memory and a processor, the memory storing code, and the processor being configured to acquire the code and execute the above-described semantically guided modulation-based building crack image segmentation method.

[0052] In some embodiments, reference Figure 5 The figure is a schematic diagram of the structure of a computer device implementing a semantically guided modulation-based building crack image segmentation method according to some embodiments of this application. The semantically guided modulation-based building crack image segmentation method in the above embodiments can... Figure 5 The computer device shown is used to implement this, and the computer device 500 includes at least one processor 501, a communication bus 502, a memory 503, and at least one communication interface 504.

[0053] The processor 501 can be a general central processing unit (CPU), or an application-specific integrated circuit (ASIC).

[0054] The communication bus 502 can be used to transmit information between the above-mentioned components.

[0055] The memory 503 can be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disk storage, a magnetic disk or other magnetic storage device, or any other medium capable of storing instructions or data that can be accessed by a computer, but not limited to. The memory 503 can exist independently and be connected to the processor 501 through the communication bus 502. The memory 503 can also be integrated with the processor 501.

[0056] The memory 503 is used to store program codes for implementing the solutions of the present application, and the processor 501 controls the execution. The processor 501 is used to execute the program codes stored in the memory 503. The program codes can include one or more software modules. The building crack image segmentation method based on semantic guidance modulation in the above-mentioned embodiments can be implemented by one or more software modules in the program codes of the processor 501 and the memory 503.

[0057] The communication interface 504 uses any transceiver-like device to communicate with other devices or communication networks, such as an Ethernet, a radio access network (RAN), a wireless local area network (WLAN), etc.

[0058] In a particular implementation, as one embodiment, the computer device can include multiple processors, each of which can be a single-CPU processor or a multi-CPU processor. A processor herein can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0059] The computer device described above can be a general-purpose computer device or a special-purpose computer device. In a particular implementation, the computer device can be a desktop computer, a laptop computer, a network server, a personal digital assistant (PDA), a mobile phone, a tablet computer, a wireless terminal device, a communication device, or an embedded device. The embodiments of the present application do not limit the type of the computer device.

[0060] In addition, the present application also provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the building crack image segmentation method based on semantic guidance modulation.

[0061] Although the preferred embodiments of the present application have been described, those skilled in the art who are informed of the basic inventive concept can make additional changes and modifications to the embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0062] Obviously, those skilled in the art can make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalent technologies, the present application also intends to include these modifications and variations.

Claims

1. A method for segmenting architectural crack images based on semantically guided modulation, characterized in that, include: Acquire visible light images of the surface of a building structure containing cracks; The visible light image is input into the encoder-decoder neural network. The encoder extracts multi-layer features and obtains deep semantic features by aggregating the high-level features at the end of the encoder. An edge-assistance branch is added to the early layers of the decoder. This branch contains a multi-branch convolutional module that enhances the gradient response to the crack edges through parallel convolutional paths with different receptive fields in order to extract the initial edge feature map. After upsampling the deep semantic features, a segmentation head consisting of a 1×1 convolutional layer and a sigmoid activation function is used to output a probability map of each pixel belonging to the background interference region. The probability map is used as a soft mask, inverted, and then multiplied element-wise with the initial edge feature map to obtain enhanced edge features that suppress non-crack edge responses. The enhanced edge features are concatenated with the features of the corresponding layer of the decoder, and then fused through a convolutional layer. Finally, the decoder outputs the crack pixel-level segmentation result.

2. The method as described in claim 1, characterized in that, The visible light image is input into a neural network with an encoder-decoder structure. The encoder extracts multi-layer features, specifically including: The input visible light image is normalized. The normalized image is then input into the backbone network of the encoder; In the backbone network, features output by at least three intermediate layers with different receptive fields in the spatial dimension are selected as the multilayer features; At least three multi-layer features are passed to the corresponding layer in the decoder via skip connections.

3. The method as described in claim 1, characterized in that, The deep semantic features obtained by aggregating high-level features at the end of the encoder specifically include: Obtain the high-level feature maps output from the last two stages of the encoder; Channel attention processing is performed on the two high-level feature maps separately to calibrate the weights of each feature channel; The two feature maps after channel attention processing are adjusted to have the same number of channels by passing them through a 1×1 convolutional layer. The two feature maps with adjusted channel numbers are added element by element to obtain a single aggregated feature map; The aggregated feature map is input into a convolutional block containing residual connections for feature refinement, and the deep semantic features are output.

4. The method as described in claim 1, characterized in that, Adding edge auxiliary branches in the early layers of the decoder specifically includes: The features prior to the first upsampling layer in the decoder are selected as the input to the edge auxiliary branch; The input features are fed into a multi-branch convolution module; The multi-branch convolution module contains three parallel convolutional paths, wherein the first path uses standard 3×3 convolution, the second path uses 3×3 dilated convolution with a dilation rate of 3, and the third path uses 3×3 dilated convolution with a dilation rate of 5. The output features of the three pathways are concatenated along the channel dimension; Perform a 1×1 convolution operation on the concatenated features to generate an initial edge feature map.

5. The method as described in claim 1, characterized in that, After upsampling the deep semantic features, a segmentation head consisting of a 1×1 convolutional layer and a sigmoid activation function is used to output a probability map of each pixel belonging to the background interference region. Specifically, this includes: The deep semantic features are upsampled using bilinear interpolation to make their height and width dimensions consistent with the initial edge feature map; The upsampled feature map is input into the 1×1 convolutional layer, and its number of channels is compressed to 1. The obtained single-channel feature map is input into the Sigmoid activation function to generate an initial probability map; The initial probability map is smoothed using Gaussian filtering to obtain the probability map used for modulation.

6. The method as described in claim 5, characterized in that, Before inputting the upsampled feature map into the 1×1 convolutional layer, the method further includes: passing the upsampled feature map through a lightweight channel attention module to enhance the response to the discriminative channel.

7. The method as described in claim 1, characterized in that, Using the probability map as a soft mask, inverting it, and multiplying it element-wise with the initial edge feature map, the enhanced edge features that suppress non-crack edge responses are obtained, specifically including: Invert each pixel value in the probability map to generate an inverted mask image; The inverted mask image is copied and expanded in the channel dimension to match the number of channels of the initial edge feature image. The copied and expanded mask image is multiplied element-wise with the initial edge feature image to obtain preliminary modulation features; The preliminary modulation features are batch normalized to obtain enhanced edge features that suppress non-crack edge responses.

8. The method as described in claim 7, characterized in that, Before performing batch normalization on the preliminary modulation features, the method further includes: performing element-weighted summation on the preliminary modulation features and the initial edge feature map, and then performing batch normalization on the features after weighted summation.

9. The method as described in claim 1, characterized in that, The enhanced edge features are concatenated with the features of the corresponding layer of the decoder through channel concatenation, and then fused through a convolutional layer. Specifically, this includes: The enhanced edge features are concatenated with the main path feature map of the corresponding scale in the decoder along the channel dimension. The stitched feature maps are fused using a fusion module, which includes a channel recalibration submodule and a depthwise separable convolution submodule. The fused feature map is then input into the next upsampling layer of the decoder.

10. The method according to any one of claims 1-9, characterized in that, The pixel-level segmentation results of the cracks output by the decoder specifically include: Repeat the decoding process until the spatial dimensions of the input image are restored; At the end of the decoder, a 1×1 convolution is used to map the number of feature channels to 2, corresponding to the background pixels and the crack pixels respectively; The Softmax function is applied to the mapped dual-channel features, and the output of the crack pixel channel is used as the final pixel-level segmentation probability map.

Images