Complex background bridge member semantic segmentation method and device based on deep learning model

Abstract

The embodiment of the invention provides a complex background bridge component semantic segmentation method and device based on a deep learning model, and the method comprises the steps: generating a span bridge component semantic segmentation data set; performing data enhancement on the semantic segmentation data set of the span bridge component for a bridge detection scene to obtain an enhanced data set; and constructing a lightweight bridge component semantic segmentation network framework based on deep learning, and training and testing a span bridge component semantic segmentation model based on the lightweight bridge component semantic segmentation network framework. According to the method, two different features of semantics and edges can be effectively fused, and finer real-time detection of bridge components is realized.

Description

Technical Field

[0001] This document relates to the technical field of semantic segmentation methods for complex background bridge components based on deep learning models, and in particular to a semantic segmentation method and device for complex background bridge components based on deep learning models. Background Art

[0002] As a key transportation infrastructure, the structural health of bridges is directly related to public safety and economic operation. As the service life of bridge structures increases, various damages will occur on their components due to factors such as environmental erosion, traffic loads, and material aging. If these damages are not discovered and repaired in time, they may cause structural function degradation or even collapse. Bridge structural components provide detailed information on the bridge structure, including the type, shape, and size of each part such as piers, main beams, and abutments, and also provide location information for damage, which is an indispensable part of realizing comprehensive bridge inspection and evaluation. Therefore, effective identification of bridge structural components is crucial to ensure the safe operation of bridges.

[0003] In traditional bridge inspection methods, algorithms such as edge detection and threshold segmentation have been successfully applied in local defect identification. However, due to the complexity of bridge structures and the changing working environment, such simple image processing techniques cannot effectively extract bridge components from complex image backgrounds. The rapid development of deep learning technology has greatly improved the robustness and automation level of image recognition, and further promoted the research and development of bridge component detection methods. The two-dimensional image bridge component recognition method based on deep learning mainly uses a data set containing bridge component images and component semantic annotations to train a deep learning model, and then uses the trained model to mark the bridge components appearing in the image. At present, the existing methods of this type can only recognize a single type of component, and multiple different types of components are merged and classified into one category, which has limited practical application value. In addition, some researchers use artificially synthesized data sets to train bridge component detection models, but the synthesized data cannot effectively simulate the complex environmental noise in different actual detection scenarios, resulting in limited generalization ability of the model. With the development of three-dimensional scanning technology, engineering structure detection technology based on point clouds has been widely studied. The three-dimensional point cloud bridge component recognition method based on deep learning requires scanning a large amount of point cloud data and making high-precision bridge component point cloud annotations, which is costly. At the same time, model training involves three-dimensional convolution operations, which are computationally expensive. Another more economical and reliable method for three-dimensional recognition of bridge components is to use a deep learning model for semantic segmentation of two-dimensional image bridge components to process image sequences, and combine multi-view geometric three-dimensional reconstruction technology to map the identified component information in the image to a three-dimensional point cloud to achieve three-dimensional detection. However, the existing two-dimensional bridge component detection model currently has rough edges and missing semantics in semantic segmentation under complex image backgrounds, and its performance needs to be improved. Therefore, how to automatically and accurately identify different bridge structure components under complex environmental noise and changeable shooting conditions is an urgent problem for both two-dimensional and three-dimensional detection.

[0004] In summary, the prior art has the following problems:

[0005] The semantic labels of bridge components in existing two-dimensional image datasets are not precise enough in specifying and classifying component categories, and are too general. For example, three important structural components, namely, crash barriers, main beams, and bridge decks, are roughly classified as main beams or superstructures, which actually cannot provide specific location information for defects. At the same time, some datasets contain too many different types of bridges and cannot guarantee that there are enough samples for special bridge components, resulting in serious weight offset problems in the trained models. Therefore, the present invention focuses on the component identification of small and medium span bridges, and specifically collects bridge detection images of three different main beam types of bridges, namely, small box beams, T-beams, and hollow slab beams, and creates semantic annotation maps of eight common and typical component types between them, thereby establishing a semantic segmentation dataset for bridge components.

[0006] 2. Existing two-dimensional bridge component recognition methods commonly use data enhancement methods to try to promote the training of their deep learning detection models. These data enhancement methods simply combine different affine transformation methods and use default parameters to enhance the dataset images. They do not consider the actual situation of the bridge detection environment and have little effect on the model gain. Therefore, the present invention proposes a data enhancement method specifically for bridge detection. This method fully considers three typical environmental interferences in actual engineering scenes, namely distance, angle and light changes, determines the optimal enhancement combination and enhancement parameters, and improves the model's anti-noise ability.

[0007] 3. Currently, most of the deep learning models for bridge component recognition on the market use traditional convolutional architectures to extract component feature information. Such models have a small receptive field and are very inefficient in extracting global context information, making it difficult to accurately identify bridge components from complex background images. At the same time, edge features are extremely important information for accurately identifying bridge components, but they have not received due attention. Therefore, there is an urgent need for a lightweight semantic segmentation network architecture for bridge components that effectively integrates two different features, semantics and edges, to achieve more sophisticated real-time detection of bridge components. Summary of the invention

[0008] The purpose of the present invention is to provide a method and device for semantic segmentation of complex background bridge components based on a deep learning model, aiming to solve the above-mentioned problems in the prior art.

[0009] The present invention provides a complex background bridge component semantic segmentation method based on a deep learning model, comprising:

[0010] Generate a semantic segmentation dataset of span bridge components;

[0011] Performing data enhancement on the span bridge component semantic segmentation dataset for a bridge detection scenario to obtain an enhanced dataset;

[0012] A deep learning-based semantic segmentation network framework for lightweight bridge components is constructed, and a semantic segmentation model for span bridge components is trained and tested based on the lightweight bridge component semantic segmentation network framework.

[0013] The present invention provides a complex background bridge component semantic segmentation device based on a deep learning model, comprising:

[0014] A dataset generation module is used to generate a semantic segmentation dataset of span bridge components;

[0015] A data enhancement module, used for performing data enhancement on the span bridge component semantic segmentation dataset for a bridge detection scenario to obtain an enhanced dataset;

[0016] The training and testing module is used to construct a semantic segmentation network framework for lightweight bridge components based on deep learning, and to train and test a semantic segmentation model for span bridge components based on the semantic segmentation network framework for lightweight bridge components.

[0017] An embodiment of the present invention also provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the above-mentioned method for semantic segmentation of complex background bridge components based on a deep learning model.

[0018] An embodiment of the present invention also provides a computer-readable storage medium, on which an implementation program for information transmission is stored. When the program is executed by a processor, the steps of the above-mentioned method for semantic segmentation of complex background bridge components based on a deep learning model are implemented.

[0019] By using the embodiments of the present invention, more professional pixel-level detection of small and medium-span bridges can be provided, and eight types of key components, including bridge decks, main beams, diaphragms and piers, can be more comprehensively identified, helping to quickly formulate targeted detection and maintenance strategies. This method has strong noise resistance and can effectively detect bridge components in complex image backgrounds with drastic changes in scale, angle and light, and has strong engineering application value. In addition, this method efficiently designs powerful feature extraction, edge and aggregation modules in a lightweight manner, successfully uses edge information to improve the performance of the detection model while occupying extremely low computing resources, and shows great potential in the direction of refined semantic segmentation and real-time detection of bridge components. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate one or more embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0021] Figure 1 is a flow chart of a method for semantic segmentation of bridge components in complex backgrounds based on a deep learning model according to an embodiment of the present invention;

[0022] Figure 2 Schematic diagram of examples of bridge components of three small and medium span bridges according to an embodiment of the present invention;

[0023] Figure 3 is a schematic diagram of an example of a bridge component dataset image and a semantic label map according to an embodiment of the present invention;

[0024] Figure 4is a schematic diagram of three typical interference examples in a bridge detection scenario according to an embodiment of the present invention;

[0025] Figure 5 is a processing schematic diagram of a data enhancement method for a bridge detection scenario according to an embodiment of the present invention;

[0026] Figure 6 This is a framework diagram of a semantic segmentation network for a double-headed lightweight bridge component according to an embodiment of the present invention;

[0027] Figure 7 is a schematic diagram of loss and accuracy curves of model training and verification in an embodiment of the present invention;

[0028] Figure 8 is a schematic diagram of visualization of detection results of a bridge component identification model according to an embodiment of the present invention;

[0029] Fig. 9 is a schematic diagram of a complex background bridge component semantic segmentation device based on a deep learning model according to an embodiment of the present invention;

[0030] Fig.10 is a schematic diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0031] In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the following will be combined with the drawings in one or more embodiments of this specification to clearly and completely describe the technical solutions in one or more embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this document.

[0032] Method Embodiment

[0033] According to an embodiment of the present invention, a method for semantic segmentation of bridge components in complex background based on a deep learning model is provided. Figure 1 is a flow chart of a method for semantic segmentation of complex background bridge components based on a deep learning model according to an embodiment of the present invention. Figure 1 As shown, the complex background bridge component semantic segmentation method based on the deep learning model according to an embodiment of the present invention specifically includes:

[0034] Step S101, generating a span bridge component semantic segmentation dataset; specifically comprising:

[0035] Collect actual inspection images of three typical small and medium span bridges: small box beam, T-beam and hollow slab beam;

[0036] Eight typical small- and medium-span bridge components, including bridge deck pavement, crash wall, expansion joint, bridge deck, main beam, diaphragm, cap beam and pier, were selected as semantic segmentation categories, and semantic label maps of these eight components were created using annotation tools.

[0037] Perform edge detection on the created semantic label map, extract semantic edges, and perform binarization processing to generate an edge label map;

[0038] The original image is placed together with the corresponding semantic and edge label maps, and the training set, validation set and test set are divided according to a certain ratio to create a semantic segmentation dataset of span bridge components. The width and height of the image and label map in the semantic segmentation dataset of span bridge components are adjusted to w and h respectively.

[0039] Step S102, performing data enhancement on the span bridge component semantic segmentation dataset for a bridge detection scenario to obtain an enhanced dataset; specifically comprising:

[0040] Step 31, randomly scaling the images, semantics and edge annotation images in the training set to any resolution within a predetermined multiple range of the original size while maintaining the aspect ratio, and using a cropping frame of size w×h to crop out the central area of ​​the scaled image;

[0041] Step 32, the cropped image is passed to a preset enhancement block for processing, so that the same image is randomly processed by at least 0 and at most 2 image enhancement technologies in the enhancement block. The enhancement block is constructed using four image processing technologies: Gaussian blur, flip, rotation, and Gamma transformation. Gaussian blur is used to simulate the noise caused by camera shake during detection; flip and rotation are used to simulate the change of camera shooting angle; Gamma transformation is used to consider the uneven light caused by various factors. Flip and rotation operations will process the image and the corresponding label map at the same time, while Gaussian blur and Gamma transformation only process the image, and the label map is directly saved in the output form;

[0042] Step 33, repeating steps 31 to 32 n times, that is, enhancing the original data n times, and saving the enhanced data each time to the same folder to form an enhanced data set.

[0043] Step S103, constructing a semantic segmentation network framework for lightweight bridge components based on deep learning, and training and testing the semantic segmentation model of span bridge components based on the semantic segmentation network framework for lightweight bridge components. The semantic segmentation network framework for lightweight bridge components specifically includes: a feature extraction module, an edge module, and an aggregation module, and outputs different results through two detection heads, wherein the feature module is constructed by 4 Mix Transformer blocks;

[0044] Specifically include:

[0045] Use cross entropy loss function to supervise the edge enhanced semantic feature map f fusion Learning, calculate the loss value between the semantic prediction value and the semantic true value, where the semantic true value is the semantic label map in the dataset;

[0046] Use binary cross entropy loss function to supervise edge features f b Learning, calculate the loss value between the edge prediction value and the edge truth value, and take the sum of the semantic loss and the edge loss as the total loss of model training, where the edge truth value is the edge label map in the dataset;

[0047] Select AdamW as the optimizer, set the training batch size, learning rate, and maximum number of training rounds;

[0048] The enhanced data set is incorporated into the span bridge component semantic segmentation data set, samples are extracted according to the training batch size and input into the lightweight bridge component semantic segmentation network framework, and steps 41 to 48 are executed to start the training of the span bridge component semantic segmentation model;

[0049] Record the training process of the semantic segmentation model for span bridge components, calculate the training accuracy and loss value in the training and verification stages, and draw a curve graph.

[0050] The semantic segmentation model of span bridge components is evaluated on the validation set every two training rounds, the IoU score is calculated, and the model with the highest IoU score is selected as the optimal model and saved.

[0051] The saved span bridge component semantic segmentation model performs semantic segmentation on different bridge components on the test set images and visualizes the detection results. Steps 41 to 48 specifically include:

[0052] Step 41, feature extraction is performed on the input image of size h×w×c through the feature extraction module, and it passes through 4 Mix Transformer blocks in sequence. The output of the previous Mix Transformer block is input into the next Mix Transformer block to extract higher-level features. Finally, the Mix Transformer blocks 1 to 4 output semantic feature maps f1, f2, f3, f4 from large to small to form a multi-scale semantic feature map, where c represents the number of image channels;

[0053] Step 42, use 1×1 convolution to compress the number of channels of semantic feature maps of different sizes to obtain compressed semantic feature maps

[0054] Step 43, extracting low-level contour features through the edge module, and inputting the low-level contour features into the low-level block for extraction, wherein the low-level block is composed of two identical sub-blocks, and each sub-block sequentially processes the input using a 3×3 convolutional layer, group regularization, and a PReLU activation function;

[0055] Step 44: compress the semantic feature map They are fed into four independent attention gating blocks together with the low-level contour features, and output four edge features f b1 ,f b2 ,f b3 ,f b4 ;

[0056] Step 45, use the bilinear interpolation upsampling method to convert the four edge features f b1 ,f b2 ,f b3 ,f b4 The resolution is adjusted to w / 2×h / 2, and the four edge features with the same adjusted resolution are concatenated together in the channel direction using the concatenation operation, and then the number of channels is adjusted to C using a 1×1 convolution. fusion , generate the final edge feature f b ;

[0057] Step 46, through the aggregation module, use 1×1 convolution to adjust the number of channels of the semantic feature map f4 to C fusion , and use the bilinear interpolation upsampling method to adjust the feature resolution to w / 2×h / 2 so that it can match the edge feature f b To carry out polymerization;

[0058] Step 47, the edge feature f b The semantic feature map f4 is input into the dynamic attention decoder for feature fusion to generate the edge-enhanced semantic feature map f fusion ;

[0059] Step 48: the edge feature f b A classification head is passed in to predict and output the edge feature map; the semantic feature map f fusion The input segmentation head predicts and outputs the semantics of the bridge components, wherein in the output semantic segmentation image, the identified background and bridge component types are marked according to the colors defined by the semantic labels.

[0060] From the above description, it can be seen that the technical solution of the embodiment of the present invention can accurately detect the position and shape of important structural components in real time under complex environmental noise and different shooting conditions. First, a component semantic segmentation dataset common to small and medium-span bridges is provided. Actual inspection images of three typical small and medium-span bridges with complex and changeable backgrounds, including small box girders, T-beams and hollow slab beams, are collected; label maps of 8 types of commonly inspected typical bridge components, including bridge deck pavement, crash barriers, expansion joints, bridge decks, main beams, diaphragms, cap beams and bridge piers, are created; binary edge label maps of components are generated based on multi-category semantic label maps.

[0061] The embodiment of the present invention also provides a data enhancement method for actual bridge detection scenarios. The method optimizes the combined use of different image transformation and processing technologies and the parameter range of these image technologies to reduce the impact of three common adverse conditions of image distance, angle and light changes on the model detection performance. The richness of samples of the same type of component area size changes in the image is effectively enhanced by scaling the image resolution and cropping the image with a fixed resolution, thereby reducing the model's attention to distance and scale changes. On the basis of scale changes, an enhancement block is designed to combine different enhancement methods to simulate a complex detection environment. Flipping and rotation produce more samples at different angles to simulate the randomness of the camera angle for on-site detection; Gaussian blur operations simulate image noise caused by accidental camera shake, and Gamma transformation simulates light changes. In order to make the enhancement effect closer to the actual situation, the image transformation technology in the enhancement block uses more refined enhancement parameters, and the same image is processed by 0 to 2 image technologies each time it enters the enhancement block.

[0062] Finally, the embodiment of the present invention provides a dual-headed lightweight bridge component semantic segmentation network based on deep learning. A novel encoder based on the Transformer structure is used to efficiently extract global bridge component feature information and more accurately process complex contextual relationships; a dual-detection head network architecture with a feature extraction module, an edge module, and an aggregation module is innovatively designed to effectively extract and fully utilize edge information, greatly improving the details of bridge component segmentation and improving detection performance; in addition, the network architecture has very few calculation parameters and can adapt to real-time lightweight deployment and detection on site, with high practical value.

[0063] Among them, the semantic segmentation dataset of components commonly used in small and medium span bridges contains complex background bridge detection images covering different detection angles of the whole bridge, and creates semantic label maps of 8 types of important structural components and edge binary label maps of components. The implementation process of the present invention involves simply adding annotation categories of small and medium span bridge components, using other annotation tools, and simply changing the name, index value or order and marking color of the category, which are all within the scope of protection.

[0064] In the data enhancement method for actual bridge detection scenarios, the method fully considers and reasonably selects different image transformation and processing technologies, optimizes the order and combination of use, and determines the optimal parameter range to improve the robustness of the detection model, targeting three common interferences: changes in component scale caused by shooting distance, changes in component position caused by shooting angle, and uneven light caused by factors such as weather and occlusion. The implementation process of the present invention involves simply adding the same type of image transformation and processing technology; changing the parameter range; modifying the upper and lower limits of the number of image transformation technologies executed in the enhancement block; and changing the number of image enhancements, all of which are within the protection range.

[0065] In the deep learning-based double-headed lightweight bridge component semantic segmentation network, a lightweight double-headed bridge component semantic segmentation framework was developed through modular design. The framework introduced a powerful Transformer architecture as the backbone network to extract global features; the attention gating block was used to efficiently extract the edge features of the bridge component in a supervised learning manner; the dynamic attention decoder was used to fully integrate the two different useful information of the bridge component semantics and edges, thereby improving the accuracy of component detection. The present invention involves simply replacing different backbone networks, attention gating blocks, and dynamic attention decoders during implementation; and simply modifying the parameters of these modules and architectures are all within the scope of protection.

[0066] The learning of edge features and semantic features is carried out in a supervised manner. Among them, the supervised learning of edge features uses a binary cross entropy loss function to calculate the edge loss, and the edge label map is used as the true value for calculating the edge loss; the supervised learning of semantic features uses a standard cross entropy function to calculate the semantic loss, and the semantic label map is used as the true value for calculating the semantic loss. The total training loss is equal to the linear sum of the edge loss and the semantic loss. In the implementation process of the present invention, simply adding auxiliary loss functions; changing the edge loss and semantic loss function types; adjusting the weights of edge loss and semantic loss in the total loss calculation are all within the protection range.

[0067] The above technical solution of the embodiment of the present invention is described in detail below in conjunction with the accompanying drawings.

[0068] This embodiment is based on Pytorch deep learning code and Anaconda environment, and relies on a platform equipped with a GeForce RTX4090 graphics card and an Intel Core i7-13700k CPU processor for calculation.

[0069] S01: Detection image collection. Collect actual detection images of three typical small and medium span bridges: small box beam, T-beam and hollow slab beam. These images cover different detection angles such as the top, side and bottom of the bridge, reflecting the shooting conditions and environmental interference of different distances, angles and obstructions. After careful screening, a total of 1220 high-definition images with a resolution of 5184×3888 were obtained. The target bridge type and target component category are as follows: Figure 2 shown.

[0070] S02: Semantic label map creation. 8 types of typical small and medium span bridge components that are frequently inspected are selected as semantic segmentation categories. The relationship between the categories and their corresponding index values ​​(including background) in this embodiment is {background: 0, bridge deck pavement: 1, crash barrier: 2, expansion joint: 3, bridge deck: 4, main beam: 5, diaphragm: 6, cap beam: 7, bridge pier: 8}. The Labelme annotation tool is used to create semantic label maps for these 8 types of components. The label map is a single-channel grayscale image, and the grayscale value corresponds to the category index. The RGB colors used for mapping (including background) are {[0,0,0],[99,0,200],[0,200,198],[198,0,200],[0,200,0],[0,0,200],[200,198,0],[200,132,0]}. The created semantic label map is as follows: Figure 3 shown.

[0071] S03: Edge label map creation. Use a custom 5×5 convolution kernel Process the semantic label map created in step 02; calculate the grayscale mean g of the processed grayscale image m ; Then extract the gray value g≥g m The grayscale value of the remaining areas is changed to 0, thereby generating a binary edge label map.

[0072] S04: Dataset establishment. The original image and the corresponding semantic and edge label images are placed together, and the training set, validation set and test set are divided into a ratio of 8:1:1 to create a bridge component semantic segmentation dataset.

[0073] S05: Pre-processing of training data. The width and height of the dataset image and label image output in step S04 are adjusted to w and h respectively, so that the image data can be input into the model at the maximum resolution allowed by the memory capacity of the 4090 graphics card, providing sufficient image information for training. In this embodiment, w is set to 1440 and h is set to 1080.

[0074] S06: Image scaling and cropping. In each enhancement, the training set images and semantic and edge annotation images obtained in step S04 are randomly scaled to any resolution in the range of 0.4 to 1.0 times the original size while maintaining the aspect ratio. Then, a cropping box of size w×h is used to crop the central area of ​​the scaled image to obtain a cropped image, thereby simulating the scale change of the detection scene.

[0075] S07: Set up the enhancement block. Use four image processing techniques, namely Gaussian blur, flip, rotation and Gamma transformation, to build the enhancement block. Gaussian blur is used to simulate the noise caused by camera shake during detection, and the blur parameter is (0.5, 5); flip and rotation simulate the change of camera shooting angle, and flip only performs horizontal flipping, the flip probability is set to 0.5, and the rotation angle range is set to (-30°, 30°). Gamma transformation is used to consider the uneven light caused by various factors, and the Gamma value range is set to (0.6, 1.2). The three detection scene changes that are considered in the data enhancement of this embodiment are as follows: Figure 4 shown.

[0076] S08: Enhancement block processes the data set. The cropped image obtained in step S06 is passed to the enhancement block in step 07 for processing. The same image is randomly processed by at least 0 and at most 2 image enhancement techniques in the enhancement block. The flip and rotation operations are set to process the image and the corresponding label map at the same time, while the Gaussian blur and gamma transformation only process the image, and its label map is directly saved in the form of the output of step S06.

[0077] S09: Output enhanced data. Repeat steps S06 to S08 n times, that is, enhance the original data n times. Save the data of each enhancement to the same folder to form an enhanced data set. In this embodiment, n is set to 4. The overall framework of the data enhancement method of this embodiment is as follows Figure 5 shown.

[0078] S10: Extract semantic features. The image of size h×w×c is first input into the feature extraction module for feature extraction. The feature module is built by 4 Mix Transformer blocks. The output of the previous block is input into the next block to extract higher-level features. In this embodiment, the sizes of the feature maps f1, f2, f3, and f4 output by Mix Transformer are respectively

[0079]

[0080] S11: Semantic feature channel compression. In this embodiment, the number of channels of the semantic feature maps of different sizes output by the 1×1 convolution compression step S10 is reduced to {16, 16, 32, 32} to generate a compressed semantic feature map.

[0081] S12: Extraction of low-level contour features. The image is input into the low-level block for extraction of low-level contour features. The low-level block consists of two identical sub-blocks. Each sub-block processes the input using a 3×3 convolution layer, group regularization, and PReLU activation function in turn. In this embodiment, the step size of the 3×3 convolution layer in the first sub-block is set to 2, and the number of output channels is 8; the step size of the second sub-block is set to 1, and the number of output channels is 16. Other parameters remain the same, and the final output is Low-level contour features of size.

[0082] S13: Edge feature extraction. The compressed semantic features output in step S11 and the low-level contour features output in step S12 are input into the attention gating block for processing to generate edge features. They are input into different attention gating blocks respectively with low-level contour features, and the output size is The four edge features f b1 ,f b2 ,f b3 ,f b4 .

[0083] S14: Adjust edge features. Use the bilinear interpolation upsampling method to adjust the resolution of the four edge features output in step S13 to w / 2×h / 2, then use the concatenation operation to splice the four edge features with the same adjusted resolution together in the channel direction, and then use 1×1 convolution to adjust the number of channels to Cfusion to generate the final edge feature f b In this embodiment, Cfusion is the sum of the number of channels of the four edge features in step S13, that is, 96.

[0084] S15: Adjust the highest level semantic features. Use 1×1 convolution to adjust the number of channels of the semantic feature map f4 in step S10 to Cfusion, and use the bilinear interpolation upsampling method to adjust the feature resolution to w / 2×h / 2 so that it can match the edge feature f output in step S14. b Perform aggregation.

[0085] S16: Feature aggregation. The edge features output from step S14 and the semantic features output from step S15 are input into the dynamic attention decoder for feature fusion to generate the edge-enhanced semantic features f fusion In this embodiment, the output f fusion The size is

[0086] S17: Output edge feature map and semantic segmentation results. The edge features of step S14 are passed into a simple classification head to predict and output the edge feature map; the refined semantic features output from step S16 are input into the segmentation head to predict and output the semantics of the bridge components. In the output semantic segmentation image, the identified background and bridge component types are marked according to the colors defined by the semantic labels in step S02. In this embodiment, the output channel of the classification head is 1, and the output channel of the segmentation head is 9. The overall framework of the deep learning model for bridge component recognition in this embodiment of the present invention is as follows: Figure 6 shown.

[0087] S18: Loss function. The cross entropy loss function is used to supervise the edge enhancement semantic features f in step S16. fusion The learning of the semantic prediction value and the semantic truth value is calculated, and the semantic truth value is the semantic label map in the data set; the edge feature f output by the supervision step S14 is used b The learning of edge prediction is to calculate the loss value between the edge truth value, where the edge truth value is the edge label map in the dataset. The sum of semantic loss and edge loss is taken as the total loss of model training.

[0088] S19: Set training hyperparameters. Select AdamW as the optimizer, set the training batch size, learning rate, and maximum training rounds. In this embodiment, the optimizer parameters are: β1=0.9, β2=0.999, the batch size is 2, the learning rate is 0.00006, and the maximum training rounds are 200.

[0089] S20: Start training. The enhanced dataset images obtained in step S09 are merged into the dataset of step S05, and then samples are extracted according to the batch size set in step S19 and fed into the model (steps S10-S17), and model training is started.

[0090] S21: Record the model training process. Calculate the training accuracy and loss value of the training and verification phases, and draw a curve graph. The loss and accuracy curves drawn in this embodiment are as follows: Figure 7 shown.

[0091] S22: Model evaluation. The model is evaluated on the validation set every two training rounds, the IoU score is calculated, and the model with the highest IoU score is selected as the optimal model for saving. The results of the semantic segmentation IoU accuracy, parameter quantity, and inference speed of different bridge components of the optimal model of this embodiment on the test set are listed in Table 1.

[0092] Table 1 Detection performance information of the model of the present invention

[0093]

[0094] S23: Result visualization. The model saved in step S22 is used to perform semantic segmentation on different bridge components on the test set image, and the detection results are visualized, such as Figure 8 shown.

[0095] In summary, the embodiment of the present invention provides an automatic and accurate semantic segmentation method for the identification of bridge detection components. Compared with existing methods, the embodiment of the present invention provides more professional pixel-level detection of small and medium-span bridges, which can more comprehensively identify 8 types of key components such as bridge decks, main beams, diaphragms and piers, helping to quickly formulate targeted detection and maintenance strategies. This method has strong noise resistance and can effectively detect bridge components in complex image backgrounds with drastic changes in scale, angle and light, and has strong engineering application value. In addition, the method efficiently designs powerful feature extraction, edge and aggregation modules in a lightweight manner, successfully uses edge information to improve the performance of the detection model while occupying extremely low computing resources, and shows great potential in the direction of refined semantic segmentation and real-time detection of bridge components.

[0096] Device Example 1

[0097] According to an embodiment of the present invention, a semantic segmentation device for complex background bridge components based on a deep learning model is provided. Fig. 9 is a schematic diagram of a complex background bridge component semantic segmentation device based on a deep learning model according to an embodiment of the present invention. Fig. 9 As shown, the complex background bridge component semantic segmentation device based on the deep learning model according to an embodiment of the present invention specifically includes:

[0098] The data set generation module 90 is used to generate a semantic segmentation data set of span bridge components; specifically, it is used to:

[0099] Collect actual inspection images of three typical small and medium span bridges: small box beam, T-beam and hollow slab beam;

[0100] Eight typical small- and medium-span bridge components, including bridge deck pavement, crash wall, expansion joint, bridge deck, main beam, diaphragm, cap beam and pier, were selected as semantic segmentation categories, and semantic label maps of these eight components were created using annotation tools.

[0101] Perform edge detection on the created semantic label map, extract semantic edges, and perform binarization processing to generate an edge label map;

[0102] The original image is placed together with the corresponding semantic and edge label maps, and the training set, validation set and test set are divided according to a certain ratio to create a semantic segmentation dataset of span bridge components. The width and height of the image and label map in the semantic segmentation dataset of span bridge components are adjusted to w and h respectively.

[0103] The data enhancement module 92 is used to perform data enhancement on the span bridge component semantic segmentation dataset for the bridge detection scene to obtain an enhanced dataset; the data enhancement module 92 is specifically used to:

[0104] Randomly scale the images, semantics, and edge annotations in the training set to any resolution within a predetermined multiple range of the original size while maintaining the aspect ratio, and use a cropping frame of size w×h to crop the central area of ​​the scaled image;

[0105] The cropped image is passed to the preset enhancement block for processing, so that the same image is randomly processed by at least 0 and at most 2 image enhancement technologies in the enhancement block. Among them, the enhancement block is built using four image processing technologies: Gaussian blur, flip, rotation, and Gamma transformation. Gaussian blur is used to simulate the noise caused by camera shake in detection; flip and rotation are used to simulate the change of camera shooting angle, and Gamma transformation is used to consider the uneven light caused by various factors. Flip and rotation operations will process the image and the corresponding label map at the same time, while Gaussian blur and Gamma transformation only process the image, and the label map is directly saved in the output form;

[0106] Repeat the above process n times, that is, enhance the original data n times, and save the enhanced data each time to the same folder to form an enhanced data set.

[0107] The training and testing module 94 is used to construct a semantic segmentation network framework for lightweight bridge components based on deep learning, and to train and test the semantic segmentation model of span bridge components based on the semantic segmentation network framework for lightweight bridge components. The semantic segmentation network framework for lightweight bridge components specifically includes: a feature extraction module, an edge module and an aggregation module, and outputs different results through two detection heads, wherein the feature module is constructed by 4 Mix Transformer blocks;

[0108] The training and testing module 94 is specifically used for:

[0109] Use cross entropy loss function to supervise the edge enhanced semantic feature map f fusion Learning, calculate the loss value between the semantic prediction value and the semantic true value, where the semantic true value is the semantic label map in the dataset;

[0110] Use binary cross entropy loss function to supervise edge features f b Learning, calculate the loss value between the edge prediction value and the edge truth value, and take the sum of the semantic loss and the edge loss as the total loss of model training, where the edge truth value is the edge label map in the dataset;

[0111] Select AdamW as the optimizer, set the training batch size, learning rate, and maximum number of training rounds;

[0112] The enhanced data set is incorporated into the span bridge component semantic segmentation data set, samples are extracted according to the training batch size and input into the lightweight bridge component semantic segmentation network framework, and steps 41 to 48 are executed to start the training of the span bridge component semantic segmentation model;

[0113] Record the training process of the semantic segmentation model for span bridge components, calculate the training accuracy and loss value in the training and verification stages, and draw a curve graph.

[0114] The semantic segmentation model of span bridge components is evaluated on the validation set every two training rounds, the IoU score is calculated, and the model with the highest IoU score is selected as the optimal model and saved.

[0115] The saved span bridge component semantic segmentation model performs semantic segmentation on different bridge components on the test set image and visualizes the detection results; wherein steps 41 to 48 specifically include:

[0116] Step 41, feature extraction is performed on the input image of size h×w×c through the feature extraction module, and it passes through 4 Mix Transformer blocks in sequence. The output of the previous Mix Transformer block is input into the next Mix Transformer block to extract higher-level features. Finally, the Mix Transformer blocks 1 to 4 output semantic feature maps f1, f2, f3, f4 from large to small to form a multi-scale semantic feature map, where c represents the number of image channels;

[0117] Step 42, use 1×1 convolution to compress the number of channels of semantic feature maps of different sizes to obtain compressed semantic feature maps

[0118] Step 43, extracting low-level contour features through the edge module, and inputting the low-level contour features into the low-level block for extraction, wherein the low-level block is composed of two identical sub-blocks, and each sub-block sequentially processes the input using a 3×3 convolutional layer, group regularization, and a PReLU activation function;

[0119] Step 44: compress the semantic feature map They are fed into four independent attention gating blocks together with the low-level contour features, and output four edge features f b1 ,f b2 ,f b3 ,f b4 ;

[0120] Step 45, use the bilinear interpolation upsampling method to convert the four edge features f b1 ,f b2 ,f b3,f b4 The resolution is adjusted to w / 2×h / 2, and the four edge features with the same adjusted resolution are concatenated together in the channel direction using the concatenation operation, and then the number of channels is adjusted to C using a 1×1 convolution. fusion , generate the final edge feature f b ;

[0121] Step 46, through the aggregation module, use 1×1 convolution to adjust the number of channels of the semantic feature map f4 to C fusion , and use the bilinear interpolation upsampling method to adjust the feature resolution to w / 2×h / 2 so that it can match the edge feature f b To carry out polymerization;

[0122] Step 47, the edge feature f b The semantic feature map f4 is input into the dynamic attention decoder for feature fusion to generate the edge-enhanced semantic feature map f fusion ;

[0123] Step 48: the edge feature f b A classification head is passed in to predict and output the edge feature map; the semantic feature map f fusion The input segmentation head predicts and outputs the semantics of the bridge components, wherein in the output semantic segmentation image, the identified background and bridge component types are marked according to the colors defined by the semantic labels.

[0124] The embodiment of the present invention is a device embodiment corresponding to the above method embodiment. The specific operations of each module can be understood by referring to the description of the method embodiment, which will not be repeated here.

[0125] Device Example 2

[0126] An embodiment of the present invention provides an electronic device, such as Fig.10 As shown, it includes: a memory 100, a processor 102, and a computer program stored in the memory 100 and executable on the processor 102, and when the computer program is executed by the processor 102, the steps described in the method embodiment are implemented.

[0127] Device Example 3

[0128] An embodiment of the present invention provides a computer-readable storage medium, on which a program for implementing information transmission is stored. When the program is executed by the processor 102, the steps described in the method embodiment are implemented.

[0129] The computer-readable storage medium in this embodiment includes, but is not limited to, ROM, RAM, magnetic disk or optical disk, etc.

[0130] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A semantic segmentation method for bridge components in complex background based on deep learning model, characterized in that: include: Generate a semantic segmentation dataset of span bridge components; Performing data enhancement on the span bridge component semantic segmentation dataset for a bridge detection scenario to obtain an enhanced dataset; A deep learning-based semantic segmentation network framework for lightweight bridge components is constructed, and a semantic segmentation model for span bridge components is trained and tested based on the lightweight bridge component semantic segmentation network framework.

2. The method according to claim 1, characterized in that Generating a span bridge component semantic segmentation dataset specifically includes: Collect actual inspection images of three typical small and medium span bridges: small box beam, T-beam and hollow slab beam; Eight typical small- and medium-span bridge components, including bridge deck pavement, crash wall, expansion joint, bridge deck, main beam, diaphragm, cap beam and pier, were selected as semantic segmentation categories, and semantic label maps of these eight components were created using annotation tools. Perform edge detection on the created semantic label map, extract semantic edges, and perform binarization processing to generate an edge label map; The original image is placed together with the corresponding semantic and edge label maps, and the training set, validation set and test set are divided according to a certain ratio to create a semantic segmentation dataset of span bridge components. The width and height of the image and label map in the semantic segmentation dataset of span bridge components are adjusted to w and h respectively.

3. The method according to claim 1, characterized in that The span bridge component semantic segmentation dataset is enhanced for the bridge detection scene, and the enhanced dataset specifically includes: Step 31, randomly scaling the images, semantics and edge annotation images in the training set to any resolution within a predetermined multiple range of the original size while maintaining the aspect ratio, and using a cropping frame of size w×h to crop out the central area of ​​the scaled image; Step 32, the cropped image is passed to a preset enhancement block for processing, so that the same image is randomly processed by at least 0 and at most 2 image enhancement technologies in the enhancement block. The enhancement block is constructed using four image processing technologies: Gaussian blur, flip, rotation, and Gamma transformation. Gaussian blur is used to simulate the noise caused by camera shake during detection; flip and rotation are used to simulate the change of camera shooting angle; Gamma transformation is used to consider the uneven light caused by various factors. Flip and rotation operations will process the image and the corresponding label map at the same time, while Gaussian blur and Gamma transformation only process the image, and the label map is directly saved in the output form; Step 33, repeating steps 31 to 32 n times, that is, enhancing the original data n times, and saving the enhanced data each time to the same folder to form an enhanced data set.

4. The method according to claim 1, characterized in that: The lightweight bridge component semantic segmentation network framework specifically includes: a feature extraction module, an edge module and an aggregation module, and outputs different results through two detection heads, wherein the feature module is constructed by four Mix Transformer blocks; The training of the semantic segmentation model of span bridge components based on the lightweight bridge component semantic segmentation network framework specifically includes: Step 41, feature extraction is performed on the input image of size h×w×c through the feature extraction module, and it passes through 4 Mix Transformer blocks in sequence. The output of the previous Mix Transformer block is input into the next Mix Transformer block to extract higher-level features. Finally, the Mix Transformer blocks 1 to 4 output semantic feature maps f1, f2, f3, f4 from large to small to form a multi-scale semantic feature map, where c represents the number of image channels; Step 42, use 1×1 convolution to compress the number of channels of semantic feature maps of different sizes to obtain compressed semantic feature maps Step 43, extracting low-level contour features through the edge module, and inputting the low-level contour features into the low-level block for extraction, wherein the low-level block is composed of two identical sub-blocks, and each sub-block sequentially processes the input using a 3×3 convolutional layer, group regularization, and a PReLU activation function; Step 44: compress the semantic feature map They are fed into four independent attention gating blocks together with the low-level contour features, and output four edge features f b1 ,f b2 ,f b3 ,f b4 ; Step 45, use the bilinear interpolation upsampling method to convert the four edge features f b1 ,f b2 ,f b3 ,f b4 The resolution is adjusted to w / 2×h / 2, and the four edge features with the same adjusted resolution are concatenated together in the channel direction using the concatenation operation, and then the number of channels is adjusted to C using a 1×1 convolution. fusion , generate the final edge feature f b ; Step 46, through the aggregation module, use 1×1 convolution to adjust the number of channels of the semantic feature map f4 to C fusion , and use the bilinear interpolation upsampling method to adjust the feature resolution to w / 2×h / 2 so that it can match the edge feature f b To carry out polymerization; Step 47, the edge feature f b The semantic feature map f4 is input into the dynamic attention decoder for feature fusion to generate the edge-enhanced semantic feature map f fusion ; Step 48: the edge feature f b A classification head is passed in to predict and output the edge feature map; the semantic feature map f fusion The input segmentation head predicts and outputs the semantics of the bridge components, wherein in the output semantic segmentation image, the identified background and bridge component types are marked according to the colors defined by the semantic labels.

5. The method according to claim 4, characterized in that The training and testing of the semantic segmentation model of span bridge components based on the lightweight bridge component semantic segmentation network framework specifically includes: Use cross entropy loss function to supervise the edge enhanced semantic feature map f fusion Learning, calculate the loss value between the semantic prediction value and the semantic true value, where the semantic true value is the semantic label map in the dataset; Use binary cross entropy loss function to supervise edge features f b Learning, calculate the loss value between the edge prediction value and the edge truth value, and take the sum of the semantic loss and the edge loss as the total loss of model training, where the edge truth value is the edge label map in the dataset; Select AdamW as the optimizer, set the training batch size, learning rate, and maximum number of training rounds; The enhanced data set is incorporated into the span bridge component semantic segmentation data set, samples are extracted according to the training batch size and input into the lightweight bridge component semantic segmentation network framework, and steps 41 to 48 are executed to start the training of the span bridge component semantic segmentation model; Record the training process of the semantic segmentation model for span bridge components, calculate the training accuracy and loss value in the training and verification stages, and draw curve graphs; The semantic segmentation model of span bridge components is evaluated on the validation set every two training rounds, the IoU score is calculated, and the model with the highest IoU score is selected as the optimal model and saved; The saved span bridge component semantic segmentation model performs semantic segmentation on different bridge components on the test set images and visualizes the detection results.

6. A semantic segmentation device for complex background bridge components based on a deep learning model, characterized in that: include: A dataset generation module is used to generate a semantic segmentation dataset of span bridge components; A data enhancement module, used for performing data enhancement on the span bridge component semantic segmentation dataset for a bridge detection scenario to obtain an enhanced dataset; The training and testing module is used to construct a semantic segmentation network framework for lightweight bridge components based on deep learning, and to train and test a semantic segmentation model for span bridge components based on the semantic segmentation network framework for lightweight bridge components.

7. The device according to claim 6, characterized in that The data set generation module is specifically used for: Collect actual inspection images of three typical small and medium span bridges: small box beam, T-beam and hollow slab beam; Eight typical small- and medium-span bridge components, including bridge deck pavement, crash wall, expansion joint, bridge deck, main beam, diaphragm, cap beam and pier, were selected as semantic segmentation categories, and semantic label maps of these eight components were created using annotation tools. Perform edge detection on the created semantic label map, extract semantic edges, and perform binarization processing to generate an edge label map; The original image and the corresponding semantic and edge label images are placed together, and the training set, the validation set and the test set are divided according to a certain ratio, so as to create a span bridge component semantic segmentation dataset, and the width and height of the image and the label image in the span bridge component semantic segmentation dataset are adjusted to w and h respectively; The data enhancement module is specifically used for: Randomly scale the images, semantics, and edge annotations in the training set to any resolution within a predetermined multiple range of the original size while maintaining the aspect ratio, and use a cropping frame of size w×h to crop the central area of ​​the scaled image; The cropped image is passed to the preset enhancement block for processing, so that the same image is randomly processed by at least 0 and at most 2 image enhancement technologies in the enhancement block. Among them, the enhancement block is built using four image processing technologies: Gaussian blur, flip, rotation, and Gamma transformation. Gaussian blur is used to simulate the noise caused by camera shake in detection; flip and rotation are used to simulate the change of camera shooting angle, and Gamma transformation is used to consider the uneven light caused by various factors. Flip and rotation operations will process the image and the corresponding label map at the same time, while Gaussian blur and Gamma transformation only process the image, and the label map is directly saved in the output form; Repeat the above process n times, that is, enhance the original data n times, and save the enhanced data each time to the same folder to form an enhanced data set.

8. The device according to claim 6, characterized in that The lightweight bridge component semantic segmentation network framework specifically includes: a feature extraction module, an edge module and an aggregation module, and outputs different results through two detection heads, wherein the feature module is constructed by four Mix Transformer blocks; The training and testing module is specifically used for: Use cross entropy loss function to supervise the edge enhanced semantic feature map f fusion Learning, calculate the loss value between the semantic prediction value and the semantic true value, where the semantic true value is the semantic label map in the dataset; Use binary cross entropy loss function to supervise edge features f b Learning, calculate the loss value between the edge prediction value and the edge truth value, and take the sum of the semantic loss and the edge loss as the total loss of model training, where the edge truth value is the edge label map in the dataset; Select AdamW as the optimizer, set the training batch size, learning rate, and maximum number of training rounds; The enhanced data set is merged into the span bridge component semantic segmentation data set, samples are extracted according to the training batch size and input into the lightweight bridge component semantic segmentation network framework, and steps 41 to 48 are executed to start the training of the span bridge component semantic segmentation model; Record the training process of the semantic segmentation model for span bridge components, calculate the training accuracy and loss value in the training and verification stages, and draw a curve graph. The semantic segmentation model of span bridge components is evaluated on the validation set every two training rounds, the IoU score is calculated, and the model with the highest IoU score is selected as the optimal model and saved. The saved span bridge component semantic segmentation model performs semantic segmentation on different bridge components on the test set image and visualizes the detection results; wherein steps 41 to 48 specifically include: Step 41, feature extraction is performed on the input image of size h×w×c through the feature extraction module, and it passes through 4 Mix Transformer blocks in sequence. The output of the previous Mix Transformer block is input into the next Mix Transformer block to extract higher-level features. Finally, the Mix Transformer blocks 1 to 4 output semantic feature maps f1, f2, f3, f4 from large to small to form a multi-scale semantic feature map, where c represents the number of image channels; Step 42, use 1×1 convolution to compress the number of channels of semantic feature maps of different sizes to obtain compressed semantic feature maps Step 43, extracting low-level contour features through the edge module, and inputting the low-level contour features into the low-level block for extraction, wherein the low-level block is composed of two identical sub-blocks, and each sub-block sequentially processes the input using a 3×3 convolutional layer, group regularization, and a PReLU activation function; Step 44: compress the semantic feature map They are fed into four independent attention gating blocks together with the low-level contour features, and output four edge features f b1 ,f b2 ,f b3 ,f b4 ; Step 45, use the bilinear interpolation upsampling method to convert the four edge features f b1 ,f b2 ,f b3 ,f b4 The resolution is adjusted to w / 2×h / 2, and the four edge features with the same adjusted resolution are concatenated together in the channel direction using the concatenation operation, and then the number of channels is adjusted to C using a 1×1 convolution. fusion , generate the final edge feature f b ; Step 46, through the aggregation module, use 1×1 convolution to adjust the number of channels of the semantic feature map f4 to C fusion , and use the bilinear interpolation upsampling method to adjust the feature resolution to w / 2×h / 2 so that it can match the edge feature f b To carry out polymerization; Step 47, the edge feature f b The semantic feature map f4 is input into the dynamic attention decoder for feature fusion to generate the edge-enhanced semantic feature map f fusion ; Step 48: the edge feature f b A classification head is passed in to predict and output the edge feature map; the semantic feature map f fusion The input segmentation head predicts and outputs the semantics of the bridge components, wherein in the output semantic segmentation image, the identified background and bridge component types are marked according to the colors defined by the semantic labels.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the method for semantic segmentation of complex background bridge components based on a deep learning model as described in any one of claims 1 to 5.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores an implementation program for information transmission, and when the program is executed by a processor, the steps of the complex background bridge component semantic segmentation method based on a deep learning model as described in any one of claims 1 to 5 are implemented.

Images