A crack image segmentation method based on residual network and round-trip sampling
By employing a residual network and a round-trip sampling encoder-decoder network structure, combined with optimized feature fusion and loss functions, the problems of deep semantic information dilution and gradient vanishing in neural networks are solved, thus improving the effect of crack image segmentation.
Patent Information
- Application Number
- CN202211620981.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-12-16
- Publication Date
- 2025-12-19
- Estimated Expiration
- 2042-12-16
AI Technical Summary
Existing neural networks suffer from the problems of deep semantic information dilution and gradient vanishing in crack image segmentation tasks, which leads to a decline in network performance and requires a large amount of sample data for training.
A codec network structure based on residual network and round-trip sampling is adopted. By combining residual network modules with hollow space pyramid pooling, the feature fusion unit and cross-entropy loss function are optimized to improve network performance and crack image segmentation effect.
It effectively preserves deep semantic information, solves the gradient vanishing problem, and improves the network's ability to process contextual information and the effect of crack edge segmentation.
Smart Images
Figure CN115880491B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application belongs to the technical field of crack detection, and particularly relates to a crack image segmentation method based on a residual network and round-trip sampling. BACKGROUND
[0002] The crack detection problem can also be regarded as a pixel-level semantic segmentation task. Since an image has more position information in a shallow convolution layer and more semantic information in a deep convolution layer, different scale features need to be fused. A typical U-net network adopts a skip connection mode for feature fusion, and good training results are obtained. However, this causes deep semantic features to be diluted layer by layer, resulting in the network degenerating into a shallow network, thereby losing the advantages of a deep network.
[0003] Deep learning is a branch of machine learning inspired by the structure of the human brain, and many neural network algorithms have been proposed for object detection and image classification tasks. At present, classical neural networks such as fully connected neural networks FCN, fully convolutional networks CNN and other neural network algorithms have obvious advantages over traditional detection methods in semantic segmentation problems, but despite this, the training of a neural network requires a large amount of sample data to obtain optimal results, and with the increase in the number of network layers, the gradient vanishing or explosion problem also occurs in the training process. SUMMARY
[0004] In order to improve the utilization of samples and solve the problem of gradient vanishing, the structure of the neural network needs to be optimized to achieve optimal training results.
[0005] Objective: In order to solve this problem and further optimize the network structure, the application proposes a crack image segmentation method based on a residual network and round-trip sampling, which helps to solve the problem of dilution of deep semantic information, improve network performance and optimize crack image segmentation effect.
[0006] To achieve the above object, the application adopts the following technical scheme:
[0007] In a first aspect, a crack image segmentation method based on a residual network and round-trip sampling is provided, comprising:
[0008] Step S1, inputting a to-be-segmented image into an encoder network to obtain a deepest feature map F1;
[0009] The encoder network comprises, in sequence, a first encoding convolution operation group, a second encoding convolution operation group, a third encoding convolution operation group, a fourth encoding convolution operation group and a fifth encoding convolution operation group connected by a maximum pooling layer;
[0010] The first encoding convolution operation group, the second encoding convolution operation group, the third encoding convolution operation group, and the fourth encoding convolution operation group each comprise two residual network modules; each residual network module comprises a convolution layer, an activation layer, and a batch normalization layer;
[0011] The fifth encoding convolution operation group comprises two combined modules, and each combined module is a combination of a residual network module and a dilated spatial pyramid pooling module.
[0012] In step S2, the feature map F1 is input into a round-trip sampling module to obtain a feature map F2; the round-trip sampling module comprises two convolution operation groups and a down-sampling layer, and each convolution operation group comprises an up-sampling layer and two convolution layers.
[0013] In step S3, the feature map F2 is input into a decoder network to obtain a feature map F3.
[0014] The decoder network comprises, in sequence, a fifth decoding convolution operation group, a fourth decoding convolution operation group, a third decoding convolution operation group, a second decoding convolution operation group, and a first decoding convolution operation group connected by up-sampling layers.
[0015] The first decoding convolution operation group, the second decoding convolution operation group, the third decoding convolution operation group, and the fourth decoding convolution operation group each comprise two residual network modules; each residual network module comprises a convolution layer, an activation layer, and a batch normalization layer.
[0016] The fifth decoding convolution operation group comprises two combined modules, and each combined module is a combination of a residual network module and a dilated spatial pyramid pooling module.
[0017] In step S4, the feature map F3 is input into a prediction network to obtain a crack segmentation image.
[0018] In some embodiments, step S1 comprises:
[0019] The image to be segmented is input into the first encoding convolution operation group to obtain a first encoding feature map.
[0020] The first encoding feature map is input into the second encoding convolution operation group after passing through a max-pooling layer to obtain a second encoding feature map.
[0021] The second encoding feature map is input into the third encoding convolution operation group after passing through a max-pooling layer to obtain a third encoding feature map.
[0022] The third encoding feature map is input into the fourth encoding convolution operation group after passing through a max-pooling layer to obtain a fourth encoding feature map.
[0023] The fourth encoding feature map is input into a fifth encoding convolution operation group after a maximum pooling layer, and a feature map F1 is obtained.
[0024] In some embodiments, step S3 comprises:
[0025] The feature map F2 is input into a fifth decoding convolution operation group to obtain a fifth decoding feature map;
[0026] The fifth decoding feature map is up-sampled and fused with the fourth encoding feature map, and then input into a fourth decoding convolution operation group to obtain a fourth decoding feature map;
[0027] The fourth decoding feature map is up-sampled and fused with the third encoding feature map, and then input into a third decoding convolution operation group to obtain a third decoding feature map;
[0028] The third decoding feature map is up-sampled and fused with the second encoding feature map, and then input into a second decoding convolution operation group to obtain a second decoding feature map;
[0029] The second decoding feature map is up-sampled and fused with the first encoding feature map, and then input into a first decoding convolution operation group to obtain a feature map F3.
[0030] In some embodiments, the fusion is performed using an optimized feature fusion unit, which includes a fusion block, a batch normalization layer, and a convolution layer.
[0031] In some embodiments, in step S4, the prediction network uses a Sigmoid function.
[0032] In some embodiments, each residual network module includes two 3*3 convolution layers, two activation layers, three batch normalization layers, and a 1*1 convolution layer.
[0033] The input feature of the residual network module is processed by a first 3*3 convolution layer, a first batch normalization layer, a first activation layer, a second 3*3 convolution layer, and a second batch normalization layer in sequence to obtain a first feature,
[0034] The input feature of the residual network module is processed by a 1*1 convolution layer and a third batch normalization layer in sequence to obtain a second feature.
[0035] The first feature and the second feature are fused and processed by a second activation layer to obtain the output feature of the residual network module.
[0036] In some embodiments, the entire codec network uses a cross-entropy loss function L ce :
[0037]
[0038] where y is the true value of the image, is a prediction value of the image, β is a weight coefficient, and γ is an adjustable focus parameter.
[0039] In a second aspect, the present application provides a crack image segmentation device based on a residual network and round-trip sampling, comprising a processor and a storage medium.
[0040] The storage medium is used to store instructions.
[0041] The processor is used to operate according to the instructions to perform the steps of the method according to the first aspect.
[0042] In a third aspect, the present application provides a storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the steps of the method according to the first aspect.
[0043] In a fourth aspect, the present application provides a computer device comprising a processor and a storage medium.
[0044] The storage medium is used to store instructions.
[0045] The processor is used to operate according to the instructions to perform the steps of the method according to the first aspect.
[0046] Compared with the prior art, the present application has the following beneficial effects:
[0047] (1) The present application provides a crack image segmentation method based on a residual network and round-trip sampling. A codec network based on a residual network and round-trip sampling is used to provide a new model for semantic segmentation tasks, which retains the structural advantages of the codec and solves the problem of dilution of deep semantic information.
[0048] (2) The present application introduces a residual network module and a hollow spatial pyramid pooling to solve the problem of gradient disappearance and improve the performance of the network. At the same time, the ability of the network to process context information is improved.
[0049] The present application uses an optimized feature fusion unit and an improved cross-entropy loss function to improve the final crack detection effect, especially the crack edge segmentation effect. BRIEF DESCRIPTION OF DRAWINGS
[0050] Figure 1 is a method flowchart of an embodiment of the present application.
[0051] Figure 2 is a codec network structure diagram of an embodiment of the present application.
[0052] Figure 3 is a ResBlock residual network module structure diagram in an embodiment of the present application.
[0053] Figure 4 This is an optimized feature fusion unit structure in the embodiments of the present invention. Detailed Implementation
[0054] The present invention will be further described below with reference to the accompanying drawings and embodiments. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention, and should not be used to limit the scope of protection of the present invention.
[0055] In the description of this invention, "several" means one or more, "multiple" means two or more, "greater than," "less than," and "exceeding" are understood to exclude the stated number, while "above," "below," and "within" are understood to include the stated number. The use of "first" and "second" in the description is merely for distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features.
[0056] In the description of this invention, the terms "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.
[0057] Example 1
[0058] like Figure 1 As shown, a crack image segmentation method based on residual networks and round-trip sampling includes:
[0059] Step S1: Input the image to be segmented into the encoder network to obtain the deepest feature map F1;
[0060] The encoder network comprises, in sequence, a first group of encoded convolutional operations, a second group of encoded convolutional operations, a third group of encoded convolutional operations, a fourth group of encoded convolutional operations, and a fifth group of encoded convolutional operations connected by max pooling layers.
[0061] The first, second, third, and fourth encoding convolution operation groups each consist of two residual network modules; each residual network module includes a convolutional layer, an activation layer, and a batch normalization layer.
[0062] The fifth group of encoding convolution operations consists of two combined modules, each of which is a combination of a residual network module and a hollow spatial pyramid pooling module;
[0063] Step S2: Input the feature map F1 into the round-trip sampling module to obtain the feature map F2; wherein the round-trip sampling module consists of two convolution operation groups and one downsampling layer, and each convolution operation group consists of one upsampling layer and two convolution layers;
[0064] Step S3: Input the feature map F2 into the decoder network to obtain the feature map F3.
[0065] The decoder network comprises, in sequence, a fifth decoding convolution operation group, a fourth decoding convolution operation group, a third decoding convolution operation group, a second decoding convolution operation group, and a first decoding convolution operation group, all connected by an upsampling layer.
[0066] The first, second, third, and fourth decoding convolutional operation groups are each composed of two residual network modules; each residual network module includes a convolutional layer, an activation layer, and a batch normalization layer.
[0067] The fifth decoding convolution operation group consists of two combined modules, each of which is a combination of a residual network module and a hollow spatial pyramid pooling module;
[0068] Step S4: Input the feature map F3 into the prediction network to obtain the crack segmentation image.
[0069] In some embodiments, such as Figure 2 As shown, step S1 includes:
[0070] The image to be segmented is processed by the first group of coding convolution operations to obtain the first coded feature map;
[0071] The first encoded feature map is passed through a max pooling layer and then input into the second encoded convolution operation group to obtain the second encoded feature map;
[0072] The second encoded feature map is passed through a max pooling layer and then input into the third encoded convolution operation group to obtain the third encoded feature map.
[0073] The third encoded feature map is passed through a max pooling layer and then input into the fourth encoded convolution operation group to obtain the fourth encoded feature map;
[0074] The fourth encoded feature map is passed through a max pooling layer and then input into the fifth encoded convolutional operation group to obtain the feature map F1.
[0075] In some embodiments, such as Figure 2 As shown, step S3 includes:
[0076] The feature map F2 is processed by the fifth decoding convolution operation group to obtain the fifth decoded feature map;
[0077] After upsampling the fifth decoded feature map and fusing it with the fourth encoded feature map, the result is input into the fourth decoded convolution operation group to obtain the fourth decoded feature map.
[0078] After upsampling the fourth decoded feature map and fusing it with the third encoded feature map, the result is input into the third decoded convolution operation group to obtain the third decoded feature map.
[0079] After upsampling the third decoded feature map and fusing it with the second encoded feature map, the result is input into the second decoded convolution operation group to obtain the second decoded feature map.
[0080] After upsampling the second decoded feature map and fusing it with the first encoded feature map, the result is input into the first decoded convolution operation group to obtain feature map F3.
[0081] In some embodiments, such as Figure 3 As shown, each residual network module includes two 3*3 convolutional layers, two activation layers, three batch normalization layers, and one 1*1 convolutional layer;
[0082] The input features of the residual network module are processed sequentially through a first 3x3 convolutional layer, a first batch of normalization layers, a first activation layer, a second 3x3 convolutional layer, and a second batch of normalization layers to obtain the first feature.
[0083] The input features of the residual network module are processed sequentially through a 1*1 convolutional layer and a third batch normalization layer to obtain the second feature;
[0084] After fusing the first and second features, the data is processed through the second activation layer to obtain the output features of the residual network module.
[0085] In some embodiments, fusion is performed using an optimized feature fusion unit, such as... Figure 4 As shown, the feature fusion unit includes a fusion block, a batch normalization layer, and a convolutional layer.
[0086] In some embodiments, in step S4, the prediction network uses the Sigmoid function.
[0087] In some embodiments, a further optimization scheme for a codec network based on residual networks and round-trip sampling replaces the loss function used by the network with an improved cross-entropy loss function.
[0088] The entire codec network uses the cross-entropy loss function L. ce for:
[0089]
[0090] Where y is the true value of the image. γ is the predicted value of the image, β is the weighting coefficient, and γ is the adjustable focusing parameter.
[0091] The present application replaces the convolutional layers in the first four groups of convolutional operations of the encoder network and the decoder network with residual network modules. Each residual network module includes a convolutional layer, an activation layer, and a batch normalization layer.
[0092] The present application replaces the convolutional layers in the fifth group of convolutional operations of the encoder network and the decoder network with a combination of a residual network module and a dilated spatial pyramid pooling module.
[0093] The present application uses an optimized feature fusion unit in place of the fusion unit in the decoding network.
[0094] The present application provides a crack image segmentation method based on a residual network and round-trip sampling. There are many ways to implement this technical solution, and the above description is only the preferred embodiment of the present application. It should be noted that for ordinary technical personnel in this technical field, without departing from the principle of the present application, several improvements and refinements can be made, which should be considered as the protection scope of the present application. The components not explicitly described in the embodiment can be implemented using existing technology.
[0095] Embodiment 2
[0096] In a second aspect, the present embodiment provides a crack image segmentation device based on a residual network and round-trip sampling, comprising a processor and a storage medium.
[0097] The storage medium is used to store instructions.
[0098] The processor is used to operate according to the instructions to perform the steps of the method according to embodiment 1.
[0099] Embodiment 3
[0100] In a third aspect, the present embodiment provides a storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the steps of the method according to embodiment 1.
[0101] Embodiment 4
[0102] In a fourth aspect, the present embodiment provides a computer device comprising a processor and a storage medium.
[0103] The storage medium is used to store instructions.
[0104] The processor is used to operate according to the instructions to perform the steps of the method according to embodiment 1.
[0105] Those skilled in the art will appreciate that embodiments of the application can be devised for a method, a system, or a computer program product. Accordingly, the present application can be embodied in the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer readable program code.
[0106] The present application is described in reference to the flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.
[0107] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.
[0108] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.
[0109] The above description is only preferred embodiments of the application. It should be pointed out that for those skilled in the art, some improvements and refinements can be made without departing from the principles of the application, and these improvements and refinements should be considered as falling within the scope of the application.
Claims
1. A crack image segmentation method based on residual network and round trip sampling, characterized in that, The method comprises the following steps: Step S1, inputting an image to be segmented into an encoder network to obtain a feature map F1 of the deepest layer; The encoder network comprises, in sequence, a first encoding convolution operation group, a second encoding convolution operation group, a third encoding convolution operation group, a fourth encoding convolution operation group and a fifth encoding convolution operation group connected by a maximum pooling layer; the first encoding convolution operation group, the second encoding convolution operation group, the third encoding convolution operation group and the fourth encoding convolution operation group each comprise two residual network modules; each residual network module comprises a convolution layer, an activation layer and a batch normalization layer; the fifth encoding convolution operation group comprises two combination modules, and each combination module is a combination of a residual network module and an atrous spatial pyramid pooling module; Step S2, inputting the feature map F1 into a round-trip sampling module to obtain a feature map F2; The round-trip sampling module comprises two convolution operation groups and a down-sampling layer, wherein each convolution operation group comprises an up-sampling layer and two convolution layers; Step S3, inputting the feature map F2 into a decoder network to obtain a feature map F3, The decoder network comprises, in sequence, a fifth decoding convolution operation group, a fourth decoding convolution operation group, a third decoding convolution operation group, a second decoding convolution operation group and a first decoding convolution operation group connected by an up-sampling layer; the first decoding convolution operation group, the second decoding convolution operation group, the third decoding convolution operation group and the fourth decoding convolution operation group each comprise two residual network modules; each residual network module comprises a convolution layer, an activation layer and a batch normalization layer; the fifth decoding convolution operation group comprises two combination modules, and each combination module is a combination of a residual network module and an atrous spatial pyramid pooling module; Step S4, inputting the feature map F3 into a prediction network to obtain a crack segmentation image. 2.The fracture image segmentation method based on the residual network and round trip sampling according to claim 1, wherein, Step S1 comprises: The image to be segmented is subjected to the first encoding convolution operation group to obtain a first encoding feature map; The first encoding feature map is input into the second encoding convolution operation group after a maximum pooling layer to obtain a second encoding feature map; The second encoding feature map is input into the third encoding convolution operation group after a maximum pooling layer to obtain a third encoding feature map; The third encoding feature map is input into the fourth encoding convolution operation group after a maximum pooling layer to obtain a fourth encoding feature map; The fourth encoding feature map is input into the fifth encoding convolution operation group after a maximum pooling layer to obtain the feature map F1. 3.The fracture image segmentation method based on the residual network and round trip sampling according to claim 2, wherein, Step S3 comprises: The feature map F2 is subjected to the fifth decoding convolution operation group to obtain a fifth decoding feature map; The fifth decoding feature map is up-sampled and fused with the fourth encoding feature map, and then input into the fourth decoding convolution operation group to obtain a fourth decoding feature map; The fourth decoding feature map is up-sampled and fused with the third encoding feature map, and then input into the third decoding convolution operation group to obtain a third decoding feature map; The third decoding feature map is up-sampled and fused with the second encoding feature map, and then input into the second decoding convolution operation group to obtain a second decoding feature map; The second decoding feature map is up-sampled and fused with the first encoding feature map, and then input into the first decoding convolution operation group to obtain the feature map F3.
4. The fracture image segmentation method based on residual network and round trip sampling according to claim 3, characterized in that, The fusion is performed using an optimized feature fusion unit, which includes a fusion block, a batch normalization layer and a convolution layer. 5.The fracture image segmentation method based on the residual network and round trip sampling according to claim 1, wherein, In step S4, the prediction network uses a Sigmoid function. 6.The fracture image segmentation method based on the residual network and round trip sampling according to claim 1, wherein, Each residual network module includes two 3*3 convolution layers, two activation layers, three batch normalization layers and a 1*1 convolution layer; The input features of the residual network module are sequentially processed by a first 3*3 convolution layer, a first batch normalization layer, a first activation layer, a second 3*3 convolution layer and a second batch normalization layer to obtain a first feature, The input features of the residual network module are sequentially processed by a 1*1 convolution layer and a third batch normalization layer to obtain a second feature; The first feature and the second feature are fused, and then processed by a second activation layer to obtain the output features of the residual network module. 7.The fracture image segmentation method based on the residual network and round trip sampling according to claim 1, wherein, The entire codec network employs a cross-entropy loss function L ce : where y is the true value of the image, is the predicted value of the image, β is a weight coefficient, and γ is an adjustable focus parameter.
8. A crack image segmentation device based on a residual network and round trip sampling, characterized by, comprising a processor and a storage medium; the storage medium is configured to store instructions; the processor is configured to operate according to the instructions to perform the steps of the method according to any one of claims 1 to 7.
9. A storage medium having stored thereon a computer program, characterized in that the computer program, when executed by the processor, implements the steps of the method according to any one of claims 1 to 7.
10. A computer device, comprising: comprising a processor and a storage medium; the storage medium is configured to store instructions; the processor is configured to operate according to the instructions to perform the steps of the method according to any one of claims 1 to 7.