An image segmentation method based on longitudinal stack reparameterization structure model
By employing a vertically stacked multi-3×3 convolutional structure in the image segmentation model, the problems of large memory consumption and simple feature fusion in existing technologies are solved, achieving higher segmentation accuracy and inference speed.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NANJING TECH UNIV
- Filing Date
- 2025-03-11
- Publication Date
- 2026-04-24
AI Technical Summary
Existing image segmentation models suffer from problems such as large memory consumption and simple feature fusion methods due to their multi-branch structure, resulting in discontinuous edge detail segmentation and insufficient accuracy in small target segmentation.
The existing 3×3 convolution is replaced by a vertically stacked multi 3×3 convolution structure. The vertical expansion is performed during the training phase, and the parameterization is reparameterized into a single 3×3 convolution during the inference phase. The Unet, SegNet and DeepLab series models are used for image segmentation.
It improves the segmentation accuracy of small targets and weak edge regions, significantly speeds up inference, and reduces the consumption of computing resources.
Smart Images
Figure CN120182601B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer vision technology, specifically relating to an image segmentation method based on a reparameterized model with a (multiple 3x3 convolution) vertical stacking structure. Background Technology
[0002] Image segmentation is the technique and process of dividing an image into several specific regions with unique properties and identifying targets of interest. It classifies pixels belonging to the same category, thereby understanding the scene and content of the image. It has applications in many fields, such as identifying roads, vehicles, and pedestrians in autonomous driving; identifying organs and tissues in CT scans and MRI images in medical image analysis; identifying terrain, vegetation, and buildings in satellite and aerial imagery in remote sensing image processing; and identifying abnormal behavior and tracking targets in video surveillance for security monitoring.
[0003] Image segmentation models allow us to classify pixels or regions in an image based on their features, dividing the image into multiple meaningful regions or objects to better understand and analyze its content. Image segmentation models play a crucial role in computer vision and image processing.
[0004] Structural reparameterization refers to the process of transforming and rearranging parameters of a neural network structure during training and inference. This converts a model with one structural form into another to improve model performance, reduce computational complexity, or increase inference efficiency. The core idea is to represent complex network structures differently during training and inference: a more complex and flexible structure is used during training to help the model learn features better and capture complex patterns in the data; while during inference, it is reparameterized into a simpler and more efficient structure to speed up inference and reduce memory usage. A common method is to change the network topology. For example, some branch structures that are multi-branch connections during training can be reparameterized into a single-path structure during inference. For instance, in ResNet, residual connections have a main path and residual paths during training. During inference, the parameters of the residual connections can be merged into the main path to form a simpler linear structure, improving inference speed.
[0005] The current mainstream methods are all horizontal expansion (for example: RepVGG[1] expands the feature representation ability during training through multi-branch structure, and DBB[2] enhances the model capacity through diversified branch combinations). The main problems of this type of method are: (1) the multi-branch structure leads to a large memory occupation during training, which limits the expansion scale of the model; (2) the feature fusion method between branches is relatively simple, and it is difficult to fully explore the complementary information between features. When applied to image segmentation, the negative impact on the image segmentation effect is mainly reflected in: (1) segmentation discontinuity is easy to occur in key areas such as edge details; (2) the segmentation accuracy of small targets or weak edge areas is insufficient. Summary of the Invention
[0006] To address the aforementioned issues, this invention proposes an image segmentation method based on a vertically stacked reparameterized structure model. This method employs a vertically expanded structure for the reparameterized model during image segmentation. Specifically, the steps of this method include: 1) acquiring the image to be processed; 2) using the image to be processed as input to the image segmentation model, with the output of the image segmentation model being the segmented image.
[0007] The image segmentation model is a reparameterized image segmentation model based on a vertical stacking structure; in the network structure of the image segmentation model, all 3×3 convolutional modules are replaced with multiple 3×3 convolutional vertical stacking structures;
[0008] (I) During the training phase of the image segmentation model, a multi-3×3 convolutional vertical stacking structure replaces the 3×3 convolution in the image segmentation model; let the input data dimensions H×W×Cin be the height, width, and number of input channels of the original image or image features, respectively, and the output data dimensions H×W×Cout be the height, width, and number of output channels of the output image or image features, respectively. Then, in the multi-3×3 convolutional vertical stacking structure, the operation process on the input data is as follows:
[0009] S1. After upsampling, 2H×2W×Cin data is obtained;
[0010] S2. After passing through one or more 3×3 convolutional layers, the number of channels changes to obtain 2H×2W×Cin;
[0011] S3. Use a 3×3 convolution with a stride of 2 to convert the resolution and number of channels to H×W×Cout and output the result.
[0012] (ii) In the inference stage, the multiple 3×3 convolution vertical stacking structure is changed to a single 3×3 convolution to perform inference on the input data.
[0013] The operation process in the training phase of the image segmentation model also includes S4, enabling skip connections: if Cout = Cin, then the processing result of step S3) is directly added to the input and used as the final output.
[0014] In the training phase of the image segmentation model, the upsampling in step S1 can be achieved using bilinear interpolation.
[0015] The image segmentation model is either the Unet network model or the SegNet network model; it is also applicable to the DeepLab series models, including DeepLabV3, DeepLabV3+, PSPNet, etc.
[0016] The reparameterization method of the multiple 3x3 convolutional vertical stacking structure of the present invention can be applied to various existing segmentation models to replace the basic 3×3 convolutional structure. During the training phase, the present invention uses upsampling layers and multiple 3×3 convolutional vertical stacking structures to replace the original simple convolution, which helps the model learn features better and capture complex patterns in the data. During the inference phase, it is reparameterized into a single 3×3 convolution to accelerate inference speed and reduce computational resource consumption.
[0017] Compared with horizontal expansion, the advantages of vertical expansion in this invention are mainly: (1) hierarchical feature extraction can better capture multi-scale information; (2) high degree of parameter sharing and less memory usage.
[0018] When applied to image segmentation, the positive effects on image segmentation are mainly: (1) improved segmentation accuracy for small targets and weak edge regions; (2) significantly improved inference speed while ensuring accuracy, which is beneficial for practical application deployment. Attached Figure Description
[0019] Figure 1(a) shows the structure of the model training phase of the present invention;
[0020] Figure 1(b) shows the structure of the reasoning stage of the model of the present invention;
[0021] Figure 2(a) shows a typical Unet network structure;
[0022] Figure 2(b) shows the Unet network structure based on the model of the embodiment of the present invention. Detailed Implementation
[0023] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.
[0024] An image segmentation method based on a vertically stacked reparameterized structure model is disclosed. This method employs a vertically expanded reparameterized model for image segmentation. Specifically, the steps include: 1) acquiring the image to be processed; 2) using the image to be processed as input to the image segmentation model, with the output of the image segmentation model being the segmented image; the image segmentation model is a vertically stacked reparameterized image segmentation model; in the network structure of the image segmentation model, all 3×3 convolutional modules are replaced with multiple 3×3 convolutional vertically stacked structures.
[0025] The design models of the multiple 3×3 convolutional vertical stacking structure are shown in Figure 1(a) and Figure 1(b).
[0026] Figure 1(a) shows the multi-3×3 convolution vertical stacking structure used in the model training stage, which is used to replace the 3×3 convolution in various existing image segmentation models. The input data dimensions H×W×Cin are the height, width and number of input channels of the original image or image features, respectively.
[0027] The image resolution is increased by the upsampling module (which can be achieved using bilinear interpolation), resulting in 2H×2W×Cin data.
[0028] After passing through one or more 3×3 convolutional layers, the number of channels changes to 2H×2W×Cin; finally, a 3×3 convolution with a stride of 2 is used to change the resolution and number of channels to H×W×Cout. The black dashed arrows indicate skip connections, which are optional and allow the processed result to be directly added to the input (requiring Cout = Cin). This multi-3×3 convolutional vertical stacking structure has many parameters and flexible configuration, which helps the model better learn features and capture complex patterns in the data.
[0029] Second, during the inference phase, the entire structure can be reparameterized into a single 3x3 convolution as shown in Figure 1(b) to speed up inference, reduce memory usage, and maintain the computational complexity of the original model.
[0030] The specific implementation process of the module reparameterization method in PyTorch is as follows:
[0031] 1. Combine the weights and biases of the 3×3 convolution conv1 and the 3×3 convolution conv2 to obtain w0 and b0.
[0032] First, perform a permute(1,0,2,3) operation on the weights w1 of the 3×3 convolution conv1 to adjust their dimension order.
[0033] Then, the processed w1 is flipped using the flip([2,3]) operation to flip its last two dimensions.
[0034] Using the processed w1 as the convolution kernel, we perform a convolution operation on the weights w2 of the 3×3 convolution conv2 to obtain the weights w0.
[0035] Multiply the bias b1 of the 3×3 convolution conv1 with w2 and sum them along the [1,2,3] dimension, then add b2 as the weight b0.
[0036] 2. The merged weights w0 are subjected to matrix multiplication and reshaping to achieve downsampling.
[0037] First, w0 is reshaped from its original shape to the shape (w_cout, w_cin, matrix_shape[0]), where matrix_shape[0] is the number of rows in the transformation matrix (i.e., 25). This reshaping operation is to make the shape of w0 match the dimensions of the transformation matrix so that matrix multiplication can be performed. (Reference)
[0038] Then, the built-in torch.matmul function of PyTorch is used to perform matrix multiplication on the reshaped w0 and the transformation matrix, and the result is reshaped again into the shape (w_cout, w_cin, 3, 3) to obtain a new w0.
[0039] 3. Merge jump connections (if jump connections are enabled)
[0040] Generate an identity mapping convolution kernel weight w3 with only the center position of the convolution kernel set to 1 and the rest set to 0, and a convolution kernel bias b3 set to 0. Use the method in step 1 to merge it with w0 and b0 to obtain new w0 and b0.
[0041] 4. Output w0 and b0 as the weights and bias parameters for merging 3×3 convolutions during the inference phase.
[0042] Taking the Unets in Figures 2(a) and 2(b) as examples, Figure 2(a) shows a typical Unet, where the 3×3 convolutions that can be replaced with the multi-3×3 convolution vertical stacking structure proposed in this invention are marked. Figure 2(b) shows the Unet replaced with the multi-3×3 convolution vertical stacking structure proposed in this invention. It can be seen that the method of this invention is simple and easy to implement, does not bring about significant changes to the original model, and is particularly convenient in programming implementation, avoiding the potential risks brought about by large-scale model modifications.
[0043] The specific implementation process of the segmentation method:
[0044] 1. Prepare the training dataset X = {x m ,z m} m m-1 Test dataset Y = {y n ,z n}n n-1 x and y are images, and z is the segmentation label.
[0045] 2. Construct the Unet network and replace all 3×3 convolutional modules with the multi-3x3 convolutional vertical stacking structure proposed in this invention.
[0046] 3. Train the network using the training dataset to obtain the trained model M.
[0047] 4. The structural reparameterization method is used to reparameterize the multiple 3×3 convolutional vertical stacking structure in the trained M to obtain the reparameterized model Mc, which has the same structure as the original Unet model.
[0048] 5. Use Mc to test the test dataset and obtain the image segmentation results.
[0049] Appendix: Transformation Matrix (25×9)
[0050]
[0051] References:
[0052] [1]Ding X, Zhang
[0053] [2]Ding X, Zhang
Claims
1. An image segmentation method based on a vertically stacked reparameterized structure model. This method employs a vertically expanded structure for the reparameterized model during image segmentation. Specifically, the steps of this method include: First, the image is processed at the acquisition point; then, the image to be processed is used as the input of the image segmentation model, and the output of the image segmentation model is the segmented image. Its characteristic is that the image segmentation model is a reparameterized image segmentation model based on a vertical stacking structure; in the network structure of the image segmentation model, all 3×3 convolutional modules are replaced with multiple 3×3 convolutional vertical stacking structures; (i) During the training phase of the image segmentation model, a multi-3×3 convolutional vertical stacking structure replaces the 3×3 convolution in the image segmentation model; let the input data dimensions H×W×Cin be the height, width, and number of input channels of the original image or image features, respectively, and the output data dimensions H×W×Cout be the height, width, and number of output channels of the output image or image features, respectively. Then, in the multi-3×3 convolutional vertical stacking structure, the operation process on the input data is as follows: S1. After upsampling, 2H×2W×Cin data is obtained; S2. After passing through one or more 3×3 convolutional layers, the number of channels changes to obtain 2H×2W×Cin; S3. Use a 3×3 convolution with a stride of 2 to convert the resolution and number of channels to H×W×Cout and output the result. (ii) During the inference stage, the multiple 3×3 convolution vertical stacking structure is changed to a single 3×3 convolution before inference is performed on the input data; The implementation method for model reparameterization is as follows: 1) Merge the weights and biases of the 3×3 convolution conv1 and 3×3 convolution conv2 respectively to obtain the weights w0 and biases b0. The steps include: 1.1) Perform the permute(1, 0, 2, 3) operation on the weights w1 of the 3×3 convolution conv1 to adjust the order of w1 dimensions; perform the flip([2, 3]) operation on the processed w1 to flip the last two dimensions of w1; 1.2) Using w1 processed in step 1.1) as the convolution kernel, perform a convolution operation on the weights w2 of the 3×3 convolution conv2 to obtain the weights w0; 1.3) Multiply the bias b1 of the 3×3 convolution conv1 with w2 and sum them along the [1, 2, 3] dimension, then add the bias b2 of the 3×3 convolution conv2 as the bias b0; 2) The merged weights w0 are subjected to matrix multiplication and reshaping to achieve downsampling. The steps include: 2.1) Reshape w0 from its original shape to the shape of (w_cout, w_cin, matrix_shape[0]); where matrix_shape[0] is the number of rows of the transformation matrix; so that the shape of w0 matches the dimension of the transformation matrix in order to perform matrix multiplication; 2.2) Use the torch.matmul function of PyTorch to perform matrix multiplication on w0 obtained in step 2.1) and the transformation matrix, and reshape the result into the shape (w_cout, w_cin, 3, 3) to obtain a new w0; Finally, w0 and b0 are used as the weights and bias parameters of the 3×3 convolution during the inference phase, respectively.
2. The image segmentation method based on a vertically stacked reparameterized structure model according to claim 1, characterized in that: The operation process in the training phase of the image segmentation model also includes S4, enabling skip connections: if Cout = Cin, then the processing result of step S3) is directly added to the input and used as the final output.
3. The image segmentation method based on a vertically stacked reparameterized structure model according to claim 2, characterized in that the model... The implementation method of reparameterization includes step 3) after step 2.2), which involves merging skip connections: Generate an identity mapping convolution kernel weight w3 with only the center position of the convolution kernel set to 1 and the rest set to 0, and a convolution kernel bias b3 set to 0; use the method in step 1) to merge w3 and b3 with w0 and b0 respectively to obtain new w0 and b0. Finally, the new w0 and b0 are used as the weights and bias parameters for merging 3×3 convolutions during the inference phase, respectively.
4. The image segmentation method based on a vertically stacked reparameterized structure model according to claim 1 or 2, characterized in that: In the training phase of the image segmentation model, the upsampling in step S1 is achieved using bilinear interpolation.
5. The image segmentation method based on a vertically stacked reparameterized structure model according to claim 1 or 2, characterized in that: The image segmentation model is either the Unet network model or the SegNet network model.
6. The image segmentation method based on a vertically stacked reparameterized structure model according to claim 1 or 2, characterized in that: The image segmentation model is a DeepLabV3, DeepLabV3+, or PSPNet network model.
Citation Information
Patent Citations
Method for segmenting vasa sanguinea retinae image
CN106651846A
Polyp segmentation algorithm based on re-parameterization and convolution block attention
CN117994273A