Training method of image segmentation model, image segmentation method and related device
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- INST OF AUTOMATION CHINESE ACAD OF SCI
- Filing Date
- 2026-01-20
- Publication Date
- 2026-06-05
AI Technical Summary
In existing technologies, image segmentation models based on the ViT architecture have high computational overhead and are prone to overfitting when performing full parameter fine-tuning. When using adapter fine-tuning, the image feature representation capability is limited, resulting in low text segmentation accuracy and poor model generalization ability.
The feature extraction module is constructed from at least two linear layers and the GELU activation function. It combines depthwise separable convolution and dual-path attention mechanism, fine-tunes parameters through internal and external adapters, generates high-resolution features, and calculates loss values to update model parameters.
It significantly reduces training costs, improves text segmentation accuracy and model generalization ability, and can effectively segment text regions in complex scenarios.
Smart Images
Figure CN122156846A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision technology, and in particular to a training method for an image segmentation model, an image segmentation method, and related apparatus. Background Technology
[0002] Scene text segmentation technology plays an important role in applications such as document analysis, scene understanding, and text removal by accurately detecting and segmenting text regions from natural images.
[0003] In recent years, the Transformer model has achieved great success in the field of natural language processing. Its ideas have been introduced into the field of computer vision, giving rise to architectures such as Vision Transformer (ViT). ViT performs well in tasks such as image classification, segmentation and object detection by modeling global context relationships.
[0004] In related technologies, zero-shot segmentation of arbitrary objects has been achieved by large-scale pre-training and prompt-driven mechanisms of the Segment Anything Model (SAM) based on the ViT architecture. However, directly fine-tuning all parameters of large-scale models such as SAM is computationally expensive and prone to overfitting. Existing technologies also use adapters to fine-tune some parameters of the model. However, traditional adapters mostly use fully connected layers, which have limited ability to model spatial features in visual tasks, resulting in limited representation ability of extracted image features, which in turn leads to low text segmentation accuracy and poor model generalization ability. Summary of the Invention
[0005] This invention provides a training method for an image segmentation model, an image segmentation method, and related apparatus to address the shortcomings of existing technologies that require fine-tuning all parameters when using SAM for image segmentation, resulting in high computational overhead and a tendency to overfit. Furthermore, when using an adapter to fine-tune some parameters, the ability to represent image features is limited, leading to low text segmentation accuracy and poor model generalization ability.
[0006] This invention provides a method for training an image segmentation model, comprising: The sample image is input into the feature extraction module of the deep learning model to obtain the first image feature; wherein, the feature extraction module is constructed by residual connection of at least two linear layers and GELU activation function; The first decoder based on the deep learning model performs depthwise separable convolution and convolution calculations on the first image features respectively, and aggregates the corresponding calculation results through a dual-path attention mechanism to obtain the decoded features; The second decoder based on the deep learning model performs upsampling and high-resolution feature refinement on the decoded features in sequence to obtain high-resolution features. The high-resolution features are then fused with the self-attention features corresponding to the decoded features to obtain the target mask features. The loss value is calculated based on the target mask features and the loss function of the deep learning model, and the parameters of the feature extraction module are updated based on the loss value. Under the condition that the iteration conditions of the deep learning model are met, an image segmentation model is obtained.
[0007] According to a training method for an image segmentation model provided by the present invention, the feature extraction module includes an internal adapter and an external adapter, wherein the internal adapter is located inside the Transformer block of the deep learning model, and the external adapter is located outside the Transformer block of the deep learning model; The internal adapter includes two linear layers and a GELU activation function, wherein the two linear layers are connected by the GELU activation function. The external adapter includes a first MLP structure and a second MLP structure. The first MLP structure includes four linear layers and four GELU activation functions, with each pair of linear layers connected by a GELU activation function. The second MLP structure includes one linear layer and one GELU activation function.
[0008] According to the training method of the image segmentation model provided by the present invention, the calculation result includes a spatial attention weight map determined by four-level depthwise separable convolution calculation and a second image feature determined by convolution calculation; The process of aggregating the corresponding calculation results through a dual-path attention mechanism to obtain the decoded features includes: The spatial attention weight map and the second image features are reconstructed respectively to obtain the reconstructed spatial attention weight map and the reconstructed second image features. The elements are then multiplied and weighted averaged to obtain the decoded features.
[0009] According to a training method for an image segmentation model provided by the present invention, the step of fusing the high-resolution features with the self-attention feature embeddings corresponding to the decoded features to obtain target mask features includes: The target mask features are calculated using the following formula: ; in, The target mask features, For the self-attention feature embedding, This refers to the high-resolution feature.
[0010] According to a training method for an image segmentation model provided by the present invention, after obtaining the image segmentation model, the method includes: The test data is input into the image segmentation model to obtain the segmentation result; The target evaluation index is calculated based on the segmentation results, and the image segmentation model is evaluated based on the target evaluation index to obtain the evaluation result; The target evaluation index includes at least one of the following: Segmentation accuracy metrics, including at least one of the fgIoU metric and the Fscore metric; Generalization ability is used to represent the robustness of the image segmentation model in deformed text and unseen scenes; Parameter efficiency, which is determined by the number of training parameters and computational overhead; Boundary consistency is used to represent the smoothness and alignment of the segmentation result with the true boundary.
[0011] The present invention also provides an image segmentation method, comprising: Obtain the image to be processed; The image to be processed is input into the image segmentation model to obtain the image segmentation result; wherein the image segmentation model is trained by the image segmentation model training method.
[0012] The present invention also provides a training apparatus for an image segmentation model, comprising: The feature extraction module is used to input the sample image into the feature extraction module of the deep learning model to obtain the first image features; wherein, the feature extraction module is constructed by at least two linear layers and GELU activation with residual connections; The first decoding module is used to perform depthwise separable convolution calculation and convolution calculation on the first image features based on the first decoder of the deep learning model, and aggregate the corresponding calculation results through a dual-path attention mechanism to obtain the decoded features; The second decoding module is used to perform upsampling and high-resolution feature refinement on the decoded features in sequence based on the second decoder of the deep learning model to obtain high-resolution features, and to fuse the high-resolution features with the self-attention features corresponding to the decoded features to obtain target mask features. The training module is used to calculate the loss value based on the target mask features and the loss function of the deep learning model, and update the parameters of the feature extraction module based on the loss value, so as to obtain an image segmentation model when the iteration conditions of the deep learning model are met.
[0013] The present invention also provides an image segmentation apparatus, comprising: The image acquisition module is used to acquire the image to be processed. The image segmentation module is used to input the image to be processed into the image segmentation model to obtain the image segmentation result; wherein the image segmentation model is trained by the image segmentation model training method.
[0014] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement a training method for an image segmentation model or an image segmentation method as described above.
[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a training method for an image segmentation model or an image segmentation method as described above.
[0016] The image segmentation model training method, image segmentation method, and related apparatus provided by this invention involve inputting sample images into the feature extraction module of a deep learning model to obtain first image features. Based on a first decoder, depthwise separable convolution and convolution calculations are performed on the first image features, and the corresponding calculation results are aggregated through a dual-path attention mechanism to obtain decoded features. Based on a second decoder, the decoded features are sequentially upsampled and refined to obtain high-resolution features. The high-resolution features are then fused with the self-attention features corresponding to the decoded features to obtain target mask features. Finally, a loss value is calculated based on the target mask features and the loss function of the deep learning model, and the parameters of the feature extraction module are updated based on the loss value to obtain the final image segmentation model. This method requires only minor parameter adjustments to achieve efficient adaptation to large-scale pre-trained models, significantly reducing training costs. Simultaneously, it enables progressive refinement of text regions, improving text segmentation accuracy and the generalization ability of the image segmentation model. Attached Figure Description
[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0018] Figure 1 This is a flowchart illustrating the training method for the image segmentation model provided by the present invention.
[0019] Figure 2 This is a schematic diagram of the working mechanism of the second decoder provided by the present invention.
[0020] Figure 3 This is a schematic diagram of the working mechanism of the external adapter provided by the present invention.
[0021] Figure 4 This is a schematic diagram of the working mechanism of the first decoder provided by the present invention.
[0022] Figure 5 This is a schematic diagram of the working mechanism of the feature extraction module provided by the present invention.
[0023] Figure 6 This is a flowchart illustrating the image segmentation method provided by the present invention.
[0024] Figure 7 This is a schematic diagram of the structure of the training device for the image segmentation model provided by the present invention.
[0025] Figure 8 This is a schematic diagram of the image segmentation device provided by the present invention.
[0026] Figure 9 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation
[0027] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.
[0028] The following is combined Figures 1-8 The present invention describes the training method, image segmentation method, and related apparatus for the image segmentation model.
[0029] Figure 1 This is a flowchart illustrating the training method of the image segmentation model provided by the present invention, as shown below. Figure 1 As shown, the method includes the following: Step 110: Input the sample image into the feature extraction module of the deep learning model to obtain the first image features; wherein, the feature extraction module is constructed by residual connection of at least two linear layers and the GELU activation function.
[0030] In this step, the sample image is the raw visual data input into the scene text segmentation system; the sample image can be a common digital image format, such as JPG (JPEG), PNG, BMP or TIFF, etc., and is not limited to a specific compression standard or encoding method.
[0031] In this embodiment, the sample images may come from publicly available datasets built for training and evaluating model performance. These datasets typically contain manually annotated scene text images. The sample images may also be video frames or still images acquired in real time by a camera, scanner, or other image acquisition device.
[0032] In this step, the feature extraction module can be an adapter in a modified Vision Transformer (ViT) based backbone network. ).
[0033] In this embodiment, the adapter can be embedded in different positions (inside and / or outside) of the Transformer block. In subsequent training, efficient fine-tuning can be achieved by iteratively updating the adapter parameters without changing the original parameters of the backbone network (frozen state), that is, while keeping the original encoder parameters frozen.
[0034] In this embodiment, the feature extraction module includes a downsampling linear layer, a GELU activation function, and an upsampling linear layer. The input and output are added through skip connections to prevent gradient vanishing and preserve the original information, thereby converting general image visual features into enhanced feature representations more suitable for scene text segmentation tasks, namely the first image features.
[0035] Step 120: The first decoder based on the deep learning model performs depthwise separable convolution and convolution calculations on the first image features respectively, and aggregates the corresponding calculation results through a dual-path attention mechanism to obtain the decoded features.
[0036] In this step, the first decoder achieves efficient feature perception through a dual-path attention mechanism.
[0037] In this step, the first decoder includes a lightweight depthwise separable convolution (DS convolution) layer, convolutional layer enhancement, and dual-path attention. The lightweight depthwise separable convolution (DS convolution) and convolutional enhancement together constitute a convolutional feature enhancement block, while the dual-path attention mechanism coordinates alternating self-attention and cross-attention for Transformer decoding. Through the synergistic effect of these components, efficient feature perception is achieved.
[0038] In this embodiment, spatial attention weights are generated by depthwise separable convolution to extract sparse embeddings from image features. Local texture enhancement is performed on the image embeddings through convolution. Finally, a dual-path attention mechanism is used to initially filter and fuse the features, extracting sparse features with long dependencies and focusing on text regions, i.e., decoding features.
[0039] Specifically, the sample image can be a billboard image. After extracting the corresponding first image features, a spatial attention weight map is generated based on the multi-level cascaded depthwise separable convolution (DS-Conv) of the first decoder. This map highlights the text area on the billboard, suppresses background noise, and calculates the sparse embedding. At the same time, the features such as the stroke edges and texture details of the text in the first image features are enhanced based on the first decoder convolutional layer (such as 3x3 convolution). Then, the features output by the two convolutional layers enter the dual-path attention module. The cross-attention mechanism is used to interact these sparse text semantic information with the dense image feature sequence. Finally, the decoded features output by aggregation contain both global text structure information and local visual details.
[0040] The first decoder in this embodiment uses depthwise separable convolution to effectively reduce computational complexity. At the same time, it utilizes spatial attention mechanism to achieve feature sparsity and focus on the foreground text region. Combined with dual-path attention mechanism, the model can simultaneously model long-range dependencies within the text (such as long text lines) and the contextual relationship between the text and the background, significantly improving the ability to perceive text in complex scenes (such as curved or occluded text).
[0041] Step 130: The second decoder based on the deep learning model performs upsampling and high-resolution feature refinement on the decoded features in sequence to obtain high-resolution features. The high-resolution features are then fused with the self-attention features corresponding to the decoded features to obtain the target mask features.
[0042] In this step, the second decoder optimizes the mask features by utilizing feature representation and high-resolution upsampling, thereby achieving fine-grained capture of the mask features of the text region.
[0043] In this step, the decoded features are upsampled through multiple transposed convolutions to gradually restore the spatial resolution of the feature map.
[0044] In this step, after upsampling, the features are further smoothed and the boundary consistency is enhanced through multiple convolutions to obtain high-resolution features.
[0045] In this step, the high-resolution features are embedded with the Transformer output after MLP transformation and then performed element-wise operations (such as the Hadamard product) to map the semantic features back to the original image size, thereby generating a fine pixel-level segmentation mask.
[0046] In this embodiment, during the upsampling process, starting from an initial mask resolution of 256×256, the features are progressively upsampled to 2048×2048 through five transposed convolutional layers. Each transposed convolutional layer has a 2×2 convolutional kernel with a stride of 2, followed by LayerNorm and an activation function. This process ensures that the features capture local details while preserving global contextual information, which is crucial for dense segmentation tasks such as scene text segmentation. The upsampling process can be represented by the following formula: ; in, For input features, These are the features after upsampling.
[0047] In this embodiment, after the upsampling path, the second decoder further enhances the fine-grained feature representation through five additional convolutional layers. Each convolutional layer has a 3×3 kernel, a stride of 1, and padding of 1 to optimize the features. These convolutional layers improve the spatial consistency and detail preservation of high-resolution features. This optimization process can be represented as follows: ; in, This is the optimized high-resolution feature.
[0048] Figure 2 This is a schematic diagram of the working mechanism of the second decoder provided by the present invention. Figure 2 In the illustrated embodiment, the sparse embedding obtained from the first decoder is first processed by the SAM decoder, and then further processed by the Transformer. The image embedding is fused with the sparse embedding to capture spatial context and semantic information. Specifically, this is achieved through positional encoding using five Transconv Blocks, providing additional spatial information for the image embedding and ensuring spatial consistency in the self-attention computation. Each transconv Block includes a Transconv layer, a Norm layer, and a GELU activation function. The Transformer output... After optimization by MLP blocks, the generated... Dimension-matched optimized embeddings and high-resolution features By combining these, the final high-resolution mask logical value is obtained, represented as Mask 2048×2048×1, which is the target mask feature.
[0049] Specifically, the self-attention features corresponding to the high-resolution features and the decoded features are fused to obtain the target mask features, which are calculated using the following formula: ; in, For target mask features, For self-attention feature embedding, It is a high-resolution feature.
[0050] In this embodiment, the second decoder achieves high-quality segmentation of text regions by refining features through high-resolution upsampling and convolution, and fusing MLP output.
[0051] Step 140: Calculate the loss value based on the target mask features and the loss function of the deep learning model, and update the parameters of the feature extraction module based on the loss value. Under the condition of satisfying the iteration conditions of the deep learning model, the image segmentation model is obtained.
[0052] In this step, the loss function is used to measure the difference between the model's prediction and the true label. This loss function can be a function suitable for segmentation tasks, such as binary cross-entropy loss (BCE Loss) or Dice Loss.
[0053] In this embodiment, the trainable parameters set by the feature extraction module are adjusted based on the calculated loss gradient using the backpropagation algorithm, while other parameters of the deep learning model remain frozen.
[0054] In this embodiment, the iteration conditions include reaching a preset number of training rounds or the loss value converging to a specific threshold, which can be set by the user according to actual needs.
[0055] In a feasible implementation, assuming training is performed using the Total-Text dataset, for an input sample image containing curved text, step 130 outputs the corresponding target mask features (prediction map), where the value of each pixel is between 0 and 1, representing the probability that the point belongs to the text. The difference between the prediction map and the corresponding ground truth label (manually annotated pixel-level binary mask) of the sample image is then calculated to obtain the loss value. Next, the optimizer (e.g., AdamW) calculates the gradient based on this loss value. During backpropagation, the gradient flows through the entire network, but only changes the relevant parameters in the feature extraction module (e.g., the Adapter), while other parameter matrices in the ViT backbone network remain unchanged. Training is set to 70 epochs on the Total-Text dataset. When these 70 epochs are completed, or the fgIOU metric on the validation set no longer improves, the model satisfies the iteration condition. At this point, the trained Adapter parameters and the frozen backbone parameters together constitute the final image segmentation model.
[0056] The image segmentation model training method provided in this invention involves inputting sample images into the feature extraction module of a deep learning model to obtain first image features. Based on a first decoder, depthwise separable convolution and convolution calculations are performed on the first image features, and the corresponding calculation results are aggregated through a dual-path attention mechanism to obtain decoded features. Based on a second decoder, the decoded features are sequentially upsampled and refined to high resolution. The high-resolution features are then fused with the self-attention features corresponding to the decoded features to obtain target mask features. Finally, a loss value is calculated based on the target mask features and the loss function of the deep learning model, and the parameters of the feature extraction module are updated based on the loss value to obtain the final image segmentation model. This method requires only minor parameter adjustments to achieve efficient adaptation to large-scale pre-trained models, significantly reducing training costs. Simultaneously, it enables progressive refinement of text regions, improving text segmentation accuracy and the generalization ability of the image segmentation model.
[0057] In some embodiments, the feature extraction module includes an internal adapter and an external adapter. The internal adapter is located inside the Transformer block of the deep learning model, and the external adapter is located outside the Transformer block of the deep learning model. The internal adapter includes two linear layers and a GELU activation function, with the two linear layers connected by a skip connection through the GELU activation function. The external adapter includes a first MLP structure and a second MLP structure. The first MLP structure includes four linear layers and four GELU activation functions, with each pair of linear layers connected by a GELU activation function. The second MLP structure includes one linear layer and one GELU activation function.
[0058] In this embodiment, the adapter can be embedded inside the Transformer block to fine-tune and adapt the input features without changing the original parameters (frozen state) of the backbone network; the adapter can also be embedded outside the Transformer block, using a deep structure of multi-layer linear transformation and GELU activation to further enhance the nonlinear expressive power of the features.
[0059] Specifically, the deep learning model is the Vision Transformer (ViT) network. MLP-Adapter modules (i.e., feature extraction modules) are inserted into each layer of the ViT network's backbone. The MLP-Adapter includes... (Internal adapter) and There are two types: (external adapters) Two linear layers are used with GELU activation and residual connections. Depend on (Four linear layers with GELU activation) and It consists of a single parameter-shared linear layer and GELU activation connected in series.
[0060] In this embodiment, the internal adapter (corresponding to) same This can be embedded inside each Transformer block, employing a classic bottleneck structure; the internal adapter consists of two linear layers connected by the GELU activation function, achieving a dimensionality compression ratio (MLP). ratio Set it to 0.25.
[0061] In this embodiment, the sample image is represented as , First, the sample image is projected into a low-dimensional space using a linear layer. (in After being processed by the GELU activation function, the original channel dimensions are restored through another linear layer for stable training. Using a skip connection, its forward process can be formally represented as: ; in and These represent the weight matrices of the downsampling and upsampling linear layers, respectively.
[0062] Figure 3 This is a schematic diagram of the working mechanism of the external adapter provided by the present invention. Figure 3 In the illustrated embodiment, the external adapter (corresponding to) same This is then inserted outside the Transformer block, employing a deeper MLP structure; the external adapter is... and It consists of two sub-modules, in which It contains four independent linear layers, each followed by the GELU activation function; It consists of a shared linear layer. Its forward process can be formally represented as: ; The image segmentation model training method provided in this embodiment of the invention uses a dual-adaptor design, where the inner adapter focuses on local features while the outer adapter is responsible for global feature transformation. During the training of the deep learning model, only the parameters of the two adapters are updated, while all other pre-trained parameters of the model remain frozen. This design makes the total number of trainable parameters significantly lower than the number of parameters of the SAM image encoder, and at the same time significantly improves the model's performance on scene text segmentation tasks.
[0063] In some embodiments, the calculation results include a spatial attention weight map determined by four-level depthwise separable convolution and a second image feature determined by convolution. Aggregating the corresponding calculation results through a dual-path attention mechanism to obtain the decoded features includes: reconstructing the spatial attention weight map and the second image feature respectively to obtain the reconstructed spatial attention weight map and the reconstructed second image feature, and then performing element-wise multiplication and weighted average calculation to obtain the decoded features.
[0064] In this embodiment, when performing modality alignment, the deep learning model can use a four-level deep separable convolution (DS-Conv) to generate a spatial attention weight map, which is specifically calculated using the following formula: ; in, This is a spatial attention weight map. For function, This is the first image feature.
[0065] In this embodiment, each depthwise separable convolution comprises a 3×3 channel independent depthwise convolution and a 1×1 channel fusion point convolution, with only a fraction of the parameters of a standard convolution. ,in For the number of channels, This design significantly reduces computational complexity while maintaining the receptive field, using the kernel size.
[0066] In this embodiment, the dimension is ( For batch size, To indicate the length, , Spatial attention weight map (for spatial dimensions) Reconstructed And apply it as a weighting factor to image embedding; first image features The initial dimension is Reconstruct and replace it with In order to perform element-wise multiplication with the spatial attention weight map, calculate the product for each spatial location, and then proceed along the spatial dimension. The weighted embeddings are averaged to obtain the final sparse embedding. The decoding features are specifically represented by the following formula: .
[0067] In one embodiment, before calculating cross-attention, a 3×3 convolutional layer is used to extract local features, i.e., second image features, from the image embedding, specifically represented by the following formula: .
[0068] in, This represents the second image feature. This convolution operation enhances local texture features such as edges and strokes, providing a fine-grained spatial prior for subsequent attention mechanisms.
[0069] In this embodiment, the self-attention component first applies sparse embeddings. Modeling is performed to capture long-range dependencies, specifically calculated using the following formula: ; In this embodiment, in the cross-attention component, Reconstructed into sequence form It interacts with sparse embeddings via cross-attention, and the interaction process is represented by the following formula: ; Finally, sparse embedding is obtained. This refers to the decoding feature, which is used in the decoding process of the second decoder.
[0070] The image segmentation model training method provided in this invention reconstructs the spatial attention weight map and the second image features respectively. The reconstructed spatial attention weight map and the reconstructed second image features are then multiplied element-wise and weighted averaged to obtain the decoded features. This method uses depthwise separable convolutions to significantly reduce the computational load, enabling the deep learning model to quickly generate high-quality spatial attention weight maps while maintaining a large receptive field. The features are then further decoded by calculating the weighted average aggregation method, which can extract effective text foreground features from complex natural scenes, further improving the efficiency and accuracy of subsequent attention mechanism processing.
[0071] Figure 4 This is a schematic diagram of the working mechanism of the first decoder provided by the present invention. Figure 4 In the illustrated embodiment, the first image feature (corresponding to the Sparse Embedding) input to the first decoder undergoes processing in two branches. In one branch, the first image feature is convolved using a four-level depthwise separable convolution (DS-ConvX4), and the output value of the convolution is compressed using the Sigmoid function to generate a spatial attention weight map. Finally, the generated weight map is multiplied by the original input feature, and the resulting sparse embedding is used to establish long-range dependencies through a self-attention mechanism. In the other branch, the first image feature is convolved using a Conv2d+GELU architecture, and the resulting second image feature and the above-mentioned sparse embedding are fused through a cross-attention mechanism to finally obtain the decoded feature (corresponding to the Sparse Embedding).
[0072] In one embodiment, after obtaining the image segmentation model, the training method of the image segmentation model includes: inputting test data into the image segmentation model to obtain segmentation results; calculating a target evaluation index based on the segmentation results, and evaluating the image segmentation model based on the target evaluation index to obtain an evaluation result; wherein, the target evaluation index includes at least one of the following: segmentation accuracy index, including at least one of fgIoU index and Fscore index; generalization ability, used to represent the robustness of the image segmentation model in deformed text and unseen scenes; parameter efficiency, which is determined by the number of training parameters and computational overhead; and boundary consistency, used to represent the smoothness and alignment of the segmentation result with the real boundary.
[0073] In this embodiment, the fgIoU (foreground intersection-union ratio) metric is used to evaluate the overlap of predicted text regions (foreground) to avoid bias caused by background dominance; the Fscore metric is the harmonic mean of precision and recall, used to comprehensively measure the accuracy of segmentation.
[0074] In this embodiment, the generalization ability of the image segmentation model can be determined by analyzing its performance on unseen data distributions (such as never-seen fonts or extremely distorted artistic fonts).
[0075] For example, poster images (containing shadows and 3D effect text) from the TextSeg dataset can be used for testing. Although the model may not have been exposed to such effects extensively during training, thanks to the powerful features of the SAM backbone and the adaptation of the Adapter, the model still accurately segmented the effect text, proving its robustness in "deformed text and unseen scenes".
[0076] In this embodiment, the corresponding parameter efficiency is determined by comparing the proportion of trainable parameters to the total number of parameters and the memory / computation consumption.
[0077] In this embodiment, boundary consistency is used to evaluate the geometric quality of the segmentation mask edges, i.e., whether the predicted text outline is smooth and closely matches the real text edges, rather than appearing jagged or blurry.
[0078] The image segmentation model training method provided in this embodiment of the invention can quantitatively verify the segmentation performance of the image segmentation model on core indicators by constructing a multi-dimensional evaluation system that includes accuracy, generalization, efficiency and boundary quality.
[0079] Figure 5 This is a schematic diagram of the working mechanism of the feature extraction module provided by the present invention. Figure 5 In the illustrated embodiment, via an internal adapter (corresponding to) (including two Linear and one GELU) and external adapter (corresponding) ,include , Extract the first image features from the sample images; during the training of the deep learning model, only update... and The parameters of the transformer network are frozen, while the parameters of other structures (attention, MLP) are frozen. Each transformer network of the deep learning model is deployed with an adapter to encode the input sample image and extract the corresponding first image features. In the hierarchical decoding stage of the model training process, sparse embeddings are extracted by Decoder1 (first decoder) and high-resolution target mask features are generated by Decoder2 (second decoder). The model loss is then calculated to achieve iterative training of the model.
[0080] The image segmentation method provided by the present invention will be described below. The image segmentation method apparatus described below and the training method of the image segmentation model described above can be referred to in correspondence.
[0081] Figure 6 This is a flowchart illustrating the image segmentation method provided by the present invention, as shown below. Figure 6 As shown, this image segmentation method includes the following steps: Step 610: Obtain the image to be processed.
[0082] In this step, the image to be processed can be a common digital image format, such as JPG (JPEG), PNG, BMP or TIFF, or other formats. This embodiment is not limited to a specific compression standard or encoding method.
[0083] In this step, the image to be processed can be obtained from a public dataset or acquired in real time via a camera, scanner, or other image acquisition device.
[0084] Step 620: Input the image to be processed into the image segmentation model to obtain the image segmentation result; wherein, the image segmentation model is trained by the image segmentation model training method.
[0085] In this step, the image segmentation model is trained through the following steps: (1) Input the sample image into the feature extraction module of the deep learning model to obtain the first image feature; wherein, the feature extraction module is constructed by residual connection of at least two linear layers and GELU activation function; (2) The first decoder based on the deep learning model performs depth-separable convolution and convolution calculations on the first image features respectively, and aggregates the corresponding calculation results through a dual-path attention mechanism to obtain the decoded features; (3) The second decoder based on the deep learning model performs upsampling and high-resolution feature refinement on the decoded features in sequence to obtain high-resolution features, and then embeds and fuses the high-resolution features with the self-attention features corresponding to the decoded features to obtain the target mask features; (4) Calculate the loss value based on the target mask features and the loss function of the deep learning model, and update the parameters of the feature extraction module based on the loss value. Under the condition of satisfying the iteration conditions of the deep learning model, the image segmentation model is obtained.
[0086] It should be noted that the implementation methods of steps (1) to (4) above correspond one-to-one with the embodiments of steps 110 to 140 above, and will not be repeated in this embodiment.
[0087] In this embodiment, the image to be processed is input into a trained image segmentation model to obtain the corresponding image segmentation result.
[0088] The image segmentation method provided in this invention improves the accuracy and efficiency of image segmentation by using an image segmentation model trained by an image segmentation model training method to process the base image to be processed.
[0089] The training apparatus for the image segmentation model provided by the present invention will be described below. The training apparatus for the image segmentation model described below can be referred to in correspondence with the training method for the image segmentation model described above.
[0090] Figure 7 This is a schematic diagram of the structure of the training device for the image segmentation model provided by the present invention, as shown below. Figure 7 As shown, the training device for this image segmentation model includes: The feature extraction module 710 is used to input the sample image into the feature extraction module of the deep learning model to obtain the first image features; wherein, the feature extraction module is constructed by at least two linear layers and GELU activation with residual connections; The first decoding module 720 is used to perform depth-separable convolution calculation and convolution calculation on the first image features based on the first decoder of the deep learning model, and aggregate the corresponding calculation results through a dual-path attention mechanism to obtain the decoded features; The second decoding module 730 is used to perform upsampling and high-resolution feature refinement on the decoded features in sequence based on the deep learning model to obtain high-resolution features, and then embed and fuse the high-resolution features with the self-attention features corresponding to the decoded features to obtain the target mask features. The training module 740 is used to calculate the loss value based on the target mask features and the loss function of the deep learning model, and update the parameters of the feature extraction module based on the loss value. Under the condition of satisfying the iteration conditions of the deep learning model, the image segmentation model is obtained.
[0091] The image segmentation model training device provided in this embodiment of the invention obtains first image features by inputting sample images into the feature extraction module of a deep learning model. Based on a first decoder, depthwise separable convolution and convolution calculations are performed on the first image features, and the corresponding calculation results are aggregated through a dual-path attention mechanism to obtain decoded features. Based on a second decoder, the decoded features are sequentially upsampled and refined to high resolution. The high resolution features are then fused with the self-attention features corresponding to the decoded features to obtain target mask features. Finally, a loss value is calculated based on the target mask features and the loss function of the deep learning model, and the parameters of the feature extraction module are updated based on the loss value to obtain the final image segmentation model. This method requires only minor parameter adjustments to achieve efficient adaptation to large-scale pre-trained models, significantly reducing training costs. Simultaneously, it enables progressive refinement of text regions, improving text segmentation accuracy and the generalization ability of the image segmentation model.
[0092] The image segmentation apparatus provided by the present invention will be described below. The image segmentation apparatus described below can be referred to in correspondence with the image segmentation method described above.
[0093] Figure 8 This is a schematic diagram of the image segmentation device provided by the present invention, as shown below. Figure 8 As shown, the image segmentation device includes an image acquisition module 810 and an image segmentation module 830.
[0094] Image acquisition module 810 is used to acquire the image to be processed; The image segmentation module 820 is used to input the image to be processed into the image segmentation model to obtain the image segmentation result; wherein, the image segmentation model is trained by the image segmentation model training method.
[0095] The image segmentation apparatus provided in this embodiment of the invention processes the base image to be processed by an image segmentation model trained by an image segmentation model training method, thereby improving the accuracy and efficiency of image segmentation.
[0096] Figure 9 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 9As shown, the electronic device may include: a processor 910, a communications interface 920, a memory 930, and a communications bus 940, wherein the processor 910, the communications interface 920, and the memory 930 communicate with each other through the communications bus 940. The processor 910 can call logic instructions in the memory 930 to execute a training method for an image segmentation model. This method includes: inputting a sample image into a feature extraction module of a deep learning model to obtain first image features; wherein the feature extraction module is constructed by residual connections of at least two linear layers and a GELU activation function; a first decoder based on the deep learning model performs depthwise separable convolution and convolution calculations on the first image features respectively, and aggregates the corresponding calculation results through a dual-path attention mechanism to obtain decoded features; a second decoder based on the deep learning model sequentially performs upsampling and high-resolution feature refinement on the decoded features to obtain high-resolution features, and fuses the high-resolution features with the self-attention features corresponding to the decoded features to obtain target mask features; calculating a loss value based on the target mask features and the loss function of the deep learning model, and updating the parameters of the feature extraction module based on the loss value; and obtaining an image segmentation model when the iteration conditions of the deep learning model are met.
[0097] Alternatively, an image segmentation method may be performed, which includes acquiring the image to be processed; inputting the image to be processed into an image segmentation model to obtain the image segmentation result; wherein the image segmentation model is trained using an image segmentation model training method.
[0098] Furthermore, the logical instructions in the aforementioned memory 930 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0099] On the other hand, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements a training method for the image segmentation model provided by the above methods. The method includes: inputting a sample image into a feature extraction module of a deep learning model to obtain a first image feature; wherein the feature extraction module is constructed by residual connections of at least two linear layers and a GELU activation function; a first decoder based on the deep learning model performs depthwise separable convolution and convolution calculations on the first image feature respectively, and aggregates the corresponding calculation results through a dual-path attention mechanism to obtain decoded features; a second decoder based on the deep learning model sequentially performs upsampling and high-resolution feature refinement on the decoded features to obtain high-resolution features, and fuses the high-resolution features with the self-attention features corresponding to the decoded features to obtain target mask features; calculating a loss value based on the target mask features and the loss function of the deep learning model, and updating the parameters of the feature extraction module based on the loss value, thereby obtaining an image segmentation model when the iteration conditions of the deep learning model are met.
[0100] Alternatively, an image segmentation method may be performed, which includes acquiring the image to be processed; inputting the image to be processed into an image segmentation model to obtain the image segmentation result; wherein the image segmentation model is trained using an image segmentation model training method.
[0101] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0102] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.
[0103] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A training method for an image segmentation model, characterized in that, include: The sample image is input into the feature extraction module of the deep learning model to obtain the first image feature; wherein, the feature extraction module is constructed by residual connection of at least two linear layers and GELU activation function; The first decoder based on the deep learning model performs depthwise separable convolution and convolution calculations on the first image features respectively, and aggregates the corresponding calculation results through a dual-path attention mechanism to obtain the decoded features; The second decoder based on the deep learning model performs upsampling and high-resolution feature refinement on the decoded features in sequence to obtain high-resolution features. The high-resolution features are then fused with the self-attention features corresponding to the decoded features to obtain the target mask features. The loss value is calculated based on the target mask features and the loss function of the deep learning model, and the parameters of the feature extraction module are updated based on the loss value. Under the condition that the iteration conditions of the deep learning model are met, an image segmentation model is obtained.
2. The training method for the image segmentation model according to claim 1, characterized in that, The feature extraction module includes an internal adapter and an external adapter. The internal adapter is located inside the Transformer block of the deep learning model, and the external adapter is located outside the Transformer block of the deep learning model. The internal adapter includes two linear layers and a GELU activation function, wherein the two linear layers are connected by the GELU activation function. The external adapter includes a first MLP structure and a second MLP structure. The first MLP structure includes four linear layers and four GELU activation functions, with each pair of linear layers connected by a GELU activation function. The second MLP structure includes one linear layer and one GELU activation function.
3. The training method for the image segmentation model according to claim 1, characterized in that, The calculation results include a spatial attention weight map determined by four-level depth-separable convolution calculation and second image features determined by convolution calculation; The process of aggregating the corresponding calculation results through a dual-path attention mechanism to obtain the decoded features includes: The spatial attention weight map and the second image features are reconstructed respectively to obtain the reconstructed spatial attention weight map and the reconstructed second image features. The elements are then multiplied and weighted averaged to obtain the decoded features.
4. The training method for the image segmentation model according to claim 1, characterized in that, The step of fusing the high-resolution features with the self-attention features corresponding to the decoded features to obtain the target mask features includes: The target mask features are calculated using the following formula: ； in, The target mask features, For the self-attention feature embedding, This refers to the high-resolution feature.
5. The training method for the image segmentation model according to any one of claims 1-4, characterized in that, After obtaining the image segmentation model, the method includes: The test data is input into the image segmentation model to obtain the segmentation result; The target evaluation index is calculated based on the segmentation results, and the image segmentation model is evaluated based on the target evaluation index to obtain the evaluation result; The target evaluation index includes at least one of the following: Segmentation accuracy metrics, including at least one of the fgIoU metric and the Fscore metric; Generalization ability is used to represent the robustness of the image segmentation model in deformed text and unseen scenes; Parameter efficiency, which is determined by the number of training parameters and computational overhead; Boundary consistency is used to represent the smoothness and alignment of the segmentation result with the true boundary.
6. An image segmentation method, characterized in that, include: Obtain the image to be processed; The image to be processed is input into the image segmentation model to obtain the image segmentation result; wherein the image segmentation model is trained by the training method of the image segmentation model as described in any one of claims 1-5.
7. A training device for an image segmentation model, characterized in that, include: The feature extraction module is used to input the sample image into the feature extraction module of the deep learning model to obtain the first image features; wherein, the feature extraction module is constructed by at least two linear layers and GELU activation with residual connections; The first decoding module is used to perform depthwise separable convolution calculation and convolution calculation on the first image features based on the first decoder of the deep learning model, and aggregate the corresponding calculation results through a dual-path attention mechanism to obtain the decoded features; The second decoding module is used to perform upsampling and high-resolution feature refinement on the decoded features in sequence based on the second decoder of the deep learning model to obtain high-resolution features, and to fuse the high-resolution features with the self-attention features corresponding to the decoded features to obtain target mask features. The training module is used to calculate the loss value based on the target mask features and the loss function of the deep learning model, and update the parameters of the feature extraction module based on the loss value, so as to obtain an image segmentation model when the iteration conditions of the deep learning model are met.
8. An image segmentation apparatus, characterized in that, include: The image acquisition module is used to acquire the image to be processed. An image segmentation module is used to input the image to be processed into an image segmentation model to obtain an image segmentation result; wherein the image segmentation model is trained by the training method of the image segmentation model as described in any one of claims 1-5.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the training method of the image segmentation model as described in any one of claims 1 to 5 or the image segmentation method as described in claim 6.
10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the training method of the image segmentation model as described in any one of claims 1 to 5 or the image segmentation method as described in claim 6.