A method for constructing a landscape painting semantic segmentation model

Abstract

The application discloses a method for constructing a landscape painting semantic segmentation model, and specifically comprises the following steps: a feature extractor and a DRA module are used to form an encoder and a decoder to combine to form a semantic segmentation model; an original sample dataset of landscape paintings is established, and reference sets are obtained by labeling the original sample dataset; part of the landscape paintings in the original sample dataset and the corresponding samples in the reference sets are selected to form a training set, and the remaining corresponding samples form a test dataset; an existing natural image semantic segmentation set is acquired, pre-training is performed on the semantic segmentation model, and thus the semantic segmentation model is adjusted; the original samples in the training set are input into the pre-trained and adjusted semantic segmentation model for training, and a landscape painting semantic segmentation model is obtained. The application can well distinguish similar details in the landscape paintings and label the similar details, and solves the problems of time-consuming and laborious manual labeling and reduced generalization caused by overfitting.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a method for constructing a semantic segmentation model for landscape paintings. Background Technology

[0002] Chinese landscape painting is an important component of traditional Chinese culture and art, possessing rich cultural connotations and artistic value, and is one of the treasures of Chinese culture. The core elements pursued in Chinese landscape painting are artistic conception and rhythm, and are also important reasons that distinguish it from Western landscape painting. Through the depiction of natural elements such as mountains, water, clouds, and trees, the grandeur and rhythm of landscapes are portrayed, imbuing the paintings with vitality and life. Research on semantic segmentation of landscape paintings is helpful for the preservation and transmission of this important cultural heritage. In the image restoration and protection of landscape paintings, for damaged or incomplete paintings, image super-resolution and restoration algorithms can restore painting techniques, protect element features, restore details and textures, and repair damage and missing parts. These algorithms all rely on landscape painting datasets with semantic segmentation and data annotation.

[0003] Existing semantic segmentation and data annotation for landscape paintings typically employ manual methods or deep learning-based algorithms. Manual semantic segmentation of landscape paintings is time-consuming and labor-intensive, and the high complexity of the segmentation process requires specialized knowledge and extensive experience. Careful verification during segmentation and annotation further complicates the handling of large-scale datasets.

[0004] Currently, deep learning-based algorithms typically employ few-shot learning. However, due to the scarcity of landscape painting datasets and the inherent limitations of few-shot learning methods, the models tend to memorize details and noise from the training samples, failing to learn broader image features. This can easily lead to overfitting and ultimately reduce the generalization ability for semantic segmentation of images. Summary of the Invention

[0005] The purpose of this invention is to provide a method for constructing a semantic segmentation model for landscape paintings, which can effectively distinguish and annotate similar details in landscape paintings, solving the problems of time-consuming and labor-intensive manual annotation and reduced generalization due to overfitting.

[0006] The technical solution adopted in this invention is a method for constructing a semantic segmentation model for landscape paintings, specifically including the following steps:

[0007] Step 1: Establish the original sample dataset of landscape paintings, and manually annotate the original sample dataset to obtain a reference set with semantic segmentation annotations;

[0008] Step 2: Select landscape paintings from a portion of the original sample dataset and their corresponding samples from the reference set to form a training set, thereby increasing the quantity of sample data in the training set. The remaining corresponding samples form the test dataset.

[0009] Step 3: Obtain an existing natural image semantic segmentation set, pre-train the natural image semantic segmentation set on the semantic segmentation model, and adjust the semantic segmentation model through the pre-training parameters. The pre-training uses an adversarial loss function in conjunction with the training.

[0010] The semantic segmentation model includes an encoder and a decoder. The encoder consists of a feature extractor MobileNet-V2 and a DRA module.

[0011] Step 4: Input the original samples in the training set into the semantic segmentation model that has been pre-trained and adjusted in Step 3 for training, and obtain a landscape painting semantic segmentation model.

[0012] The invention is further characterized in that,

[0013] Furthermore, grayscale values ​​are filled around the images in the reference set from step 1 to make all images in the reference set the same size.

[0014] Furthermore, the sample enhancement in step 2 includes processing the samples using at least one transformation operation: image distortion, image blurring, image tone transformation, image brightness transformation, and image grayscale transformation.

[0015] Furthermore, step 3, which involves pre-training the natural image semantic segmentation set on the semantic segmentation model, specifically includes:

[0016] S301, Input the natural image M into the semantic segmentation model, perform 4x downsampling on the natural image M to obtain image p, perform 16x downsampling on the natural image M to obtain image p3, and use the feature extractor MobileNet-V2 to extract features from the natural image M to obtain primary feature maps p1 and p2, and save the weight and bias parameter information of the convolutional layer in the feature extractor MobileNet-V2;

[0017] S302, the primary feature map p1 is iteratively processed by a convolutional network to form a secondary feature map p'1; the primary feature map p2 is input into CBAM for feature detail attention discrimination and then input into ASPP; ASPP iterates through convolution on the primary feature map p2 and outputs feature map p'2; the ASPP iterates through convolution on the primary feature map p2 using the weight and bias parameter information of the convolutional layer saved in S301.

[0018] S303, concatenate feature map p'2 with image p3, and perform convolution iteration on the concatenated feature map to form feature map p'23;

[0019] S404, Upsample the feature map p'23, and then concatenate the upsampled feature map p'23 with the secondary feature map p'1 and the image p to obtain the image T;

[0020] In step S305, image T is input into the decoder. The decoder first performs convolution iteration on image T and then upsamples it to output the semantic segmentation image of image T. Steps S301 to S304 are repeated to complete the processing of all natural images M. The convolution iteration parameters in steps S302, S303 and S305 are integrated and processed, and the parameters with the highest running frequency are retained to complete the pre-training of the semantic segmentation model.

[0021] Furthermore, the adjustment of the semantic segmentation model through pre-trained parameters in step 3 specifically involves: loading CBAM and ASPP with the convolution iteration parameters retained in S305, so that the weights and bias parameters of CBAM and ASPP are initialized according to a normal distribution.

[0022] Furthermore, step 3, which involves using an adversarial loss function in conjunction with training, specifically includes: employing the Focal loss function and the Dice loss function, and combining the Focal loss function and the Dice loss function with weights to form an adversarial loss function, as shown in the following formula:

[0023]

[0024] Where θ represents the convolution iteration parameters of the semantic segmentation model. For the input image, Given the input set of images, The training set consists of each input image and its accompanying annotations. For the desired output value, The output value predicted by the semantic segmentation model is α, which is a fixed parameter value ranging from 0 to 1.

[0025] Furthermore, in step 3, the DRA module includes the adjusted ASPP and the residual block. Specifically, the adjusted ASPP is: adding a spatial channel attention mechanism module CBAM before the ASPP, and adjusting the expansion rate of each layer of the original ASPP, adjusting the expansion rate of the second layer to 4 times, the expansion rate of the third layer to 6 times, the expansion rate of the fourth layer to 8 times, and keeping the expansion rates of the first and fifth layers unchanged.

[0026] The residual block includes a downsampled residual block with cross-layer skip connections and convolution operations, and a direct residual block that provides information to the decoder.

[0027] Furthermore, step 4, which involves inputting the original samples from the training set into the semantic segmentation model that has been pre-trained and adjusted in step 3, specifically includes:

[0028] S401, input the original sample I in the training set into the semantic segmentation model, and obtain image f by downsampling the original sample I by 4 times, obtain image f3 by downsampling the original sample I by 16 times, and obtain primary feature maps f1 and f2 by using the feature extractor MobileNet-V2 to extract features from the original sample I.

[0029] S402, the primary feature map f1 is iteratively processed by a convolutional network to form a secondary feature map f'1; the primary feature map f2 is input into CBAM for feature detail attention discrimination and then input into ASPP. ASPP convolves and iterates on the primary feature map f2 and outputs the feature map f'2.

[0030] The feature map f'2 is concatenated with the image f3, and the concatenated feature map is then subjected to convolution iteration to form the feature map f'23;

[0031] Upsample the feature map f'23, and then concatenate the upsampled feature map f'23 with the secondary feature map f'1 and the image f to obtain the image F;

[0032] S403, input image F into decoder to obtain semantic segmentation image of image F, compare the semantic segmentation image with the corresponding semantic segmentation labeled image in training set, when the similarity between the semantic segmentation image and the semantic segmentation labeled image is not less than 90%, retain the convolution iteration parameters of encoder and decoder when processing the original sample I, and reload encoder and decoder based on the convolution iteration parameters;

[0033] S404, repeating S401-S403, processes all original samples in the training set to make the convolution iteration parameters of the semantic segmentation model converge and fix, thus obtaining a landscape painting semantic segmentation model.

[0034] Compared with the prior art, the beneficial effects of the present invention are:

[0035] (1) In order to enable the network to better distinguish the relationship between details such as figures, buildings, and bridges in landscape paintings, this invention uses MobileNet-V2 as the backbone network of the feature extractor, and cooperates with the DRA module to improve the segmentation of feature details and the coordination of the weight combination of Focal loss function and Dice loss function. This solves the problem of low distinguishability of details such as figures, buildings, and bridges in landscape paintings that conform to traditional Chinese elements. It can extract the key features of landscape paintings while capturing the charm and vividness of the picture, and produce more expressive semantic segmentation.

[0036] (2) The spatial-channel domain self-attention block (CBAM) in the DRA module of this invention can, to a certain extent, reduce the dependence on labeled data by adaptively adjusting the network's attention, thereby improving the understanding of the vividness of landscape painting; the spatial convolution pooling pyramid can reduce the model degradation phenomenon, while retaining more information and gradients, it helps the network to better capture details such as lines and edges, so that the network can learn the semantic features of landscape painting more accurately.

[0037] (3) This invention optimizes the network by fusion of the weights of the Dice loss function and the Focal loss function to perform gradient descent, so that the semantic segmentation model can more accurately capture the boundaries and relationships between elements when segmenting landscape paintings, better preserve the details and vividness of landscape paintings, and adjust the contribution of different loss functions according to the actual situation, so that the model can better adapt to the characteristics of landscape paintings and segmentation needs, and improve the artistic expression and realism of the segmentation results.

[0038] (4) The residual block in the DRA module of this invention combines skip connections and residual learning mechanisms, allowing information to be directly transmitted through the network. This facilitates better learning and representation of input features, improves the network's ability to perceive features of different scales and complexities, and enhances the accuracy and detail preservation of semantic segmentation. It preserves more information and gradients, while helping the network capture details such as lines and edges and accurately learn the semantic features of landscape paintings. Attached Figure Description

[0039] Figure 1 This is a flowchart illustrating the construction method of a semantic segmentation model for landscape painting according to the present invention.

[0040] Figure 2 This is a schematic diagram of the training process of the semantic segmentation model after inputting the training set into the pre-trained and adjusted model.

[0041] Figure 3 This is a comparison between the original sample dataset in the training set and the reference set in the embodiment;

[0042] Figure 4 This is a comparison chart of semantic segmentation results of common semantic segmentation methods in the present invention;

[0043] Figure 5 This is a comparison chart of ablation experiment results using different improved methods in this invention;

[0044] Figure 6 This is a comparison chart of ablation experiment results using different loss functions in this invention;

[0045] Figure 7 This is a comparison chart showing the results of using Xception and MobileNet-V2 as backbone networks in this invention. Detailed Implementation

[0046] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0047] like Figures 1-2 As shown, the present invention provides a method for constructing a semantic segmentation model for landscape paintings, which specifically includes the following steps:

[0048] Step 1: Establish the original sample dataset of landscape paintings, manually annotate the original sample dataset to obtain a reference set with semantic segmentation annotations, and fill the area around the images in the reference set with gray values ​​to make all images in the reference set the same size.

[0049] Step 2: Select landscape paintings from a portion of the original sample dataset and their corresponding samples from the reference set to form a training set, thereby increasing the number of samples in the training set.

[0050] Specifically, one or more of the following methods are used to increase the number of sample data in the training set: image distortion, image blurring, image tone transformation, image brightness transformation, and image grayscale transformation. This is to better train the semantic segmentation model and make the final landscape painting semantic segmentation model more accurate. The remaining corresponding samples form the test dataset.

[0051] Step 3: Obtain an existing natural image semantic segmentation set, pre-train the natural image semantic segmentation set on the semantic segmentation model, and adjust the semantic segmentation model through the pre-training parameters. The pre-training uses an adversarial loss function in conjunction with the training.

[0052] Specifically, it includes the following steps:

[0053] S301, Input the natural image M into the semantic segmentation model, perform 4x downsampling on the natural image M to obtain image p, perform 16x downsampling on the natural image M to obtain image p3, and use the feature extractor MobileNet-V2 to extract features from the natural image M to obtain primary feature maps p1 and p2, and save the weight and bias parameter information of the convolutional layer in the feature extractor MobileNet-V2;

[0054] S302, the primary feature map p1 is iteratively processed by a convolutional network to form a secondary feature map p'1; the primary feature map p2 is input into CBAM for feature detail attention discrimination and then input into ASPP; ASPP iterates through convolution on the primary feature map p2 and outputs feature map p'2; the ASPP iterates through convolution on the primary feature map p2 using the weight and bias parameter information of the convolutional layer saved in S301.

[0055] S303, concatenate feature map p'2 with image p3, and perform convolution iteration on the concatenated feature map to form feature map p'23;

[0056] S304, Upsample the feature map p'23, and then concatenate the upsampled feature map p'23 with the secondary feature map p'1 and the image p to obtain the image T;

[0057] In step S305, image T is input into the decoder. The decoder first performs convolution iteration on image T and then upsamples it to output the semantic segmentation image of image T. Steps S301 to S304 are repeated to complete the processing of all natural images M. The convolution iteration parameters in steps S302, S303 and S305 are integrated and processed, and the parameters with the highest running frequency are retained to complete the pre-training of the semantic segmentation model.

[0058] Then, the convolution iteration parameters retained in S305 are used to load CBAM and ASPP, so that the weights and bias parameters of CBAM and ASPP are initialized according to a normal distribution, thereby adjusting the semantic segmentation model.

[0059] The semantic segmentation model includes an encoder and a decoder that incorporates image concatenation and upsampling operations. The encoder is composed of the feature extractor MobileNet-V2 and the DRA module.

[0060] Step 4: Input the original samples in the training set into the semantic segmentation model that has been pre-trained and adjusted in Step 3 for training, and obtain a landscape painting semantic segmentation model.

[0061] Specifically, the following steps are included:

[0062] S401, input the original sample I in the training set into the semantic segmentation model, and obtain image f by downsampling the original sample I by 4 times, obtain image f3 by downsampling the original sample I by 16 times, and obtain primary feature maps f1 and f2 by using the feature extractor MobileNet-V2 to extract features from the original sample I.

[0063] S402, the primary feature map f1 is iteratively processed by a convolutional network to form a secondary feature map f'1; the primary feature map f2 is input into CBAM for feature detail attention discrimination and then input into ASPP. ASPP convolves and iterates on the primary feature map f2 and outputs the feature map f'2.

[0064] The feature map f'2 is concatenated with the image f3, and the concatenated feature map is then subjected to convolution iteration to form the feature map f'23;

[0065] Upsample the feature map f'23, and then concatenate the upsampled feature map f'23 with the secondary feature map f'1 and the image f to obtain the image F;

[0066] S403, input image F into decoder to obtain semantic segmentation image of image F, compare the semantic segmentation image with the corresponding semantic segmentation labeled image in training set, when the similarity between the semantic segmentation image and the semantic segmentation labeled image is not less than 90%, retain the convolution iteration parameters of encoder and decoder when processing the original sample I, and reload encoder and decoder based on the convolution iteration parameters;

[0067] S404, repeating S401-S403, processes all original samples in the training set to make the convolution iteration parameters of the semantic segmentation model converge and fix, thus obtaining a landscape painting semantic segmentation model.

[0068] The DRA module includes the adjusted ASPP and residual blocks. Specifically, the adjusted ASPP is based on the existing ASPP structure, adjusting the expansion rate of each layer of the original five-layer pooling pyramid. While the first and last layers remain unchanged, the expansion rate of the second layer is reduced from 6 times to 4 times, the third layer from 12 times to 6 times, and the fourth layer from 18 times to 8 times, as shown in Table 1.

[0069] Table 1 ASPP Hollow Convolution Dilation Rate

[0070]

[0071]

[0072] The adjusted ASPP is more in line with the characteristics of the landscape painting dataset, ensuring better classification results with fewer parameters and lower algorithm complexity.

[0073] This method introduces CBAM between the feature extractor MobileNet-V2 and ASPP, enabling the encoder to focus more on high-frequency details of trees, mountains, buildings, and people, reducing the dilatation rate of ASPP (Spatial Convolutional Pooling Pyramid), and capturing large-scale semantic information features with less parameter information.

[0074] ASPP stands for Atrous Spatial Pyramid Pooling, a convolutional neural network module used for image segmentation. It employs a series of dilated convolutional and pooling layers to capture feature information from multiple receptive fields. ASPP captures contextual information at different scales by applying multiple dilated convolutional kernels at varying sampling rates. This allows the model to capture the relationship between local details and global context, thus improving image segmentation performance. These dilated convolutional layers sample in parallel, capturing objects and image context at multiple scales, enabling ASPP to expand the receptive field while maintaining resolution, thereby better capturing multi-scale information.

[0075] CBAM, or Channel-Spatial Attention Module, is an attention mechanism in deep learning that strengthens the correlation between channel and spatial features in convolutional neural networks. Given an intermediate feature map, CBAM infers attention maps sequentially along two independent dimensions (channel and space), then multiplies these attention maps with the input feature map for adaptive feature optimization. CBAM consists of two sub-modules: a channel attention module and a spatial attention module. The channel attention module calculates the importance of each channel by performing global pooling and applies it to each location on the channel feature map. The spatial attention module calculates the importance of each spatial location by calculating the average and maximum values ​​for each location, thus reinforcing useful spatial locations. By combining these two sub-modules, CBAM can simultaneously focus on both channel and spatial features, thereby improving network performance.

[0076] The residual block consists of a downsampled residual block with cross-layer skip connections and convolution operations, and a direct residual block that provides information to the decoder. The downsampled residual block is concatenated with CBAM (Spatial-Channel Domain Self-Attention Block) and ASPP (Spatial Convolutional Pooling Pyramid). The original information is fused into the secondary feature space through convolutional layers. After upsampling and processing by the MobileNet-V2 feature extractor, the feature maps are then concatenated and fused through a 1×1 convolutional layer. The direct residual block is directly introduced before the decoder to provide feature information to the decoder.

[0077] Two types of residual blocks combine skip connections and residual learning mechanisms, allowing information to be passed directly through the network. This facilitates better learning and representation of input features, improves the network's ability to perceive features of different scales and complexities, and enhances the accuracy and detail preservation of semantic segmentation. It retains more information and gradients, while helping the network capture details such as lines and edges and accurately learn the semantic features of landscape paintings.

[0078] Specifically, step 3, which uses an adversarial loss function in conjunction with training, includes: using the Focal loss function and the Dice loss function, and combining the Focal loss function and the Dice loss function with weights to form an adversarial loss function, as shown in the following formula:

[0079]

[0080] Where θ represents the convolution iteration parameters of the semantic segmentation model. For the input image, Given the input set of images, The training set consists of each input image and its accompanying annotations. For the desired output value, The output value predicted by the semantic segmentation model is α, which is a fixed parameter value ranging from 0 to 1.

[0081] The Dice loss function is defined as follows:

[0082]

[0083] In this context, TP indicates that the label is true and the prediction is true; TN indicates that the label is false and the prediction is false; FP indicates that the label is false but the prediction is true; FN indicates that the label is false and the prediction is true; and Card(·) represents the count. The Dice loss function calculates the similarity between the predicted segmentation result and the actual segmentation result, with a value between 0 and 1. The closer the value is to 1, the higher the similarity between the predicted result and the actual result.

[0084] The Focal loss function is defined as follows:

[0085]

[0086] Where k is the number of categories, This represents the expected output value. This represents the output predicted by the model. By using the Focal loss function, the model's ability to learn minority classes and difficult-to-classify samples can be improved, thereby enhancing model performance under class imbalance problems.

[0087] By fusing the weights of the Dice loss function and the Focal loss function to optimize the network through gradient descent, the semantic segmentation model can more accurately capture the boundaries and relationships between elements when segmenting landscape paintings, better preserve the details and vividness of the landscape paintings, and adjust the contribution of different loss functions according to the actual situation, so that the model can better adapt to the characteristics of landscape paintings and segmentation needs, and improve the artistic expression and realism of the segmentation results.

[0088] After the semantic segmentation model for landscape paintings is constructed, landscape paintings from the test set are input into it. The output semantically segmented images are compared with manually annotated images from the test set, and the accuracy of the semantic segmentation model for landscape paintings in this invention is evaluated based on the similarity between the two.

[0089] Example

[0090] Step 1: Select 709 landscape paintings to establish an original sample dataset. Manually annotate the original sample dataset to obtain a reference set with semantic segmentation annotations. The landscape paintings in the original sample dataset are in JPEG format, which are then read into RGB format, as shown below. Figure 3 As shown, the images with semantic segmentation in the reference set are output as lossless compressed PNG single-channel grayscale images. Since the landscape paintings are not painted by the same artist and vary in size, all images are adjusted to a size of 512×512 by filling the area around the images with grayscale values.

[0091] Step 2: Select landscape paintings from a portion of the original sample dataset and their corresponding samples from the reference set to form a training set, thereby increasing the number of samples in the training set. Five methods are used: image distortion, image blurring, image tone transformation, image brightness transformation, and image grayscale transformation, to expand the number of samples in the training set to five times the original size, in order to alleviate the network overfitting problem. The remaining corresponding samples form the test dataset.

[0092] This invention categorizes the content of landscape painting images into seven categories, and their labels and dataset distribution are shown in Table 2:

[0093] Table 2 Semantic Segmentation Tags for Landscape Paintings

[0094]

[0095] Step 3: The PASCAL VOC 2012 dataset is used as the natural image semantic segmentation set. This dataset contains 17,125 images (2,913 of which are used for segmentation) and 27,450 labeled anchor boxes. The semantic segmentation model is pre-trained on the natural image semantic segmentation set, and the semantic segmentation model is adjusted through the pre-trained parameters. The pre-training steps are as follows:

[0096] S301, Input the natural image M into the semantic segmentation model, process the natural image M in three ways respectively, perform 4x downsampling to obtain image p, and directly introduce image p into the decoder to provide low-level detailed features for the decoder;

[0097] The natural image M is downsampled by 16 times to obtain image p3, and image p3 is input into ASPP;

[0098] The MobileNet-V2 feature extractor is used to extract features from the natural image M to obtain primary feature maps p1 and p2. The weight and bias parameter information of the convolutional layer in the MobileNet-V2 feature extractor is saved.

[0099] S302, the primary feature map p1 is iterated through a 1×1 convolutional network to form a secondary feature map p'1; the primary feature map p2 is input into CBAM for feature detail attention discrimination and then input into ASPP. ASPP iterates the primary feature map p2 through a 1×1 convolutional network and outputs feature map p'2. The ASPP uses the weight and bias parameter information of the convolutional layer saved in S301 for the convolutional iteration of the primary feature map p2.

[0100] S303, concatenate feature map p'2 with image p3, and iterate the concatenated feature map through a 1×1 convolutional network to form feature map p'23;

[0101] S304, Upsample the feature map p'23 by 4 times, and then concatenate the upsampled feature map p'23 with the secondary feature map p'1 and the image p to obtain the image T;

[0102] In step S305, image T is input into the decoder. The decoder first performs a 3×3 convolutional network iteration on image T, and then performs a 4x upsampling to output the semantic segmentation image of image T. Steps S301 to S304 are repeated to complete the processing of all natural images M. The convolutional iteration parameters in steps S302, S303 and S305 are integrated and processed, and the parameters with the highest running frequency are retained to complete the pre-training of the semantic segmentation model.

[0103] Then, the convolution iteration parameters retained in S305 are used to load CBAM and ASPP, so that the weights and bias parameters of CBAM and ASPP are initialized according to a normal distribution, thereby adjusting the semantic segmentation model.

[0104] The Focal loss function and the Dice loss function are used, and the Focal loss function and the Dice loss function are combined with weights to form an adversarial loss function for pre-training. The adversarial loss function is shown in the following formula:

[0105]

[0106] Where θ represents the convolution iteration parameters of the semantic segmentation model. For the input image, Given the input set of images, The training set consists of each input image and its accompanying annotations. For the desired output value, The output value predicted by the semantic segmentation model is α, which is a fixed parameter value with a range of 0.8.

[0107] Step 4: Input the original samples in the training set into the semantic segmentation model that has been pre-trained and adjusted in Step 3 for training, and obtain a landscape painting semantic segmentation model.

[0108] Specifically, the following steps are included:

[0109] S401, input the original sample I (C×H×W) from the training set into the semantic segmentation model, where C represents the number of channels, H represents the length, and W represents the width. Then, downsample the original sample I by a factor of 4 to obtain the image f. It provides low-level detailed features to the decoder before being directly introduced into the decoder.

[0110] Image f3 is obtained by downsampling the original sample I by 16 times.

[0111] The original sample I was subjected to feature extraction using the MobileNet-V2 feature extractor to obtain the primary feature map f1. and f2

[0112] S402, the primary feature map f1 is iteratively processed through a 1×1 convolutional network to form the secondary feature map f'1. The primary feature map f2 is input into CBAM for feature detail attention discrimination, and then input into ASPP. ASPP performs convolutional iterations on the primary feature map f2 and outputs feature map f'2.

[0113] Feature map f'2 is concatenated with image f3, and the concatenated feature map is then subjected to convolution iteration to form feature map f'23.

[0114] Upsample the feature map f'23, and then concatenate the upsampled feature map f'23 with the secondary feature map f'1 and the image f to obtain the image F. Enlarged to the same size F (C×H×W) as the original image using bilinear interpolation.

[0115] S403, input image F into decoder to obtain semantic segmentation image of image F, compare the semantic segmentation image with the corresponding semantic segmentation labeled image in training set, when the similarity between the semantic segmentation image and the semantic segmentation labeled image is not less than 90%, retain the convolution iteration parameters of encoder and decoder when processing the original sample I, and reload encoder and decoder based on the convolution iteration parameters.

[0116] S404, repeating S401-S403, processes all original samples in the training set to make the convolution iteration parameters of the semantic segmentation model converge and fix, thus obtaining a landscape painting semantic segmentation model.

[0117] After the semantic segmentation model for landscape paintings is constructed, landscape paintings from the test set are input into it. The output semantically segmented images are compared with manually annotated images from the test set, and the accuracy of the semantic segmentation model for landscape paintings in this invention is evaluated based on the similarity between the two.

[0118] Table 3 shows the semantic segmentation performance scores for seven types of elements on three evaluation metrics. Specifically, the Intersection over Union (IoU) for mountains and trees reached 66.59% and 66.67%, respectively; the weighted average class pixel accuracy (mAP) reached 82.43% and 82.18%, respectively; and the accuracy (Acc) for buildings and trees reached 80.49% and 77.94%, respectively. This demonstrates that the proposed semantic segmentation model for landscape paintings can achieve high-speed automatic annotation and batch acquisition of large amounts of labeled segmented datasets.

[0119] Table 3. Evaluation Indicators for Semantic Segmentation and Classification

[0120]

[0121]

[0122] Table 4 compares our DRA model with commonly used network models, including U-Net, UNet++, PSPNet, LPSNet, DDRNet, and SegFormer, all trained in the same manner. Here, GFLOPs represents the number of floating-point operations in billions, and #P(M) represents the number of parameters in the model in millions. As can be seen from the table, DDRNet has the lowest computational complexity (only 5 GFLOPs) and fewer parameters, but its performance is slightly worse. Our DRAFormer has the fewest parameters, but on the SegCLP dataset, it significantly improves mIoU to 62.3%, mAP to 77.04%, and Acc to 84.06%. Our DRA achieves better performance on key evaluation metrics (mIoU, mAP, and Acc.), more accurately segments different categories in images, and demonstrates excellent pixel-level classification accuracy. Figure 4Qualitative comparisons clearly show that our method is more accurate, has richer labeled elements, and exhibits semantic consistency. As shown in the first row, even though the "water" portion of the painting is generally almost entirely blank, the fusion network can still correctly identify it. This is because the network considers the overall composition of the painting, inferring its category from the features of the boats on the water and the surrounding shore. In the second and third rows, the classification of trees is more detailed, and the classification of water surfaces and bridges is also more accurate.

[0123] Table 4 shows the scores of DRA compared to other common semantic segmentation models.

[0124]

[0125] To further demonstrate the effectiveness of the method of this invention, ablation experiments were conducted with different network structures. Table 5 shows the network structure evaluation metrics results of different improvement methods. The baseline represents the experimental method of changing the feature extractor of the DeepLabV3+ model to MobileNet-V2, and the loss function is Focal Loss+αDice Loss. The remaining ablation experiments are all improvements based on the baseline. Analysis shows that introducing residual blocks can also result in a 7.93% improvement in mean intersection-over-union (mIoU) performance compared to the baseline, and introducing an attention mechanism can bring a 9.46% improvement in mIoU compared to the baseline. Considering that residual blocks may provide stronger potential for training networks on small datasets with higher granularity semantic segmentation, and that the CBAM module can simultaneously focus on channel and spatial features to enhance network performance, the DRA network of this invention combines these two components. Compared to the baseline, DRA achieves a 10.14% improvement in mIoU.

[0126] Table 5 Evaluation index results for different network structures

[0127]

[0128] Figure 5This section presents the visualization results of ablation experiments under different network structures. From left to right, the results are: Input image, Baseline, Increase Dilation rate, Decrease Dilation rate, Baseline+Residual, Baseline+CBAM, Final fusion network result, and manually annotated image (Ground Truth). The second row of manually annotated images contains errors; mountains and trees are blended together in the painting, making them difficult to distinguish. However, this invention correctly recognized and corrected these errors. To describe the vividness and spirit of the landscape painting, analyzing the semantic features of each part mainly requires considering the overall macroscopic composition of the image. In the feature extractor, different preceding convolutional layers are used as texture filters to find the indistinguishable parts, and then the compositional characteristics are analyzed in subsequent convolutional layers for accurate segmentation. In the third row, the fusion network's detection of mountains and trees is relatively more complete and accurate, which helps to convey the natural landscape expressed in the landscape painting, allowing viewers to better appreciate its artistic conception and vividness.

[0129] This invention conducted ablation experiments with different loss functions, and the evaluation results are shown in Table 6. It can be seen from all the evaluation indicators that using the weight combination of Focal Loss and Dice Loss can improve the network's average crossover ratio performance by at least 0.25%. It is believed that using both of these loss functions simultaneously has greater potential in finer-grained small sample datasets.

[0130] Table 6 Evaluation index results for different loss functions

[0131]

[0132] Figure 6The results of ablation experiments using different loss functions are shown. From left to right: Input image, manually labeled image (Ground Truth), using cross-entropy loss function (CE Loss), using both cross-entropy and Dice loss functions (CELoss+Dice Loss), using Focal loss function (Focal Loss), and using both Focal and Dice loss functions (Focal Loss+αDice Loss). In landscape paintings, artists use minimal ink and blank space, employing a few strokes to condense mountains and flowing water into rhythmic and vibrant images. These elements occupy a large proportion of the painting, while figures and buildings often occupy a small portion, expressing the harmonious relationship between humanity and nature. This emphasizes the integration of nature and humanity, giving the work vitality and emotional resonance. Because the Focal loss function is more suitable for combating class imbalance problems, the Dice loss function is robust to small objects. From... Figure 6 It can be seen that the segmentation result obtained by combining the Focal loss function and the Dice loss function is more in line with the vivid and expressive characteristics of landscape painting.

[0133] This invention conducted ablation experiments on the backbone networks of different feature extractors. Table 7 shows the computational cost, parameter count, and evaluation metrics for Xception and MobileNet-V2 as backbone networks, respectively. The computational cost (FLOPs, Floating Point Operations) measures the total number of floating-point operations performed by the model during inference, expressed in gigabit-level (GFLOPs) units. Higher computational cost results in more computations and higher time complexity. The number of parameters (#P) ​​refers to the parameters the model needs to train, reflecting its spatial complexity; more parameters mean higher GPU and system memory usage. The data in the table shows that the method of this invention has a lightweight advantage, achieving similar accuracy to Xception as the backbone network while requiring nearly three times less computation (GFLOPs) and 11 times fewer parameters. The final model using MobileNet-V2 has low computational complexity and high computation speed. It can be seen that Xception has a strong overall feature extraction capability, but Xception introduces more parameters and computational cost, the network is slightly slower than MobileNet-V2 as a whole, and consumes more GPU memory.

[0134] Table 7. Computational requirements, parameter count, and scoring for different backbone networks.

[0135]

[0136] Figure 7 This section compares the results of using Xception and MobileNet-V2 as the backbone network. The first row shows the results with Xception as the backbone network, and the second row shows the results with MobileNet-V2. Visually, Xception as the extractor seems to perform better than MobileNet-V2 in fine-grained processing, but it performs poorly for some small targets. As seen in the first and second columns, MobileNet-V2 is more accurate in detecting mountain peaks, able to detect smaller distant peaks in landscape paintings, or peaks that are difficult to capture and connected to trees. In summary, while Xception performs slightly better as the backbone network than MobileNet-V2, its parameter count and computational complexity increase dramatically, while MobileNet-V2 provides better performance while maintaining lower model complexity.

Claims

1. A method for constructing a semantic segmentation model for landscape paintings, characterized in that, Specifically, the following steps are included: Step 1: Establish the original sample dataset of landscape paintings, and manually annotate the original sample dataset to obtain a reference set with semantic segmentation annotations; Step 2: Select landscape paintings from a portion of the original sample dataset and their corresponding samples from the reference set to form a training set, thereby increasing the quantity of sample data in the training set. The remaining corresponding samples form the test dataset. Step 3: Obtain an existing natural image semantic segmentation set, pre-train the natural image semantic segmentation set on the semantic segmentation model, and adjust the semantic segmentation model through the pre-training parameters. The pre-training uses an adversarial loss function in conjunction with the training. The semantic segmentation model includes an encoder and a decoder. The encoder consists of a feature extractor MobileNet-V2 and a DRA module. Step 4: Input the original samples in the training set into the semantic segmentation model that has been pre-trained and adjusted in Step 4 for training, and obtain a landscape painting semantic segmentation model. Step 3, which involves pre-training the natural image semantic segmentation set on the semantic segmentation model, specifically includes: S301, Input the natural image M into the semantic segmentation model, perform 4x downsampling on the natural image M to obtain image p, perform 16x downsampling on the natural image M to obtain image p3, and use the feature extractor MobileNet-V2 to extract features from the natural image M to obtain primary feature maps p1 and p2, and save the weight and bias parameter information of the convolutional layer in the feature extractor MobileNet-V2; S302, the primary feature map p1 is iteratively processed by a convolutional network to form a secondary feature map p'1; the primary feature map p2 is input into CBAM for feature detail attention discrimination and then input into ASPP. ASPP performs convolutional iteration on the primary feature map p2 and outputs feature map p'2. The ASPP performs convolutional iteration on the primary feature map p2 using the weight and bias parameter information of the convolutional layer saved in S301. S303, concatenate feature map p'2 with image p3, and perform convolution iteration on the concatenated feature map to form feature map p'23; S304, Upsample the feature map p'23, and then concatenate the upsampled feature map p'23 with the secondary feature map p'1 and the image p to obtain the image T; S305, input image T into the decoder. The decoder first performs convolution iteration on image T and then upsamples it to output the semantic segmentation image of image T. Repeat S301 to S304 to complete the processing of all natural images M. Integrate the convolution iteration parameters in S302, S303 and S305 and retain the parameters with the highest running frequency to complete the pre-training of the semantic segmentation model. In step 3, the DRA module includes the adjusted ASPP and the residual block. Specifically, the adjusted ASPP is: adding a spatial channel attention mechanism module CBAM before the ASPP, and adjusting the expansion rate of each layer of the original ASPP, adjusting the expansion rate of the second layer to 4 times, the expansion rate of the third layer to 6 times, the expansion rate of the fourth layer to 8 times, and keeping the expansion rates of the first and fifth layers unchanged. The residual block includes a downsampled residual block with cross-layer skip connections and convolution operations, and a direct residual block that provides information to the decoder.

2. The method for constructing a semantic segmentation model for landscape paintings according to claim 1, characterized in that, The grayscale values ​​around the images in the reference set of step 1 are filled to make all images in the reference set the same size.

3. The method for constructing a semantic segmentation model of landscape painting according to claim 1, characterized in that, The sample enhancement in step 2 includes processing the samples using at least one transformation operation: image distortion, image blurring, image tone transformation, image brightness transformation, and image grayscale transformation.

4. The method for constructing a semantic segmentation model of landscape painting according to claim 1, characterized in that, In step 3, adjusting the semantic segmentation model using pre-trained parameters specifically involves loading CBAM and ASPP with the convolution iteration parameters retained in S305, so that the weights and bias parameters of CBAM and ASPP are initialized according to a normal distribution.

5. The method for constructing a semantic segmentation model for landscape paintings according to claim 1, characterized in that, Step 3, which involves using an adversarial loss function in conjunction with training, specifically includes: using the Focal loss function and the Dice loss function, and combining the Focal loss function and the Dice loss function with weights to form an adversarial loss function, as shown in the following formula: ； in, These are the convolution iteration parameters for the semantic segmentation model. For the input image, Given the input set of images, The training set consists of each input image and its accompanying annotations. For the desired output value, The output value predicted by the semantic segmentation model. This is a fixed parameter value, ranging from 0 to 1.

6. The method for constructing a semantic segmentation model for landscape paintings according to claim 1, characterized in that, Step 4, which involves inputting the original samples from the training set into the semantic segmentation model that has been pre-trained and adjusted in step 3, specifically includes: S401, input the original sample I in the training set into the semantic segmentation model, and obtain image f by downsampling the original sample I by 4 times, obtain image f3 by downsampling the original sample I by 16 times, and obtain primary feature maps f1 and f2 by using the feature extractor MobileNet-V2 to extract features from the original sample I. S402, the primary feature map f1 is iteratively processed by a convolutional network to form a secondary feature map f'1; the primary feature map f2 is input into CBAM for feature detail attention discrimination and then input into ASPP. ASPP convolves and iterates on the primary feature map f2 and outputs feature map f'2. The feature map f'2 is concatenated with the image f3, and the concatenated feature map is then subjected to convolution iteration to form the feature map f'23; Upsample the feature map f'23, and then concatenate the upsampled feature map f'23 with the secondary feature map f'1 and the image f to obtain the image F; S403, input image F into the decoder to obtain the semantic segmentation image of image F. Compare the semantic segmentation image with the corresponding semantic segmentation labeled image in the training set. When the similarity between the semantic segmentation image and the semantic segmentation labeled image is not less than 90%, retain the convolution iteration parameters of the encoder and decoder when processing the original sample I, and based on the... The convolution iteration parameters are used to reload the encoder and decoder; S404, repeating S401-S403, processes all original samples in the training set to make the convolution iteration parameters of the semantic segmentation model converge and fix, thus obtaining a landscape painting semantic segmentation model.

Images