Semantic segmentation model construction method and device, electronic equipment and storage medium

Abstract

The present disclosure provides a semantic segmentation model construction method and device, electronic equipment and storage medium, the method comprises: acquiring a first data set and a second data set; the first data set refers to a training set without carrying label information, and the second data set refers to a training set carrying labels; based on the first data set, a first pre-training model is constructed by using a self-supervised contrast learning framework; based on the first pre-training model and the second data set, a first semantic segmentation model is constructed.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of image segmentation, and particularly relates to a semantic segmentation model construction method and device, electronic equipment and storage medium. BACKGROUND

[0002] Semantic segmentation refers to a process of associating each pixel in an image to a class label using machine vision algorithms. In the related art, a semantic segmentation model can be used to implement semantic segmentation of an image. However, when constructing a semantic segmentation model, a large number of samples with label information are usually required to train the model, and the samples are difficult to obtain. SUMMARY

[0003] The present disclosure provides a semantic segmentation model construction method and device, electronic equipment and storage medium to solve the problems in the related art.

[0004] A first aspect embodiment of the present disclosure provides a semantic segmentation model construction method, which comprises:

[0005] obtaining a first data set and a second data set; the first data set refers to a training set without label information, and the second data set refers to a training set with labels;

[0006] constructing a first pre-training model based on the first data set using a self-supervised contrast learning framework;

[0007] constructing a first semantic segmentation model based on the first pre-training model and the second data set.

[0008] In some embodiments of the present disclosure, the constructing a pre-training model based on the first data set using a self-supervised contrast learning method comprises:

[0009] randomly enhancing each image sample in the first data set;

[0010] extracting local features of each image sample and the randomly enhanced image sample using a backbone network of the self-supervised contrast learning framework;

[0011] constructing a loss function with the goal of maximizing the similarity of positive samples and benchmark image samples and minimizing the similarity of negative samples and benchmark image samples; the positive samples refer to local features belonging to the benchmark image samples, the negative samples refer to local features not belonging to the benchmark image samples, and the benchmark image samples refer to any image sample in the first data set;

[0012] training the self-supervised contrast learning framework based on the loss function to obtain the first pre-training model.

[0013] In some embodiments of the present disclosure, the first semantic segmentation model is constructed based on the first pre-trained model and the second data set, including:

[0014] The first feature extraction model is constructed by using the backbone network of the first pre-trained model as an input layer of the first feature extraction model.

[0015] The first semantic segmentation model is constructed by using the first feature extraction model as an input layer of the first semantic segmentation model.

[0016] In some embodiments of the present disclosure, the first feature extraction model includes:

[0017] a first input layer, a plurality of first convolutional layers, a plurality of first normalization layers, a plurality of first pooling layers, and a first fully connected layer.

[0018] The first input layer includes the backbone network of the first pre-trained model.

[0019] The plurality of first convolutional layers, the plurality of first normalization layers, and the plurality of first pooling layers are sequentially and circularly connected in the order of the first convolutional layers, the first normalization layers, and the first pooling layers.

[0020] The first fully connected layer is connected to the last first pooling layer in the plurality of first pooling layers.

[0021] In some embodiments of the present disclosure, the first pooling layer includes a maximum pooling layer and an average pooling layer.

[0022] In some embodiments of the present disclosure, the first semantic segmentation model is constructed by using the first feature extraction model as an output layer of the first semantic segmentation model, including:

[0023] The second semantic segmentation model is constructed by using the first feature extraction model as an input layer of the second semantic segmentation model.

[0024] The second semantic segmentation model is trained by using the second data set, and the first semantic segmentation model is obtained.

[0025] In some embodiments of the present disclosure, the second semantic segmentation model includes a second input layer, an encoding layer, a decoding layer, and an output layer.

[0026] The encoding layer includes at least two second convolutional layers, a same number of first up-sampling layers as the second convolutional layers, and a second fully connected layer, each of the at least two second convolutional layers has a different step length, the first up-sampling layers are used for up-sampling image samples processed by the second convolutional layers, and the second fully connected layer is used for splicing image samples processed by each first up-sampling layer to obtain a high-level feature matrix.

[0027] In some embodiments of the present disclosure, the decoding layer includes a concatenation layer and a second up-sampling layer.

[0028] The concatenation layer is configured to concatenate image samples input by the second input layer and the high-level feature matrix.

[0029] In some embodiments of the present disclosure, the training of the second semantic segmentation model using the second data set and the obtaining of the first semantic segmentation model include:

[0030] The second data set is divided into a second training set, a second validation set and a second test set according to a preset sample ratio, the second semantic segmentation model is trained using the second training set, the second validation set and the second test set, and the first semantic segmentation model is obtained.

[0031] The second aspect embodiment of the present disclosure provides a semantic segmentation model construction device, which includes:

[0032] A first obtaining unit is configured to obtain a first data set and a second data set; the first data set refers to a training set without label information, and the second data set refers to a training set with labels.

[0033] A first construction unit is configured to construct a first pre-training model based on the first data set and using a self-supervised contrast learning framework.

[0034] A second construction unit is configured to construct a first semantic segmentation model based on the first pre-training model and the second data set.

[0035] The third aspect embodiment of the present disclosure provides an electronic device, which includes:

[0036] At least one processor, and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect embodiment of the present disclosure.

[0037] The fourth aspect embodiment of the present disclosure provides a non-transitory computer readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to perform the method described in the first aspect embodiment of the present disclosure.

[0038] In summary, the present disclosure provides a semantic segmentation model construction method and device, an electronic device, and a storage medium. The method comprises: obtaining a first data set and a second data set; the first data set refers to a training set without label information, and the second data set refers to a training set with labels; constructing a first pre-training model based on the first data set using a self-supervised contrast learning framework; and constructing a first semantic segmentation model based on the first pre-training model and the second data set.

[0039] The semantic segmentation model construction method provided by the embodiments of the present disclosure fully utilizes the first data set without label information, solves the problem of insufficient samples of the second data set with label information. The first pre-training model is constructed using the self-supervised contrast learning framework, which can directly learn the feature information of the samples in the first data set, and thus the first semantic segmentation model only needs to include a small amount of first data set with samples to achieve high-accuracy segmentation effect.

[0040] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0041] The accompanying drawings incorporated in the specification and forming a part thereof illustrate embodiments consistent with the present disclosure and together with the description, serve to explain the principles of the present disclosure, and do not limit the present disclosure.

[0042] Figure 1 A flowchart of the semantic segmentation model construction method provided by the embodiments of the present disclosure is shown in the following figure;

[0043] Figure 2 A flowchart of the method for constructing the first pre-training model provided by the embodiments of the present disclosure is shown in the following figure;

[0044] Figure 3 A flowchart of the semantic segmentation model construction method provided by the application example of the present disclosure is shown in the following figure;

[0045] Figure 4 An architecture diagram of the first feature extraction model provided by the application example of the present disclosure is shown in the following figure;

[0046] Figure 5 An architecture diagram of the first semantic segmentation model provided by the application example of the present disclosure is shown in the following figure;

[0047] Figure 6 A result diagram of the medical image segmented by the first semantic segmentation model in the application example of the present disclosure is shown in the following figure;

[0048] Figure 7 A structure diagram of the semantic segmentation model construction device provided by the embodiments of the present disclosure is shown in the following figure;

[0049] Figure 8 A schematic diagram of a hardware composition structure of an electronic device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0050] Embodiments of the present disclosure are described in detail below with reference to the accompanying drawings, examples of which are shown in the drawings, wherein the same or similar notations represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by reference to the drawings are exemplary and are intended to explain the present disclosure, and cannot be understood as a limitation of the present disclosure.

[0051] Semantic segmentation refers to a process of associating each pixel in an image to a class label using machine vision algorithms. The class label can be a person, a car, a flower, a piece of furniture, etc. At present, semantic segmentation algorithms have been applied to many fields, such as autonomous driving, face detection, agricultural soil detection, medical image segmentation, etc. In the related art, a semantic segmentation model can be used to implement semantic segmentation of an image. However, when constructing a semantic segmentation model, a large number of samples with label information are usually required to train the model, and the sample acquisition is difficult.

[0052] The following briefly introduces several semantic segmentation schemes in the related art:

[0053] Scheme A: Scheme A provides a medical image segmentation method based on deep learning. The method is based on a semantic segmentation model U-Net, and fuses a multi-scale framework, a dense convolutional network, an attention mechanism, a pyramid model, small sample enhancement and other technologies, which helps to realize feature reuse, recover lost context information, suppress the response of irrelevant areas, improve the performance of the region of interest, and solve the pain point problems of ultrasound image sample shortage, low pixel, fuzzy boundary, large difference, etc.

[0054] Scheme B: Scheme B provides a dental tooth image segmentation method based on artificial intelligence. The method combines a deep convolutional model and a digital oral image together, uses the deep convolutional model to predict the boundary box of the tooth region, can realize accurate division of different tooth tissues, provides a reference for clinical oral physicians in disease diagnosis and treatment, and obtains more fine tooth classification and discrimination.

[0055] Scheme C: Scheme C provides a lung CT image segmentation algorithm. The method constructs an improved pyramid scene parsing network (PSPNet), uses a lightweight network Mobilenet series MobilenetV3 algorithm to extract feature information, then introduces a core module pyramid pooling module to improve the ability to extract global information, and finally performs upsampling and other operations to make the classified image more close to the real outline of the target.

[0056] However, in the above schemes, whether it is the semantic segmentation model of scheme A, the deep convolutional model of scheme B, or PSPNet of scheme C, all need to use a labeled dataset to train the model in order to obtain the optimal model parameters and thus obtain better image segmentation results.

[0057] However, the label information of samples in the dataset usually needs to be manually labeled. Therefore, when building an image semantic segmentation model, the method of training the model with a large number of labeled samples is difficult to obtain, which increases the difficulty and cost of building the model.

[0058] To address the shortcomings in related technologies, this disclosure provides a method for constructing a semantic segmentation model.

[0059] The present disclosure will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0060] like Figure 1 As shown, Figure 1 This is a flowchart illustrating the semantic segmentation model construction method provided in this embodiment of the disclosure. The semantic segmentation model construction method provided in this embodiment of the disclosure includes the following steps:

[0061] Step 101: Obtain the first dataset and the second dataset; the first dataset refers to the training set without label information, and the second dataset refers to the training set with labels.

[0062] In one embodiment, the first dataset can be crawled from the web. Specifically, a first dataset without label information can be randomly obtained from a publicly available dataset.

[0063] In one embodiment, the second dataset can be obtained in the same way as the first dataset, and then a deep learning annotation tool can be used to annotate the samples that do not carry label information to obtain the second dataset.

[0064] In one embodiment, the first dataset and the second dataset can be obtained from the same publicly available dataset or from different publicly available datasets.

[0065] Step 102: Based on the first dataset, construct a first pre-trained model using a self-supervised contrastive learning framework;

[0066] In one embodiment, the self-supervised contrastive learning framework can be the MoCo framework, the SimCLR framework, the BYOL framework, or the Dino framework, and this disclosure does not limit this.

[0067] Step 103, constructing a first semantic segmentation model based on the first pre-training model and the second data set.

[0068] In an embodiment, the first semantic segmentation model can directly load the weights of the first pre-training model.

[0069] The semantic segmentation model construction method provided by the embodiments of the present disclosure comprises: obtaining a first data set and a second data set; the first data set refers to a training set without label information, and the second data set refers to a training set with labels; constructing a first pre-training model based on the first data set by using a self-supervised contrast learning framework; and constructing a first semantic segmentation model based on the first pre-training model and the second data set.

[0070] The semantic segmentation model construction method provided by the embodiments of the present disclosure fully utilizes the first data set without label information, solves the problem of insufficient samples of the second data set with label information, and enables the first semantic segmentation model to achieve high-accuracy segmentation effect only by containing a small amount of first data set.

[0071] In an embodiment, when the first pre-training model is constructed by using the self-supervised contrast learning method, the self-supervised contrast learning framework is further trained by using the samples in the first data set to ensure the effect of feature learning.

[0072] Based on this, in an embodiment, as shown in Figure 2 The step 102 comprises:

[0073] Step 201, randomly enhancing each image sample in the first data set;

[0074] In an embodiment, the random enhancement at least comprises one of the following methods:

[0075] Contrast enhancement, brightness enhancement, saturation enhancement, random rotation, random cropping, random scaling.

[0076] In an embodiment, for example, the initial size of the image sample is 800*800, which can be cropped to 512*512 or 400*400.

[0077] Step 202, extracting local features of each image sample and each image sample after random enhancement by using the backbone network of the self-supervised contrast learning framework;

[0078] In an embodiment, the backbone network can be a residual network (ResNet), a visual geometry group network (VGGNet), or the like deep learning network.

[0079] Step 203: Constructing a loss function with the goal of maximizing the similarity between positive samples and reference image samples and minimizing the similarity between negative samples and reference image samples; the positive samples refer to local features belonging to the reference image samples, the negative samples refer to local features not belonging to the reference image samples, and the reference image samples refer to any image sample in the first data set;

[0080] In an embodiment, the maximization of the similarity between positive samples and reference image samples and the minimization of the similarity between negative samples and reference image samples can be expressed as a mathematical expression as follows:

[0081] Sim(f(x),f(x + ))>>Sim(f(x),f(x - ));

[0082] Wherein, x refers to a reference image sample, x + refers to a positive sample having similar characteristics with x, x - refers to a negative sample not having similar characteristics with x, Sim() is a metric function for measuring the similarity of sample characteristics, Sim() generally uses Euclidean distance or cosine similarity, f() refers to an encoder of the self-supervised contrast learning framework, i.e., a backbone network, and the result of f() is expressed in the form of a matrix.

[0083] In an embodiment, the mathematical expression of the loss function L simCV is as follows:

[0084]

[0085] Wherein, the loss function L SimCV fuses the ideas of the classifier and cross-entropy, and is more suitable for the goal of pixel-level different view contrast learning and more suitable for downstream semantic segmentation tasks.

[0086] Step 204: Training the self-supervised contrast learning framework based on the loss function to obtain the first pre-training model.

[0087] In the above embodiment, based on the contrast learning algorithm, by learning the similar or dissimilar representations of two different samples in the first training set, it is ensured that similar samples are as close to each other as possible and dissimilar samples are as far away from each other as possible after being extracted by the backbone network, and finally the first pre-training model beneficial to the downstream semantic segmentation task is obtained.

[0088] In an embodiment, due to the semantic segmentation model in the related art, there are often problems of excessive parameters and low segmentation accuracy. For example, the semantic segmentation model based on the VGGNet has excessive parameters, resulting in slow training speed and consuming more computing resources. Although the lightweight semantic segmentation model has fast training speed, the segmentation accuracy is sacrificed, resulting in low segmentation accuracy.

[0089] Therefore, the first semantic segmentation model is constructed based on the first pre-training model and the second data set, which includes:

[0090] The backbone network of the first pre-training model is used as the input layer of the first feature extraction model to construct the first feature extraction model.

[0091] The first feature extraction model is used as the input layer of the first semantic segmentation model to construct the first semantic segmentation model.

[0092] Since the input layer of the first feature extraction model is the backbone network of the first pre-training model, the first feature extraction model only needs a small number of samples for training to achieve good feature extraction effect. Not only effectively shortens the training time of the model, but also effectively makes up for the problem of insufficient samples carrying label information.

[0093] In an embodiment, although the backbone network of the first pre-training model is used as the input layer of the first feature extraction model to speed up the training speed of the first feature extraction model and reduce the sample quantity of the first feature extraction model. However, in order to obtain better semantic segmentation effect, it is also necessary to improve the correctness of the feature extraction of the first feature extraction model.

[0094] Therefore, in an embodiment, the first feature extraction model includes:

[0095] a first input layer, a plurality of first convolutional layers, a plurality of first standard layers, a plurality of first pooling layers, and a first fully connected layer;

[0096] The first input layer includes the backbone network of the first pre-training model.

[0097] In an embodiment, the first feature extraction model can directly load the weights of the backbone network of the first pre-training model.

[0098] The plurality of first convolutional layers, the plurality of first normalization layers and the plurality of first pooling layers are sequentially and circularly connected in the order of the first convolutional layers, the first normalization layers and the first pooling layers.

[0099] In an embodiment, the first convolutional layers and the first normalization layers both adopt a residual network structure. Specifically, the first convolutional layers and the first normalization layers both include a first connection layer, which is used to splice the input information and the output information of the first convolutional layers or the first normalization layers to make up for the semantic information lost in the feature extraction process.

[0100] The first connection layer can be represented by the following mathematical expression:

[0101]

[0102] Wherein, y represents the output information of the first convolutional layers or the first normalization layers, F() represents an activation function ReLU, W i represents the weight of the i-th layer (the serial number of the first convolutional layer or the first normalization layer), W 1×1 represents the weight of the convolutional layer with a kernel size of 1*1 in the first convolutional layer or the first normalization layer.

[0103] The use of the first connection layer helps to solve the problems of gradient disappearance and gradient explosion in the deep learning network.

[0104] In an embodiment, the first convolutional layers or the first normalization layers can be composed of three or other number of convolutional layers with different sizes of kernel and different number of filters.

[0105] The first fully connected layer connects the last one of the plurality of first pooling layers.

[0106] In an embodiment, in order to obtain better feature extraction effect, not only the global features of the input image should be focused on, but also the local features of the input image should be focused on.

[0107] Based on this, in an embodiment, the first pooling layer includes a max pooling layer and an average pooling layer.

[0108] Since the max pooling layer pays more attention to important local features, and the average pooling layer pays more attention to global features of the image, therefore, in the embodiment of the present disclosure, the way of simultaneously using the max pooling layer and the average pooling layer can help the first feature extraction model to reduce unnecessary redundant parameters, and further improve the speed and accuracy of downstream semantic segmentation.

[0109] In an embodiment, constructing the first semantic segmentation model by using the first feature extraction model as the output layer of the first semantic segmentation model includes:

[0110] constructing a second semantic segmentation model by taking the first feature extraction model as an input layer of the second semantic segmentation model;

[0111] training the second semantic segmentation model by using the second data set, and obtaining the first semantic segmentation model.

[0112] In an embodiment, when encoding the image, different stride convolution layers can be used to encode the image, so as to more comprehensively extract multi-scale feature information, global features and local features in the image.

[0113] Based on this, in an embodiment, the second semantic segmentation model comprises a second input layer, an encoding layer, a decoding layer and an output layer.

[0114] The encoding layer comprises at least two second convolution layers, a same number of first up-sampling layers as the second convolution layers, and a second full connection layer, each of the at least two second convolution layers has a different stride, the first up-sampling layers are used to up-sample the image samples processed by the second convolution layers, and the second full connection layer is used to splice the image samples processed by each first up-sampling layer to obtain a high-level feature matrix.

[0115] The up-sampling manner of the first up-sampling layer to the image samples processed by the second convolution layers can minimize the loss of semantic information.

[0116] In an embodiment, the decoding layer comprises a splicing layer and a second up-sampling layer.

[0117] The splicing layer is used to splice the image samples input by the second input layer and the high-level feature matrix.

[0118] In an embodiment, the training of the second semantic segmentation model by using the second data set and the obtaining of the first semantic segmentation model comprise:

[0119] dividing the second data set into a second training set, a second validation set and a second test set according to a preset sample ratio, training the second semantic segmentation model by using the second training set, the second validation set and the second test set, and obtaining the first semantic segmentation model.

[0120] In an embodiment, the second semantic segmentation model can be trained on a PyTorch-based open source semantic segmentation toolbox MMSegmentation platform. Specifically, a stochastic gradient descent is used as an optimizer, an initial learning rate is set to 0.01, a cosine decay strategy is added, the maximum training round is 100, and the model with the lowest loss in the training process is selected as the first semantic segmentation model.

[0121] The semantic segmentation model construction method provided in this disclosure will be further described below with an application example.

[0122] This application example focuses on semantic segmentation of medical images, such as segmenting lesions and background in CT images. With the rapid development of medical imaging and computer technologies, medical image analysis has become an indispensable part of the medical field. In medicine, lesion extraction and disease diagnosis are inseparable from the analysis and processing of medical images. In the field of medical image processing, accurate segmentation of specific structures and tissues is crucial, as subsequent processing is performed within the segmented target region. Therefore, obtaining accurate segmented medical images is a pressing problem to be solved.

[0123] like Figure 3 As shown, the semantic segmentation model construction method provided in this application example includes the following steps:

[0124] Step 301: Obtain the first dataset and the second dataset; the first dataset refers to the training set without label information, and the second dataset refers to the training set with labels.

[0125] In this public application example, the first and second datasets may use publicly available datasets or medical image data.

[0126] Step 302: Based on the first dataset, construct a first pre-trained model using a self-supervised contrastive learning framework;

[0127] Step 303: Use the backbone network of the first pre-trained model as the input layer of the first feature extraction model to construct the first feature extraction model;

[0128] like Figure 4 As shown, Figure 4 A schematic diagram of the architecture of the first feature extraction model provided for the application example of this disclosure.

[0129] Figure 4 In this process, the first convolutional layer (Conv Block) or the first identity layer (Identity Block) can be composed of three convolutional kernels of different sizes, with the sizes of the three convolutional kernels being 1*1, 3*3, and 1*1, respectively.

[0130] Figure 4 The first pooling block includes a max pooling layer and an average pooling layer.

[0131] Figure 4In the first aspect, the first convolutional layer (Conv Block) or the first identity layer (Identity Block) further comprises a first connection layer (Add1), and the first feature extraction model further comprises a first full connection layer (Fc Block).

[0132] In step 304, the first feature extraction model is used as an input layer of the second semantic segmentation model to construct the second semantic segmentation model.

[0133] As shown in Figure 5 , Figure 5 The first semantic segmentation model is provided in the architecture schematic diagram of the application example of the present disclosure.

[0134] Figure 5 In the first aspect, the first semantic segmentation model comprises a second input layer (Atros Conv), an encoding layer, a decoding layer and an output layer.

[0135] Figure 5 In the first aspect, the encoding layer comprises four second convolutional layers, four first up-sampling layers (Upsampling) and a second full connection layer (Add2). The kernel sizes of the four second convolutional layers (Conv) are 1*1, 3*3, 3*3 and 3*3 respectively, and the step sizes of the four second convolutional layers (Conv) are 1, 6, 12 and 18 respectively.

[0136] Figure 5 In the first aspect, the decoding layer comprises a concatenation layer (Concatenate) and a second up-sampling layer. The concatenation layer (Concatenate) is used to concatenate the output information of the second full connection layer (Add2) and the input information of the second input layer (Atros Conv).

[0137] Figure 5 In the first aspect, the output layer adopts a Softmax-based classifier.

[0138] In step 305, the second data set is divided into a second training set, a second validation set and a second test set according to a preset sample ratio, the second semantic segmentation model is trained by using the second training set, the second validation set and the second test set, and the first semantic segmentation model is obtained.

[0139] As shown in Figure 6 , Figure 6 The result of the medical image segmented by the first semantic segmentation model in the application example of the present disclosure is shown in the schematic diagram. Figure 6 In the first aspect, the left side is an input medical image, and the right side is an output image segmentation result.

[0140] In summary, the semantic segmentation model construction method provided by the examples of the present disclosure can obtain a first semantic segmentation model with faster training speed and better segmentation result through the combination of the second data set without carrying label information and the first data set carrying label information.

[0141] Firstly, the problem of few medical image labels and time-consuming and laborious professional label annotation is solved, and the first pre-training model constructed using the second data set without carrying label information can accelerate the training speed of the first semantic segmentation model.

[0142] Secondly, the first pooling layer in the first feature extraction module includes a maximum pooling layer and an average pooling layer, which reduces information redundancy, improves the size invariance of the first semantic segmentation model, and prevents overfitting.

[0143] Secondly, the encoding layer of the first semantic segmentation model uses a second convolution layer with different scales to expand the feature extraction range, and the first up-sampling layer can better fuse the global information and local information of the image, ensuring the integrity of the image segmentation in the global and the accuracy of the image segmentation in the local.

[0144] Secondly, the decoding layer of the first semantic segmentation model uses a second up-sampling layer, which can reduce feature loss to the greatest extent, helping to improve the robustness of the first semantic segmentation model and the accuracy of image segmentation.

[0145] To implement the semantic segmentation model construction method provided by the embodiments of the present disclosure, the embodiments of the present disclosure further provide a semantic segmentation model construction device, as shown in Figure 7 The structure of the semantic segmentation model construction device provided by the embodiments of the present disclosure is shown in the figure. Figure 7 The structure of the semantic segmentation model construction device provided by the embodiments of the present disclosure is shown in the figure.

[0146] The first acquisition unit 701 is configured to acquire a first data set and a second data set; the first data set refers to a training set without carrying label information, and the second data set refers to a training set carrying label information;

[0147] The first construction unit 702 is configured to construct a first pre-training model based on the first data set and using a self-supervised contrast learning framework.

[0148] The second construction unit 703 is configured to construct a first semantic segmentation model based on the first pre-training model and the second data set.

[0149] In an embodiment, the first construction unit 702 is specifically configured to:

[0150] randomly enhance each image sample in the first data set;

[0151] extract local features of each image sample and each image sample after random augmentation by using a backbone network of the self-supervised contrast learning framework;

[0152] construct a loss function aiming to maximize similarity between positive samples and the reference image sample and minimize similarity between negative samples and the reference image sample, wherein the positive samples refer to local features belonging to the reference image sample, the negative samples refer to local features not belonging to the reference image sample, and the reference image sample refers to any image sample in the first data set;

[0153] train the self-supervised contrast learning framework based on the loss function to obtain the first pre-training model.

[0154] In an embodiment, the second construction unit 703 is specifically configured to:

[0155] construct a first feature extraction model by using a backbone network of the first pre-training model as an input layer of the first feature extraction model;

[0156] construct a first semantic segmentation model by using the first feature extraction model as an input layer of the first semantic segmentation model.

[0157] In an embodiment, the first feature extraction model comprises:

[0158] a first input layer, a plurality of first convolutional layers, a plurality of first normalization layers, a plurality of first pooling layers, and a first fully connected layer;

[0159] The first input layer comprises a backbone network of the first pre-training model.

[0160] The plurality of first convolutional layers, the plurality of first normalization layers, and the plurality of first pooling layers are sequentially and circularly connected in the order of first convolutional layers, first normalization layers, and first pooling layers.

[0161] The first fully connected layer is connected to a last first pooling layer in the plurality of first pooling layers.

[0162] In an embodiment, the first pooling layer comprises a max-pooling layer and an average-pooling layer.

[0163] In an embodiment, the second construction unit 703 is specifically configured to:

[0164] construct a second semantic segmentation model by using the first feature extraction model as an input layer of the second semantic segmentation model;

[0165] train the second semantic segmentation model by using the second data set to obtain the first semantic segmentation model.

[0166] In an embodiment, the second semantic segmentation model comprises a second input layer, an encoding layer, a decoding layer and an output layer.

[0167] The encoding layer comprises at least two second convolution layers, a same number of first up-sampling layers as the number of the second convolution layers, and a second full connection layer, each of the at least two second convolution layers has a different step length, the first up-sampling layers are used to up-sample the image samples processed by the second convolution layers, and the second full connection layer is used to splice the image samples processed by each first up-sampling layer to obtain a high-level feature matrix.

[0168] In an embodiment, the decoding layer comprises a splicing layer and a second up-sampling layer.

[0169] The splicing layer is used to splice the image samples input by the second input layer and the high-level feature matrix.

[0170] In an embodiment, the second construction unit 703 is further configured to:

[0171] divide the second data set into a second training set, a second validation set and a second test set according to a preset sample ratio, train the second semantic segmentation model by using the second training set, the second validation set and the second test set, and obtain the first semantic segmentation model.

[0172] It should be noted that the semantic segmentation model construction apparatus provided in the above embodiments is only taken as an example for the division of the above program modules, and in actual application, the above processing can be completed by different program modules according to needs, that is, the internal structure of the semantic segmentation model construction apparatus is divided into different program modules to complete all or part of the above processing. In addition, the semantic segmentation model construction apparatus provided in the above embodiments and the semantic segmentation model construction method provided in the embodiments of the present disclosure belong to the same concept, and the specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0173] Figure 8 The hardware composition structure diagram of the electronic device provided in the embodiments of the present disclosure is shown in FIG. 8. Figure 8 As shown in FIG. 8, the electronic device 800 comprises at least one processor 802, and a memory 801 connected with the at least one processor 802 in communication; wherein the memory 801 stores instructions executable by the at least one processor 802, and the instructions are executed by the at least one processor 802 to implement the steps of the image segmentation method provided in the embodiments of the present disclosure.

[0174] Optionally, the electronic device can be specifically an image segmentation apparatus of the embodiments of the present application, and the electronic device can implement the corresponding processes implemented by the image segmentation apparatus in each method of the embodiments of the present application. For brevity, details are not described herein again.

[0175] It can be understood that the electronic device further includes a communication interface 803. The various components in the electronic device are coupled together through a bus system 804. It can be understood that the bus system 804 is used to realize the connection communication between the components. In addition to including a data bus, the bus system 804 also includes a power supply bus, a control bus, and a status signal bus. However, for the purpose of clear illustration, all the various buses are marked as the bus system 804 in the Figure 8

[0176] ​It can be appreciated that the memory 801 can be a volatile memory or a nonvolatile memory, and can also include both volatile and nonvolatile memory. Among them, the nonvolatile memory can be a Read Only Memory (ROM), a Programmable Read Only Memory (PROM), an Erasable Programmable Read Only Memory (EPROM), an Electrically Erasable Programmable Read Only Memory (EEPROM), a ferromagnetic random access memory (FRAM), a Flash Memory, a magnetic surface memory, an optical disc, or a Compact Disc Read Only Memory (CDROM); the magnetic surface memory can be a disk memory or a tape memory. The volatile memory can be a Random Access Memory (RAM) used as an external cache. By way of example but not limitation, many forms of RAM can be used, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDR SDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), Sync Link Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The memory 801 described in the embodiments of the present application is intended to include, but not limited to, these and any other suitable type of memory.

[0177] The method disclosed in the embodiments of the present application can be applied to the processor 802 or implemented by the processor 802. The processor 802 can be an integrated circuit chip having a processing capability of signals. In the implementation process, each step of the above method can be completed by the integrated logic circuit of hardware in the processor 802 or the instruction in the form of software. The processor 802 described above can be a general processor, a DSP, or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, etc. The processor 802 can implement or execute the disclosed methods, steps and logic block diagrams in the embodiments of the present application. The general processor can be a microprocessor or any conventional processor, etc. In combination with the steps of the method disclosed in the embodiments of the present application, the hardware decoding processor can be directly embodied to execute the above-mentioned method, or the combination of hardware and software modules in the decoding processor can be executed. The software module can be located in the storage medium, which is located in the memory 801, and the processor 802 reads the information in the memory 801 to complete the above-mentioned method in combination with the hardware thereof.

[0178] In the exemplary embodiments, the electronic device can be implemented by one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), FPGAs, general-purpose processors, controllers, MCUs, microprocessors (Microprocessors), or other electronic elements, for executing the above-mentioned method.

[0179] The embodiments also provide a non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the steps of the image segmentation method disclosed in the embodiments of the present application when the computer executes the computer instructions.

[0180] Optionally, the computer-readable storage medium can be applied to the image segmentation device in the embodiments of the present application, and the computer instructions cause the computer to execute the corresponding processes implemented by the image segmentation device in each method of the embodiments of the present application. For brevity, the details are not described here.

[0181] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other manners. The described device embodiments are merely illustrative, for example, the division of the units is only a logical function division, and there can be another division manner in actual implementation, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed coupling, or direct coupling or communication connection between the components can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0182] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, can be located in one place or distributed on a plurality of network units; some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0183] In addition, each functional unit in each embodiment of the present application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be realized in the form of hardware or in the form of hardware plus software functional unit.

[0184] Those of ordinary skill in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by a program instructing related hardware, and the foregoing program can be stored in a computer readable storage medium, and the program executes the steps of the above-mentioned method embodiments when executed; and the foregoing storage medium includes mobile storage devices, ROM, RAM, magnetic discs or optical discs, and various media that can store program codes.

[0185] Alternatively, the integrated unit of the present application, if implemented in the form of a software function module and sold or used as an independent product, can also be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present application can be embodied in the form of a software product, and the computer software product is stored in a storage medium, and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in the embodiments of the present application. The foregoing storage medium includes mobile storage devices, ROM, RAM, magnetic discs or optical discs, and various media that can store program codes.

[0186] The above merely illustrates the specific embodiments of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of the changes or replacements within the technical range disclosed by the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for constructing a semantic segmentation model, characterized in that, include: Obtain the first and second datasets; The first dataset refers to the training set without label information, and the second dataset refers to the training set with labels. Based on the first dataset, a first pre-trained model is constructed using a self-supervised contrastive learning framework. Based on the first pre-trained model and the second dataset, a first semantic segmentation model is constructed; The step of constructing a first semantic segmentation model based on the first pre-trained model and the second dataset includes: The first feature extraction model is constructed by using the backbone network of the first pre-trained model as the input layer of the first feature extraction model. The first semantic segmentation model is constructed by using the first feature extraction model as the input layer of the first semantic segmentation model; the first semantic segmentation model includes a second input layer, an encoding layer, a decoding layer and an output layer. The first feature extraction model includes: A first input layer, multiple first convolutional layers, multiple first standard layers, multiple first pooling layers, and a first fully connected layer; The first input layer includes the backbone network of the first pre-trained model; The plurality of first convolutional layers, the plurality of first standard layers, and the plurality of first pooling layers are sequentially and cyclically connected in the order of first convolutional layers, first standard layers, and first pooling layers; The first fully connected layer connects to the last of the plurality of first pooling layers.

2. The method according to claim 1, characterized in that, The step of constructing a pre-trained model based on the first dataset using self-supervised contrastive learning includes: Random augmentation is performed on each image sample in the first dataset; The backbone network of the self-supervised contrastive learning framework is used to extract the local features of each image sample and the randomly enhanced local features of each image sample. A loss function is constructed with the goal of maximizing the similarity between positive samples and benchmark image samples and minimizing the similarity between negative samples and benchmark image samples. Positive samples refer to local features belonging to benchmark image samples, and negative samples refer to local features not belonging to benchmark image samples. Benchmark image samples refer to any image sample in the first dataset. Based on the loss function, the self-supervised contrastive learning framework is trained to obtain the first pre-trained model.

3. The method according to claim 1, characterized in that, The first pooling layer includes a max pooling layer and an average pooling layer.

4. The method according to claim 3, characterized in that, The step of using the first feature extraction model as the output layer of the first semantic segmentation model to construct the first semantic segmentation model includes: The first feature extraction model is used as the input layer of the second semantic segmentation model to construct the second semantic segmentation model; The second semantic segmentation model is trained using the second dataset to obtain the first semantic segmentation model.

5. The method according to claim 4, characterized in that, The second semantic segmentation model includes a second input layer, an encoding layer, a decoding layer, and an output layer; The encoding layer includes at least two second convolutional layers, a first upsampling layer with the same number of second convolutional layers, and a second fully connected layer. Each of the at least two second convolutional layers has a different stride. The first upsampling layer is used to upsample the image samples processed by the second convolutional layer, and the second fully connected layer is used to concatenate the image samples processed by each first upsampling layer to obtain a high-level feature matrix.

6. The method according to claim 5, characterized in that, The decoding layer includes a splicing layer and a second upsampling layer; The stitching layer is used to stitch together the image samples input from the second input layer with the high-level feature matrix.

7. The method according to claim 4, characterized in that, The step of training the second semantic segmentation model using the second dataset to obtain the first semantic segmentation model includes: The second dataset is divided into a second training set, a second validation set, and a second test set according to a preset sample ratio. The second semantic segmentation model is trained using the second training set, the second validation set, and the second test set to obtain the first semantic segmentation model.

8. A semantic segmentation model construction device, characterized in that, include: The first acquisition unit is used to acquire the first dataset and the second dataset; The first dataset refers to the training set without label information, and the second dataset refers to the training set with labels. The first building unit is used to build a first pre-trained model based on the first dataset using a self-supervised contrastive learning framework. The second building unit is used to build a first semantic segmentation model based on the first pre-trained model and the second dataset; The step of constructing a first semantic segmentation model based on the first pre-trained model and the second dataset includes: The first feature extraction model is constructed by using the backbone network of the first pre-trained model as the input layer of the first feature extraction model. The first semantic segmentation model is constructed by using the first feature extraction model as the input layer of the first semantic segmentation model; the first semantic segmentation model includes a second input layer, an encoding layer, a decoding layer and an output layer. The first feature extraction model includes: A first input layer, multiple first convolutional layers, multiple first standard layers, multiple first pooling layers, and a first fully connected layer; The first input layer includes the backbone network of the first pre-trained model; The plurality of first convolutional layers, the plurality of first standard layers, and the plurality of first pooling layers are sequentially and cyclically connected in the order of first convolutional layers, first standard layers, and first pooling layers; The first fully connected layer connects to the last of the plurality of first pooling layers.

9. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 7.

Images