A method and system for semantic segmentation based on image generation

Abstract

The application discloses a semantic segmentation method based on image generation, comprising the following steps: acquiring an image to be segmented, performing normalization processing on the image to be segmented, converting the image to be segmented after the normalization processing into a tensor format image, inputting the obtained tensor format image into a pre-trained semantic segmentation model to obtain a semantic segmentation result. The application can solve the technical problems that the existing semantic segmentation method based on a discriminative model cannot achieve the best precision due to insufficient knowledge and insufficient information, and thus the accuracy and generalization are poor, and the technical problem that the output channel number of a pixel-level classifier must be modified along with the increase and decrease of the total number of categories, and frequent model modification will increase the time cost.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image data processing, and more particularly, relates to a semantic segmentation method and system based on image generation. BACKGROUND

[0002] Nowadays, semantic segmentation has been increasingly widely applied in the field of computer vision, including automatic driving, robots, matting software, etc. Since each image contains rich semantic entities, how to store more rich semantic knowledge in the model becomes the key to improving the detection performance of the semantic model.

[0003] The existing semantic segmentation method is to use a discriminative model, which first uses a feature extraction network to obtain the features of the entire image, and then uses a pixel-level classifier to convert the features of each pixel position on the features into class probability values. The number of probability values is equal to the total number of classes, and finally, the cross-entropy loss function is directly used to supervise the offline training of the model.

[0004] However, the existing semantic segmentation method based on the discriminative model has some defects that cannot be ignored: first, since the pixel-level classifier in the discriminative semantic segmentation model essentially learns the features of different classes and their discriminative boundaries, it does not learn enough rich semantic information and detailed features, which makes it difficult for the existing discriminative semantic segmentation model to achieve the best mean Intersection over Union (mIoU) precision, and thus leads to poor accuracy and generalization. Second, the existing discriminative model must modify the output channel number of the pixel-level classifier as the total number of classes increases or decreases, but frequent model modification increases the time cost. SUMMARY

[0005] In view of the above defects or improvement needs of the prior art, the present application provides a semantic segmentation method and system based on image generation, which aims to solve the technical problems that the existing semantic segmentation method based on the discriminative model cannot achieve the best precision due to insufficient knowledge and insufficient information, and thus leads to poor accuracy and generalization, and the technical problem that the method must modify the output channel number of the pixel-level classifier as the total number of classes increases or decreases, and frequent model modification leads to an increase in time cost.

[0006] To achieve the above-mentioned purpose, according to one aspect of the present application, a semantic segmentation method based on image generation is provided, comprising the following steps:

[0007] (1) Obtain an image to be segmented, normalize the image to be segmented, and convert the normalized image to be segmented into a tensor format image.

[0008] (2) Input the tensor format image obtained in step (1) into a pre-trained semantic segmentation model to obtain a semantic segmentation result.

[0009] Preferably, the semantic segmentation model comprises an image feature extractor, a discrete feature sequence number classifier, a discrete feature code table, a semantic image decoder, a color-class mapping module, a class-color mapping module, and a semantic image encoder connected in sequence, wherein the class-color mapping module and the semantic image encoder are only used for offline training of the semantic segmentation model.

[0010] The image feature extractor is a Swin Transformer network, which comprises 24 Swin Transformer modules connected in sequence. The image feature extractor receives an image with an input dimension of bs×3×h×w, and outputs four feature matrices with dimensions of and wherein bs is a batch data size pre-set in the offline training process, h and w are the number of pixels of the long side and the number of pixels of the short side of the image, respectively.

[0011] Preferably, the discrete feature sequence number classifier comprises a feature aggregation layer, a feature processing layer, and a classifier module.

[0012] The feature aggregation layer is a convolution module, which receives the four feature matrices output by the Swin Transformer network as input, and outputs an aggregated feature matrix with a dimension of

[0013] The feature processing layer is composed of two Swin Transformer modules, which receives the aggregated feature matrix output by the feature aggregation layer as input, and outputs a feature matrix with a dimension of

[0014] The input of the classifier module is the feature matrix output by the feature processing layer, and the output is a probability matrix with a dimension of Each element in the probability matrix is the probability of the occurrence of 8192 discrete features in the element.

[0015] Preferably, the discrete feature code table is a collection of multiple discrete features, each discrete feature has a dimension of 128, and each discrete feature has a unique sequence number in the discrete feature code table, i.e., a discrete feature sequence number.

[0016] The input of the discrete feature code table is the probability matrix, and the output is a discrete feature matrix with a dimension of Specifically, the first step is to find the discrete feature index corresponding to the discrete feature with the highest probability in each element of the probability matrix. The indexes corresponding to all elements in the probability matrix form an index matrix. Then, based on each element in the index matrix, the corresponding discrete feature is queried in the discrete feature code table. All discrete features corresponding to all elements constitute a discrete feature matrix.

[0017] The semantic image decoder specifically adopts the DALL-E VQ-VAE model decoder, and the input is a discrete feature matrix output from a discrete feature code table, with a dimension of [missing information]. The output is the predicted semantic image, with dimensions bs×3×h×w.

[0018] The weights of the semantic image decoder are the decoder weights of the DALL-E VQ-VAE model downloaded from the network and loaded into the semantic image decoder.

[0019] The color-category mapping module is an embedding module, whose weights are a C×3 category-color array. The input to the color-category mapping module is the semantic image to be predicted, with dimensions bs×3×h×w, output from the semantic image decoder. The output is a semantic segmentation result with dimensions bs×1×h×w. Here, C represents the total number of categories, and the semantic segmentation result is the predicted semantic mask, i.e., the category corresponding to each pixel in the image. Specifically, the color-category mapping module first calculates the mapping from pixel L(i,j) in the i-th row and j-th column of the semantic image to each category c. k The absolute distance of the preset RGB colors Where k∈[1, C], where Let R, G, and B represent the R, G, and B color values ​​at pixel L(i,j) of the predicted semantic image, respectively. k G k B k Representing category c k The preset R, G, and B color values ​​are then used. Next, the category corresponding to the minimum of the C absolute distances is taken as the category k corresponding to pixel L(i,j). L =argmi k (d L，k Finally, the category k corresponding to all pixels L(i,j) in the semantic image is... L The semantic segmentation results are obtained by splicing the components according to their spatial location.

[0020] The class-color mapping module is used to obtain the color corresponding to each pixel according to the class of the pixel, and the input is the ground truth of the semantic mask with the dimension of bs×1×h×w, and the output is the ground truth of the semantic image with the dimension of bs×3×h×w. Wherein, the ground truth of the semantic segmentation mask comes from the dataset used in the offline training.

[0021] The semantic image encoder specifically adopts the encoder of the VQ-VAE model of DALL-E, and the input is the ground truth of the semantic image output by the class-color mapping module, with the dimension of bs×3×h×w, and the output is the ground truth of the equivalent feature matrix with the dimension of

[0022] Preferably, the semantic segmentation model is trained by the following steps:

[0023] (1) Obtain the ADE20K dataset, and divide 25574 images and their corresponding ground truth of the semantic mask of the ADE20K dataset into a training set, and divide 2000 images and their corresponding ground truth of the semantic mask of the ADE20K dataset into a validation set.

[0024] (2) Generate a class-color array with the dimension of C×3.

[0025] (3) Initialize the weights of the image feature extractor, the discrete feature sequence classifier, the discrete feature code table, the class-color mapping module, the color-class mapping module and the semantic image encoder to obtain the initialized image feature extractor, the discrete feature sequence classifier, the discrete feature code table, the class-color mapping module, the color-class mapping module and the semantic image encoder.

[0026] (4) Use the training set of the ADE20K dataset obtained in step (1), the initialized discrete feature code table obtained in step (3), the class-color mapping module and the semantic image encoder to train the initialized image feature extractor and the discrete feature sequence classifier obtained in step (3) to obtain the trained image feature extractor and the discrete feature sequence classifier.

[0027] (5) Save the weights of the trained image feature extractor and the discrete feature sequence classifier obtained in step (4), the initialized discrete feature code table obtained in step (3), the weights of the color-class mapping module, and the weights of the semantic image decoder to obtain the weights of the semantic segmentation model. Wherein, the weights of the semantic image decoder are the weights of the decoder of the VQ-VAE model of DALL-E downloaded from the network.

[0028] Preferably, step (2) comprises the following sub-steps:

[0029] (2-1) Generate three one-dimensional arrays A​R A G A B Preferably, each element of the array is... and And k1∈[1, array A] R The total number of elements in array A], k2∈[1, array A G The total number of elements in array A], k3∈[1, array A B The total number of elements in the [data].

[0030] (2-2) Set counters k1 = 1, k2 = 1, k3 = 1, and initialize the RGB color array A. RGB It is an empty array.

[0031] (2-3) Determine if k1 is greater than the preset maximum number of loops J (its value is equal to that of array A). R If the total number of elements in the array is 1, proceed to step (2-13); otherwise, proceed to step (2-4).

[0032] (2-4) Determine if k2 is greater than the preset maximum number of loops K (its value is equal to that of array A). G If the total number of elements in the array is equal to the total number of elements in the array, proceed to step (2-3); otherwise proceed to step (2-5).

[0033] (2-5) Determine if k3 is greater than the preset maximum number of loops Q (its value is equal to that of array A). B If the total number of elements in the array is equal to the total number of elements in the array, proceed to step (2-4); otherwise proceed to step (2-6).

[0034] (2-6) Generate a random integer r between -15 and 15. R and update A R The k1th element of the array To obtain the updated A R The k1th element of the array

[0035] (2-7) Generate a random integer r between -15 and 15. G and update A G The k2th element of the array To obtain the updated A G The k2th element of the array

[0036] (2-8) Generate a random integer r between -15 and 15. B and update A B The k3th element of the array To obtain the updated A Bthe k1th element of the array

[0037] the updated RGB color array A obtained in step (2-9) R the k1th element of the array the updated RGB color array A obtained in step (2-7) G the k2th element of the array the updated RGB color array A obtained in step (2-8) B the k3th element of the array a three-dimensional element and the three-dimensional element is inserted into the RGB color array A RGB at the end to obtain an updated RGB color array A RGB .

[0038] (2-10) setting k1 = k1 + 1, and returning to step (2-5).

[0039] (2-11) setting k2 = k2 + 1, and returning to step (2-4).

[0040] (2-12) setting k3 = k3 + 1, and returning to step (2-3).

[0041] (2-13) obtaining the first C three-dimensional elements in the updated RGB color array A RGB obtained in step (2-9) to obtain a category color array.

[0042] Preferably, step (3) comprises the following sub-steps:

[0043] (3-1) loading the pre-trained weights of the image feature extractor to the image feature extractor.

[0044] (3-2) initializing the weights of the discrete feature sequence number classifier to random values, and setting the weights of the discrete feature sequence number classifier to gradient-enabled weights.

[0045] (3-3) loading the preset category color array obtained in step (2) into the color-category mapping module, and setting the weights of the color-category mapping module to gradient-disabled weights.

[0046] (3-4) loading the preset category color array obtained in step (2) into the category-color mapping module, and setting the weights of the category-color mapping module to gradient-disabled weights.

[0047] (3-5) loading the pre-trained weights of the semantic image encoder to the semantic image encoder, and setting the weights of the semantic image encoder to gradient-disabled weights.

[0048] Step (4) comprises the following sub-steps:

[0049] (4-1) Set a counter i = 1 and initialize the hyperparameters of the training process to obtain the initialized hyperparameters of the training process.

[0050] (4-2) Obtain a plurality of images and their corresponding semantic mask ground truths from the training set of the ADE20K dataset obtained in step (1).

[0051] (4-3) Perform data preprocessing on the plurality of images and semantic mask ground truths obtained in step (4-2) to obtain a plurality of preprocessed images and semantic mask ground truths.

[0052] (4-4) Map the plurality of preprocessed semantic mask ground truths obtained in step (4-3) to a plurality of discrete feature sequence number matrix ground truths using the class-color mapping module, the semantic image encoder, and the discrete feature code table.

[0053] (4-5) Map the plurality of preprocessed images obtained in step (4-3) to a plurality of discrete feature sequence number probability matrices using the image feature extractor and the discrete feature sequence number classifier connected in sequence, and the dimension of the probability matrix is

[0054] (4-6) Input the plurality of discrete feature sequence number probability matrices obtained in step (4-5) and the plurality of discrete feature sequence number matrix ground truths obtained in step (4-4) into the cross-entropy loss function to obtain a semantic feature loss value.

[0055] (4-7) Perform backpropagation using the semantic feature loss value obtained in step (4-6) to obtain the gradients of the image feature extractor and the discrete feature sequence number classifier.

[0056] (4-8) Update the weights of the image feature extractor and the discrete feature sequence number classifier using the learning rate set in step (4-1), the AdamW optimizer, and the weight gradients obtained in step (4-7) to obtain new weights of the image feature extractor and the discrete feature sequence number classifier.

[0057] (4-9) Set the counter i = i + 1 and set the gradients of the image feature extractor and the discrete feature sequence number classifier to 0.

[0058] (4-10) Determine whether i is greater than the maximum number of iterations n set in advance, if yes, the process ends, otherwise return to step (4-2).

[0059] Preferably, step (4-4) comprises the following sub-steps:

[0060] (4-4-1) input the plurality of data pre-processed semantic mask true values obtained in step (4-3) into a category-color mapping module to obtain a plurality of semantic image true values.

[0061] (4-4-2) input the plurality of semantic image true values obtained in step (4-4-1) into the initialized semantic image encoder obtained in step (3) to obtain a plurality of equivalent feature matrix true values.

[0062] (4-4-3) query the true value of each equivalent feature from the initialized discrete feature code table obtained in step (3) according to the plurality of equivalent feature matrix true values obtained in step (4-4-2), and obtain the discrete feature sequence number matrix true value by splicing all the queried discrete feature sequence numbers according to the spatial position, wherein the dimension of the discrete feature sequence number matrix true value is

[0063] Step (4-5) includes the following sub-steps:

[0064] (4-5-1) input each pre-processed image obtained in step (4-3) into an image feature extractor to obtain four different spatial scales and channel numbers of image features corresponding to each image.

[0065] (4-5-2) input the four different spatial scales and channel numbers of image features corresponding to each image obtained in step (4-5-1) into a discrete feature sequence number classifier to obtain a probability matrix of each discrete feature sequence number.

[0066] According to another aspect of the present application, a semantic segmentation system based on image generation is provided, comprising:

[0067] The first module is configured to acquire an image to be segmented, perform normalization processing on the image to be segmented, and convert the normalized image to be segmented into a tensor format image.

[0068] The second module is configured to input the tensor format image obtained by the first module into a pre-trained semantic segmentation model to obtain a semantic segmentation result.

[0069] Overall, the above technical solutions conceived by the present application can achieve the following beneficial effects compared with the prior art:

[0070] (1) Since the application adopts a discrete feature sequence number classifier, a discrete feature code table, a semantic image decoder and a color-category mapping module. It models semantic segmentation as image generation, effectively utilizing the advantage that generative models can contain more rich semantic information. Therefore, the application can solve the technical problem that the pixel-level classifier in the existing discriminative semantic segmentation model cannot learn enough rich semantic information and detailed features, resulting in the existing discriminative semantic segmentation model being difficult to achieve the best mIoU precision, and further leading to poor accuracy and generalization.

[0071] (2) Since the application adopts a color-category mapping module and a category-color mapping module. It presets colors for categories. When performing pixel-level classification, the application does not need the pixel-level classifier used by the existing discriminative semantic segmentation, but only needs to calculate the distance between the color of the semantic image predicted by the model and the preset category color to predict the pixel category. Therefore, it can solve the technical problem of modifying the output channel number of the pixel-level classifier due to the increase or decrease of the number of categories in the existing discriminative semantic segmentation model, avoid frequent model modification, and reduce the labor and time cost of business landing BRIEF DESCRIPTION OF DRAWINGS

[0072] Figure 1 is a flowchart of the semantic segmentation method based on image generation of the application.

[0073] Figure 2 is a structural schematic diagram of the semantic segmentation model adopted by the application. DETAILED DESCRIPTION

[0074] In order to make the purpose, technical scheme and advantages of the application clearer, the application will be further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the application and do not limit the application. In addition, the technical features involved in each embodiment of the application described below can be combined with each other as long as they do not conflict with each other.

[0075] As shown in Figure 1 , the application provides a semantic segmentation method based on image generation, comprising the following steps:

[0076] (1) obtaining an image to be segmented, normalizing the image to be segmented, and converting the normalized image to be segmented into a tensor format image.

[0077] (2) inputting the tensor format image obtained in step (1) into a pre-trained semantic segmentation model to obtain a semantic segmentation result.

[0078] As shown in Figure 2As shown, the semantic segmentation model of the present application comprises an image feature extractor, a discrete feature serial number classifier, a discrete feature code table, a semantic image decoder, a color-class mapping module, a class-color mapping module, and a semantic image encoder connected in sequence, wherein the last two modules are only used for offline training of the semantic segmentation model.

[0079] The image feature extractor is specifically a Swin Transformer network, which comprises 24 Swin Transformer modules connected in sequence. The image feature extractor receives an image with an input dimension of bs×3×h×w, and outputs four feature matrices with dimensions of and wherein bs is a batch size preset in the offline training process, h and w are respectively the number of pixels of the long side and the number of pixels of the short side of the image.

[0080] The discrete feature serial number classifier comprises a feature aggregation layer, a feature processing layer, and a classifier module. The feature aggregation layer is a convolution module, the input of which is the four feature matrices output by the Swin Transformer network, and the output of which is an aggregated feature matrix with a dimension of The feature processing layer is composed of two layers of Swin Transformer modules, the input of which is the aggregated feature matrix output by the feature aggregation layer, and the output dimension of which is a feature matrix with a dimension of

[0081] The input of the classifier module is the feature matrix output by the feature processing layer, and the output is a probability matrix with a dimension of Each element in the probability matrix is the probability of the occurrence of 8192 discrete features in the element. The discrete features are a set of independent features stored in the discrete feature code table. Preferably, the total number of discrete features is 8192, so the number of channels of the probability matrix is 8192.

[0082] The advantage of using the discrete feature serial number classifier here is that the spatial scale of the output of the discrete feature serial number classifier is smaller, and the computational resources required are less. Specifically, the spatial scale of the discrete feature serial number classifier is which is equivalent to the spatial scale of the existing pixel-level classifier (h×w). Therefore, the technical problem of excessive prediction density of the existing discriminative semantic segmentation, resulting in large computational overhead, is solved.

[0083] The discrete feature code table is a collection of a plurality of discrete features (in this embodiment, it contains 8192 discrete features), each discrete feature has a dimension of 128, and each discrete feature has a unique serial number in the discrete feature code table, referred to as a discrete feature serial number. The input of the discrete feature code table is a probability matrix, and the output is a discrete feature matrix, which has a dimension of Specifically, first, find the serial number of the discrete feature with the maximum probability in each element of the probability matrix in the discrete feature code table (i.e., the discrete feature serial number), and the serial numbers corresponding to all elements in the probability matrix form a serial number matrix, then, according to each element in the serial number matrix, query the corresponding discrete feature in the discrete feature code table, and all discrete features corresponding to all elements form a discrete feature matrix. The weight of the discrete feature code table is the weight of the decoder of the DALL-E VQ-VAE model downloaded from the network and loaded into the discrete feature code table.

[0084] The semantic image decoder specifically adopts the decoder of the DALL-E VQ-VAE model, and the input is the discrete feature matrix output from the discrete feature code table, which has a dimension of The output is a predicted semantic image, which has a dimension of bsx3xhwxw. Specifically, the weight of the semantic image decoder is the weight of the decoder of the DALL-E VQ-VAE model downloaded from the network and loaded into the semantic image decoder.

[0085] The advantage of using the semantic image decoder here is that:

[0086] (1) The semantic image decoder maps the low spatial resolution discrete feature matrix to the high resolution semantic image, and further obtains the high resolution semantic segmentation result. The semantic image decoder contains a large amount of detailed texture information and has fine decoding ability for class boundaries. Therefore, it solves the technical problem that some existing semantic segmentation directly upsamples the low resolution pixel-level classification result to obtain a rough high resolution semantic segmentation result, which further leads to unclear class boundaries of the semantic segmentation result.

[0087] (2) The weight of the semantic image decoder is the pre-trained weight of the image generation model on large-scale unsupervised data, which performs well on image generation tasks containing boundary details and textures. However, existing discriminative semantic segmentation cannot utilize the pre-trained weight of the image generation model. Therefore, the semantic image decoder loaded with the pre-trained weight of the image generation model on large-scale unsupervised data solves the technical problem of inaccurate boundary details and textures of existing discriminative semantic segmentation.

[0088] The color-category mapping module is an Embedding module whose weights are a category-color array of dimension Cx3, and the input of the color-category mapping module is the to-be-predicted semantic image of dimension bsx3xhxw output by the semantic image decoder, and the output is a semantic segmentation result of dimension bsx1xhwx, where C represents the total number of categories.

[0089] Specifically, the semantic segmentation result is a predicted semantic mask, i.e., the category corresponding to each pixel in the image, and each category is represented by a serial number.

[0090] Specifically, the color-category mapping module first calculates the absolute distance of the pixel L(i,j) of the i-th row and j-th column of the semantic image to the preset RGB color of each category c k where k∈[1,C], where R(i,j), G(i,j), and B(i,j) respectively represent the R, G, and B color values of the pixel L(i,j) of the predicted semantic image. k k k Rc, Gc, and Bc respectively represent the preset R, G, and B color values of the category c k Then, the category corresponding to the minimum value among the C obtained absolute distances is obtained as the category k L =argmin k (d L，k ), and finally, the categories k L corresponding to all pixels L(i,j) in the semantic image are spliced according to the spatial positions to obtain the semantic segmentation result.

[0091] Specifically, the sequentially connected image feature extractor, discrete feature serial number classifier, discrete feature code table, semantic image decoder color-category mapping module are used in the online semantic segmentation process, and the purpose is to map the tensor format image to the semantic segmentation result.

[0092] The category-color mapping module is used to obtain the color corresponding to each pixel according to the category of the pixel, and the input of the category-color mapping module is the true value of the semantic mask of dimension bsx1xhwx, and the output is the true value of the semantic image of dimension bsx3xhwx. Wherein, the true value of the semantic segmentation mask comes from the dataset used in offline training.

[0093] The semantic image encoder specifically adopts the encoder of the VQ-VAE model of DALL-E, and the input of the semantic image encoder is the true value of the semantic image output by the category-color mapping module, which is of dimension bsx3xhwx, and the output is the true value of the equivalent feature matrix of dimension . ​​​

[0094] Specifically, the class-color mapping module and the semantic image encoder are auxiliary modules used only in the offline training process. The purpose is to map the true value of the semantic mask to the true value of the equivalent feature matrix, and obtain the true value of the discrete feature serial number through the similarity query processing of the discrete feature code table. The true value of the discrete feature serial number is used as a supervision signal to supervise the offline training of the image feature extractor and the discrete feature serial number classifier.

[0095] The semantic segmentation model of the present application is trained by the following steps:

[0096] (1) Obtain the ADE20K dataset, divide 25574 images of the ADE20K dataset and their corresponding true values of the semantic mask into a training set, and divide 2000 images of the ADE20K dataset and their corresponding true values of the semantic mask into a validation set.

[0097] (2) Generate a class color array with a dimension of Cx3.

[0098] Specifically, the class color array is an array of C RGB color values preset for C classes respectively.

[0099] This step includes the following sub-steps:

[0100] (2-1) Generate three one-dimensional arrays A R , A G , A B . Preferably, each element of the array is and and k1∈[1, the total number of elements in array A R ], k2∈[1, the total number of elements in array A G ], k3∈[1, the total number of elements in array A B ].

[0101] Specifically, in this example, the total number of elements in array A R , A G , A B is 5, 6, and 5 respectively.

[0102] The advantages of this step are: first, the array A R , A G , A B of each channel is an arithmetic sequence, and the elements are uniformly distributed between 0 and 255, which fully utilizes the color space to maximize the absolute distance of the class color and reduces the possibility of pixel error classification. Second, the array A R , A G , A BThe initial elements are different, and the differences in the sequences are also different, which avoids the same value for different array elements and reduces the possibility of pixel misclassification.

[0103] (2-2) Set counters k1 = 1, k2 = 1, k3 = 1, and initialize the RGB color array A. RGB It is an empty array.

[0104] (2-3) Determine if k1 is greater than the preset maximum number of loops J (its value is equal to that of array A). R If the total number of elements in the array is 1, proceed to step (2-13); otherwise, proceed to step (2-4).

[0105] (2-4) Determine if k2 is greater than the preset maximum number of loops K (its value is equal to that of array A). G If the total number of elements in the array is equal to the total number of elements in the array, proceed to step (2-3); otherwise proceed to step (2-5).

[0106] (2-5) Determine if k3 is greater than the preset maximum number of loops Q (its value is equal to that of array A). B If the total number of elements in the array is equal to the total number of elements in the array, proceed to step (2-4); otherwise proceed to step (2-6).

[0107] (2-6) Generate a random integer r between -15 and 15. R and update A R The k1th element of the array To obtain the updated A R The k1th element of the array

[0108] (2-7) Generate a random integer r between -15 and 15. G and update A G The k2th element of the array To obtain the updated A G The k2th element of the array

[0109] (2-8) Generate a random integer r between -15 and 15. B and update A B The k3th element of the array To obtain the updated A B The k3th element of the array

[0110] The advantage of steps (2-6) to (2-8) above is that each element is inserted into the RGB color array A. RGB Previously, independent random integers were added to avoid duplicate element values ​​and reduce the possibility of pixel misclassification.

[0111] (2-9) Set k1 = k1 + 1, and return to step (2-5). R the k1th element of the array the updated A obtained in step (2-7) G the k2th element of the array the updated A obtained in step (2-8) B the k3th element of the array a three-dimensional element and insert the three-dimensional element into the end of the RGB color array A RGB to obtain an updated RGB color array A RGB .

[0112] (2-10) Set k1 = k1 + 1, and return to step (2-5).

[0113] (2-11) Set k2 = k2 + 1, and return to step (2-4).

[0114] (2-12) Set k3 = k3 + 1, and return to step (2-3).

[0115] (2-13) Obtain the first C three-dimensional elements in the updated RGB color array A RGB obtained in step (2-9) to obtain a category color array.

[0116] (3) Initialize the weights of the image feature extractor, the discrete feature sequence number classifier, the discrete feature code table, the category-color mapping module, the color-category mapping module, and the semantic image encoder to obtain an initialized image feature extractor, discrete feature sequence number classifier, discrete feature code table, category-color mapping module, color-category mapping module, and semantic image encoder.

[0117] Step (3) includes the following sub-steps:

[0118] (3-1) Load the pre-trained weights of the image feature extractor into the image feature extractor.

[0119] Preferably, the backbone network structure of the large weight version (Large version) of the Swin Transformer is used as the structure of the image feature extractor. The weights of the ImageNet-22K pre-trained Swin Transformer backbone network are downloaded from the network and loaded into the image feature extractor, and the weights of the image feature extractor are set to be gradient weights (i.e., the weights of the image feature extractor will be optimized in the offline training process).

[0120] (3-2) Initialize the weights of the discrete feature number classifier to random values, and set the weights of the discrete feature number classifier to have gradients (i.e., the weights of the image feature extractor will be optimized during the offline training process).

[0121] (3-3) Load the preset category color array obtained in step (2) into the color-category mapping module, and set the weights of the color-category mapping module to have no gradients.

[0122] (3-4) Load the preset category color array obtained in step (2) into the category-color mapping module, and set the weights of the category-color mapping module to have no gradients.

[0123] (3-5) Load the pre-trained weights of the semantic image encoder into the semantic image encoder, and set the weights of the semantic image encoder to have no gradients.

[0124] Specifically, this step is to download the weights of the encoder of the VQ-VAE model of DALL-E from the network and load them into the semantic image encoder.

[0125] (4) Use the training set of the ADE20K dataset obtained in step (1), the initialized discrete feature code table obtained in step (3), the category-color mapping module, and the semantic image encoder to train the initialized image feature extractor and the discrete feature number classifier obtained in step (3) to obtain the trained image feature extractor and the discrete feature number classifier.

[0126] This step includes the following sub-steps:

[0127] (4-1) Set the counter i = 1, and initialize the hyperparameters of the training process to obtain the initialized hyperparameters of the training process.

[0128] The hyperparameters of the training process include the maximum number of iterations, the batch size bs, the learning rate, and the optimizer.

[0129] Specifically, the batch size bs is set to 32, the initial learning rate is set to 0.00001, and the optimizer is set to the AdamW optimizer.

[0130] (4-2) Obtain a plurality of images and their corresponding semantic mask ground truths from the training set of the ADE20K dataset obtained in step (1).

[0131] (4-3) Data preprocessing is performed on the plurality of images and the semantic mask ground truths obtained in step (4-2) to obtain a plurality of preprocessed images and semantic mask ground truths.

[0132] Specifically, the preprocessing procedure of this step includes, in sequence, interpolation of uniform scale, random horizontal flip, random cropping, and normalization, wherein the dimensions of the plurality of preprocessed images are bs x 3 x h x w, and the dimensions of the corresponding preprocessed semantic masks are bs x 1 x h x w, bs being the batch size set in step (4-1).

[0133] (4-4) Mapping the true values of the plurality of preprocessed semantic masks obtained in step (4-3) into true values of a plurality of discrete feature number matrices using the class-color mapping module, the semantic image encoder, and the discrete feature code table.

[0134] Specifically, this step is to map the true values of the plurality of preprocessed semantic masks with dimensions of bs x 1 x h x w into true values of a plurality of discrete feature number matrices with dimensions of .

[0135] This step includes the following sub-steps:

[0136] (4-4-1) Inputting the true values of the plurality of data preprocessed semantic masks obtained in step (4-3) into the class-color mapping module to obtain the true values of a plurality of semantic images.

[0137] (4-4-2) Inputting the true values of the plurality of semantic images obtained in step (4-4-1) into the initialized semantic image encoder obtained in step (3) to obtain the true values of a plurality of equivalent feature matrices.

[0138] (4-4-3) Querying, in the initialized discrete feature code table obtained in step (3), the discrete feature and its discrete feature number that are closest to the true value of each equivalent feature from the true values of the plurality of equivalent feature matrices obtained in step (4-4-2), and splicing all the queried discrete feature numbers according to spatial positions to obtain the true values of a discrete feature number matrix, which has dimensions of

[0139] (4-5) Mapping the plurality of preprocessed images obtained in step (4-3) into a plurality of probability matrices of discrete feature numbers using the image feature extractor and the discrete feature number classifier connected in sequence, which have dimensions of

[0140] This step includes the following sub-steps:

[0141] (4-5-1) Inputting each preprocessed image obtained in step (4-3) into the image feature extractor to obtain image features corresponding to each image at four different spatial scales and channel numbers.

[0142] (4-5-2) input the image features of each image corresponding to four different spatial scales and channel numbers obtained in step (4-5-1) into the discrete feature sequence number classifier to obtain a probability matrix of each discrete feature sequence number.

[0143] (4-6) input the probability matrix of the plurality of discrete feature sequence numbers obtained in step (4-5) and the true value of the plurality of discrete feature sequence number matrices obtained in step (4-4) into a cross-entropy loss function to obtain a semantic feature loss value.

[0144] (4-7) perform back propagation using the semantic feature loss value obtained in step (4-6) to obtain the gradient of the image feature extractor and the discrete feature sequence number classifier.

[0145] (4-8) update the weights of the image feature extractor and the discrete feature sequence number classifier using the learning rate set in step (4-1), the AdamW optimizer, and the weight gradient obtained in step (4-7) to obtain the weights of the new image feature extractor and the discrete feature sequence number classifier.

[0146] (4-9) set the counter i = i + 1 and set the gradient of the image feature extractor and the discrete feature sequence number classifier to 0.

[0147] (4-10) determine whether i is greater than the maximum iteration number n set in advance, if yes, the process ends, otherwise return to step (4-2).

[0148] Specifically, the maximum number of loops n set in advance ranges from 1000 to 1000000, and is preferably 160000.

[0149] (5) save the weights of the trained image feature extractor, the discrete feature sequence number classifier obtained in step (4), the initialized discrete feature code table obtained in step (3), the weights of the color-class mapping module, and the weights of the semantic image decoder to obtain the weights of the semantic segmentation model. The weights of the semantic image decoder are the weights of the decoder of the DALL-E VQ-VAE model downloaded from the network.

[0150] Those skilled in the art will readily understand that the above description is only the preferred embodiment of the present application and is not intended to limit the present application. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A semantic segmentation method based on image generation, characterized in that, Includes the following steps: Step 1: Obtain the image to be segmented, normalize the image to be segmented, and convert the normalized image to be segmented into a tensor format image; Step 2: Input the tensor format image obtained in Step 1 into the pre-trained semantic segmentation model to obtain the semantic segmentation result. The semantic segmentation model includes an image feature extractor, a discrete feature index classifier, a discrete feature code table, a semantic image decoder, a color-category mapping module, a category-color mapping module, and a semantic image encoder, which are connected in sequence. The category-color mapping module and the semantic image encoder are only used for offline training of the semantic segmentation model. The discrete feature code table is a collection of multiple discrete features. The dimension of the discrete features is 128. Each discrete feature has a unique index in the discrete feature code table, which is called the discrete feature index. The semantic segmentation model is trained using the following steps: (1) Obtain the ADE20K dataset, divide the 25574 sets of images and their corresponding semantic masks of the ADE20K dataset into the training set, and divide the 2000 sets of images and their corresponding semantic masks of the ADE20K dataset into the validation set. (2) Generation dimension is The category color array; step (2) includes the following sub-steps: (2-1) Generate three one-dimensional arrays , , Each element of the array is... , 40 and 45, and k1∈[1, array The total number of elements in the array], k2∈[1, array The total number of elements in the array], k3∈[1, array [Total number of elements in]; (2-2) Set counters k1=1, k2=1, k3=1, and initialize the RGB color array. An empty array; (2-3) Determine if k1 is greater than the preset maximum number of loops J, whose value is equal to the array If the total number of elements in the array is 1, proceed to step (2-13); otherwise, proceed to step (2-4). (2-4) Determine if k2 is greater than the preset maximum number of loops K, whose value is equal to the array The total number of elements in the array is determined. If the total number of elements is found, proceed to step (2-3); otherwise, proceed to step (2-5). (2-5) Determine if k3 is greater than the preset maximum number of loops Q, whose value is equal to the array The total number of elements in the array is determined. If the total number of elements is found, proceed to step (2-4); otherwise, proceed to step (2-6). (2-6) Generate a random integer between -15 and 15. and update The k1th element of the array To obtain the updated The k1th element of the array ; (2-7) Generate a random integer between -15 and 15. and update The k2th element of the array To obtain the updated The k2th element of the array ; (2-8) Generate a random integer between -15 and 15. and update The k3th element of the array To obtain the updated The k3th element of the array ; (2-9) The updated version obtained in step (2-6) The k1th element of the array The updated version obtained in steps (2-7) The k2th element of the array and the updated version obtained in steps (2-8) The k3th element of the array To form a three-dimensional element ( , , ), and three-dimensional elements ( , , Insert into the RGB color array At the end, to obtain the updated RGB color array. ; (2-10) Set k1=k1+1 and return to step (2-5); (2-11) Set k2=k2+1 and return to step (2-4); (2-12) Set k3=k3+1 and return to step (2-3); (2-13) Obtain the updated RGB color array obtained in step (2-9). The first C three-dimensional elements are used to obtain the category color array; (3) Initialize the weights of the image feature extractor, discrete feature index classifier, discrete feature code table, category-color mapping module, color-category mapping module, and semantic image encoder to obtain the initialized image feature extractor, discrete feature index classifier, discrete feature code table, category-color mapping module, color-category mapping module, and semantic image encoder; step (3) includes the following sub-steps: (3-1) Load the pre-trained weights of the image feature extractor into the image feature extractor; (3-2) Initialize the weights of the discrete feature index classifier to random values, and set the weights of the discrete feature index classifier to gradient weights; (3-3) Load the preset category color array obtained in step (2) into the color-category mapping module, and set the weight of the color-category mapping module to a weight without gradient; (3-4) Load the preset category color array obtained in step (2) into the category-color mapping module, and set the weight of the category-color mapping module to a weight without gradient; (3-5) Transfer the pre-trained weights of the semantic image encoder to the semantic image encoder, and set the weights of the semantic image encoder to gradient-free weights; (4) Using the training set of the ADE20K dataset obtained in step (1), the initialized discrete feature code table obtained in step (3), the category-color mapping module, and the semantic image encoder, train the initialized image feature extractor and discrete feature index classifier obtained in step (3) to obtain the trained image feature extractor and discrete feature index classifier; step (4) includes the following sub-steps: (4-1) Set the counter i=1 and initialize the hyperparameters of the training process to obtain the initialized hyperparameters of the training process; (4-2) Obtain the ground truth values ​​of multiple images and their corresponding semantic masks from the training set of the ADE20K dataset obtained in step (1); (4-3) Perform data preprocessing on the ground truth values ​​of multiple images and semantic masks obtained in step (4-2) to obtain multiple preprocessed ground truth values ​​of images and semantic masks; (4-4) Using the category-color mapping module, semantic image encoder, and discrete feature code table, the ground truth values ​​of the multiple preprocessed semantic masks obtained in step (4-3) are mapped to the ground truth values ​​of multiple discrete feature index matrices; (4-5) Using a sequentially connected image feature extractor and discrete feature index classifier, the multiple preprocessed images obtained in step (4-3) are mapped into a probability matrix of multiple discrete feature indices, the dimension of which is... ,in, The pre-set batch data size for offline training. and These are the number of pixels on the long side and the number of pixels on the short side of the image, respectively. (4-6) Input the probability matrix of multiple discrete feature indices obtained in step (4-5) and the true value of multiple discrete feature index matrices obtained in step (4-4) into the cross-entropy loss function to obtain the semantic feature loss value; (4-7) Backpropagation is performed using the semantic feature loss value obtained in step (4-6) to obtain the gradients of the image feature extractor and the discrete feature index classifier; (4-8) Using the learning rate set in step (4-1), the AdamW optimizer, and the weight gradient obtained in step (4-7), update the weights of the image feature extractor and the discrete feature index classifier to obtain new weights for the image feature extractor and the discrete feature index classifier. (4-9) Counter i = i + 1, and set the gradients of the image feature extractor and the discrete feature index classifier to 0; (4-10) Determine if i is greater than the preset maximum number of iterations n. If yes, the process ends; otherwise, return to step (4-2). (5) Save the weights of the trained image feature extractor and discrete feature number classifier obtained in step (4), the weights of the initialized discrete feature code table, the weights of the color-class mapping module and the weights of the semantic image decoder obtained in step (3) to obtain the weights of the semantic segmentation model; wherein, the weights of the semantic image decoder are the weights of the decoder of the DALL-E VQ-VAE model downloaded from the network.

2. The semantic segmentation method based on image generation according to claim 1, characterized in that, The image feature extractor is a Swing Transformer network, which consists of 24 Swing Transformer modules connected in sequence. The image feature extractor receives input dimension as follows: The image is output as four feature matrices; the dimensions of the feature matrices are respectively... , , and .

3. The semantic segmentation method based on image generation according to claim 2, characterized in that, The discrete feature index classifier includes a feature aggregation layer, a feature processing layer, and a classifier module; The feature aggregation layer is a convolutional module that takes four feature matrices from the Swing Transformer network as input and outputs an aggregated feature matrix with dimension 1. ; The feature processing layer consists of two Swin Transformer modules, whose input is feature aggregation. The feature matrix after layer aggregation is output as a feature matrix with dimension 1. ; The classifier module takes the feature matrix output from the feature processing layer as input and outputs a probability matrix with dimension 1. Each element in the probability matrix represents the probability of 8192 discrete features appearing in that element.

4. The semantic segmentation method based on image generation according to claim 3, characterized in that, The input to the discrete feature code table is the probability matrix output by the classifier module, and the output is the discrete feature matrix, whose dimension is... Specifically, the first step is to find the discrete feature index corresponding to the discrete feature with the highest probability in each element of the probability matrix. The indexes corresponding to all elements in the probability matrix are combined to form an index matrix. Then, based on each element in the index matrix, the corresponding discrete feature is queried in the discrete feature code table. All discrete features corresponding to all elements constitute a discrete feature matrix. The semantic image decoder specifically adopts the DALL-E VQ-VAE model decoder, and the input is a discrete feature matrix output from a discrete feature code table, with a dimension of [missing information]. The output is the predicted semantic image, with dimensions of . ; The weights of the semantic image decoder are the weights of the decoder of the DALL-E VQ-VAE model downloaded from the network and loaded into the semantic image decoder; The color-category mapping module is an embedding module, and the weights of this embedding module are in dimension 1. The category-color array, the input to the color-category mapping module is the semantic image decoder output dimension. The semantic image to be predicted, the output is a dimensionless image. The semantic segmentation result is given by C, where C represents the total number of categories. The semantic segmentation result is the predicted semantic mask, i.e., the category corresponding to each pixel in the image. Specifically, the color-category mapping module first calculates the mapping from pixel L(i,j) in the i-th row and j-th column of the semantic image to each category. The absolute distance of the preset RGB colors , where k∈[1,C], where , , Let R, G, and B represent the R, G, and B color values ​​at pixel L(i,j) of the predicted semantic image, respectively. , , Representing categories The preset R, G, and B color values ​​are used; then, the category corresponding to the minimum value among the C absolute distances is taken as the category corresponding to pixel L(i,j). Finally, the categories corresponding to all pixels L(i,j) in the semantic image are... The semantic segmentation results are obtained by splicing the components according to their spatial location.

5. The semantic segmentation method based on image generation according to claim 4, characterized in that, The category-color mapping module is used to obtain the color corresponding to each pixel based on its category. Its input is a dimension-1232 ... The truth value of the semantic mask, the output is a dimensionless... The ground truth of the semantic image; where the ground truth of the semantic segmentation mask comes from the dataset used during offline training; The semantic image encoder specifically adopts the DALL-E VQ-VAE model encoder, whose input is the ground truth of the semantic image output by the category-color mapping module, and its dimension is... The output is of dimension 1. The true value of the equivalent characteristic matrix.

6. The semantic segmentation method based on image generation according to claim 5, characterized in that, Step (4-4) includes the following sub-steps: (4-4-1) Input the true values ​​of the semantic masks obtained in step (4-3) after multiple data preprocessing into the category-color mapping module to obtain the true values ​​of multiple semantic images; (4-4-2) Input the truth values ​​of the multiple semantic images obtained in step (4-4-1) into the initialized semantic image encoder obtained in step (3) to obtain the truth values ​​of multiple equivalent feature matrices; (4-4-3) Take the truth values ​​of the multiple equivalent feature matrices obtained in step (4-4-2), and in the initialized discrete feature code table obtained in step (3), find the discrete feature whose truth value is closest to that of each equivalent feature and its discrete feature index. Then, concatenate all the retrieved discrete feature indices according to their spatial positions to obtain the truth value of the discrete feature index matrix, the dimension of which is... ; Steps (4-5) include the following sub-steps: (4-5-1) Input each preprocessed image obtained in step (4-3) into the image feature extractor to obtain image features of four different spatial scales and number of channels corresponding to each image; (4-5-2) Input the image features corresponding to the four different spatial scales and number of channels for each image obtained in step (4-5-1) into the discrete feature index classifier to obtain the probability matrix of each discrete feature index.

7. A semantic segmentation system based on image generation, characterized in that, include: The first module is used to acquire the image to be segmented, normalize the image to be segmented, and convert the normalized image to be segmented into a tensor format image. The second module is used to input the tensor format image obtained from the first module into the pre-trained semantic segmentation model to obtain the semantic segmentation result. The semantic segmentation model includes an image feature extractor, a discrete feature index classifier, a discrete feature code table, a semantic image decoder, a color-category mapping module, a category-color mapping module, and a semantic image encoder, which are connected in sequence. The category-color mapping module and the semantic image encoder are only used for offline training of the semantic segmentation model. The discrete feature code table is a collection of multiple discrete features. The dimension of the discrete features is 128. Each discrete feature has a unique index in the discrete feature code table, which is called the discrete feature index. The semantic segmentation model is trained using the following steps: (1) Obtain the ADE20K dataset, divide the 25574 sets of images and their corresponding semantic masks of the ADE20K dataset into the training set, and divide the 2000 sets of images and their corresponding semantic masks of the ADE20K dataset into the validation set. (2) Generation dimension is The category color array; step (2) includes the following sub-steps: (2-1) Generate three one-dimensional arrays , , Each element of the array is , 40 and 45, and k1∈[1, array The total number of elements in the array], k2∈[1, array The total number of elements in the array], k3∈[1, array [Total number of elements in]; (2-2) Set counters k1=1, k2=1, k3=1, and initialize the RGB color array. An empty array; (2-3) Determine if k1 is greater than the preset maximum number of loops J, whose value is equal to the array If the total number of elements in the array is 1, proceed to step (2-13); otherwise, proceed to step (2-4). (2-4) Determine if k2 is greater than the preset maximum number of loops K, whose value is equal to the array The total number of elements in the array is determined. If the total number of elements is found, proceed to step (2-3); otherwise, proceed to step (2-5). (2-5) Determine if k3 is greater than the preset maximum number of loops Q, whose value is equal to the array The total number of elements in the array is determined. If the total number of elements is found, proceed to step (2-4); otherwise, proceed to step (2-6). (2-6) Generate a random integer between -15 and 15. and update The k1th element of the array To obtain the updated The k1th element of the array ; (2-7) Generate a random integer between -15 and 15. and update The k2th element of the array To obtain the updated The k2th element of the array ; (2-8) Generate a random integer between -15 and 15. and update The k3th element of the array To obtain the updated The k3th element of the array ; (2-9) The updated version obtained in step (2-6) The k1th element of the array The updated version obtained in steps (2-7) The k2th element of the array and the updated version obtained in steps (2-8) The k3th element of the array To form a three-dimensional element ( , , ), and three-dimensional elements ( , , Insert into the RGB color array At the end, to obtain the updated RGB color array. ; (2-10) Set k1=k1+1 and return to step (2-5); (2-11) Set k2=k2+1 and return to step (2-4); (2-12) Set k3=k3+1 and return to step (2-3); (2-13) Obtain the updated RGB color array obtained in step (2-9). The first C three-dimensional elements are used to obtain the category color array; (3) Initialize the weights of the image feature extractor, discrete feature index classifier, discrete feature code table, category-color mapping module, color-category mapping module, and semantic image encoder to obtain the initialized image feature extractor, discrete feature index classifier, discrete feature code table, category-color mapping module, color-category mapping module, and semantic image encoder; step (3) includes the following sub-steps: (3-1) Load the pre-trained weights of the image feature extractor into the image feature extractor; (3-2) Initialize the weights of the discrete feature index classifier to random values, and set the weights of the discrete feature index classifier to gradient weights; (3-3) Load the preset category color array obtained in step (2) into the color-category mapping module, and set the weight of the color-category mapping module to a weight without gradient; (3-4) Load the preset category color array obtained in step (2) into the category-color mapping module, and set the weight of the category-color mapping module to a weight without gradient; (3-5) Transfer the pre-trained weights of the semantic image encoder to the semantic image encoder, and set the weights of the semantic image encoder to gradient-free weights; (4) Using the training set of the ADE20K dataset obtained in step (1), the initialized discrete feature code table obtained in step (3), the category-color mapping module, and the semantic image encoder, train the initialized image feature extractor and discrete feature index classifier obtained in step (3) to obtain the trained image feature extractor and discrete feature index classifier; step (4) includes the following sub-steps: (4-1) Set the counter i=1 and initialize the hyperparameters of the training process to obtain the initialized hyperparameters of the training process; (4-2) Obtain the ground truth values ​​of multiple images and their corresponding semantic masks from the training set of the ADE20K dataset obtained in step (1); (4-3) Perform data preprocessing on the ground truth values ​​of multiple images and semantic masks obtained in step (4-2) to obtain multiple preprocessed ground truth values ​​of images and semantic masks; (4-4) Using the category-color mapping module, semantic image encoder, and discrete feature code table, the ground truth values ​​of the multiple preprocessed semantic masks obtained in step (4-3) are mapped to the ground truth values ​​of multiple discrete feature index matrices; (4-5) Using a sequentially connected image feature extractor and discrete feature index classifier, the multiple preprocessed images obtained in step (4-3) are mapped into a probability matrix of multiple discrete feature indices, the dimension of which is... ,in, The pre-set batch data size for offline training. and These are the number of pixels on the long side and the number of pixels on the short side of the image, respectively. (4-6) Input the probability matrix of multiple discrete feature indices obtained in step (4-5) and the true value of multiple discrete feature index matrices obtained in step (4-4) into the cross-entropy loss function to obtain the semantic feature loss value; (4-7) Backpropagation is performed using the semantic feature loss value obtained in step (4-6) to obtain the gradients of the image feature extractor and the discrete feature index classifier; (4-8) Using the learning rate set in step (4-1), the AdamW optimizer, and the weight gradient obtained in step (4-7), update the weights of the image feature extractor and the discrete feature index classifier to obtain new weights for the image feature extractor and the discrete feature index classifier. (4-9) Counter i = i + 1, and set the gradients of the image feature extractor and the discrete feature index classifier to 0; (4-10) Determine if i is greater than the preset maximum number of iterations n. If yes, the process ends; otherwise, return to step (4-2). (5) Save the weights of the trained image feature extractor and discrete feature number classifier obtained in step (4), the weights of the initialized discrete feature code table, the weights of the color-class mapping module and the weights of the semantic image decoder obtained in step (3) to obtain the weights of the semantic segmentation model; wherein, the weights of the semantic image decoder are the weights of the decoder of the DALL-E VQ-VAE model downloaded from the network.

Images