Self-supervised urban scene image segmentation method based on color and texture features

Abstract

The application discloses a self-supervised urban scene image segmentation method based on color and texture features, which comprises three stages: in the self-supervised pre-training stage, for a large number of unmarked urban scene images, pixel points with similar color or texture features are aggregated to generate image segmentation pseudo labels, then an image segmentation model based on an improved U-Net is constructed, the unmarked urban scene images and the pseudo labels thereof are used for self-supervised pre-training, and a pre-training model is obtained; in the supervised fine-tuning stage, the pre-training model is fine-tuned using a small amount of labeled urban scene images to obtain a target model; and in the prediction stage, an urban scene image to be segmented is input into the target model, and a segmentation result is output. The application uses the generated image segmentation pseudo labels for self-supervised pre-training, so that the pre-training process is closely related to the subsequent image segmentation task, and when deep and shallow features are fused, the features are enhanced by using spatial and channel attention, so that the segmentation precision of the model is improved.

Description

Technical Field

[0001] This invention belongs to the field of image segmentation technology, specifically relating to a self-supervised urban scene image segmentation method based on color and texture features. Background Technology

[0002] Image segmentation is a crucial task in computer vision, assigning specific semantic labels to each pixel in an image and providing technical support for intelligent image understanding and analysis. Currently, deep convolutional neural networks are widely used in image segmentation, driving the rapid development of this field.

[0003] Despite significant progress in previous research, deep network learning requires a large number of labeled image samples, a need that is often difficult to meet in practical applications due to the high cost of manual annotation. To address this issue, self-supervised learning techniques have emerged.

[0004] Existing self-supervised learning methods can be divided into two main categories: image reconstruction-based self-supervised learning and contrastive learning-based self-supervised learning. However, during self-supervised pre-training, image reconstruction-based self-supervised learning methods focus more on the similarity between the images generated by the pre-trained model and the original images; contrastive learning-based self-supervised learning methods focus more on the similarity between features extracted by the pre-trained model after different preprocessing methods of the same image. This results in a low correlation between the self-supervised pre-training process and the subsequent image segmentation task, thus affecting the image segmentation performance. Summary of the Invention

[0005] The problem to be solved by this invention is to provide a self-supervised urban scene image segmentation method based on color and texture features to address the shortcomings of existing technologies. By generating image segmentation pseudo-labels for self-supervised pre-training, the pre-training process is closely related to the subsequent image segmentation task. When fusing deep and shallow features, spatial and channel attention are used to enhance features and improve the segmentation accuracy of the model.

[0006] This invention adopts the following technical solution: a self-supervised urban scene image segmentation method based on color and texture features, comprising six steps in three stages:

[0007] Phase 1: Self-supervised pre-training phase

[0008] Step 1: Obtain an unlabeled urban scene image dataset and perform data preprocessing to use it as a self-supervised pre-training dataset;

[0009] Step 2: Construct an image segmentation model based on the improved U-Net;

[0010] Step 3: Use the self-supervised pre-training dataset obtained in Step 1 to train the image segmentation model based on the improved U-Net to obtain the pre-trained model;

[0011] Phase 2: Supervised Fine-tuning Phase

[0012] Step 4: Obtain the labeled urban scene image dataset and perform data preprocessing, dividing the dataset into training set, validation set and test set according to the proportions;

[0013] Step 5: Fine-tune the pre-trained model using the training and validation sets obtained in Step 4 to obtain the target model;

[0014] Phase 3, Prediction Phase

[0015] Step 6: Input the test set image obtained in Step 4 as the urban scene image to be segmented into the target model obtained in Step 5 to obtain the urban scene image segmentation result of the image.

[0016] Furthermore, in step 1, acquiring an unlabeled urban scene image dataset and performing data preprocessing specifically includes the following steps:

[0017] Step 1-1: Download or create an unlabeled urban scene image dataset from the internet;

[0018] Step 1-2: For each unlabeled image in the dataset, generate pseudo-labels for image segmentation;

[0019] Steps 1-3: Perform preprocessing on the images and their pseudo-labels in the dataset, including random color jittering, scaling, horizontal flipping, and cropping.

[0020] Furthermore, in steps 1-2, for each unlabeled image in the dataset, pseudo-labels for image segmentation are generated, specifically including the following steps:

[0021] Step 1-2-1: Let It is a 3D pixel value matrix of an unmarked urban scene image, with a size of ,in, Indicates the height of the image. The width of the image is represented by 3, and the number 3 represents the three color channels: red, green, and blue. Pixel values ​​from 0 to 255 are divided into c intervals. Take the median value of each interval as the representative pixel value for that interval, and define the pixel value interval as follows: The representative pixel value is ,in, It is a preset integer. ;

[0022] Step 1-2-2: Generate a size of A three-dimensional matrix A, any element in matrix A correspond pixel values ,like The range of pixel values ​​is ,make ,in, , , ;

[0023] Steps 1-2-3: Generate three units of size... Given two-dimensional matrices B, C, and D, any element in matrix B, C, and D. , and correspond pixels on ,in, , Initialize all elements in matrix B to False, and any element in matrix C... ;

[0024] Step 1-2-4: Select an element in matrix B whose value is False. ,make , , Represented by pixels Centered Local Binary Pattern (LBP) features of the region ,in, It is a collection of pixels. Calculated based on matrix C, It is a pre-defined odd number, and d is a temporary variable;

[0025] Step 1-2-5: If pixel points The pixel value ranges for the red, green, and blue color channels are respectively Centered on this pixel The LBP characteristics of the region are ,from Surrounding distance equals Choose two types of pixels from the available pixels:

[0026] One type is pixels with similar colors, referring to pixels whose red, green, and blue color channels are located in the same range. The pixels;

[0027] Another type is pixels with similar textures, requiring that the pixel be centered on the desired pixel. LBP characteristics of the region and The correlation is greater than the preset threshold;

[0028] Change the element value of matrix B corresponding to the selected pixel to True, and add that pixel to the set. ,in, , , LBP features are calculated based on matrix C;

[0029] Step 1-2-6: If the set in step 1-2-5 If no new pixels are added, proceed to step 1-2-7; otherwise, let Return to steps 1-2-5;

[0030] Step 1-2-7: Set the element values ​​of matrix D ,in, corresponding pixels , Represents a set The number of pixels in the image; if matrix B has no elements with a value of False, then matrix D is the image. If the image segmentation pseudo-label is not found, return to step 1-2-4.

[0031] Furthermore, in step 2, constructing an image segmentation model based on the improved U-Net specifically includes the following steps:

[0032] Step 2-1: Replace the shrinkage path of U-Net with an encoder consisting of one convolutional module and four downsampled convolutional modules connected in sequence; the convolutional module outputs feature map F1, and the first downsampled convolutional module, the second downsampled convolutional module, the third downsampled convolutional module, and the fourth downsampled convolutional module output feature maps F2, F3, F4, and F5 in sequence;

[0033] Step 2-2: Replace the U-Net's expansion path with a decoder consisting of four sequentially connected multi-attention feature fusion modules; the first multi-attention feature fusion module receives feature maps F4 and F5 and outputs feature map F6; the second multi-attention feature fusion module receives feature maps F3 and F6 and outputs feature map F7; the third multi-attention feature fusion module receives feature maps F2 and F7 and outputs feature map F8; the fourth multi-attention feature fusion module receives feature maps F1 and F8 and outputs feature map F9.

[0034] Steps 2-3: The final U-Net... The convolution is replaced by a segmentation result generation module in the self-supervised pre-training stage. This module receives the feature map F9 and outputs the segmentation result of the urban scene image.

[0035] Further, in step 3, the image segmentation model based on the improved U-Net is trained using the self-supervised pre-training dataset obtained in step 1 to obtain the pre-trained model, specifically including the following steps:

[0036] Step 3-1: Set the initial parameters for model training;

[0037] Step 3-2: Calculate the L2 norm loss during training; calculate the gradient based on the loss, update the network model parameters using the stochastic gradient descent (SGD) optimizer, and use the poly strategy for learning rate decay;

[0038] Step 3-3: Use L2 norm loss to evaluate the model. The smaller the loss value, the more accurate the segmentation result obtained by the model.

[0039] Step 3-4: Repeat the training process from Step 3-2 to Step 3-3. After each training round, use the L2 norm loss to evaluate the network model until the preset total number of training rounds is completed or the L2 norm loss value reaches the preset threshold. Save the optimal network model as the pre-trained model.

[0040] Further, in step 4, a labeled urban scene image dataset is obtained and preprocessed. The dataset is then divided into a training set, a validation set, and a test set according to a set ratio. This specifically includes the following steps:

[0041] Step 4-1: Download or create a labeled urban scene image dataset from an open-source dataset website;

[0042] Step 4-2: Perform preprocessing on the images and their labels in the dataset, including random color jittering, scale adjustment, horizontal flipping, and cropping.

[0043] Step 4-3: Divide the images and their labels in the preprocessed dataset into training set, validation set and test set according to the proportion.

[0044] Furthermore, in step 5, the pre-trained model is fine-tuned using the training set and validation set obtained in step 4 to obtain the target model, specifically including the following steps:

[0045] Step 5-1: Replace the segmentation result generation module in the final self-supervised pre-training stage of the model with the segmentation result generation module in the supervised fine-tuning and prediction stages. This module receives the feature map F9 and outputs the final urban scene image segmentation result; set the initial parameters for model training.

[0046] Step 5-2: Calculate the cross-entropy loss during training; calculate the gradient based on the loss, update the network model parameters using the stochastic gradient descent (SGD) optimizer, and use the poly strategy for learning rate decay;

[0047] Step 5-3: Use the average intersection-union ratio of the validation set ( The model is evaluated. A higher value indicates a more accurate segmentation result obtained by the model;

[0048] Step 5-4: Repeat the training process from Step 5-2 to Step 5-3, using the validation set after each training round. The network model is evaluated until the preset total number of training rounds is completed, and the optimal network model is saved as the target model.

[0049] Further, in step 6, the test set image obtained in step 4 is input as the urban scene image to be segmented into the target model obtained in step 5 to obtain the urban scene image segmentation result of the image. This specifically includes the following steps:

[0050] Step 6-1: Import the target model saved in Step 5, input the test set image obtained in Step 4 as the urban scene image to be segmented into the target model, and output the final urban scene image segmentation result.

[0051] Step 6-2: Calculation Save the test results.

[0052] The present invention also provides: an electronic device, comprising:

[0053] One or more processors;

[0054] A storage device on which one or more programs are stored;

[0055] When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned self-supervised urban scene image segmentation method.

[0056] Compared with the prior art, the present invention, employing the above technical solution, has the following technical effects:

[0057] 1. For a large number of unlabeled urban scene images, this invention generates pseudo-labels for image segmentation by aggregating pixels with similar color or texture features. These pseudo-labels are used for self-supervised pre-training of the image segmentation model based on the improved U-Net, making the self-supervised pre-training process closely related to the subsequent image segmentation task and improving the segmentation accuracy of the model.

[0058] 2. When fusing deep and shallow features, this invention utilizes spatial attention and channel attention to enhance features, further improving the segmentation accuracy of the model. Attached Figure Description

[0059] Figure 1 This is a flowchart of the self-supervised urban scene image segmentation method based on color and texture features of the present invention;

[0060] Figure 2 Pixels in embodiments of the present invention Surrounding distance equals A diagram illustrating the pixels;

[0061] Figure 3 This is a schematic diagram of the image segmentation model structure based on the improved U-Net of this invention;

[0062] Figure 4 This is a schematic diagram of the convolution module structure of the present invention;

[0063] Figure 5 This is a schematic diagram of the downsampling convolution module structure of the present invention;

[0064] Figure 6 This is a schematic diagram of the multi-attention feature fusion module structure of the present invention;

[0065] Figure 7 This is a schematic diagram of the segmentation result generation module in the self-supervised pre-training stage of the present invention.

[0066] Figure 8 This is a schematic diagram of the segmentation result generation module structure for the supervised fine-tuning stage and the prediction stage of the present invention. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of the application will be further described in detail below with reference to the accompanying drawings. The described embodiments are only a part of the embodiments involved in this invention. All non-innovative embodiments based on these embodiments by other researchers in the art are within the protection scope of this invention. Furthermore, the step numbers in the embodiments of this invention are only set for ease of explanation and do not limit the order of the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0068] In one embodiment of the present invention, a self-supervised urban scene image segmentation method based on color and texture features is provided, the specific process of which is as follows: Figure 1 As shown, the steps include:

[0069] Step 1: Obtain an unlabeled urban scene image dataset and perform data preprocessing to serve as a self-supervised pre-training dataset.

[0070] Furthermore, step 1 specifically includes the following steps:

[0071] Step 1-1: Download or create an unlabeled urban scene image dataset from the internet; the unlabeled urban scene image dataset used in this embodiment contains 50,000 unlabeled urban scene images collected from the internet.

[0072] Step 1-2: For each unlabeled image in the dataset, generate pseudo-labels for image segmentation.

[0073] Furthermore, steps 1-2 specifically include the following steps:

[0074] Step 1-2-1: Let img be a 3D pixel value matrix of an unmarked urban scene image, with a size of... ,in, Indicates the height of the image. The width of the image is represented by 3, and the number 3 represents the three color channels: red, green, and blue. Pixel values ​​from 0 to 255 are divided into c intervals. Take the median value of each interval as the representative pixel value for that interval, and define the pixel value interval as follows: The representative pixel value is ,in, It is a preset integer;

[0075] Preferably, this embodiment is set as follows: , .

[0076] Step 1-2-2: Generate a size of A three-dimensional matrix A, where any element in matrix A correspond pixel values ,like The range of pixel values ​​is ,make ,in, , , ;

[0077] Steps 1-2-3: Generate three units of size... Given two-dimensional matrices B, C, and D, any element in matrix B, C, and D. , and correspond pixels on ,in, , Initialize all elements in matrix B to False, and any element in matrix C... ;

[0078] Step 1-2-4: Select the elements in matrix B whose value is False. ,make , , Represented by pixels Centered Local Binary Pattern (LBP) features of the region ,in, It is a collection of pixels. Calculated based on matrix C, It is a pre-defined odd number, and d is a temporary variable;

[0079] Preferably, this embodiment is set as follows: .

[0080] Step 1-2-5: If pixel points The pixel value ranges for the red, green, and blue color channels are respectively Centered on this pixel The LBP characteristics of the region are ,from Surrounding distance equals Choose two types of pixels from the available pixels:

[0081] One type is pixels with similar colors, referring to pixels whose red, green, and blue color channels are located in the same range. The pixels;

[0082] Another type is pixels with similar textures, requiring that the pixel be centered on the desired pixel. LBP characteristics of the region and The correlation is greater than a preset threshold, and in this embodiment, the threshold is set to 0.7;

[0083] Change the element value of matrix B corresponding to the selected pixel to True, and add that pixel to the set. ,in, , , LBP features are calculated based on matrix C;

[0084] In this embodiment, Surrounding distance equals pixels such as Figure 2 As shown, the dark gray area is Surrounding distance equals The pixels; the light gray area is Surrounding distance equals The pixels; The same principle applies to other situations.

[0085] Step 1-2-6: If the set in step 1-2-5 If no new pixels are added, proceed to step 1-2-7; otherwise, let Return to steps 1-2-5;

[0086] Step 1-2-7: Set the element values ​​of matrix D ,in, corresponding pixels , Represents a set The number of pixels in the image; if matrix B has no element with a value of False, then matrix D is the pseudo-label for image segmentation of image img; otherwise, return to step 1-2-4;

[0087] Steps 1-3: Preprocess the dataset, including random color jitter (i.e., randomly adjusting the brightness, contrast, and saturation of the image), random resizing between 0.5 and 2.0, random horizontal flipping with a probability of 0.5, and random cropping at an image resolution of 1024×512.

[0088] Step 2: Construct an image segmentation model based on the improved U-Net.

[0089] Furthermore, step 2 specifically includes the following steps:

[0090] Step 2-1: As Figure 3 As shown, the shrinkage path of U-Net is replaced with an encoder consisting of a convolutional module and four downsampling convolutional modules connected in sequence.

[0091] The convolutional module outputs feature map F1, and the first downsampling convolutional module, the second downsampling convolutional module, the third downsampling convolutional module, and the fourth downsampling convolutional module output feature maps F2, F3, F4, and F5 respectively.

[0092] The structure of the convolutional module is as follows Figure 4 As shown, it contains two groups: The system consists of convolutional layers, batch normalization layers, and ReLU activation function layers. The padding of the convolutional layers is set to 1, and the number of channels is set to 64.

[0093] The structure of the downsampling convolution module is as follows: Figure 5 As shown, it contains one The system consists of a max pooling layer, two sets of 3×3 convolutional layers, a batch normalization layer, and a ReLU activation function layer. The stride of the max pooling layer is set to 2, and the padding of the convolutional layers is set to 1. The number of channels in the convolutional layers of the first, second, third, and fourth downsampling convolutional modules are set to 128, 256, 512, and 512, respectively.

[0094] Step 2-2: As Figure 3 As shown, the expansion path of U-Net is replaced with a decoder consisting of four sequentially connected multi-attention feature fusion modules:

[0095] The first multi-attention feature fusion module receives feature maps F4 and F5 and outputs feature map F6.

[0096] The second multi-attention feature fusion module receives feature maps F3 and F6 and outputs feature map F7.

[0097] The third multi-attention feature fusion module receives feature maps F2 and F7 and outputs feature map F8.

[0098] The fourth multi-attention feature fusion module receives feature maps F1 and F8 and outputs feature map F9.

[0099] The structure of the multi-attention feature fusion module is as follows: Figure 6 As shown, the input shallow and deep feature maps are multiplied by their respective spatial attention values. The deep feature map is then upsampled. The shallow and deep feature maps are then concatenated along the channels. The concatenated feature map is multiplied by its channel attention value, and then passed sequentially through... Convolutional layers, batch normalization layers, ReLU activation function layers, Convolutional layers, batch normalization layers, and ReLU activation function layers yield the output feature map of the multi-attention feature fusion module. Among these, the spatial attention submodule... Convolution padding is set to 3, and the number of channels is set to 1; in the channel attention submodule... The number of channels in the convolution is the same as the number of channels in the input feature map; the final... The number of channels in the convolution is the same as the number of channels in the input shallow feature map. Convolution padding is set to 1, and the number of channels is the same as the number of channels in the input shallow feature map.

[0100] Steps 2-3: The final U-Net... The convolution is replaced by a segmentation result generation module in the self-supervised pre-training stage. This module receives the feature map F9 and outputs the segmentation result of the urban scene image.

[0101] The segmentation result generation module in the self-supervised pre-training stage, such as... Figure 7 As shown, the input feature map F9 passes through sequentially Convolutional layers, batch normalization layers, ReLU activation function layers, Convolutional layers, batch normalization layers, and ReLU activation function layers output urban scene image segmentation results. Convolution padding is set to 1, and the number of channels is set to 64. The number of convolution channels is set to 1.

[0102] Step 3: Use the self-supervised pre-training dataset obtained in Step 1 to train the image segmentation model based on the improved U-Net to obtain the pre-trained model.

[0103] Furthermore, step 3 specifically includes the following steps:

[0104] Step 3-1: Set the initial parameters for model training as follows:

[0105] Initial learning rate: 0.01;

[0106] Optimizer: SGD optimizer;

[0107] Learning rate decay: a poly strategy;

[0108] Weight decay: 0.0004;

[0109] Momentum: 0.9;

[0110] Batch size: 16;

[0111] Training epochs: 300;

[0112] L2 norm loss threshold: 0.1.

[0113] Step 3-2: Calculate the L2 norm loss during training. The calculation formula is as follows:

[0114] ;

[0115] in, For the true value, This is a predicted value; and The more similar the two samples are, the smaller the loss value.

[0116] The gradient is calculated based on the loss function, and the network model parameters are updated using the stochastic gradient descent (SGD) optimizer. A poly strategy is used for learning rate decay, and the decay formula is as follows:

[0117] ;

[0118] in, The initial learning rate, This is the current training round number. This is the total number of training rounds set.

[0119] Step 3-3: Use L2 norm loss to evaluate the model. The smaller the loss value, the more accurate the segmentation result obtained by the model.

[0120] Steps 3-4: Repeat the training process from steps 3-2 to 3-3. After each training round, use the L2 norm loss to evaluate the network model until the preset total number of training rounds of 300 rounds is completed or the L2 norm loss value reaches the threshold of 0.1. Save the optimal network model as the pre-trained model.

[0121] Step 4: Obtain the labeled urban scene image dataset and perform data preprocessing, dividing the dataset into training set, validation set and test set according to the proportion.

[0122] Furthermore, step 4 specifically includes the following steps:

[0123] Step 4-1: Download the Cityscapes dataset from the official Cityscapes dataset website.

[0124] Specifically, the Cityscapes dataset contains 5,000 finely annotated city scene images and 20,000 coarsely annotated city scene images, with a total of 30 segmentation categories. This embodiment uses 5,000 finely annotated images, which include 19 segmentation categories, 2,975 training set images, 500 validation set images, and 1,525 test set images.

[0125] Step 4-2: Preprocess the Cityscapes dataset by random color jittering (i.e., randomly adjusting the brightness, contrast, and saturation of the image), random resizing between 0.5 and 2.0, random horizontal flipping with a probability of 0.5, and random cropping at an image resolution of 1024×512.

[0126] Step 4-3: Place the preprocessed images and their labels into the corresponding training set, validation set, and test set.

[0127] Step 5: Use the training set and validation set obtained in Step 4 to fine-tune the pre-trained model to obtain the target model.

[0128] Furthermore, step 5 specifically includes the following steps:

[0129] Step 5-1: Replace the segmentation result generation module in the final self-supervised pre-training stage of the model with the segmentation result generation module in the supervised fine-tuning and prediction stages. This module receives the feature map F9 and outputs the final urban scene image segmentation result.

[0130] The segmentation result generation module for the supervised fine-tuning and prediction stages, such as Figure 8 As shown, the input feature map F9 passes through sequentially Convolutional layers, batch normalization layers, ReLU activation function layers, Convolutional layers, batch normalization layers, and ReLU activation function layers output urban scene image segmentation results; among them... Convolution padding is set to 1, and the number of channels is set to 64. The number of convolution channels is set to the number of segmentation categories in the labeled urban scene image dataset.

[0131] The initial parameters for model training are set as follows:

[0132] Initial learning rate: 0.01;

[0133] Optimizer: SGD optimizer;

[0134] Learning rate decay: a poly strategy;

[0135] Weight decay: 0.0004;

[0136] Momentum: 0.9;

[0137] Batch size: 16;

[0138] Training epochs: 300.

[0139] Step 5-2: Calculate the cross-entropy loss during training. The calculation formula is as follows:

[0140] ;

[0141] in, For the true value, For predicted values; when When =1, ,at this time The closer the value is to 1, the smaller the loss value. The closer to 0, the greater the loss value; when When =0, ,at this time The closer the value is to 1, the greater the loss. The closer it is to 0, the smaller the loss value.

[0142] The gradient is calculated based on the loss function, and the network model parameters are updated using the stochastic gradient descent (SGD) optimizer. The learning rate decay is performed using a poly strategy.

[0143] Step 5-3: Use the average intersection-union ratio of the validation set ( The model is evaluated. A higher value indicates a more accurate segmentation result obtained by the model. The calculation formula is as follows:

[0144] ;

[0145] in, Represents the number of categories; Representing the kind; (True Positive) indicates a true positive instance, meaning that both the model prediction and the actual label are positive. Indicates the first The number of true instances in each category; (False Positive) is a false positive, meaning the model predicts the sample as positive, but the true label indicates the sample is negative. Indicates the first The number of false positives in each category; (False Negative) indicates a false negative, meaning the model predicts the sample as a negative, but the true label indicates the sample as a positive. Indicates the first The number of false counterexamples in each category.

[0146] Step 5-4: Repeat the training process from Step 5-2 to Step 5-3, using the validation set after each training round. The network model is evaluated based on the value until the preset total of 300 training epochs are completed, and then the optimal value is used. The optimal network model is saved as the target model.

[0147] Step 6: Input the test set image obtained in Step 4 as the urban scene image to be segmented into the target model obtained in Step 5 to obtain the urban scene image segmentation result of the image.

[0148] Furthermore, step 6 specifically includes the following steps:

[0149] Step 6-1: Import the target model saved in Step 5, input the test set image obtained in Step 4 as the urban scene image to be segmented into the target model, and output the final urban scene image segmentation result.

[0150] Step 6-2: Calculation Save the test results.

[0151] In this embodiment of the invention, an electronic device is also provided, comprising: one or more processors; a storage device having one or more programs stored thereon; when the one or more programs are executed by the one or more processors, the one or more processors implement the self-supervised urban scene image segmentation method based on color and texture features described in any of the above embodiments.

[0152] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A self-supervised urban scene image segmentation method based on color and texture features, characterized in that, It includes the following three stages: Phase 1: Self-supervised pre-training phase Step 1: Obtain an unlabeled urban scene image dataset and perform data preprocessing to use it as a self-supervised pre-training dataset; Step 2: Construct an image segmentation model based on the improved U-Net; Step 3: Use the self-supervised pre-training dataset obtained in Step 1 to train the image segmentation model based on the improved U-Net to obtain the pre-trained model; Phase 2: Supervised Fine-tuning Phase Step 4: Obtain the labeled urban scene image dataset and perform data preprocessing, dividing the dataset into training set, validation set and test set according to the proportions; Step 5: Fine-tune the pre-trained model using the training and validation sets obtained in Step 4 to obtain the target model; Phase 3, Prediction Phase Step 6: Use the test set image obtained in Step 4 as the urban scene image to be segmented, input it into the target model obtained in Step 5, and obtain the urban scene image segmentation result of the image.

2. The self-supervised urban scene image segmentation method according to claim 1, characterized in that, Step 1 involves acquiring an unlabeled urban scene image dataset and performing data preprocessing, including the following sub-steps: Step 1-1: Download or create an unlabeled urban scene image dataset from the internet; Step 1-2: For each unlabeled image in the dataset, generate pseudo-labels for image segmentation; Steps 1-3: Preprocess the images and their pseudo-labels in the dataset, including: random color jittering, scale adjustment, horizontal flipping, and cropping.

3. The self-supervised urban scene image segmentation method according to claim 2, characterized in that, In steps 1-2, for each unlabeled image in the dataset, pseudo-labels for image segmentation are generated, including the following sub-steps: Step 1-2-1: Let It is a 3D pixel value matrix of an unmarked urban scene image, with a size of ,in, Indicates the height of the image. The width of the image is represented by 3, and the number 3 represents the three color channels: red, green, and blue. Pixel values ​​from 0 to 255 are divided into c intervals. Take the median value of each interval as the representative pixel value for that interval, and define the pixel value interval as follows: The representative pixel value is ,in, For a preset integer, ; Step 1-2-2: Generate a size of A three-dimensional matrix A, where any element in matrix A correspond pixel values ,like The range of pixel values ​​is ,make ,in, , , ; Steps 1-2-3: Generate three units of size... Given two-dimensional matrices B, C, and D, any element in matrices B, C, and D. , and correspond pixels on ,in, , Initialize all element values ​​in matrix B to False, and represent any element in matrix C as... ; Step 1-2-4: Select the elements in matrix B whose value is False. ,make , , Represented by pixels Centered Local binary pattern characteristics of the region ,in, It is a collection of pixels. Calculated based on matrix C, It is a pre-defined odd number, and d is a temporary variable; Step 1-2-5: If pixel points The pixel value ranges for the red, green, and blue color channels are respectively Centered on this pixel The local binary pattern characteristics of the region are ,from Surrounding distance equals Select pixels with similar colors and textures from the set; change the element values ​​of matrix B corresponding to the selected pixels to True, and add them to the set. ,in, , , The local binary pattern features are calculated based on matrix C; Step 1-2-6: If the set in step 1-2-5 If no new pixels are added, proceed to step 1-2-7; otherwise, let Return to steps 1-2-5; Step 1-2-7: Set the element values ​​of matrix D ,in, corresponding pixels , Represents a set The number of pixels in the image; if matrix B has no elements with a value of False, then matrix D is the image. If the image segmentation pseudo-label is not found, return to step 1-2-4.

4. The self-supervised urban scene image segmentation method according to claim 3, characterized in that, The pixels with similar colors refer to pixels whose red, green, and blue channel values ​​are located in the following ranges respectively. The pixels; The pixels with similar textures refer to those centered around that pixel. Local binary pattern characteristics of the region and The pixels whose correlation is greater than a preset threshold.

5. The self-supervised urban scene image segmentation method according to claim 1, characterized in that, Step 2 involves constructing an image segmentation model based on the improved U-Net, which includes the following sub-steps: Step 2-1: Replace the shrinkage path of U-Net with an encoder consisting of one convolutional module and four downsampled convolutional modules connected in sequence; the convolutional module outputs feature map F1, and the first downsampled convolutional module, the second downsampled convolutional module, the third downsampled convolutional module, and the fourth downsampled convolutional module output feature maps F2, F3, F4, and F5 in sequence; Step 2-2: Replace the U-Net's expansion path with a decoder consisting of four sequentially connected multi-attention feature fusion modules; the first multi-attention feature fusion module receives feature maps F4 and F5 and outputs feature map F6; The second multi-attention feature fusion module receives feature maps F3 and F6 and outputs feature map F7; the third multi-attention feature fusion module receives feature maps F2 and F7 and outputs feature map F8. The fourth multi-attention feature fusion module receives feature maps F1 and F8 and outputs feature map F9; Steps 2-3: The final U-Net... The convolution is replaced by a segmentation result generation module in the self-supervised pre-training stage, which receives feature map F9 and outputs the urban scene image segmentation result.

6. The self-supervised urban scene image segmentation method according to claim 5, characterized in that, In step 3, the image segmentation model based on the improved U-Net is trained using the self-supervised pre-training dataset obtained in step 1 to obtain the pre-trained model, including the following sub-steps: Step 3-1: Set the initial parameters for model training; Step 3-2: Calculate the L2 norm loss during training; calculate the gradient based on the loss, update the network model parameters using a stochastic gradient descent optimizer, and use a poly strategy to decay the learning rate; Step 3-3: Use L2 norm loss to evaluate the model. The smaller the loss value, the more accurate the segmentation result obtained by the model. Step 3-4: Repeat the training process from Step 3-2 to Step 3-3. After each training round, use the L2 norm loss to evaluate the network model until the preset total number of training rounds is completed or the L2 norm loss value reaches the preset threshold. Save the optimal network model as the pre-trained model.

7. The self-supervised urban scene image segmentation method according to claim 1, characterized in that, In step 4, a labeled urban scene image dataset is obtained and preprocessed. The dataset is then divided into a training set, a validation set, and a test set according to a set ratio. This includes the following sub-steps: Step 4-1: Download or create a labeled urban scene image dataset from an open-source dataset website; Step 4-2: Preprocess the images and their labels in the dataset, including: random color jittering, scale adjustment, horizontal flipping, and cropping; Step 4-3: Divide the images and their labels in the preprocessed dataset into training set, validation set and test set according to the proportion.

8. The self-supervised urban scene image segmentation method according to claim 7, characterized in that, In step 5, the pre-trained model is fine-tuned using the training set and validation set obtained in step 4 to obtain the target model, including the following sub-steps: Step 5-1: Replace the segmentation result generation module in the final self-supervised pre-training stage of the model with the segmentation result generation module in the supervised fine-tuning and prediction stages, receive the feature map F9, and output the final urban scene image segmentation result; set the initial parameters for model training. Step 5-2: Calculate the cross-entropy loss during training; calculate the gradient based on the loss, update the network model parameters using a stochastic gradient descent optimizer, and use a poly strategy to decay the learning rate; Step 5-3: Use the mean intersection-over-union ratio (MIR) of the validation set to evaluate the model. A higher MIR indicates that the model obtains more accurate segmentation results. Step 5-4: Repeat the training process from Step 5-2 to Step 5-3. After each training round, use the average intersection-union ratio of the validation set to evaluate the network model until the preset total number of training rounds is completed. Save the optimal network model as the target model.

9. The self-supervised urban scene image segmentation method according to claim 7, characterized in that, In step 6, the test set image obtained in step 4 is input as the urban scene image to be segmented into the target model obtained in step 5 to obtain the urban scene image segmentation result of the image, including the following sub-steps: Step 6-1: Import the target model saved in Step 5, input the test set image obtained in Step 4 as the urban scene image to be segmented into the target model, and output the final urban scene image segmentation result. Step 6-2: Calculate the average crossover ratio and save the test results.

10. An electronic device, characterized in that, include: One or more processors; A storage device on which one or more programs are stored; When the one or more programs are executed by the one or more processors, the one or more processors implement the self-supervised urban scene image segmentation method as described in any one of claims 1 to 9.

Images