An Extraterrestrial Image Segmentation Method and System Combining Self-Supervised Learning and Semi-Supervised Learning

Abstract

The present invention relates to a method and system for extraterrestrial image segmentation that combines self-supervised learning and semi-supervised learning. This method performs self-supervised pre-training on extraterrestrial images by means of mask image modeling, enabling the model to learn diverse and robust feature space representations. Then, the model is fine-tuned in a semi-supervised manner to fully utilize the information in the unlabeled part of the extraterrestrial data, so that the model can have better performance in the downstream task, namely semantic segmentation. Through self-supervised learning pre-training, the network of the present invention can learn good feature representations from unlabeled data, and improve the pseudo-label prediction quality in the fine-tuning stage of the semi-supervised method. At the same time, in the fine-tuning stage, the supervision information of the unlabeled area is utilized by generating pseudo-labels, making the prediction results output by the model more accurate. The present invention can significantly improve the performance of image segmentation.

Description

Technical Field

[0001] The present invention belongs to the fields of image segmentation, self-supervised learning, and semi-supervised learning, and specifically relates to an extraterrestrial image segmentation method and system that combines self-supervised learning and semi-supervised learning. Background Art

[0002] Image segmentation aims to divide an image into several specific regions with separate semantic information, classify each pixel point of the input image, determine the category of each pixel, and thus perform regional division to extract the parts of interest. Image segmentation is an important branch in the field of artificial intelligence currently.

[0003] Existing image segmentation methods can be generally divided into the following two categories. The first category is traditional image segmentation algorithms, including: threshold method, which simply divides into two categories based on the size relationship between the pixel values of the image and a set threshold to separate the foreground and background; pixel clustering method, which assumes there are K categories in the image and uses iterative clustering methods such as the K-means clustering algorithm to classify each pixel point of the image; edge segmentation method, which segments different regions in the image according to the results of image edge detection; graph cut algorithm, which associates image segmentation with the minimum cut problem of a graph by constructing a weighted graph between image pixels, etc. These algorithms use traditional construction or iterative algorithms for image segmentation. The second category is deep learning-based image segmentation algorithms, which output a classification result for each pixel point through a feed-forward network, and typical network structures include fully convolutional, dilated convolutional, etc.

[0004] However, in the above methods, traditional algorithms cannot make full use of existing dataset resources and cannot achieve good results. The accuracy in the segmentation of complex scenes often cannot meet the requirements; on the one hand, existing deep learning methods based on feed-forward are overly dependent on supervised training with a large amount of labeled data, and current extraterrestrial image segmentation datasets (such as the Mars image segmentation dataset) cannot meet the data requirements for high-performance pure supervised training; on the other hand, current methods do not fully consider the characteristics of extraterrestrial data, such as sparse labeling and a large number of similar image data, and there is still a large room for improvement in the segmentation results, and the overall performance cannot meet the needs of practical applications. Summary of the Invention

[0005] In view of the above technical problems, the present invention proposes an extraterrestrial image segmentation method that combines self-supervised learning and semi-supervised learning, which reduces the dependence on a large amount of labeled data and at the same time combines the characteristics of extraterrestrial images to constrain the feature space learned by the model.

[0006] The technical solution adopted by the present invention is as follows:

[0007] An extraterrestrial image segmentation method that combines self-supervised learning and semi-supervised learning, comprising the following steps:

[0008] Collect extraterrestrial image training data;

[0009] In the self-supervised pre-training stage, use the extraterrestrial image training data for self-supervised pre-training to learn the color features and texture features of the image mask area, and obtain the neural network model in the self-supervised pre-training stage;

[0010] In the semi-supervised fine-tuning stage, use the information of the unlabeled part of the extraterrestrial images to fine-tune and train the neural network model in the self-supervised pre-training stage, and obtain the image segmentation neural network model in the semi-supervised fine-tuning stage;

[0011] Input the image to be segmented into the trained image segmentation neural network model in the semi-supervised fine-tuning stage to obtain the semantic segmentation result.

[0012] Further, the collection of extraterrestrial image training data is to collect a large number of extraterrestrial images and their corresponding segmentation labels to form a training dataset.

[0013] Further, the neural network model in the self-supervised pre-training stage includes a feature extraction network B seg , an output module H RGB , an output module H LBP ; Randomly mask the extraterrestrial image data and input it into the feature extraction network B seg , and the feature extraction network B seg combines the deep features and shallow features of the network to obtain the feature representation of the image, and this feature representation is respectively input into the output module H RGB , the output module H LBP , to predict the color features and texture features of the original image mask area, that is, to predict in the RGB color space and the local binary pattern feature space respectively.

[0014] Further, the following loss function is used for end-to-end joint optimization in the self-supervised pre-training stage:

[0015] L rgb = ||g(f(x⊙M)))-x||2

[0016] L lbp = ||h(f(x⊙M)))-s||2

[0017] L pre-train = λ1L rgb + λ2L lbp

[0018] In the formula, L rgb is the loss function term for RGB color prediction, L lbpFor the loss function term of local binary pattern feature prediction, x is the input sample image, s is the local binary pattern histogram calculated before masking x, g and h respectively represent output module H RGB and output module H LBP , f represents the feature extraction network; M represents a randomly generated mask, where 1 indicates that the area is valid; ⊙ represents the operation of element-wise product, L pre-train is the total loss function term, and λ1 and λ2 are weight values.

[0019] Furthermore, the image segmentation neural network model in the semi-supervised fine-tuning stage includes a feature extraction network f, an output module H seg , a discriminator d; the discriminator predicts whether each pixel is labeled by learning the annotation uncertainty of each pixel in the image, and selects the region with high confidence as the pseudo-label by setting a threshold; the pseudo-label is fused with the true label of the labeled region in the original data, and the semantic segmentation prediction result of the network is constrained by using the fused label for end-to-end training.

[0020] Furthermore, the training process in the semi-supervised fine-tuning stage is divided into two steps:

[0021] The first step, that is, the total function loss term in the early stage of training is:

[0022] L = λ ce L ce + λ dice L dice ,

[0023] where L ce is the segmentation prediction cross-entropy loss function term, L dice is the loss function term for optimizing the discriminator's binary classification prediction, and λ ce and λ dice are weight values;

[0024] The second step, that is, adding the L pseudo loss in the later stage of training, that is, the total function loss term is:

[0025] L = λ ce L ce + λ dice L dice + λ pseudo L pseudo ,

[0026] where L pseudo is the cross-entropy loss function term for predicting unlabeled regions based on pseudo-labels, and λ pseudo is the weight value.

[0027] An extraterrestrial image segmentation system that combines self-supervised learning and semi-supervised learning, which includes:

[0028] A training data collection module for collecting extraterrestrial image training data;

[0029] A self-supervised pre-training module for performing self-supervised pre-training using the extraterrestrial image training data to learn the color features and texture features of the image mask region, and obtaining a neural network model in the self-supervised pre-training stage;

[0030] A semi-supervised fine-tuning module for fine-tuning and training the neural network model in the self-supervised pre-training stage using the information of the unlabeled part in the extraterrestrial image, and obtaining an image segmentation neural network model in the semi-supervised fine-tuning stage;

[0031] A semantic segmentation module for inputting the image to be segmented into the trained image segmentation neural network model in the semi-supervised fine-tuning stage to obtain a semantic segmentation result.

[0032] Through self-supervised learning pre-training, the network of the present invention can learn good feature representations from unlabeled data, and improve the pseudo-label prediction quality in the fine-tuning stage in the semi-supervised manner. At the same time, in the fine-tuning stage, the supervision information of the unlabeled region is utilized by generating pseudo-labels, making the prediction results output by the model more accurate. Compared with the prior art, the present invention significantly improves the segmentation performance of extraterrestrial images (including Mars images, etc.). On the AI4MARS large-scale Mars image segmentation benchmark test set, the present invention improves the Frequency Weighted Intersection over Union index from 83.23% to 88.82%, and the Mean Intersection over Union index from 68.73% to 70.64%; on the S 5 On the Mars image segmentation benchmark test set of Mars, the present invention improves the Frequency Weighted Intersection over Union index from 76.47% to 87.18%, and the Mean Intersection over Union index from 76.38% to 77.20%. Description of the Drawings

[0033] Figure 1 It is a structural diagram of the image segmentation neural network framework used in the embodiment of the present invention. The upper part is the self-supervised pre-training process, and the lower part is the semi-supervised fine-tuning process.

[0034] Figure 2A 、 Figure 2B They are respectively the input image and the model prediction segmentation result diagram of the embodiment of the present invention. Detailed Embodiments

[0035] To make the above features and advantages of the present invention more obvious and understandable, specific embodiments are given below and will be described in detail in conjunction with the accompanying drawings. It should be noted that the specific number of layers, number of modules, number of functions, and settings of certain layers given in the following embodiments are only a preferred implementation manner and are not used for limitation. Those skilled in the art can select the number and set certain layers according to actual needs, which should be understandable.

[0036] A method for segmenting extraterrestrial images that combines self-supervised learning and semi-supervised learning. Under the input of a given extraterrestrial surface image, self-supervised pre-training is carried out by means of masked image modeling, so that the model learns a diverse and robust feature space representation. Next, the model is fine-tuned in a semi-supervised manner, making full use of the information in the unlabeled part of the extraterrestrial data, so that the model can have better performance in the downstream task, namely semantic segmentation. The method includes the following steps:

[0037] 1) Collect training data of extraterrestrial images.

[0038] 2) First, in the self-supervised pre-training stage, the extraterrestrial image data is randomly masked and input into the feature extraction network (baseline model). The deep features and shallow features of the network are combined to obtain the feature representation of the image.

[0039] 3) The feature representation obtained in the previous step is respectively input into two output modules to predict the color features and texture features of the masked area of the original image, that is, the prediction is carried out respectively in the RGB color space and the local binary pattern (LBP) feature space.

[0040] 4) After the self-supervised pre-training is completed, the weight parameter information of the feature extraction network in the obtained model is used to initialize a standard segmentation model, and the model is fine-tuned in a semi-supervised manner on the downstream task.

[0041] 5) Input the extraterrestrial image data, and the standard segmentation model outputs the predicted semantic segmentation result. At the same time, a discriminator is trained. Since the image itself has labeled areas and unlabeled areas, indicating the labeling uncertainty of this area, the discriminator predicts whether each pixel is labeled by learning this uncertainty.

[0042] 6) After completing step 5), each time the training data is input, the semantic segmentation result corresponding to the entire image (including the labeled area and the unlabeled area) can be obtained, as well as the labeling uncertainty of each pixel in the image. The discriminator selects the regions with higher confidence by setting a threshold as the candidate pseudo-labels, and fuses them with the true labels of the labeled areas in the original data to obtain more supervision information.

[0043] 7) Use the fused tags to constrain the semantic segmentation prediction results of the network and perform end-to-end training.

[0044] 8) Input the extraterrestrial image to be segmented into the trained image segmentation model to obtain the semantic segmentation result.

[0045] An embodiment of the present invention discloses a method for segmenting extraterrestrial images by combining self-supervised learning and semi-supervised learning, which is specifically described as follows:

[0046] Step 1: Collect and build a training data set consisting of a large number of extraterrestrial images and their corresponding segmentation tags.

[0047] Step 2: Perform the self-supervised pre-training stage and build a neural network model for image segmentation.

[0048] The network structure in the self-supervised pre-training stage is as shown in the appendix Figure 1 and the model consists of three sub-networks: a feature extraction network B seg , an output module H RGB , and an output module H LBP .

[0049] The feature extraction network generally adopts a backbone network part similar to ResNet-101, including 34 consecutive convolutional layers. A rectified linear unit (ReLU) follows each convolutional layer. A 3x3 max pooling with a stride of 2 follows the first convolutional layer for downsampling. The final output feature of this network is the feature map corresponding to the input image, which is the result of downsampling the original input size by 16 times. Among them, the feature extraction network changes the stride of the 2nd - 4th convolutional layers and the 32nd - 34th convolutional layers of the original ResNet model to 1, and the output feature is the combination of the output of the 4th convolutional layer and the output of the last convolutional layer.

[0050] The output module H RGB and the output module H LBP both adopt the Head part of the DeepLabV3+ network, including: a projection module, which consists of a convolutional layer followed by batch normalization (Batch Normalization) and a rectified linear unit; a multi-scale aggregation pooling layer. After the input is processed by 4 dilated convolutional layers with different dilation rates and an average pooling layer + convolutional layer, the features they output are combined and input into a projection module to obtain the final output. This projection module includes a convolutional layer followed by batch normalization and a rectified linear unit, and dropout is applied for regularization; finally, a classifier consisting of a convolutional layer outputs the final prediction result or feature map. Among them, the output module H RGBAll convolutional layers are replaced with gated convolution layers. Output module H RGB Finally, the network outputs the RGB color prediction results for the masked area of the input image; output module H LBP Finally, the network outputs the prediction results of the local binary pattern features for the masked area of the input image. The local binary pattern features are given by the statistical histogram of the local binary patterns of different divided blocks of the image, and the dimension of the histogram is 25.

[0051] Step 3: Pre-train the image segmentation neural network model in the self-supervised learning stage.

[0052] There are two loss functions for end-to-end joint optimization:

[0053] L rgb = ||g(f(x⊙M)) - x||2

[0054] L lbp = ||h(f(x⊙M)) - s||2

[0055] L pre-train = λ1L rgb + λ2L lbp

[0056] In the formula, L rgb is the loss function term for RGB color prediction, L lbp is the loss function term for local binary pattern feature prediction, x is the input sample image, s is the local binary pattern histogram calculated before masking x, g and h respectively represent output module H RGB and output module H LBP , f represents the feature extraction network, M represents a randomly generated mask, where 1 indicates that the area is valid, ⊙ represents the element-wise product operation, and L pre-train is the total loss function term. Usually, the weights λ1 and λ2 are set to 0.5 and 0.5.

[0057] Step 4: Extract the feature extraction network in Step 3 and fine-tune the model for the semantic segmentation task in a semi-supervised manner. First, build the network model for the semi-supervised fine-tuning stage.

[0058] The network structure in the semi-supervised fine-tuning stage is as Figure 1 shown. The entire network consists of the feature extraction network f, output module H seg , and discriminator d.

[0059] The feature extraction network structure in the semi-supervised fine-tuning stage is the same as that in Step 3; output module H seg , discriminator d are both the same as output module H LBP in Step 3.

[0060] Step 5: Train the model in the semi-supervised fine-tuning stage.

[0061] The training process in this stage is divided into two steps:

[0062] In the first step, the total function loss term in the early stage of training is:

[0063] L = λ ce L ce + λ dice L dice ,

[0064] L ce is the cross-entropy loss function term for segmentation prediction, L dice is the loss function term for optimizing the discriminator's binary classification prediction, and λ ce and λ dice are weight values.

[0065]

[0066] In the formula represents the probability that the prediction probability of the pixel at position (h, w) by the network belongs to class c j , c i is the true label of the pixel at this position, E x represents the mean of the pixels of all training sample images, and E h,w represents the mean of all pixels of a training sample image (strictly speaking, L ce is only calculated for the pixels in the labeled area, and the subsequent L pseudo is only calculated for the pixels in the unlabeled area), and C represents the number of different classes in the labels defined by the dataset.

[0067]

[0068] In the formula, p h,w is the certainty of the pixel at position (h, w) output by the discriminator d, q is the uncertainty label for the entire image, and q h,w is the label value of the pixel at position (h, w), which is 1 if the pixel is labeled, otherwise 0.

[0069] In the second step, in the later stage of training, add the L pseudo loss:

[0070] L = λ ce L ce + λ dice L dice + λ pseudo L pseudo ,

[0071] L pseudo is the cross-entropy loss function term for predicting unlabeled regions based on pseudo-labels:

[0072]

[0073] In the formula represents the probability that the prediction probability of the pixel at position (h, w) by the network belongs to class c j , is the predicted pseudo-label of the pixel at this position, represents the probability that the prediction probability of the pixel at position (h, w) by the network belongs to class , and λ pseudo is the weight value.

[0074] The entire network is jointly optimized end-to-end.

[0075] Step 6: In the inference stage, use the feature extraction network f and the output module H of the semi-supervised fine-tuning stage that have been trained seg , input the reference extraterrestrial image to be tested (see the Mars image shown Figure 2A ), and finally output the corresponding semantic segmentation result (see Figure 2B ).

[0076] Based on the same inventive concept, another embodiment of the present invention is an extraterrestrial image segmentation system that combines self-supervised learning and semi-supervised learning, which includes:

[0077] A training data collection module for collecting extraterrestrial image training data;

[0078] A self-supervised pre-training module for performing self-supervised pre-training using extraterrestrial image training data to learn the color features and texture features of the image mask region, and obtaining a neural network model in the self-supervised pre-training stage;

[0079] A semi-supervised fine-tuning module for fine-tuning and training the neural network model in the self-supervised pre-training stage using the information of the unlabeled part of the extraterrestrial image, and obtaining an image segmentation neural network model in the semi-supervised fine-tuning stage;

[0080] A semantic segmentation module for inputting the image to be segmented into the trained image segmentation neural network model in the semi-supervised fine-tuning stage to obtain a semantic segmentation result.

[0081] For the specific implementation process of each module, refer to the description of the method of the present invention above.

[0082] Based on the same inventive concept, another embodiment of the present invention provides an electronic device (such as a computer, a server, a smart phone, etc.), which includes a memory and a processor. The memory stores a computer program, and the computer program is configured to be executed by the processor. The computer program includes instructions for performing each step in the method of the present invention.

[0083] Based on the same inventive concept, another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, a magnetic disk, an optical disc). When the computer program stored in the computer-readable storage medium is executed by a computer, each step of the method of the present invention is implemented.

[0084] The extraterrestrial pictures in the present invention may be pictures of Mars or pictures in other extraterrestrial exploration fields, mainly focusing on the terrain segmentation task of extraterrestrial. The present invention can be used for picture segmentation in scenarios similar to the extraterrestrial terrain segmentation task.

[0085] The above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit them. Those of ordinary skill in the art can modify the technical solutions of the present invention or make equivalent substitutions without departing from the spirit and scope of the present invention. The protection scope of the present invention shall be subject to what is described in the claims.

Claims

1. An extraterrestrial image segmentation method combining self-supervised learning and semi-supervised learning, characterized in that, It includes the following steps: Collect training data of extraterrestrial images; In the self-supervised pre-training stage, use the training data of extraterrestrial images for self-supervised pre-training to learn the color features and texture features of the masked regions of the images, and obtain the neural network model in the self-supervised pre-training stage; In the semi-supervised fine-tuning stage, use the information of the unlabeled part of the extraterrestrial images to fine-tune and train the neural network model in the self-supervised pre-training stage, and obtain the image segmentation neural network model in the semi-supervised fine-tuning stage; Input the image to be segmented into the trained image segmentation neural network model in the semi-supervised fine-tuning stage to obtain the semantic segmentation result; The neural network model in the self-supervised pre-training stage includes a feature extraction network B seg , an output module H RGB , an output module H LBP ; randomly mask the extraterrestrial image data and input it into the feature extraction network B seg , and the feature extraction network B seg combines the deep features and shallow features of the network to obtain a feature representation of the image, and this feature representation is respectively input into the output module H RGB , the output module H LBP to predict the color features and texture features of the masked area of the original image, that is, to make predictions in the RGB color space and the local binary pattern feature space respectively; In the self-supervised pre-training stage, the following loss function is used for end-to-end joint optimization: L rgb = ||g(f(x⊙M)))-x||2 L lbp = ||h(f(x ⊙ M)) - s||² L pre-train = λ1L rgb + λ2L lbp Among them, L rgb is the loss function term for RGB color prediction, and L lbp is the loss function term for local binary pattern feature prediction. x is the input sample image, s is the local binary pattern histogram calculated before masking x, g and h respectively represent the output module H RGB and the output module H LBP , and f represents the feature extraction network; M represents a randomly generated mask, where 1 indicates that the area is valid; ⊙ represents the operation of element-wise product, and L pre-train is the total loss function term, and λ1 and λ2 are weight values; The image segmentation neural network model in the semi-supervised fine-tuning stage includes a feature extraction network f and an output module H seg , and a discriminator d; the discriminator predicts whether each pixel is labeled by learning the annotation uncertainty of each pixel in the image, and selects the region with high confidence as the pseudo-label by setting a threshold; fuses the pseudo-label with the true label of the labeled region in the original data, and uses the fused label to constrain the semantic segmentation prediction result of the network for end-to-end training.

2. The method according to claim 1, wherein The collection of the training data of extraterrestrial images is to collect a large number of extraterrestrial images and their corresponding segmentation labels to form a training dataset.

3. The method according to claim 1, characterized in that, The training process in the semi-supervised fine-tuning stage is divided into two steps: The total function loss term in the first step, i.e., in the early stage of training, is: L = λ ce L ce + λ dice L dic e, Among them, L ce is the segmentation prediction cross-entropy loss function term, and L dice is the loss function term for optimizing the discriminator binary classification prediction. λ ce and λ dice为 are weight values; Among them, represents the probability that the prediction probability of the pixel at the position (h, w) by the network belongs to the category c j where c i is the true label of the pixel at this position, and E x represents the mean value of the pixels of all training sample images, and E h,w represents the mean value of all pixels of a training sample image, and C represents the number of different categories in the labels defined by the dataset; where p h,w is the certainty of the pixel at (h, w) output by the discriminator d, q is the uncertainty label for the entire image, and q h,w is the label value of the pixel at (h, w), which is 1 if the pixel is labeled and 0 otherwise; The second step is to add L in the later stage of training pseudo Loss, that is, the loss term of the total function is as follows: L = λ ce L ce + λ dice L dice + λ pseudo L pseudo , Among them, L pseudo is the cross-entropy loss function term for predicting unlabeled regions based on pseudo-labels, and λ pseudo is the weight value.

4. The method according to claim 3, characterized in that The said L pseudo The loss is: Among them, represents the probability that the prediction probability of the pixel at the position (h, w) by the network belongs to class c j of the probability, represents the probability that the prediction probability of the pixel at the position (h, w) by the network belongs to the class of the probability, is the predicted pseudo-label of the pixel at this position.

5. An extraterrestrial image segmentation system that combines self-supervised learning and semi-supervised learning, characterized in that, It includes: A training data collection module for collecting training data of extraterrestrial images; A self-supervised pre-training module for using the training data of extraterrestrial images for self-supervised pre-training to learn the color features and texture features of the masked regions of the images, and obtaining the neural network model in the self-supervised pre-training stage; A semi-supervised fine-tuning module for using the information of the unlabeled part of the extraterrestrial images to fine-tune and train the neural network model in the self-supervised pre-training stage, and obtaining the image segmentation neural network model in the semi-supervised fine-tuning stage; A semantic segmentation module for inputting the image to be segmented into the trained image segmentation neural network model in the semi-supervised fine-tuning stage to obtain the semantic segmentation result; The neural network model in the self-supervised pre-training stage includes a feature extraction network B seg , an output module H RGB , an output module H LBP ; randomly mask the extraterrestrial image data and input it into the feature extraction network B seg , the feature extraction network B seg combines the deep features and shallow features of the network to obtain the feature representation of the image, and this feature representation is respectively input into the output module H RGB , the output module H LBP to predict the color features and texture features of the masked area of the original image, that is, to make predictions in the RGB color space and the local binary pattern feature space respectively; In the self-supervised pre-training stage, the following loss function is used for end-to-end joint optimization: L rgb = ||g(f(x⊙M)))-x||2 L lbp = ||h(f(x⊙M)) - s||² L pre-train = λ1L rgb + λ2L lbp Among them, L rgb is the loss function term for RGB color prediction, and L lbp is the loss function term for local binary pattern feature prediction. x is the input sample image, s is the local binary pattern histogram calculated before masking x, g and h respectively represent the output module H RGB and the output module H LBP , f represents the feature extraction network; M represents a randomly generated mask, where 1 indicates that the area is valid; ⊙ represents the operation of element-wise product, and L pre-train is the total loss function term, and λ1 and λ2 are weight values; The image segmentation neural network model in the semi-supervised fine-tuning stage includes a feature extraction network f and an output module H seg , and a discriminator d; the discriminator predicts whether each pixel is labeled by learning the annotation uncertainty of each pixel in the image, and selects the region with high confidence as the pseudo-label by setting a threshold; fuses the pseudo-label with the true label of the labeled region of the original data, and uses the fused label to constrain the semantic segmentation prediction result of the network for end-to-end training.

6. An electronic device, characterized in that, It includes a memory and a processor. The memory stores a computer program, and the computer program is configured to be executed by the processor. The computer program includes instructions for executing the method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and when the computer program is executed by a computer, the method according to any one of claims 1 to 4 is implemented.

Images