A method for segmenting a Mars terrain based on a double-branch input neural network

Abstract

This invention discloses a Martian terrain segmentation method based on a dual-branch input neural network. After samples are input into the network, they undergo joint training through two branches: a regular branch responsible for extracting information from the target's interior region, and a boundary branch responsible for extracting the target's edge information. The information extracted by both branches is then processed by a fusion module, combining region and boundary features to output a refined semantic segmentation result. This method effectively segments inaccurately labeled Martian terrain, achieving accurate boundary segmentation. By introducing dual-branch information, this invention simultaneously utilizes both precisely labeled target regions and inaccurately labeled target boundaries, solving the problem of inaccurate labeling and saving significant time and manpower costs.

Description

Technical Field

[0001] This invention belongs to the field of image recognition and relates to a Martian terrain segmentation method based on a dual-branch input neural network. Background Technology

[0002] Conducting large-scale surface exploration of extraterrestrial objects using unmanned or manned methods is one of the future development directions of my country's deep space exploration. The ability to efficiently and accurately identify unknown and complex surface environments is a key issue affecting the effectiveness of the exploration.

[0003] Unlike the unknown environments of Earth, the terrain of extraterrestrial surfaces such as the Moon, Mars, and asteroids is characterized by indistinct features and limited prior knowledge. Traditional perception methods reconstruct elevation maps using stereo vision and determine whether a terrain feature is an obstacle based on its height. These methods are complex, use relatively limited features, and are easily affected by lighting conditions. Interference light in the image leads to low terrain recognition efficiency and poor robustness. In reality, in addition to obstacle geometry, features such as color and texture should be considered. Therefore, the most direct and effective method is obstacle detection through images, as image data contains the most feature information.

[0004] In recent years, deep learning methods have achieved successful applications in image-based object recognition and detection due to their excellent performance and unlimited potential. The development of computer, big data, and GPU technologies in recent years has made it possible to implement computationally intensive deep learning methods. While these methods have achieved great success in object recognition and detection across various terrestrial domains thanks to their superior performance and unlimited potential, they have not yet been effectively applied in deep space exploration. Deep learning requires training samples and the preparation of labeled terrain features within known Martian scenes. However, due to the complex terrain of Mars, it is difficult to clearly delineate and label various terrain features, leading to inaccurate intelligent segmentation results based on deep learning. Furthermore, the complexity and time cost of fine terrain segmentation techniques are very high, and the corresponding learning network size would also increase significantly. For deep space probes with limited computing resources and sample data, this makes the project less feasible. Summary of the Invention

[0005] The purpose of this invention is to overcome the aforementioned shortcomings and provide a Martian terrain segmentation method based on a dual-branch input neural network. After samples are input into the network, they undergo joint training through two branches: a regular branch responsible for extracting information from the target's interior region, and a boundary branch responsible for extracting the target's edge information. The information extracted by the two branches is then processed by a fusion module, combining region and boundary features to output a refined semantic segmentation result. This achieves accurate segmentation of Martian terrain with inaccurate labeling, thus resolving the problem of inaccurate labeling by introducing dual-branch information and simultaneously utilizing both precisely labeled target regions and inaccurately labeled target boundaries, thereby saving significant time and manpower costs.

[0006] To achieve the above-mentioned objectives, the present invention provides the following technical solution:

[0007] A Martian terrain segmentation method based on a dual-branch input neural network includes the following steps:

[0008] (1) Manually label the obstacle areas in the Mars images to obtain a dataset of Mars terrain segmentation, and divide the dataset into a training set and a test set;

[0009] (2) Establish a dual-branch neural network model for Mars terrain segmentation;

[0010] (3) The dual-branch neural network Mars terrain segmentation model is trained using the training set to obtain the trained dual-branch neural network Mars terrain segmentation model. The specific method is as follows:

[0011] (31) Divide the obstacle regions in the training set images into internal regions and edge regions, and label them respectively;

[0012] (32) Train the marked internal and edge regions using regular and boundary branches respectively to obtain feature maps of the internal and edge regions;

[0013] (33) Merge the feature maps of the internal and peripheral regions to obtain the terrain segmentation results;

[0014] (34) Repeat steps (31) to (33) to train the Mars terrain segmentation model by combining the loss function to obtain the trained model;

[0015] (4) Input the test set images into the trained Mars terrain segmentation model for semantic segmentation to complete the Mars terrain segmentation task.

[0016] Furthermore, in step (31), the inner region refers to the set of pixels whose adjacent pixels are all obstacle pixels, and the edge region refers to the set of obstacle pixels in the obstacle region other than the inner region;

[0017] In step (31), the obstacle region in the training set image is divided into internal region and edge region by using the transfer learning method.

[0018] Furthermore, in step (31), the boundary reduced to 0.9 times the original boundary of the obstacle area is defined as the regular boundary, and the area with the regular boundary at the center of the target is selected as the internal area; the boundary reduced to 0.85 times the original boundary of the obstacle area is defined as the edge boundary, and the area from the edge boundary to the original boundary is selected as the edge area.

[0019] Furthermore, in step (32), during regular branch training, based on the ResNet-18 network structure, four UNet networks are connected to construct four internal region feature maps of different sizes. During boundary branch training, VGG-16 is used as the model, and four pooling layers are connected to generate four edge region feature maps of different sizes. The sizes of the internal region feature maps and the edge region feature maps correspond one-to-one.

[0020] Furthermore, in step (33), an attention mechanism is used to fuse the feature maps of the internal and edge regions.

[0021] Furthermore, in step (33), during the process of fusing the feature maps of the internal region and the edge region, a convolutional layer of size 1x1 is used to adjust the number of channels of the feature maps of the internal region and the edge region to be equal.

[0022] Furthermore, in step (34), during the training of the Mars terrain segmentation network model in conjunction with the loss function, the stochastic gradient descent method is used to optimize the loss function, and the Mars terrain segmentation model is trained until the loss function converges.

[0023] Furthermore, in step (34), the loss function used for the regular branch is the Generalized Diceloss, and the loss function used for the boundary branch is the Focal loss.

[0024] Furthermore, the dual-branch neural network Mars terrain segmentation model includes an input layer, a feature extraction layer, an information processing layer, and an output layer;

[0025] The input layer receives the image and divides the image into internal and edge regions;

[0026] The feature extraction layer includes a conventional branch encoder and a boundary flow encoder. The conventional branch encoder and the boundary flow encoder are used to train the internal region and the edge region respectively, so as to obtain the feature maps of the internal region and the edge region respectively.

[0027] The information processing layer consists of decoders, which share feature maps of the internal and edge regions, obtain the segmentation results, and output them through the output layer.

[0028] Furthermore, in step (1), the number of obstacle types contained in the Mars image is ≥3; the ratio of the training set to the test set is 7:3 to 9:1.

[0029] Compared with the prior art, the present invention has the following advantages:

[0030] (1) The present invention provides a neural network-based Mars terrain segmentation method based on dual-branch input. It designs a method for dividing the target's internal region and edge region, and trains the model by combining the target's internal and edge information. Compared with manual annotation, the trained model is more accurate in segmenting the Martian terrain boundary.

[0031] (2) The present invention provides a neural network-based Mars terrain segmentation method based on dual-branch input. When extracting feature maps, a multi-scale structure is adopted to generate multiple feature maps of different sizes, thereby improving the generalization ability of the model in complex scenes.

[0032] (3) In the present invention, a neural network-based Mars terrain segmentation method based on dual-branch input is adopted, which uses a combination of multiple loss functions to make the network model results more accurate, while improving the segmentation ability of targets of different sizes and complex shapes, and adapting to the segmentation of various terrains on Mars. Attached Figure Description

[0033] Figure 1 This is a schematic diagram of the dual-branch input neural network for Mars terrain segmentation according to the present invention;

[0034] Figure 2 This is a structural diagram of the ResNet-18 convolutional neural network used in this invention;

[0035] Figure 3 This is a structural diagram of the VGG-16 convolutional neural network used in this invention. Detailed Implementation

[0036] The features and advantages of the present invention will become clearer and more apparent from the following detailed description.

[0037] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments. Although various aspects of embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless specifically indicated otherwise.

[0038] This invention addresses the inaccuracies of manual image segmentation annotation by designing a dual-branch neural network. The regular branch employs a classic semantic segmentation network to calculate the interior regions of the target, while the boundary branch calculates the target's boundaries. A fusion module combines the outputs of the two branches. This design ensures accurate segmentation of correctly annotated target interiors while avoiding the impact of incorrect boundary labels on boundary classification.

[0039] This invention discloses a Martian terrain segmentation method based on a dual-branch input neural network, comprising the following steps:

[0040] 1) Manually label obstacle areas in Mars images, select labeled samples to form a Mars terrain segmentation dataset, and select a portion of images as the training set and a portion of images as the test set.

[0041] 2) Input the training set images into the dual-branch neural network Mars terrain segmentation model for specific processing;

[0042] 3) Using specially processed images, train regular branches and boundary branches respectively to obtain preliminary segmentation results of Martian terrain;

[0043] 4) Integrate the outputs of the dual-branch network;

[0044] 5) The Mars terrain segmentation network is optimized by combining the loss function to obtain a trained model;

[0045] 6) Input the test set images into the network model trained in step 5) for semantic segmentation to complete the Mars terrain segmentation task.

[0046] Step 2) involves specific image processing: dividing the manually labeled obstacle pixel regions into internal regions and edge regions. Internal regions are sets of pixels where every adjacent pixel is an obstacle pixel, while edge regions are sets of obstacle pixels outside the internal regions. Internal regions are used as labels for regular branches, and edge regions are used as labels for boundary branches.

[0047] Step 3) The regular branch is based on the ResNet-18 network structure. After the feature map is extracted, four UNet networks are connected to construct four feature maps with different widths and heights to adapt to the varied Martian terrain.

[0048] In step 3), the boundary branch network uses VGG-16 as the model. After the obtained feature map, four pooling layers are connected to generate four feature maps with different widths and heights, resulting in a feature map with the same width and height as the regular branch.

[0049] Step 4) uses a convolutional layer to adjust the number of channels in the output feature maps (i.e., internal region feature maps and edge region feature maps) of the regular branch and the boundary branch to be equal;

[0050] Furthermore, the convolutional layers used in the channel adjustment are 1x1 convolutional layers.

[0051] In step 4), the fusion process uses an attention mechanism to aggregate the feature maps of the regular branches and the boundary branches to obtain a refined segmentation result.

[0052] Step 5) uses different loss functions for the two branches, enabling the Mars terrain segmentation network to learn information about the interior and boundaries of the target region. The generalized Dice loss function is used for the regular branch, while the Focal loss function is used for the boundary branch.

[0053] Furthermore, the mathematical description of the generalized Dice loss function is as follows:

[0054]

[0055] In the formula, r ln Let P represent the true pixel class of category l at position n, and P represent the true pixel class of category l at position n. ln w represents the corresponding predicted probability value. l This represents the weight of each category.

[0056] Furthermore, the boundary flow loss function is Focal loss, where the mathematical description is as follows:

[0057]

[0058] The predicted probability for each category, γ is a value greater than 0, and α is a decimal between [0, 1]. Both γ and α are fixed values ​​and do not participate in training. For the foreground class, The larger the value, the easier it is to predict foreground classes, and the smaller their weight. For background classes, The smaller the value, the easier it is to predict the background category, and the smaller the weight.

[0059] Example 1:

[0060] The following examples illustrate a Martian terrain segmentation method based on a dual-branch input neural network according to the present invention:

[0061] (1) Overall System Composition

[0062] The system for Martian terrain segmentation of this invention consists of two parts. The first part is the training part, which trains the network first. This is done offline before the system is used. The second part is the testing part, which uses the trained network to discriminate input Martian terrain images. The processing steps of the two parts are the same, as detailed in (4). The difference between the two parts lies in the input data. The input data for the first part is a training dataset prepared in advance as needed. The input data for the second part is a test dataset. The test dataset can be untrained images prepared in advance.

[0063] (2) Dataset

[0064] The dataset used in this invention consists of real Martian terrain scene images captured by navigation cameras. Three types of labeled images are used for training and testing, including land rocks, gravel, and vehicle bodies. The ratio of the training dataset to the testing dataset is set to 8:2. The training and testing datasets are randomly partitioned. The real Martian terrain scene images need to be preprocessed before being input into the convolutional neural network model.

[0065] The input image size is standardized to 1024×1024 RGB images. The contrast and brightness of the images are then normalized.

[0066] (3) Structure of a dual-branch neural network for Mars terrain segmentation model

[0067] This invention addresses the inaccuracies of manual image segmentation annotation by re-analyzing the causes of segmentation results and redesigning a dual-branch neural network. The dual-branch neural network Mars terrain segmentation model comprises four layers: an input layer, a feature extraction layer, an information processing layer, and an output layer.

[0068] The input layer is the image input part. The training set images are input into the dual-branch neural network Mars terrain segmentation model for specific processing.

[0069] The feature extraction layer is a feature encoder generated by a convolutional neural network, divided into a regular branch encoder and a boundary flow encoder. The regular branch encoder is based on the ResNet-18 network structure, while the boundary branch encoder uses VGG-16 as the model. It extracts rich image features required by the information processing layer from the image of the input layer.

[0070] The information processing layer consists of decoders, which share the image feature information generated by the feature extraction layer to complete the Mars terrain segmentation task.

[0071] The output layer receives the output of the convolutional neural network.

[0072] (4) Specific steps

[0073] Due to the small size of the target training dataset, data augmentation was employed to preprocess the data and maximize accuracy with limited data. The network was then trained using transfer learning. The basic idea is to train the network using both regular flow and boundary flow methods on specially processed images to achieve initial Martian terrain segmentation. The outputs of the dual-branch network are then fused, and their loss function is combined to perform fine-grained Martian terrain segmentation, ultimately defining semantic regions.

[0074] The network is trained using transfer learning. For image processing, the label boundary region is reduced to 0.9 times the original boundary based on the annotation, defined as the regular boundary. The region at the center of the target image with this regular boundary is selected as the input to the regular branch network. Similarly, the boundary is reduced to 0.85 times the original boundary based on the annotation, defined as the edge boundary. The region from the edge boundary to the original label boundary is selected as the input to the edge branch network. Stochastic gradient descent is used to optimize the target loss function J during network training. After the training set images are shuffled, training images are extracted in the shuffled order, ensuring that each image is used only once during iteration. The network is trained until the loss function converges.

[0075] like Figure 1 The specific processing steps are as follows:

[0076] Step 1: Perform data augmentation on the training images to expand the training set.

[0077] Step 2: Input Image. Based on the annotations, the label boundary region is reduced to 0.9 times the original boundary and defined as the regular boundary. The region at the center of the target object with the regular boundary is selected as the input to the regular flow network. Based on the label boundary, it is reduced to 0.85 times the original boundary and defined as the edge boundary. The region from the edge boundary to the original label boundary is selected as the input to the boundary flow network. The input size is uniformly 1024x1024 RGB images.

[0078] Step 3: Use a two-branch neural network. For example... Figure 2 The ResNet-18 network structure is used as a regular stream encoder to encode the input image within a regular bound. After extracting the regular feature maps through encoding, four UNet networks are created to construct four network modules at different scales, adapting to the diverse Martian terrain categories. For example... Figure 3 The VGG-16 model is used as a boundary flow encoder to encode the input image at the edge boundary.

[0079] Step 4: Employ an attention mechanism to fuse the feature maps output from both branches to obtain the segmentation result.

[0080] The segmentation performance of the model is measured by two metrics: Intersection over Union (IOU) and Average Intersection over Union (mIOU). In image segmentation tasks, IOU represents the overlap rate between the "predicted pixel class" and the "ground pixel class," measuring the model's segmentation performance for a single class. MIOU, on the other hand, represents the average IOU across all classes, measuring the model's overall performance. The formulas for IOU and mIOU are shown below:

[0081]

[0082]

[0083] Where TP represents the number of correctly detected samples, FP represents the number of correctly detected samples that were not detected, and FN represents the number of samples that were incorrectly detected.

[0084] Evaluating the segmentation results is essentially evaluating pixel-level classification. The predicted classification results are compared pixel-by-pixel with the labeled images, evaluating the prediction output for each pixel. Let classifying a pixel as a road surface pixel be "positive," and classifying it as background pixel be "negative." The classification result for each pixel is then categorized as one of four: True Positive (TP), False Positive (FP), True Negative (TN), or False Negative (FN). Here, TP represents a positive classification, FP represents a negative classification, TN represents a negative classification, and FN represents a positive classification. Furthermore, the Intersection over Union (IoU) and Average Intersection over Union (AIO) ratios for each image on the entire test set can be calculated.

[0085] The segmentation performance values ​​IOU and MIOU obtained by the method of this invention are shown in Table 1:

[0086] Table 1. Segmentation Performance Values ​​(IOU and MIOU) Results

[0087] Model Name Crushed Stone IOU Lu Yan IOU Vehicle body IOU MIOU Unet 79.1％ 95.5％ 87.3％ 87.3％ Unet++ 81.4％ 95.8％ 90.5％ 89.2％ Method of the present invention 85.7％ 96.1％ 92.7％ 91.5％

[0088] The present invention has been described in detail above with reference to specific embodiments and exemplary examples; however, these descriptions should not be construed as limiting the present invention. Those skilled in the art will understand that various equivalent substitutions, modifications, or improvements can be made to the technical solutions and embodiments of the present invention without departing from the spirit and scope of the invention, and all such modifications and improvements fall within the scope of the present invention. The scope of protection of the present invention is defined by the appended claims.

[0089] The contents not described in detail in this specification are common knowledge to those skilled in the art.

Claims

1. A Martian terrain segmentation method based on a dual-branch input neural network, characterized in that, Includes the following steps: (1) Manually label the obstacle areas in the Mars images to obtain a dataset of Mars terrain segmentation, and divide the dataset into a training set and a test set; (2) Establish a dual-branch neural network model for Mars terrain segmentation; (3) The dual-branch neural network Mars terrain segmentation model is trained using the training set to obtain the trained dual-branch neural network Mars terrain segmentation model. The specific method is as follows: (31) Divide the obstacle regions in the training set images into internal regions and edge regions, and label them respectively; (32) Train the marked internal and edge regions using regular and boundary branches respectively to obtain feature maps of the internal and edge regions; (33) Merge the feature maps of the internal and peripheral regions to obtain the terrain segmentation results; (34) Repeat steps (31) to (33) to train the Mars terrain segmentation model by combining the loss function to obtain the trained model; (4) Input the test set images into the trained Mars terrain segmentation model for semantic segmentation to complete the Mars terrain segmentation task; In step (31), the inner region refers to the set of pixels whose adjacent pixels are all obstacle pixels, and the edge region refers to the set of obstacle pixels in the obstacle region other than the inner region. In step (31), the obstacle region in the training set image is divided into an internal region and an edge region using the transfer learning method; In step (31), the boundary reduced to A times the original boundary of the obstacle area is defined as the regular boundary, and the area with the regular boundary at the center of the target is selected as the internal area; the boundary reduced to B times the original boundary of the obstacle area is defined as the edge boundary, and the area from the edge boundary to the original boundary is selected as the edge area; A = 0.9, B = 0.85; In step (34), the loss function used for the regular branch is the Generalized Dice loss, and the loss function used for the boundary branch is the Focalloss loss function.

2. The Martian terrain segmentation method based on a dual-branch input neural network according to claim 1, characterized in that, In step (32), during regular branch training, four UNet networks are connected based on the ResNet-18 network structure to construct four internal region feature maps of different sizes. During boundary branch training, VGG-16 is used as the model, and four pooling layers are connected to generate four edge region feature maps of different sizes. The sizes of the internal region feature maps and the edge region feature maps correspond one-to-one.

3. The Martian terrain segmentation method based on a dual-branch input neural network according to claim 1, characterized in that, In step (33), an attention mechanism is used to fuse the feature maps of the internal and edge regions.

4. The Martian terrain segmentation method based on a dual-branch input neural network according to claim 1, characterized in that, In step (33), a 1x1 convolutional layer is used to adjust the number of channels of the feature maps of the internal region and the edge region to be equal during the fusion of feature maps of the internal region and the edge region.

5. The Martian terrain segmentation method based on a dual-branch input neural network according to claim 1, characterized in that, In step (34), during the training of the Mars terrain segmentation network model in conjunction with the loss function, the stochastic gradient descent method is used to optimize the loss function, and the Mars terrain segmentation model is trained until the loss function converges.

6. The Martian terrain segmentation method based on a dual-branch input neural network according to claim 1, characterized in that, The dual-branch neural network Mars terrain segmentation model consists of an input layer, a feature extraction layer, an information processing layer, and an output layer. The input layer receives the image and divides the image into internal and edge regions; The feature extraction layer includes a conventional branch encoder and a boundary flow encoder. The conventional branch encoder and the boundary flow encoder are used to train the internal region and the edge region respectively, so as to obtain the feature maps of the internal region and the edge region respectively. The information processing layer consists of decoders, which share feature maps of the internal and edge regions, obtain the segmentation results, and output them through the output layer.

7. The Martian terrain segmentation method based on a dual-branch input neural network according to claim 1, characterized in that, In step (1), the number of obstacle types contained in the Mars image is ≥3; the ratio of the training set to the test set is 7:3 to 9:1.

Images