Method and device for identifying obstacles on celestial body surface

Abstract

The present invention relates to the field of deep space exploration technology, and provides a method and device for identifying obstacles on the surface of a celestial body, including: marking obstacles on an original image set obtained by a deep space exploration rover to obtain a marked sample set; performing feature fusion on each image in the marked sample set to obtain a fused marked sample set; constructing a convolutional neural network, inputting the fused marked sample set into the convolutional neural network, and obtaining a training model with a minimum loss function; performing feature fusion on an image to be identified obtained by a deep space exploration rover, and inputting the fused image to be identified into the training model to obtain a segmentation and identification result of the obstacle. The present invention overcomes the problem of insufficient training data for an extraterrestrial rover, and realizes the ability to detect obstacles on an extraterrestrial body through a single image with high accuracy.

Description

Technical Field

[0001] The present invention relates to the field of deep space exploration technology, and in particular to a method and device for identifying obstacles on the surface of a celestial body. Background Art

[0002] The exploration of the surface of extraterrestrial bodies by patrollers is an important part of future deep space exploration. As extraterrestrial bodies are often far away from the Earth and communication delays are large, it is impossible to complete the exploration mission solely by remote control from the ground. The patrollers themselves need to have considerable autonomous exploration capabilities. For patrollers, moving in the unknown environment of the surface of extraterrestrial bodies is very uncertain. They need to autonomously perceive their surroundings and identify obstacles that are dangerous to their movement (such as rocks, deep pits, etc.), so as to develop a safe route and successfully complete the surface exploration mission.

[0003] At present, the rovers that humans have successfully launched and landed include lunar rovers and Mars rovers, which mainly use stereo vision sensors to detect and identify obstacles. The Spirit and Opportunity Mars rovers, which landed on January 3 and 25, 2004, respectively, mainly used remote operation and semi-autonomous control methods, using stereo vision to draw three-dimensional maps for obstacle detection and navigation; the Curiosity Mars rover, which successfully landed on the surface of Mars in August 2012, also used stereo vision as the main technology for obstacle avoidance, path planning, and navigation and positioning; in December 2013, the "Yutu" rover carried by my country's Chang'e 3 adopted a remote operation working mode and also used stereo vision technology to achieve three-dimensional reconstruction of the unknown environment on the lunar surface, realizing a local autonomous obstacle avoidance method based on stereo vision.

[0004] However, on the one hand, due to the limited baseline length of the patroller, the traditional 3D reconstruction accuracy is low, and the obstacle detection accuracy is not high. The effective obstacle detection range provided by the current stereo camera is generally only within ten meters. Therefore, the 3D reconstruction accuracy problem in medium and long distances may lead to obstacle avoidance and path planning failures; on the other hand, in order to obtain a dense 3D reconstruction map, all image pixels need to be matched, and the stereo matching calculation is large. At the same time, a larger disparity search range is required during the matching search, which further increases the calculation amount. Summary of the invention

[0005] Based on this, an embodiment of the present invention provides a method and device for identifying obstacles on the surface of a celestial body, so as to solve the problem of low accuracy and large amount of calculation in the obstacle identification method of the extraterrestrial rover in the prior art.

[0006] According to a first aspect of an embodiment of the present invention, a method for identifying obstacles on the surface of a celestial body is provided, comprising:

[0007] Obstacles are annotated on the original image set obtained by the deep space exploration rover to obtain an annotated sample set;

[0008] Performing feature fusion on each image in the labeled sample set to obtain the fused labeled sample set;

[0009] Constructing a convolutional neural network, inputting the fused labeled sample set into the convolutional neural network, and obtaining a training model with the minimum loss function;

[0010] The image to be identified obtained by the deep space exploration rover is subjected to feature fusion, and the fused image to be identified is input into the training model to obtain the segmentation and identification result of the obstacle.

[0011] Optionally, the method for performing the feature fusion on each image to be fused includes:

[0012] Calculate the circular LBP feature map of the image to be fused by a pixel sampling method of a circular neighborhood of a first area, and obtain a first window LBP feature map of the corresponding image;

[0013] Calculate the circular LBP feature map of the image to be fused by a pixel sampling method of a circular neighborhood of a second area, and obtain a second window LBP feature map of the corresponding image;

[0014] The image to be fused, the LBP feature map of the first window and the LBP feature map of the second window are respectively used as channel maps of the RGB channels to obtain an image after the features of the image to be fused.

[0015] Optionally, constructing a convolutional neural network, inputting the fused labeled sample set into the convolutional neural network, and obtaining a training model with a minimum loss function includes:

[0016] A convolutional neural network is constructed based on the DeepLabv3+ network, and the fused labeled sample set is input into the convolutional neural network to obtain a training model with the minimum loss function.

[0017] Optionally, the convolutional neural network is constructed based on the DeepLabv3+ network, and the fused labeled sample set is input into the convolutional neural network to obtain a training model with the minimum loss function, including:

[0018] Building a backbone network based on the DeepLabv3+ network, and inputting each fused image in the fused labeled sample set into the backbone network to obtain first-layer features and second-layer features corresponding to the fused image; wherein the backbone network is a deep residual network or a lightweight network;

[0019] The atrous spatial pyramid pooling sequentially convolves and pools each of the first-layer features to obtain a first feature map corresponding to each of the first-layer features;

[0020] Upsampling each of the first feature maps by a residual upsampling conversion method to obtain a corresponding second feature map, and concatenating each of the second feature maps with the corresponding second-layer feature to obtain a concatenated feature map;

[0021] The spliced ​​feature map is upsampled to obtain a segmentation result of the training sample, and a training model with a minimum loss function is determined according to the segmentation result.

[0022] Optionally, upsampling each of the first feature maps by a residual upsampling conversion method to obtain a corresponding second feature map includes:

[0023] Pass in sequence

[0024]

[0025] S i,j =F sim (Q j ,K i )=-||Q j -K i || 2

[0026] W i,j =F w (S i,j )

[0027]

[0028]

[0029] Get the corresponding second feature map f l j Among them, C q , C k , C v Each represents a convolutional layer. Represented by the first feature map f h The feature map obtained by upsampling the preset multiple, express The j-th feature position of Denotes the first feature map f h The i-th feature position, F w represents the sigmoid function, F mul Represents the dot product function.

[0030] Optionally, constructing a convolutional neural network, inputting the fused labeled sample set into the convolutional neural network, and obtaining a training model with a minimum loss function includes:

[0031] Construct a convolutional neural network, input the fused labeled sample set into the convolutional neural network, calculate the loss function of the convolutional neural network, and train the convolutional neural network using a gradient descent method to obtain a training model with the minimum loss function.

[0032] Optionally, the loss function is a smooth IOU loss function, where the expression of IOU is:

[0033]

[0034] Smooth represents the unique hot encoding P of the label in the segmentation task m Perform smoothing as follows:

[0035]

[0036] Among them, M represents the total number of classification categories, m represents one of all classification categories, y represents the label category, and ε is a preset hyperparameter.

[0037] According to a second aspect of an embodiment of the present invention, there is provided a device for identifying obstacles on the surface of a celestial body, comprising:

[0038] An image annotation module is used to annotate obstacles in the original image set obtained by the deep space exploration rover to obtain an annotation sample set;

[0039] A feature fusion module, used for performing feature fusion on each image in the annotated sample set to obtain the fused annotated sample set;

[0040] A model building module, used to construct a convolutional neural network, input the fused labeled sample set into the convolutional neural network, and obtain a training model with the minimum loss function;

[0041] The segmentation and recognition module is used to perform feature fusion on the image to be recognized obtained by the deep space exploration rover, and input the fused image to be recognized into the training model to obtain the segmentation and recognition result of the obstacle.

[0042] According to a third aspect of an embodiment of the present invention, there is provided a device for identifying obstacles on the surface of a celestial body, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the method for identifying obstacles on the surface of a celestial body are implemented as described in any one of the steps provided in the first aspect of the above-mentioned embodiment.

[0043] A fourth aspect of an embodiment of the present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps of the method for identifying celestial body surface obstacles as described in any one of the first aspect of the embodiment.

[0044] Compared with the prior art, the method and device for identifying obstacles on the surface of a celestial body according to the embodiment of the present invention have the following beneficial effects:

[0045] The present invention firstly performs obstacle marking on the original image set obtained by the deep space exploration rover, and then performs feature fusion to obtain a fused marked sample set, then constructs a convolutional neural network, inputs the fused marked sample set into the convolutional neural network, obtains a training model with the minimum loss function, performs feature fusion on the to-be-recognized image obtained by the deep space exploration rover, and inputs the fused to-be-recognized image into the training model to obtain the segmentation and recognition results of the obstacles, thus overcoming the problem of insufficient training data for the extraterrestrial rover, and realizing the detection of obstacles of extraterrestrial celestial bodies through a single image, having high obstacle segmentation and recognition accuracy and fast recognition speed, and being able to accurately segment obstacles that are dangerous to the patrol movement of the celestial body rover, and being suitable for the segmentation and recognition of obstacles on the surfaces of various celestial bodies. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a schematic diagram of the implementation flow of a method for identifying obstacles on the surface of a celestial body provided by an embodiment of the present invention;

[0047] Figure 2 It is a schematic diagram of the implementation process of feature fusion provided by an embodiment of the present invention;

[0048] Figure 3 yes Figure 1 Schematic diagram of the specific implementation process of step S103;

[0049] Figure 4 It is a schematic diagram of the implementation flow of another method for identifying obstacles on the surface of a celestial body provided by an embodiment of the present invention;

[0050] Figure 5 is a schematic diagram of the structure of a convolutional neural network provided by an embodiment of the present invention;

[0051] Figure 6 1 is a schematic diagram of the implementation flow of residual upsampling conversion provided by an embodiment of the present invention;

[0052] Figure 7 It is a structural schematic diagram of a celestial body surface obstacle identification device provided by an embodiment of the present invention;

[0053] Figure 8 It is a schematic diagram of the structure of another celestial body surface obstacle identification device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0054] In the following description, for the purpose of illustration rather than limitation, specific details such as specific system structures and technologies are provided to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may also be implemented in other embodiments without these specific details.

[0055] In order to illustrate the technical solution of the present invention, a specific embodiment is provided below for illustration.

[0056] See also Figure 1 , which is a schematic diagram of an implementation flow of a method for identifying obstacles on the surface of a celestial body provided in this embodiment, is described in detail as follows:

[0057] Step S101, annotate obstacles on the original image set obtained by the deep space exploration rover to obtain a labeled sample set.

[0058] Due to the limited baseline length of the patroller, the traditional three-dimensional reconstruction accuracy is low, and the obstacle detection accuracy is not high. The effective obstacle detection range provided by the current stereo camera is generally only within ten meters. Therefore, in the case of medium and long distances, the three-dimensional reconstruction accuracy problem may lead to the failure of obstacle avoidance and path planning. On the other hand, in order to obtain a dense three-dimensional reconstruction map, it is necessary to match all image pixels. The stereo matching calculation is large, and a large parallax search range is required during the matching search, which further increases the calculation amount. On the other hand, the amount of data from the extraterrestrial patroller is insufficient, which makes the current matching algorithm not very accurate. Therefore, this embodiment provides a method for identifying obstacles on the surface of celestial bodies based on deep learning, which can be used for the segmentation of rocks and craters on the surface of celestial bodies by deep space exploration patrollers, and can also be used for obstacle segmentation of field robots.

[0059] In recent years, deep learning has made rapid progress in the field of computer vision, and in some areas has even shown the potential to surpass the human eye and achieved great success. Unlike traditional recognition algorithms in the field of computer vision, deep learning methods do not require manual construction and feature screening, and perform well in problems such as target detection, segmentation, and classification.

[0060] Specifically, the standard for labeling images in this embodiment can be: (1) for the near view, label all obstacles that threaten the patrol movement of the patroller, that is, obstacles that exceed the designed obstacle-crossing capability of the patroller; (2) for the far view, label only large obstacles. Furthermore, this embodiment can also divide the labeled sample set into training samples, verification samples, and test samples, for example, randomly divide the labeled sample set into training set, verification set, and test set in a ratio of 6:2:2.

[0061] Optionally, this embodiment also cleans the image data to delete dirty data and improve the accuracy of segmentation and recognition. It should be understood that this embodiment does not limit the method of data cleaning.

[0062] Step S102: performing feature fusion on each image in the annotated sample set to obtain the fused annotated sample set.

[0063] Exemplarily, in this embodiment, the original images obtained by the deep space exploration rover can be feature fused to obtain a feature fused labeled sample set, or each image in the labeled sample set can be feature fused to obtain the fused labeled sample set, and then the fused labeled samples can be divided into training samples, verification samples and test samples, such as Figure 4 shown.

[0064] Furthermore, since the sample set images are generally grayscale images, in order to improve the segmentation and recognition effect, all images input to the network model must go through the feature fusion step, that is, the single-channel original image and the different LBP feature maps obtained by the LBP operator are fused. The LBP operator has the advantages of rotation invariance and grayscale invariance. It is a texture description algorithm that describes the local features of the relationship between the central pixel of the image and the neighboring pixels. It is applied to multiple fields such as face, expression, and terrain recognition.

[0065] Optional, see Figure 2 The specific implementation process of the method of performing the feature fusion on each image to be fused in step S102 may include:

[0066] Step S201, calculating the circular LBP feature map of the image to be fused by a pixel sampling method of a circular neighborhood of a first area, and obtaining a first window LBP feature map of the corresponding image.

[0067] Step S202, calculating the circular LBP feature map of the image to be fused by a pixel sampling method of a circular neighborhood of a second area, and obtaining a second window LBP feature map of the corresponding image.

[0068] Step S203: The image to be fused, the LBP feature map of the first window, and the LBP feature map of the second window are respectively used as channel maps of RGB channels to obtain an image after the features of the image to be fused.

[0069] Exemplarily, a pixel sampling method of a circular neighborhood is adopted, and the radius of the circle is set to 8, and a circular LBP feature map of the original image is calculated, which is called a small window LBP feature map (first window LBP feature map). The LBP feature of the small window is conducive to the segmentation and identification of smaller obstacles such as rocks; then a pixel sampling method of a circular neighborhood is adopted, and the radius of the circle is set to 16, and a circular LBP feature map of the original image is calculated, which is called a large window LBP feature map (second window LBP feature map). The LBP feature of the large window is conducive to the segmentation and identification of larger obstacles such as craters; finally, the original image, the small window LBP feature map and the large window LBP feature map are respectively used as channel maps of the RGB channels to obtain a feature-fused image.

[0070] Optionally, the present embodiment may also resample the fused image into an image of 513×513 pixels (RGB three channels) to meet the limitations of training hardware and recognition effect.

[0071] Step S103, constructing a convolutional neural network, inputting the fused labeled sample set into the convolutional neural network, and obtaining a training model with the minimum loss function.

[0072] Optionally, the specific steps of constructing a convolutional neural network in this embodiment may include: constructing a convolutional neural network based on a DeepLabv3+ network, inputting the fused labeled sample set into the convolutional neural network, and obtaining a training model with a minimum loss function, that is, improving the traditional DeepLabv3+ network to obtain the constructed convolutional neural network of this embodiment.

[0073] In one embodiment, see Figure 3 The specific implementation process of building a convolutional neural network based on the DeepLabv3+ network in this embodiment, inputting the fused labeled sample set into the convolutional neural network, and obtaining a training model with the minimum loss function may include:

[0074] Step S301, construct a backbone network based on the DeepLabv3+ network, and input each fused image in the fused labeled sample set into the backbone network to obtain the first layer features and the second layer features corresponding to the fused image; wherein the backbone network is a deep residual network or a lightweight network.

[0075] Specifically, the backbone network of this embodiment may include two forms. One is a deep residual network, which is the basic version of the model of this embodiment. The deep residual network has high obstacle segmentation and recognition accuracy and can accurately segment obstacles that are dangerous to the patrol movement of the celestial body rover. It is suitable for obstacle segmentation and recognition on the surfaces of various celestial bodies. The other is a lightweight network, which is a fast version of the model of this embodiment. It greatly improves the recognition speed while sacrificing a certain recognition accuracy and the recognition accuracy can meet the basic task requirements.

[0076] In step S302, atrous spatial pyramid pooling performs convolution and pooling on each of the first-layer features in sequence to obtain a first feature map corresponding to each of the first-layer features.

[0077] Step S303: upsample each of the first feature maps through a residual upsampling conversion method to obtain a corresponding second feature map, and concatenate each of the second feature maps with the corresponding second-layer feature to obtain a concatenated feature map.

[0078] Step S304, upsampling the spliced ​​feature map to obtain a segmentation result of the training sample, and determining a training model with a minimum loss function according to the segmentation result.

[0079] Exemplarily, the image after feature fusion is input into the backbone network to obtain deep hierarchical features, i.e., the first layer features. After feature extraction by the backbone network, the resolution becomes one sixteenth of the original, and the low-level features obtained in the intermediate process are output with a resolution of one quarter of the original, i.e., the second layer features. Then, atrous spatial pyramid pooling (ASPP) is performed on the output features. Atrous spatial pyramid pooling can achieve multiple effective receptive fields through convolution and pooling operations with different expansion rates, obtain multi-resolution features, and mine multi-scale contextual information. After this process, the encoded high-dimensional features are obtained.

[0080] Furthermore, this embodiment uses a residual upsampling transformer (RUT) to upsample the feature map output by the ASPP by a preset multiple, for example, upsampling by 4 times. Figure 5 As shown, the spliced ​​feature map is obtained by splicing with the low-level features at the same time. That is, this embodiment redesigns the upsampling of DeepLabv3+, designs the residual upsampling conversion, and performs upsampling of the feature map output by ASPP by a preset multiple, and splices it with the low-level features at the same time. Then, the spliced ​​feature map is upsampled again, for example, 4 times upsampling (bilinear interpolation), to obtain the final segmentation result output.

[0081] Optionally, upsampling each of the first feature maps by a residual upsampling conversion method to obtain a corresponding second feature map includes:

[0082] Pass in sequence

[0083]

[0084] S i,j =F sim (Q j ,K i )=-||Q j -K i || 2

[0085] W i,j =F w (S i,j )

[0086]

[0087]

[0088] Get the corresponding second feature map f l j Among them, C q , C k , C v Each represents a convolutional layer. Represented by the first feature map f h The feature map obtained by upsampling the preset multiple, express The j-th feature position of Denotes the first feature map f h The i-th feature position, F w represents the sigmoid function, F mul Represents the dot product function.

[0089] Specifically, Figure 6 As shown, the low-dimensional and high-resolution feature map From the high-dimensional low-resolution feature map f h Obtained by upsampling 4 times by interpolation, and querying key value Among them, l represents a low-dimensional high-resolution feature map, h represents a high-dimensional low-resolution feature map, and C q , C k , C v All of them pass through one convolution layer. The parameters of the three convolution layers may be different, so different symbols are used in this embodiment. The feature map The j-th feature position of is the feature map f h The i-th feature position of .

[0090] According to the above Q i , K i 、V i Calculate the similarity S i,j =F sim (Q j ,K i )=-||Q j -K i || 2 , and then calculate the weight W by the similarity i,j =F w (S i,j ), F w is the sigmoid function, F sim Represents the similarity function; Finally, multiply each channel (the number of channels in this embodiment can be 256) to obtain the residual output F mul is the dot product, and the final output feature is

[0091] The convolutional neural network established by the above method, in order to reduce the amount of calculation, adopts block calculation instead of overall calculation during calculation, so as to retain the context information to the maximum extent, with high recognition accuracy and fast speed.

[0092] In another embodiment, the specific implementation process of constructing a convolutional neural network, inputting the fused labeled sample set into the convolutional neural network, and obtaining a training model with a minimum loss function includes:

[0093] Construct a convolutional neural network, input the fused labeled sample set into the convolutional neural network, calculate the loss function of the convolutional neural network, and train the convolutional neural network using a gradient descent method to obtain a training model with the minimum loss function.

[0094] Optionally, the loss function used in this embodiment is a smooth IOU loss function, where the expression of IOU is:

[0095]

[0096] In order to suppress overfitting, this embodiment introduces label smoothing processing commonly used in classification tasks, that is, the one-hot encoding P of the label in the segmentation task m(one-hot encoding) is smoothed, so in the smooth IOU loss function, smooth represents the one-hot encoding P of the label in the segmentation task m Perform smoothing as follows:

[0097]

[0098] Where M represents the total number of categories, m represents one of all the categories, y represents the label category, and ε is a preset hyperparameter, a smaller hyperparameter. For example, when the backbone network is a deep residual network, ε = 10 -6 , or, when the backbone network is a lightweight network, ε=10 -5 .

[0099] Step S104, performing feature fusion on the image to be identified obtained by the deep space exploration rover, and inputting the fused image to be identified into the training model to obtain the segmentation and identification result of the obstacle.

[0100] Exemplarily, the test samples after the above-mentioned feature fusion can be input into a trained convolutional network, and the trained convolutional network is used to test the test sample set after the feature fusion to obtain the segmentation and recognition results of the obstacles.

[0101] This embodiment uses deep learning to identify obstacles on the surface of celestial bodies. After the network model training is completed, the entire segmentation and recognition process is fully automatic and does not require human intervention. It is suitable for the recognition of obstacles on the surfaces of various celestial bodies. Among them, the original image is fused with the LBP feature map of the original image by utilizing the advantages of rotation invariance and grayscale invariance of the LBP operator. The fused image is used as the input of the entire network, which improves the recognition and segmentation accuracy of obstacles. In addition, this embodiment proposes two versions of backbone networks. The basic version of the backbone network ResNet has the characteristics of high segmentation and recognition accuracy and fast segmentation and recognition speed. The fast version of the backbone network MobileNet has the characteristics of fast segmentation and recognition speed, high recognition accuracy and accuracy. In addition, this embodiment uses a pre-trained model to avoid random initialization of model parameters in the initial stage of model training, which is conducive to targeted training of various extraterrestrial environments, so that the generalization performance of the network model is good. In addition, this embodiment is an end-to-end network model, the early annotation and training are simple, easy to understand and convenient to deploy, reducing the complexity of the project, and is suitable for deep space exploration patrol applications.

[0102] For example, this embodiment can use images taken by the Yutu lunar rover carried by Chang'e 3 as samples. There are three types of segmentation, namely background, rock, and crater. The specific process is as follows:

[0103] (1) Obtain image data taken by the Chang'e-3 lander and Yutu rover, and clean the data (for example, delete duplicate photos and photos that cannot be recognized by the naked eye). After data cleaning, the number of images decreased from 541 to 334.

[0104] (2) The obtained sample set is randomly divided into training set, validation set and test set in the ratio of 6:2:2. Thus, 200 training set samples, 67 validation set samples and 67 test set samples are obtained.

[0105] (3) Calculate the small window LBP feature map and large window LBP feature map of all images respectively, and fuse the RGB three-channel features of the original image, small window LBP feature map, and large window LBP feature map to obtain the feature fusion map. For convenience, all images input to the network are scaled to a size of 513×513 pixels (three channels). To avoid overfitting, all training images are scaled, center cropped, random Gaussian noise, normalized, etc. To calculate the loss, the labels are also processed in the same way (only geometric transformation processing is performed, Gaussian noise and normalization are not required).

[0106] (4) Model construction:

[0107] 1) In the feature extraction process, for the fast segmentation and recognition scenario, mobilenet is used as the backbone for feature extraction, while the basic version uses resnet as the backbone for feature extraction. In the intermediate process, a 24×129×129 feature map is output for subsequent splicing, while compensating for the information loss of high-dimensional features. The final output of feature extraction is a 320×33×33 high-dimensional feature map.

[0108] 2) After the Atrous Spatial Pyramid Pooling (ASPP) module, the feature map becomes 256×33×33.

[0109] 3) After the residual upsampling conversion, it is concatenated with the feature map output by the feature extraction module and passes through a convolution layer, and the feature map becomes 3×129×129 in size. Specifically, F sim =-||Q j -K i || 2 , F w is the sigmoid function, F mul is the dot product.

[0110] 4) Finally, after bilinear interpolation, the output image is 3×513×513 in size, where 3 represents three categories.

[0111] (5) Input the training samples into the constructed neural network, calculate the smoothIOUloss, train the neural network using the gradient descent method, and save the training model with the highest mIOU on the validation set.

[0112] (6) Input the test sample into the trained network model, and use the trained network model to test the test set samples to obtain the segmentation and recognition results of obstacles such as rocks and craters. Table 1 shows the comparison indicators and recognition speed of DeepLabv3+ and the method of the present invention. The backbone networks use mobilenet and resnet respectively, where f / s represents the number of frames recognized per second, and the graphics card used is NVIDIA 2080TI.

[0113] Table 1 Comparison between the traditional DeepLabv3+ recognition method and the method of the present invention

[0114]

[0115] In the above-mentioned method for identifying obstacles on the surface of celestial bodies, obstacles are annotated on the original image set obtained by the deep space exploration rover, and features are fused to obtain a fused annotated sample set, then a convolutional neural network is constructed, the fused annotated sample set is input into the convolutional neural network to obtain a training model with the smallest loss function, features are fused on the images to be identified obtained by the deep space exploration rover, the fused images to be identified are input into the training model, and obstacle segmentation and identification results are obtained, which overcomes the problem of insufficient training data for extraterrestrial rover and realizes the detection of obstacles on extraterrestrial bodies through a single image. The obstacle segmentation and identification has high accuracy and fast identification speed, and can accurately segment obstacles that are dangerous to the patrol movement of the celestial rover, and is suitable for the segmentation and identification of obstacles on the surfaces of various celestial bodies.

[0116] Those skilled in the art will appreciate that the sequence numbers of the steps in the above embodiments do not imply a sequence of execution, and the execution sequence of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0117] Corresponding to the celestial body surface obstacle identification method described in the above embodiment, this embodiment provides a celestial body surface obstacle identification device. Figure 7 , is a schematic diagram of the structure of the celestial body surface obstacle identification device in this embodiment. For the convenience of explanation, only the parts related to this embodiment are shown.

[0118] The celestial body surface obstacle recognition device mainly includes: an image annotation module 110 , a feature fusion module 120 , a model building module 130 and a segmentation and recognition module 140 .

[0119] The image annotation module 110 is used to annotate obstacles on the original image set obtained by the deep space exploration rover to obtain an annotated sample set.

[0120] The feature fusion module 120 is used to perform feature fusion on each image in the annotated sample set to obtain the fused annotated sample set.

[0121] The model building module 130 is used to construct a convolutional neural network, and input the fused labeled sample set into the convolutional neural network to obtain a training model with the minimum loss function.

[0122] The segmentation and recognition module 140 is used to perform feature fusion on the image to be recognized obtained by the deep space exploration rover, and input the fused image to be recognized into the training model to obtain the segmentation and recognition result of the obstacle.

[0123] The above-mentioned celestial body surface obstacle recognition device mainly first performs obstacle annotation on the deep space image set, then performs feature fusion to obtain a fused annotation sample set, then constructs a convolutional neural network, inputs the fused annotation sample set into the convolutional neural network, obtains a training model with the smallest loss function, and finally performs feature fusion on the image to be identified, and inputs the fused image to be identified into the training model to obtain the segmentation and recognition result of the obstacle, which overcomes the problem of insufficient training data for extraterrestrial body patrols, realizes the detection of extraterrestrial obstacles through a single image, has high obstacle segmentation and recognition accuracy, fast recognition speed, and can accurately segment obstacles that are dangerous to the patrol movement of celestial body patrols, and is suitable for the segmentation and recognition of obstacles on the surfaces of various celestial bodies.

[0124] This embodiment also provides a schematic diagram of a celestial body surface obstacle identification device 100. Figure 8 As shown, the celestial body surface obstacle identification device 100 of this embodiment includes: a processor 150, a memory 160, and a computer program 161 stored in the memory 160 and executable on the processor 150, such as a program of a celestial body surface obstacle identification method.

[0125] The processor 150 implements the steps in the above-mentioned celestial body surface obstacle identification method embodiment when executing the computer program 161 in the memory 160, for example Figure 1 Alternatively, when the processor 150 executes the computer program 161, the functions of each module / unit in the above-mentioned device embodiments are realized, for example, Figure 7 The functions of the modules 110 to 140 are shown.

[0126] Exemplarily, the computer program 161 may be divided into one or more modules / units, which are stored in the memory 160 and executed by the processor 150 to complete the present invention. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program 161 in the celestial body surface obstacle identification device 100. For example, the computer program 161 may be divided into an image annotation module 110, a feature fusion module 120, a model building module 130, and a segmentation and identification module 140, and the specific functions of each module are as follows:

[0127] The image annotation module 110 is used to annotate obstacles on the original image set obtained by the deep space exploration rover to obtain an annotated sample set.

[0128] The feature fusion module 120 is used to perform feature fusion on each image in the annotated sample set to obtain the fused annotated sample set.

[0129] The model building module 130 is used to construct a convolutional neural network, and input the fused labeled sample set into the convolutional neural network to obtain a training model with the minimum loss function.

[0130] The segmentation and recognition module 140 is used to perform feature fusion on the image to be recognized obtained by the deep space exploration rover, and input the fused image to be recognized into the training model to obtain the segmentation and recognition result of the obstacle.

[0131] The celestial body surface obstacle identification device 100 may include, but is not limited to, a processor 150 and a memory 160. Those skilled in the art will appreciate that Figure 8 It is only an example of the celestial body surface obstacle identification device 100 and does not constitute a limitation of the celestial body surface obstacle identification device 100. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the celestial body surface obstacle identification device 100 may also include input and output devices, network access devices, buses, etc.

[0132] The processor 150 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0133] The memory 160 may be an internal storage unit of the celestial body surface obstacle identification device 100, such as a hard disk or memory of the celestial body surface obstacle identification device 100. The memory 160 may also be an external storage device of the celestial body surface obstacle identification device 100, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. equipped on the celestial body surface obstacle identification device 100. Further, the memory 160 may also include both the internal storage unit of the celestial body surface obstacle identification device 100 and an external storage device. The memory 160 is used to store the computer program and other programs and data required by the celestial body surface obstacle identification device 100. The memory 160 may also be used to temporarily store data that has been output or is to be output.

[0134] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the division of the above-mentioned functional units and models is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0135] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0136] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0137] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.

Claims

1. A method for identifying obstacles on the surface of a celestial body, characterized in that: include: Obstacles are annotated on the original image set obtained by the deep space exploration rover to obtain an annotated sample set; Performing feature fusion on each image in the labeled sample set to obtain the fused labeled sample set; Constructing a convolutional neural network, inputting the fused labeled sample set into the convolutional neural network, and obtaining a training model with the minimum loss function; Performing feature fusion on the image to be identified obtained by the deep space exploration rover, and inputting the fused image to be identified into the training model to obtain the segmentation and identification result of the obstacle; The convolutional neural network is constructed, and the fused labeled sample set is input into the convolutional neural network to obtain a training model with a minimum loss function, including: Building a backbone network based on the DeepLabv3+ network, and inputting each fused image in the fused labeled sample set into the backbone network to obtain first-layer features and second-layer features corresponding to the fused image; wherein the backbone network is a deep residual network or a lightweight network; The atrous spatial pyramid pooling sequentially convolves and pools each of the first-layer features to obtain a first feature map corresponding to each of the first-layer features; Upsampling each of the first feature maps by a residual upsampling conversion method to obtain a corresponding second feature map, and concatenating each of the second feature maps with the corresponding second-layer feature to obtain a concatenated feature map; The spliced ​​feature map is upsampled to obtain a segmentation result of the training sample, and a training model with a minimum loss function is determined according to the segmentation result.

2. The method for identifying obstacles on the surface of a celestial body according to claim 1, characterized in that: The method for performing the feature fusion on each image to be fused includes: Calculate the circular LBP feature map of the image to be fused by a pixel sampling method of a circular neighborhood of a first area, and obtain a first window LBP feature map of the corresponding image; Calculate the circular LBP feature map of the image to be fused by a pixel sampling method of a circular neighborhood of a second area, and obtain a second window LBP feature map of the corresponding image; The image to be fused, the LBP feature map of the first window and the LBP feature map of the second window are respectively used as channel maps of the RGB channels to obtain an image after the features of the image to be fused.

3. The method for identifying obstacles on the surface of a celestial body according to claim 1, characterized in that: The method of upsampling each of the first feature maps by the residual upsampling conversion method to obtain a corresponding second feature map includes: Pass in sequence Get the corresponding second feature map Among them, C q , C k , C v Each represents a convolutional layer. Represented by the first feature map f h The feature map obtained by upsampling the preset multiple, express The j-th feature position of Denotes the first feature map f h The i-th feature position, F w represents the sigmoid function, F mul Represents the dot product function.

4. The method for identifying obstacles on the surface of a celestial body according to claim 1, characterized in that: The convolutional neural network is constructed, and the fused labeled sample set is input into the convolutional neural network to obtain a training model with a minimum loss function, including: Construct a convolutional neural network, input the fused labeled sample set into the convolutional neural network, calculate the loss function of the convolutional neural network, and train the convolutional neural network using a gradient descent method to obtain a training model with the minimum loss function.

5. The method for identifying obstacles on the surface of a celestial body according to claim 4, characterized in that: The loss function is the smoothIOU loss function, where the expression of IOU is: Smooth represents the unique hot encoding P of the label in the segmentation task m Perform smoothing as follows: Among them, M represents the total number of classification categories, m represents one of all classification categories, y represents the label category, and ε is a preset hyperparameter.

6. A device for identifying obstacles on the surface of a celestial body, characterized in that: include: An image annotation module is used to annotate obstacles in the original image set obtained by the deep space exploration rover to obtain an annotation sample set; A feature fusion module, used for performing feature fusion on each image in the annotated sample set to obtain the fused annotated sample set; A model building module, used to construct a convolutional neural network, input the fused labeled sample set into the convolutional neural network, and obtain a training model with the minimum loss function; A segmentation and recognition module is used to perform feature fusion on the image to be recognized obtained by the deep space exploration rover, and input the fused image to be recognized into the training model to obtain the segmentation and recognition result of the obstacle; The model building module is specifically used for: Building a backbone network based on the DeepLabv3+ network, and inputting each fused image in the fused labeled sample set into the backbone network to obtain first-layer features and second-layer features corresponding to the fused image; wherein the backbone network is a deep residual network or a lightweight network; The atrous spatial pyramid pooling sequentially convolves and pools each of the first-layer features to obtain a first feature map corresponding to each of the first-layer features; Upsampling each of the first feature maps by a residual upsampling conversion method to obtain a corresponding second feature map, and concatenating each of the second feature maps with the corresponding second-layer feature to obtain a concatenated feature map; The spliced ​​feature map is upsampled to obtain a segmentation result of the training sample, and a training model with a minimum loss function is determined according to the segmentation result.

7. A device for identifying obstacles on the surface of a celestial body, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method for identifying obstacles on the surface of a celestial body as described in any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method for identifying obstacles on the surface of a celestial body as described in any one of claims 1 to 5 are implemented.

Images