Mountain top recognition method and system based on multimodal convolutional neural network

Through the method based on multimodal convolutional neural network, combined with Faster R-CNN and fuzzy fusion technology, the problem of poor mountain vertex recognition in the existing technology is solved, and more accurate and robust mountain vertex recognition is achieved.

CN118552857BActive Publication Date: 2024-11-05NAT UNIV OF DEFENSE TECH +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411019632.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-29
Publication Date
2024-11-05
Estimated Expiration
2044-07-29

AI Technical Summary

Technical Problem

The existing mountain peak recognition method needs to be improved in terms of recognition effect, and it is easy to cause missed or incorrect raising, especially when the terrain is complex and the DEM data accuracy is limited.

Method used

The multimodal convolutional neural network method is used to calculate the terrain parameter model through DEM data, and multimodal images are produced. Faster R-CNN deep learning network is used for training, and the fuzzy fusion method is used to fuse the recognition results of different modal images. Extract accurate mountain apex.

Benefits of technology

It reduces the phenomenon of missing and mis-lifting of mountain tops, improves the accuracy and robustness of recognition, and can more effectively identify mountain top features in the terrain.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118552857B_ABST
    Figure CN118552857B_ABST
Patent Text Reader

Abstract

The present invention proposes a mountain top recognition method and system based on a multimodal convolutional neural network. The method includes: based on DEM data, a terrain parameter model is calculated by a terrain analysis method, and a multimodal image is produced by using different visualization methods; then the mountain top area is annotated; based on a deep learning network, different modal images after annotation are used for training to obtain mountain top recognition models corresponding to different modalities; mountain tops are recognized on different modal images by using the mountain top recognition models corresponding to different modalities; mountain tops recognized by different modal images are fused by a fuzzy fusion method; according to the fused mountain top recognition result, the identified mountain top area is mapped onto the corresponding DEM data, and the extreme elevation points in the mountain top area are extracted as accurate mountain tops. The present invention uses a fuzzy fusion method to achieve the fusion of different modal image recognition results, and obtains the most accurate mountain top recognition result.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of mountain top recognition, and in particular to a mountain top recognition method and system based on a multimodal convolutional neural network. Background Art

[0002] The mountain top point, also known as the peak or peak, refers to the maximum point of elevation in a local area, which is convex in all directions. The mountain top point is one of the most controlling feature points of the surface morphology, and has an important influence on landform morphology analysis, hydrological basin analysis, vegetation distribution analysis, and military key target selection.

[0003] The mountain top is usually understood as the highest point of a mountain. It is a common geographical phenomenon that is understandable but difficult to quantify accurately. A mountain in geomorphological terms is defined as a highland with an altitude of more than 500m, a relative relief of more than 200m, and a steep slope. From top to bottom, a mountain is divided into three parts: the mountain top, the slope, and the foothills. When this definition is introduced into the field of DEM digital terrain analysis based on quantitative analysis, there are some difficult threshold issues, especially the relative relief within a certain range and the definition of the range of the mountain.

[0004] DEM is the data basis of conventional mountain top extraction algorithms. It can be used to analyze the terrain characteristics in the local area and extract basic features such as mountain tops, saddles, valley points, ridge lines, valley lines, etc. Therefore, using DEM data to calculate basic terrain parameters, such as slope, aspect, undulation, etc., and extracting mountain tops through threshold setting in the local analysis area is one of the commonly used algorithms. This type of method determines the mountain top by calculating the changing relationship between the terrain parameters between the neighborhoods in the analysis window, and can achieve rapid extraction of mountain tops, but they mostly focus on the local characteristics of the mountain top, which is prone to false extraction or omission. In order to improve the limitations of the mountain top extraction algorithm based on DEM, Zhong Teng et al. [Zhong Teng, Tang Guoan, Zhou Yi, et al. Automatic extraction of mountain tops based on inverse terrain DEM [J]. Bulletin of Surveying and Mapping, 2009 (4): 35-37.)] and Liu Hongjian et al. [Liu Hongjian, Liu Jianzhong, Cai Zhongxiang, et al. A mountain top extraction method based on watershed division [J]. Journal of Surveying and Mapping Science and Technology, 2014 (2): 119-122.] adopted the hydrological watershed analysis method, that is, by simulating surface water flow and constructing a watershed calculation model to identify mountain tops, although this method can effectively improve the limitations of the analysis window, it is still affected by factors such as terrain complexity and DEM data accuracy, is more sensitive to terrain micro-undulations, and is prone to false mountain tops.

[0005] In recent years, deep learning technologies represented by convolutional neural networks (CNNs) have shown great advantages in target detection. Target detection is a behavior that finds and marks areas of interest in digital images or videos. It can extract features through specific algorithms and identify and locate objects of a specific category. If a DEM is visualized as a terrain image, then the identification of a mountain top is similar to finding a specific target in an image, which is the mountain top.

[0006] CNN-based object detection algorithms include two technical development routes: Anchor-based and Anchor-free. Anchor-based methods include One-Stage object detection algorithms and Two-Stage object detection algorithms. The Two-stage object detection algorithm divides the detection problem into two stages: first, a candidate region containing the approximate location information of the target is generated, and then the candidate region is classified and the position is refined. R-CNN, Fast R-CNN, Faster R-CNN, etc. are typical representatives of the Two-stage object detection algorithm. The One-Stage object detection algorithm does not require a candidate region stage. It directly generates the category probability and position coordinate value of the object through a stage. YOLO, SSD, RetinaNet, etc. are typical representatives of the One-stage object detection algorithm.

[0007] Du et al. [Du L, You X, Li K, et al. Multi-modal deep learning for landform recognition[J].ISPRS Journal of Photogrammetry and Remote Sensing, 2019, 158:63-75.] used a deep learning network to extract the physical and visual features of landforms from three different data types, including shading maps, elevation and slope. The residual learning model was used to mine the correlation information between different modal features, learn the multimodal landform data features, fuse and generate the deep features of the landforms, and realize the joint expression of landform data. The experimental results show that compared with other landform recognition algorithm methods, the multimodal landform recognition algorithm based on deep learning has better performance; however, loess landforms, karst landforms, etc. are more descriptions of smaller-scale surface morphology, which leads to fatal flaws in this method. Summary of the invention

[0008] In view of the problem that the recognition effect of existing mountain top point recognition methods needs to be improved, the present invention proposes a mountain top point recognition method and system based on a multimodal convolutional neural network, studies the recognition effects of different modal data, and then uses a fuzzy fusion method to realize the fusion of different modal image recognition results to obtain the most accurate mountain top point recognition result, reducing the phenomenon of missed and wrong mountain top points.

[0009] In order to achieve the above object, the present invention adopts the following technical solutions:

[0010] On the one hand, the present invention proposes a mountain top recognition method based on a multimodal convolutional neural network, comprising:

[0011] Step 1: Based on DEM data, the terrain parameter model is calculated through terrain analysis methods, and different visualization methods are used to produce multimodal images;

[0012] Step 2: Annotate the mountain top area of ​​the multimodal image;

[0013] Step 3: Based on the Faster R-CNN deep learning network, the labeled images of different modalities are used for training to obtain the peak point recognition model corresponding to different modalities; Faster R-CNN uses ResNet50 for feature extraction;

[0014] Step 4: Identify the peak points of images of different modalities through the peak point recognition models corresponding to different modalities;

[0015] Step 5: Use the fuzzy fusion method to fuse the peak points identified by different modal images;

[0016] Step 6: Based on the fused mountain top recognition results, the identified mountain top area is mapped to the corresponding DEM data, and the extreme elevation points in the mountain top area are extracted as the accurate mountain top.

[0017] Furthermore, the multimodal image includes: a contour map, a layered color map, a gradient layered color map, a grayscale shading map, a color shading map, and a slope map.

[0018] Furthermore, the step 1 comprises:

[0019] Based on the terrain analysis and mapping functions provided by Golden Software Surfer, the original image data of various modes of the experimental area were produced, and the layered color map, gradient layered color map and grayscale shading map were superimposed with contour line information at the same time.

[0020] Furthermore, in step 2, labelImg software is used to label the mountain top area, and the labeled area contains at least two or three closed contour lines.

[0021] Furthermore, the step 5 comprises:

[0022] First, determine the position of the rectangular recognition area. If there is only one modal recognition result and the confidence level is greater than 0.8, the recognition result is identified as the peak point. If there are multiple modal recognition results, determine the degree of overlap of the rectangular recognition areas of different modalities. If the overlap exceeds 80%, it is determined that the different modal recognitions are a peak point. Second, for overlapping recognition results, calculate the area of ​​the overlapping area as A0, and then use the following formula for calculation:

[0023]

[0024] Where p represents the recognition confidence corresponding to the overlapping area, A i represents the area of ​​the peak point of the i-th modality image recognition in the overlapping recognition results, p i represents the recognition confidence corresponding to the i-th modality image in the overlapping recognition results, and n is the total number of modality image types in the overlapping recognition results;

[0025] If p is greater than 0.8, the center point of the rectangular area where the overlapping area is located is determined as the mountain top recognition result.

[0026] Another aspect of the present invention provides a mountain top recognition system based on a multimodal convolutional neural network, comprising:

[0027] A multimodal image production unit is used to calculate the terrain parameter model based on DEM data through terrain analysis methods, and to produce multimodal images using different visualization methods;

[0028] A mountain top point region labeling unit, used for labeling the mountain top point region of the multimodal image;

[0029] The mountain top recognition model training unit is used to train the annotated images of different modalities based on the Faster R-CNN deep learning network to obtain the mountain top recognition models corresponding to different modalities; the Faster R-CNN uses ResNet50 for feature extraction;

[0030] A mountain top point recognition unit, used for recognizing mountain top points of images of different modalities by using mountain top point recognition models corresponding to different modalities;

[0031] A different modality recognition result fusion unit is used to fuse the peak points of different modality image recognitions by using a fuzzy fusion method;

[0032] The mountain top extraction unit is used to map the identified mountain top area onto the corresponding DEM data according to the fused mountain top recognition result, and extract the extreme elevation point in the mountain top area as the accurate mountain top.

[0033] Furthermore, the multimodal image includes: a contour map, a layered color map, a gradient layered color map, a grayscale shading map, a color shading map, and a slope map.

[0034] Furthermore, the multimodal image production unit is specifically used for:

[0035] Based on the terrain analysis and mapping functions provided by Golden Software Surfer, the original image data of various modes of the experimental area were produced, and the layered color map, gradient layered color map and grayscale shading map were superimposed with contour line information at the same time.

[0036] Furthermore, in the mountain top point area labeling unit, labelImg software is used to implement the labeling of the mountain top point area, and the labeled area contains at least two or three closed contour lines.

[0037] Furthermore, the different modality recognition result fusion unit is specifically used for:

[0038] First, determine the position of the rectangular recognition area. If there is only one modal recognition result and the confidence level is greater than 0.8, the recognition result is identified as the peak point. If there are multiple modal recognition results, determine the degree of overlap of the rectangular recognition areas of different modalities. If the overlap exceeds 80%, it is determined that the different modal recognitions are a peak point. Second, for overlapping recognition results, calculate the area of ​​the overlapping area as A0, and then use the following formula for calculation:

[0039]

[0040] Where p represents the recognition confidence corresponding to the overlapping area, A i represents the area of ​​the peak point of the i-th modality image recognition in the overlapping recognition results, p i represents the recognition confidence corresponding to the i-th modality image in the overlapping recognition results, and n is the total number of modality image types in the overlapping recognition results;

[0041] If p is greater than 0.8, the center point of the rectangular area where the overlapping area is located is determined as the mountain top recognition result.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] The present invention is based on DEM data, calculates terrain parameter models through terrain analysis methods, and uses different visualization methods to produce multimodal images; annotates the mountain top area of ​​the multimodal image; based on the Faster R-CNN deep learning network, uses the annotated different modal images for training to obtain mountain top recognition models corresponding to different modalities; recognizes mountain tops of different modal images through mountain top recognition models corresponding to different modalities; uses a fuzzy fusion method to fuse mountain tops recognized by different modal images; maps the recognized mountain top area to the corresponding DEM data according to the fused mountain top recognition result, extracts the extreme elevation points in the mountain top area as accurate mountain tops. The present invention studies the recognition effect of different modal data, and then uses a fuzzy fusion method to realize the fusion of different modal image recognition results, obtains the most accurate mountain top recognition result, and reduces the phenomenon of missing and wrong mountain tops. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 This is a flowchart of a method for identifying mountain tops based on a multimodal convolutional neural network according to an embodiment of the present invention;

[0045] Figure 2 This is an example diagram of the experimental area used in the embodiment of the present invention;

[0046] Figure 3 The sample data types and production processes used in the embodiments of the present invention;

[0047] Figure 4 An example diagram of sample data produced for an embodiment of the present invention;

[0048] Figure 5 An example diagram of sample data annotation provided by an embodiment of the present invention;

[0049] Figure 6 A training batch graph provided by an embodiment of the present invention;

[0050] Figure 7 A loss function diagram provided by an embodiment of the present invention;

[0051] Figure 8 The recognition effect diagram provided by the embodiment of the present invention;

[0052] Fig. 9 This is an example diagram of a test data set used in an embodiment of the present invention;

[0053] Fig.10 A confirmed mountain top point of the Motianling test area provided in an embodiment of the present invention;

[0054] Fig.11 The grayscale terrain shading image recognition result provided by the embodiment of the present invention;

[0055] Fig.12 The Motianling test area recognition result provided by the embodiment of the present invention;

[0056] Fig.13 The present invention is a schematic diagram of the architecture of a mountain top recognition system based on a multimodal convolutional neural network according to an embodiment of the present invention. DETAILED DESCRIPTION

[0057] The present invention will be further explained below with reference to the accompanying drawings and specific embodiments:

[0058] This paper proposes a mountain top recognition method based on multimodal convolutional neural network, and attempts to use FasterR-CNN deep learning network to realize automatic recognition and extraction of mountain tops based on DEM visualization image data of different models. The basic steps are as follows: Figure 1 As shown, including:

[0059] The first step is to create sample data. Based on DEM data, terrain parameter models such as slope, aspect, curvature, etc. are calculated through terrain analysis methods, and multi-modal images such as surface elevation, surface slope, and surface aspect are created using different visualization methods. They are the data basis for deep learning sample data creation;

[0060] The second step is to mark the mountain top. Since each DEM data is expanded into contour maps, layered color maps, gradient layered color maps, grayscale shading maps, color shading maps, as well as slope maps, aspect maps, curvature maps and other multi-modal images, marking the mountain top point on a contour map is equivalent to completing the marking of other modal images;

[0061] The third step is to train the Faster R-CNN model. The present invention uses the Faster R-CNN deep learning network as the model for mountain top recognition. Since the image data of different modalities corresponding to the same DEM data have the same mountain top annotation information, the model training process is based on the same modality data for training, that is, a model is trained for the contour map, and a model is trained for the layered color map. The recognition effect of different modality data is studied, and then the recognition data of different modalities are integrated to obtain the most accurate recognition result. This is a key idea proposed by the present invention;

[0062] Step 4: Predict the mountain top. According to the training results of the model, predict the mountain top of the test set sample data respectively, and obtain the prediction results of the mountain top area of ​​different modal images, that is, use the prediction models of different modalities to predict the corresponding modal data. For example, the contour map prediction set data of the same DEM data uses the contour map training model to predict the prediction results under the contour map training model, and the layered color map prediction set data of the same DEM data uses the layered color map training model to predict the prediction results under the layered color map training model, and so on to obtain the mountain top prediction results of multiple modalities;

[0063] The fifth step is to fuse the peak points of different modal image recognition. In theory, the prediction results of multiple different modalities are similar, but due to differences in visual expression effects, the experimental results are not consistent. This is because the differences in the feature maps of different modalities lead to differences in recognition prediction effects. Modality A can recognize the peak point, but modality B may not be able to recognize it. Here, the confidence value of the prediction result is required. The fuzzy fusion method is used to achieve the fusion of different modal image recognition results, reducing the phenomenon of missing or mistaking the peak point;

[0064] Step 6: Extract the mountain top. Based on the prediction results, map the identified mountain top area to the corresponding DEM data, and extract the extreme elevation points in the mountain top area as the accurate mountain top value.

[0065] 1. Preparation of sample set data

[0066] 1.1 Sample data source

[0067] In order to better realize the extraction of terrain features based on deep learning, SRTM DEM data of five experimental areas with different terrain features were selected as the data source for the sample data set, including an experimental area in Zhejiang Province representing hilly terrain (such as Figure 2 (a) in the middle), and a certain experimental area in Henan Province showing low mountain plain terrain (such as Figure 2 (b)), representing a certain experimental area in Shaanxi with low mountain loess terrain (such as Figure 2 (c)), representing an experimental area in Guizhou with medium mountainous terrain (such as Figure 2 d), and an experimental area in Tibet representing high mountain terrain (such as Figure 2 (shown in (e)).

[0068] The mountain top extraction algorithm based on deep learning is more about judging the terrain features in the appropriate area from a local perspective. In essence, there is little practical significance in selecting experimental areas with different terrains. However, in terms of the integrity of the experimental samples, selecting sample data from different experimental areas will help to make the experimental results more robust.

[0069] 1.2 Sample data preparation

[0070] The mountain top sample data set should reflect the topographic features of the mountain top as much as possible, and allow deep learning to mine the most advantageous feature maps. To this end, according to the experimental settings, 9 different modal images such as contour map, layered color map, gradient layered color map, grayscale shading map, color shading map, slope map, slope aspect map, profile curvature map, and plane curvature map are selected as the basis for sample set production ( Figure 3 ).

[0071] Golden Software Surfer was used to produce various modal data. First, the mapping function provided by GoldenSoftware Surfer was used to produce the original image data of various modes of the five experimental areas. At the same time, in order to enrich the graphic information, the layered color map, gradient layered color map and grayscale shading map were superimposed with contour information; secondly, the terrain analysis function provided by Golden Software Surfer was used to produce slope data, aspect data and curvature data respectively, and then the mapping function of Golden Software Surfer was used to produce image data of the corresponding mode; finally, the corresponding small area image data was randomly selected and used as sample data for deep learning. Obviously, in this process, the range of different modal data selected in the same experimental area is consistent. Figure 4 As shown, only 6 types of image data are shown here. Figure 4 (a) is a contour map, (b) is a layered color map, (c) is a gradient layered color map, (d) is a grayscale shading map, (e) is a color shading map, and (f) is a slope map. This is because in the actual training process, the slope direction, curvature and other effects are not good, so the corresponding experimental operations were not continued.

[0072] 1.3 Sample Data Labeling

[0073] After completing the mountain top sample set, it is necessary to label the mountain top area of ​​each sample data. Use labelImg software to label the mountain top, such as Figure 5 As shown in the figure, the mountain top area of ​​the two sample data is marked.

[0074] Compared with the annotation of people, animals, vehicles, etc., the annotation of the mountain top area is a representation of professional knowledge. For the annotation of the mountain top area, if the area is set too large, it may contain interference information. If the annotation is too small, it may lead to insufficient feature information. Therefore, in the annotation process, we should follow the principle of "the annotation area contains at least two or three closed contour lines". This is why the contour line information is superimposed in the sample data set production process.

[0075] In addition, the same annotation data is directly used for the annotation of other modal data in the same DEM area.

[0076] 2 Faster R-CNN Model

[0077] The present invention uses Faster R-CNN as a convolutional neural network model for deep identification of mountain peaks. Faster R-CNN was proposed by Ross B. Girshick in 2016. It integrates feature extraction, proposal extraction, bounding box regression, and classification into one network, greatly improving the overall performance, especially in terms of detection speed.

[0078] Faster R-CNN mainly consists of 4 parts.

[0079] (1) Convolutional layers. As a CNN network object detection method, Faster R-CNN uses a set of basic Conv+Relu+Pooling layers to extract feature maps of images; the feature maps will be shared for subsequent RPN layers and fully connected layers. As an implementable method, ResNet50 is specifically used to extract feature maps of images.

[0080] (2) Region Proposal Networks (RPN). RPN is used to generate proposed regions. This layer uses Softmax to determine whether the anchors are positive or negative, and then uses bounding box regression to refine the anchors to obtain accurate candidates.

[0081] (3) Region of Interest Pooling (ROI Pooling). It collects the input feature map and the proposed region, extracts the proposed feature map after integrating this information, and sends it to the subsequent fully connected layer to determine the target category.

[0082] (4) Classification: The proposed feature map is used to calculate the category of the proposed area, and the bounding box is regressed again to obtain the final precise position of the detection box.

[0083] 3 Faster R-CNN model training

[0084] The sample data set of the same modality includes a total of 3320 sample data, of which 3320*0.9=2988 are used for deep learning model training, and the remaining 332 are used for testing. The number of training set sample data and test set sample data is randomly selected by the system.

[0085] The system has been tested for 100 batches ( Figure 6 , where mAP stands for mean average precision, and mAP 0.5 stands for mean average precision when the intersection-over-union threshold is 0.5). After about 24 hours of model training, we obtained the following Figure 7 The loss function graph shown shows that the training fit gradually slows down.

[0086] 4. Fusion of recognition results of different modal data

[0087] Since the image data of different modalities contain different information, the recognition results may be different. It is necessary to fuse the recognition results of different modal image data in the same area to improve the overall recognition quality. In theory, the prediction results of the six different modalities are similar, but due to the difference in visual expression effects, the experimental results are not consistent. This is because the difference in the feature maps of different modalities leads to differences in recognition prediction effects. Modality A can identify the peak point, but modality B may not be able to recognize it. Here, the confidence value of the prediction result is required. The fuzzy fusion method is used to achieve the fusion of different modal image recognition results, reducing the phenomenon of missing and mistaking the peak point. First, determine the position of the rectangular recognition area. If there is only one modal recognition result and the confidence is greater than 0.8, it can be identified as the peak point; if there are multiple modal recognition results, then determine the overlap of the different modal recognition rectangles. If the overlap exceeds 80%, it can be determined that the different modal recognitions are a peak point; secondly, for the overlapping recognition results, calculate the area of ​​the overlapping area as A0, and then use the following formula to calculate:

[0088]

[0089] Where p represents the confidence of the identification corresponding to the overlapping area (the mountain top area), A i represents the area of ​​the peak point of the i-th modality image recognition in the overlapping recognition results, p i It represents the confidence of the recognition (for the mountain top area) corresponding to the i-th modal image in the overlapping recognition results, and n is the total number of modal image types in the overlapping recognition results. If p is greater than 0.8, then the center point of the rectangular area where the overlapping area is located can be determined as the mountain top point recognition result.

[0090] For the peak point area recognition result data, it is reflected in the recognized peak point rectangular area and recognition confidence, that is, the recognized peak point area is displayed in two pairs of (x, y) coordinates, supplemented by the confidence of the recognized peak point area. Figure 8 As shown in the figure, the area of ​​the identified mountain top is [121,78]\[247,216], and the confidence level is 0.99. This almost confirms that the center point of this area is the mountain top. From a more rigorous perspective, the coordinate point pairs of the image can be mapped to the DEM data, and the extreme high point can be found in the identified area to determine the mountain top.

[0091] To verify the effect of the present invention, the following experiment was performed:

[0092] (1) Test data source

[0093] Various types of terrain data in Dengfeng area of ​​Zhengzhou City, Henan Province were selected as the test data source for deep learning recognition, such as Fig. 9 As shown, (a) is the Motianling test area, (b) is the Yuzhaishan test area, (c) is the Songshan test area, (d) is the Baishijian test area, (e) is the Jianshao-Laohutouzhai test area, and (f) is the Dafangshan-Laolongtou test area.

[0094] for Fig. 9 In the Motianling test area shown in (a), it can be clearly considered that "Xiaoelingkou", "Xuanyuan Mountain", "Motianling", "near Tangyao Village", "Dharma Cave", "Jiming Mountain", "886 Highland" and "903 Highland" are the topographic feature points of the mountain top. Fig.10 shown.

[0095] for Fig. 9 In the Yuzhaishan test area shown in (b), it can be clearly identified that “Yuzhaishan”, “Jiuduolianhuashan”, “903 Highland”, “963 Highland”, “960 Highland”, “1241 Highland”, “1088 Highland”, “1499 Highland”, “1485 Highland”, “1487 Highland”, “1387 Highland”, “1159 Highland”, “1082 Highland”, “1137 Highland” and “977 Highland” are the topographic feature points of the mountain top.

[0096] for Fig. 9 In the Songshan test area shown in (c), it can be clearly seen that "Songshan", "691 Highland", "1171 Highland", "1441 Highland", "892 Highland" and "779 Highland" are the topographic feature points of the mountain top.

[0097] for Fig. 9 In the Baishijian experimental area shown in (d), it can be clearly seen that "Baishijian", "930 Highland" and "Xiezi Mountain" are the topographic feature points of the mountain top.

[0098] for Fig. 9 In the Jianshao-Laohutouzhai test area shown in (e), it can be clearly identified that "Jianshao", "Laohutouzhai", "Lintai Mountain", "744 Highland", "713 Highland", "623 Highland", "758 Highland", "522 Highland", "652 Highland", "592 Highland", "723 Highland", "891 Highland", "701 Highland", "881 Highland" and "768 Highland" are the topographic feature points of the mountain top.

[0099] for Fig. 9 In the Dafangshan-Laolongtou test area shown in (f), it can be clearly identified that "Hutou", "Laolongtou", "Dafangshan", "Guogongling", "847 Highland", "971 Highland" and "788 Highland" are the topographic feature points of the mountain top.

[0100] (2) Identification results

[0101] According to the sample data production process, the present invention produces test image data of different modes in the above six test areas. The size of the test data is 1000*1000 pixels. Then, six different modes of training, such as contour map, layered color map, gradient layered color map, grayscale shading map, color shading map, and slope map, are used for corresponding recognition. Taking the grayscale terrain shading map as an example, the effect map obtained is as follows: Fig.11 shown.

[0102] for Fig.11 In the Motianling test area shown in (a), the deep learning model identified the topographic feature points of the mountain tops of "Xiaoelingkou", "Xuanyuan Mountain", "Motianling", "near Tangyao Village", "Dharma Cave", "Jiming Mountain", "886 Highland", and "903 Highland". It also identified two more topographic feature points of "Xuanyuangou" and "Liang B343". However, it is a pity that "Dharma Cave" and "Jiming Mountain" were not distinguished. Overall, for the Motianling test area, the recognition accuracy of the terrain feature recognition algorithm based on deep learning reached 120%.

[0103] for Fig.11In the Yuzhaishan test area shown in (b), the deep learning model identified the topographic feature points of the tops of Yuzhaishan, Jiuduolianhuashan, 963 Highland, 1241 Highland, 1088 Highland, 1499 Highland, 1485 Highland, 1487 Highland, 1387 Highland, 1159 Highland, and 977 Highland, and also identified the topographic feature point of 886 Highland. However, 903 Highland and 960 Highland were not identified, and since 1082 Highland and 1137 Highland were close to each other, they were identified as one topographic feature point, and the approximate range was correctly marked. Two saddle points were also identified. Overall, for the Motianling test area, the recognition accuracy of the terrain feature recognition algorithm based on deep learning reached 83%.

[0104] for Fig.11 In the Songshan test area shown in (c), the deep learning model identified the topographic feature points of the topography of "Songshan", "1171 Highland", "1441 Highland", and "892 Highland", and also identified the topographic feature point of "1073 Highland". However, "691 Highland" and "779 Highland" were not identified. A saddle point was also identified. Overall, for the Songshan test area, the recognition accuracy of the terrain feature recognition algorithm based on deep learning reached 83%.

[0105] for Fig.11 In the Baishijian test area shown in (d), the deep learning model identified "Baishijian" and "930 Highland" as topographic feature points on the mountain tops, and also identified topographic feature points near "930 Highland". However, "Xiezi Mountain" was not identified. Overall, for the Baishijian test area, the recognition accuracy of the terrain feature recognition algorithm based on deep learning reached 75%.

[0106] for Fig.11 In the Jianshao-Laohutou Village test area shown in (e), the deep learning model identified "Jianshao", "Laohutou Village", "Lintai Mountain", "744 Highland", "713 Highland", "758 Highland", "522 Highland", "652 Highland", "592 Highland", "891 Highland", "701 Highland", "881 Highland", and "768 Highland" as mountaintop terrain feature points, and also identified more mountaintop feature points near "Miaopojian" and "768 Highland". However, "623 Highland" and "723 Highland" were not identified. Overall, for the Jianshao-Laohutou Village test area, the recognition accuracy of the terrain feature recognition algorithm based on deep learning reached 87%.

[0107] for Fig.11In the Dafangshan-Laolongtou test area shown in (f), the deep learning model identified "Tiger Head", "Laolongtou", "Dafangshan", "Guogongling", "847 Highland", "971 Highland", and "788 Highland" as mountaintop terrain feature points, and also identified more mountaintop feature points near "847 Highland". Overall, for the Dafangshan-Laolongtou test area, the recognition accuracy of the terrain feature recognition algorithm based on deep learning reached 100%.

[0108] Compare the recognition results of different modal data in the same test area horizontally. Taking the Motianling test area as an example, the actual confirmed mountain top points are as follows: Fig.10 As shown in Figure 2, the recognition results of different modal data are as follows: Fig.12 As shown, Fig.12 (a) is the recognition result corresponding to the contour map, (b) is the recognition result corresponding to the grayscale shading map, (c) is the recognition result corresponding to the gradient layered color map, (d) is the recognition result corresponding to the layered color map, (e) is the recognition result corresponding to the color shading map, and (f) is the recognition result corresponding to the slope map.

[0109] There are differences in the recognition results of different modal data, which just shows that there are differences in the information provided by different modal data. It is very necessary to fuse the recognition results of different modal data.

[0110] Based on the above embodiments, Fig.13 As shown, the present invention also proposes a mountain top recognition system based on a multimodal convolutional neural network, comprising:

[0111] A multimodal image production unit is used to calculate the terrain parameter model based on DEM data through terrain analysis methods, and to produce multimodal images using different visualization methods;

[0112] A mountain top point region labeling unit, used for labeling the mountain top point region of the multimodal image;

[0113] The mountain top recognition model training unit is used to train the annotated images of different modalities based on the Faster R-CNN deep learning network to obtain the mountain top recognition models corresponding to different modalities; the Faster R-CNN uses ResNet50 for feature extraction;

[0114] A mountain top point recognition unit, used for recognizing mountain top points of images of different modalities by using mountain top point recognition models corresponding to different modalities;

[0115] A different modality recognition result fusion unit is used to fuse the peak points of different modality image recognitions by using a fuzzy fusion method;

[0116] The mountain top extraction unit is used to map the identified mountain top area onto the corresponding DEM data according to the fused mountain top recognition result, and extract the extreme elevation point in the mountain top area as the accurate mountain top.

[0117] Furthermore, the multimodal image includes: a contour map, a layered color map, a gradient layered color map, a grayscale shading map, a color shading map, and a slope map.

[0118] Furthermore, the multimodal image production unit is specifically used for:

[0119] Based on the terrain analysis and mapping functions provided by Golden Software Surfer, the original image data of various modes of the experimental area were produced, and the layered color map, gradient layered color map and grayscale shading map were superimposed with contour line information at the same time.

[0120] Furthermore, in the mountain top point area labeling unit, labelImg software is used to implement the labeling of the mountain top point area, and the labeled area contains at least two or three closed contour lines.

[0121] Furthermore, the different modality recognition result fusion unit is specifically used for:

[0122] First, determine the position of the rectangular recognition area. If there is only one modal recognition result and the confidence level is greater than 0.8, the recognition result is identified as the peak point. If there are multiple modal recognition results, determine the degree of overlap of the rectangular recognition areas of different modalities. If the overlap exceeds 80%, it is determined that the different modal recognitions are a peak point. Second, for overlapping recognition results, calculate the area of ​​the overlapping area as A0, and then use the following formula for calculation:

[0123]

[0124] Where p represents the recognition confidence corresponding to the overlapping area, A i represents the area of ​​the peak point of the i-th modality image recognition in the overlapping recognition results, p i represents the recognition confidence corresponding to the i-th modality image in the overlapping recognition results, and n is the total number of modality image types in the overlapping recognition results;

[0125] If p is greater than 0.8, the center point of the rectangular area where the overlapping area is located is determined as the mountain top recognition result.

[0126] In summary, the present invention studies the recognition effects of different modal data, and then uses the fuzzy fusion method to achieve the fusion of different modal image recognition results, thereby obtaining the most accurate mountain top point recognition result and reducing the phenomenon of missed and wrong mountain top points.

[0127] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A mountain top recognition method based on multimodal convolutional neural network, It is characterized in that include: Step 1: Based on DEM data, the terrain parameter model is calculated through terrain analysis methods, and different visualization methods are used to produce multimodal images; Step 2: Annotate the mountain top area of ​​the multimodal image; Step 3: Based on the Faster R-CNN deep learning network, the labeled images of different modalities are used for training to obtain the peak point recognition model corresponding to different modalities; Faster R-CNN uses ResNet50 for feature extraction; Step 4: Identify the peak points of images of different modalities through the peak point recognition models corresponding to different modalities; Step 5: Use the fuzzy fusion method to fuse the peak points identified by different modal images; Step 6: Based on the fused mountain top recognition results, the identified mountain top area is mapped onto the corresponding DEM data, and the extreme elevation points in the mountain top area are extracted as the accurate mountain top points; The step 5 comprises: First, determine the position of the rectangular recognition area. If there is only one modal recognition result and the confidence level is greater than 0.8, then the recognition result is identified as the peak point. If there are multiple modal recognition results, then determine the degree of overlap of the rectangular recognition areas of different modalities. If the overlap exceeds 80%, then determine that the different modal recognitions are a peak point. Second, for overlapping recognition results, calculate the area of ​​the overlapping area as A. 0 , and then use the following formula to calculate: Where p represents the recognition confidence corresponding to the overlapping area, A i represents the area of ​​the peak point of the i-th modality image recognition in the overlapping recognition results, p i represents the recognition confidence corresponding to the i-th modality image in the overlapping recognition results, and n is the total number of modality image types in the overlapping recognition results; If p is greater than 0.8, the center point of the rectangular area where the overlapping area is located is determined as the mountain top recognition result.

2. According to the multimodal convolutional neural network mountain top recognition method according to claim 1, It is characterized in that The multimodal images include: contour map, layered color map, gradient layered color map, grayscale shading map, color shading map, and slope map.

3. The mountain top recognition method based on multimodal convolutional neural network according to claim 1, It is characterized in that The step 1 comprises: Based on the terrain analysis and mapping functions provided by Golden Software Surfer, the original image data of various modes of the experimental area were produced, and the layered color map, gradient layered color map and grayscale shading map were superimposed with contour line information at the same time.

4. The mountain top recognition method based on multimodal convolutional neural network according to claim 1, It is characterized in that In step 2, labelImg software is used to label the mountain top area, and the labeled area contains at least two or three closed contour lines.

5. A mountain top recognition system based on multimodal convolutional neural network, It is characterized in that include: A multimodal image production unit is used to calculate the terrain parameter model based on DEM data through terrain analysis methods, and to produce multimodal images using different visualization methods; A mountain top point region labeling unit, used for labeling the mountain top point region of the multimodal image; The mountain top recognition model training unit is used to train the annotated images of different modalities based on the Faster R-CNN deep learning network to obtain the mountain top recognition models corresponding to different modalities; the Faster R-CNN uses ResNet50 for feature extraction; A mountain top point recognition unit, used for recognizing mountain top points of images of different modalities by using mountain top point recognition models corresponding to different modalities; A different modality recognition result fusion unit is used to fuse the peak points of different modality image recognitions by using a fuzzy fusion method; The mountain top point extraction unit is used to map the identified mountain top point area onto the corresponding DEM data according to the fused mountain top point recognition result, and extract the extreme elevation point in the mountain top point area as the accurate mountain top point; The different modality recognition result fusion unit is specifically used for: First, determine the position of the rectangular recognition area. If there is only one modal recognition result and the confidence level is greater than 0.8, then the recognition result is identified as the peak point. If there are multiple modal recognition results, then determine the degree of overlap of the rectangular recognition areas of different modalities. If the overlap exceeds 80%, then determine that the different modal recognitions are a peak point. Second, for overlapping recognition results, calculate the area of ​​the overlapping area as A. 0 , and then use the following formula to calculate: Where p represents the recognition confidence corresponding to the overlapping area, A i represents the area of ​​the peak point of the i-th modality image recognition in the overlapping recognition results, p i represents the recognition confidence corresponding to the i-th modality image in the overlapping recognition results, and n is the total number of modality image types in the overlapping recognition results; If p is greater than 0.8, the center point of the rectangular area where the overlapping area is located is determined as the mountain top recognition result.

6. The mountain top recognition system based on multimodal convolutional neural network according to claim 5, It is characterized in that The multimodal images include: contour map, layered color map, gradient layered color map, grayscale shading map, color shading map, and slope map.

7. The mountain top recognition system based on multimodal convolutional neural network according to claim 5, It is characterized in that The multimodal image production unit is specifically used for: Based on the terrain analysis and mapping functions provided by Golden Software Surfer, the original image data of various modes of the experimental area were produced, and the layered color map, gradient layered color map and grayscale shading map were superimposed with contour line information at the same time.

8. The mountain top recognition system based on multimodal convolutional neural network according to claim 5, It is characterized in that In the mountain top point area labeling unit, labelImg software is used to implement the labeling of the mountain top point area, and the labeled area contains at least two or three closed contour lines.