Method and device for fine type recognition of remote sensing image target under guidance of domain knowledge

Abstract

The application provides a remote sensing image target fine type recognition method and device under the guidance of domain knowledge, which comprises the following steps: obtaining an original remote sensing image to be recognized, and performing multi-dimensional feature extraction on the original remote sensing image to obtain an edge feature image and a texture feature image of the original remote sensing image; performing feature extraction and fusion of a multi-stream convolutional neural network on the original remote sensing image, the edge feature image and the texture feature image to obtain a fusion feature image of the original remote sensing image; identifying an initial category of the target to be recognized based on the fusion feature image; obtaining a target knowledge template library corresponding to the initial category, and identifying the fine type of the target to be recognized according to the target knowledge template library. Through the extraction and fusion of multi-dimensional features of the original remote sensing image, target recognition is performed based on the fusion features, and the preliminary recognition result is refined and reconfirmed in combination with prior knowledge, so as to improve the recognition accuracy of the target and meet the high-precision recognition requirement of the fine type of the target.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of remote sensing image recognition, and in particular to a remote sensing image target fine type recognition method and device guided by domain knowledge. BACKGROUND

[0002] Target detection and recognition has become one of the important tasks of high-resolution remote sensing image interpretation, which specifically refers to the process of detecting the target of interest category in the remote sensing image. The information to be detected and recognized includes the position and category information of the target. With the rapid development of artificial intelligence technology, the mode of target detection and recognition has changed from automation to intelligence. Therefore, a target recognition method based on deep learning is generated, which does not require professional knowledge and experience for manual feature design, can automatically learn how to extract features from data, gradually combines low-level features into high-level (more abstract and semantically meaningful) features, and obtains a feature representation with good generalization, strong robustness and high discrimination, which can provide an effective framework for target extraction in images.

[0003] Although the target recognition method based on deep learning has a good performance in many open source target detection data sets, for remote sensing images, due to the influence of many factors such as large imaging scene, low target appearance frequency, obvious scale effect, large observation angle difference, large intra-class difference, and high inter-class similarity, the recognition accuracy of the target is low, and it cannot fully meet the needs of current practical applications. In this case, simply relying on the features of the image itself for target detection and recognition cannot meet the requirements of practicality, especially for more fine-grained categories (such as models), more external knowledge input and prior constraints need to be considered to improve the accuracy, rationality and usability of the final result, and a remote sensing image target fine type recognition method guided by domain knowledge is urgently needed. SUMMARY

[0004] The present application provides a remote sensing image target fine type recognition method and device guided by domain knowledge, which solves the problem of insufficient target recognition accuracy based on remote sensing images in the prior art, and cannot meet the detection requirements of high precision.

[0005] The present application provides a remote sensing image target fine type recognition method guided by domain knowledge, comprising:

[0006] An original remote sensing image to be recognized is obtained, and multi-dimensional feature extraction is performed on the original remote sensing image to obtain an edge feature image and a texture feature image of the original remote sensing image;

[0007] The original remote sensing image, the edge feature image and the texture feature image are subjected to feature extraction and fusion of a multi-stream convolutional neural network to obtain a fusion feature image of the original remote sensing image;

[0008] identify an initial category of the target to be identified in the original remote sensing image based on the fused feature image;

[0009] obtain a target knowledge template library corresponding to the initial category, and identify a fine type of the target to be identified according to the target knowledge template library.

[0010] In one embodiment, the step of performing multi-stream convolutional neural network feature extraction and fusion on the original remote sensing image, the edge feature image and the texture feature image to obtain the fused feature image of the original remote sensing image comprises:

[0011] performing overlapping grid division processing on the original remote sensing image according to a preset size of a sliding window and a preset sliding step to obtain a plurality of grid images corresponding to the original remote sensing image; the edge feature image comprises an edge feature sub-image corresponding to each of the grid images, and the texture feature image comprises a texture feature sub-image corresponding to each of the grid images;

[0012] inputting a target grid image, an edge feature sub-image corresponding to the target grid image and a texture feature sub-image corresponding to the target grid image into a pre-trained multi-stream convolutional neural network model, and performing multi-stream convolutional neural network feature extraction and fusion on the target grid image, the edge feature sub-image corresponding to the target grid image and the texture feature sub-image corresponding to the target grid image by using the multi-stream convolutional neural network model to obtain the fused feature image of the original remote sensing image; the target grid image is any one of the plurality of grid images;

[0013] The multi-stream convolutional neural network model comprises a plurality of convolutional neural networks sharing weights and an attention mechanism, and the plurality of convolutional neural networks comprises a first convolutional neural network, a second convolutional neural network and a third convolutional neural network.

[0014] The input of the first convolutional neural network is the grid image, the input of the second convolutional neural network is the edge feature sub-image, and the input of the third convolutional neural network is the texture feature sub-image; and the attention mechanism is used for feature fusion on the outputs of the plurality of convolutional neural networks.

[0015] In one embodiment, before the step of performing multi-stream convolutional neural network feature extraction and fusion on the original remote sensing image, the edge feature image and the texture feature image to obtain the fused feature image of the original remote sensing image, the method further comprises:

[0016] generating a fine classification system according to each preset category; obtaining an initial remote sensing image under the fine classification system; the initial remote sensing image contains classification targets under each of the preset categories.

[0017] obtain annotation information of the initial remote sensing image, and perform overlapping cropping processing on the initial remote sensing image based on the annotation information, and determine a sample image containing the classification target from the cropped image;

[0018] construct a first sample data set based on the sample image, and iteratively train a preset basic multi-stream convolutional neural network model using the first sample data set;

[0019] based on the image block corresponding to the classification target in each of the preset categories in the sample image, construct a target template image corresponding to the fine classification system in the target knowledge template library;

[0020] perform binaryzation processing on the mean image of the image block corresponding to the classification target in each of the preset categories in the sample image, to construct a target template binaryzation image corresponding to the fine classification system in the target knowledge template library; the target knowledge template library includes the target template image and the target template binaryzation image in each of the preset categories, and attribute information and global geospatial distribution range information of the classification target corresponding to the target template image and the target template binaryzation image;

[0021] perform rotation expansion processing on the target image block in the target template image according to a preset angle interval, to obtain a plurality of angle images corresponding to the target template image; the target image block is an image block in the target template image, for which the classification recognition result obtained by performing classification recognition on the target image block using the multi-stream convolutional neural network model is inconsistent with the true value;

[0022] construct a second sample data set based on the target template image and the angle image;

[0023] iteratively train a preset basic fine classification model using the second sample data set.

[0024] In one embodiment, after the fine type of the to-be-identified target is identified according to the target knowledge template library, the method further includes:

[0025] convert the target pixel coordinates in each of the grid images into pixel coordinates under the original remote sensing image; the target pixel coordinates are pixel coordinates corresponding to the to-be-identified target in the grid image;

[0026] obtain projection parameter information of the original remote sensing image, and determine geospatial position information of the to-be-identified target according to the projection parameter information and the pixel coordinates;

[0027] Based on the target knowledge template library, the global geospatial distribution range of the target to be identified is determined, and based on the geospatial location information and the global geospatial distribution range, it is determined whether the geospatial location of the target to be identified is within the global geospatial distribution range;

[0028] If so, based on the preset nonmaximum suppression algorithm, the grid images are deduplicated and stitched together.

[0029] In one embodiment, the step of identifying the initial category of the target to be identified in the original remote sensing image based on the fused feature image includes:

[0030] The bounding rectangle of the target to be identified in the original remote sensing image is determined based on the fused feature image, as well as the longitudinal angle of the bounding rectangle; the longitudinal angle is the angle between the long side of the bounding rectangle and the calibration direction.

[0031] Based on the longitudinal angle and the calibration direction, the image block covered by the outer rectangle is rotated and cropped to obtain a target image containing the target to be identified;

[0032] The target image is input into a pre-trained fine-grained target classification model to obtain the category prediction value output by the fine-grained target classification model; the category prediction value is the probability value of the target to be identified in the original remote sensing image as each preset category;

[0033] The predicted category values ​​are sorted in descending order, and a preset number of categories are selected as the initial categories of the target to be identified according to the sorting order.

[0034] In one embodiment, the initial categories include multiple categories, and the step of identifying the fine-grained type of the target to be identified based on the target knowledge template base includes:

[0035] Obtain the prior target from the target knowledge template library;

[0036] The deviation of the mean size between the target to be identified and the prior target is determined based on the spatial resolution of the original remote sensing image.

[0037] The initial categories are filtered based on the mean deviation of the size to determine the first target category of the target to be identified;

[0038] The target image is binarized to obtain a target binarized image corresponding to the target image;

[0039] calculate a similarity between the target to be identified and a prior target in a first template binary image based on the target binary image; the first template binary image is a template binary image corresponding to the first target category in the target knowledge template library;

[0040] determine the fine type of the target to be identified according to the similarity.

[0041] In one embodiment, the step of calculating the similarity between the target to be identified and the prior target in the first template binary image based on the target binary image comprises:

[0042] respectively calculate a target hash value of the target binary image and a template hash value of the first template binary image based on a difference hash algorithm;

[0043] calculate a Hamming distance between the target hash value and the template hash value;

[0044] determine the similarity between the target to be identified and the prior target in the first template binary image according to the Hamming distance.

[0045] The application also provides a device for identifying a fine type of a target in a remote sensing image under guidance of domain knowledge, comprising:

[0046] a feature extraction module configured to acquire an original remote sensing image to be identified and perform multi-dimensional feature extraction on the original remote sensing image to obtain an edge feature image and a texture feature image of the original remote sensing image;

[0047] a feature fusion module configured to perform feature extraction and fusion on the original remote sensing image, the edge feature image and the texture feature image by using a multi-stream convolutional neural network to obtain a fusion feature image of the original remote sensing image;

[0048] a first identification module configured to identify an initial category of a target to be identified in the original remote sensing image based on the fusion feature image;

[0049] a second identification module configured to acquire a target knowledge template library corresponding to the initial category and identify a fine type of the target to be identified according to the target knowledge template library.

[0050] The application also provides an electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method for identifying a fine type of a target in a remote sensing image under guidance of domain knowledge as described above.

[0051] The application further provides a non-transitory computer-readable storage medium, which stores a computer program, and the computer program, when executed by a processor, implements the steps of the remote sensing image target fine type recognition method under the guidance of the domain knowledge.

[0052] The application further provides a computer program product, which comprises a computer program, and the computer program, when executed by a processor, implements the steps of the remote sensing image target fine type recognition method under the guidance of the domain knowledge.

[0053] The application provides a remote sensing image target fine type recognition method and device under the guidance of domain knowledge, which extracts multi-dimensional features of the original remote sensing image to obtain an edge feature image and a texture feature image of the original remote sensing image, extracts and fuses features of the original remote sensing image, the edge feature image and the texture feature image through a multi-stream convolutional neural network to obtain a fusion feature image of the original remote sensing image, identifies an initial category of a target to be recognized in the original remote sensing image based on the fusion feature image, acquires a target knowledge template library corresponding to the initial category, and identifies a fine type of the target to be recognized according to the acquired target knowledge template library. Through multi-dimensional feature extraction and fusion of the original remote sensing image, such as edge features and texture features, preliminary identification of the target is performed based on the fusion features, and the prior knowledge of the target knowledge template library is combined to refine and reconfirm the preliminary identification result, so as to improve the identification accuracy of the target and meet the high-precision identification requirement of the fine type of the target. BRIEF DESCRIPTION OF DRAWINGS

[0054] In order to more clearly illustrate the technical solutions of the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0055] Figure 1 is one of the flowcharts of the remote sensing image target fine type recognition method under the guidance of the domain knowledge provided by the application;

[0056] Figure 2 is another flowchart of the remote sensing image target fine type recognition method under the guidance of the domain knowledge provided by the application;

[0057] Figure 3 is a structural schematic diagram of the remote sensing image target fine type recognition device under the guidance of the domain knowledge provided by the application;

[0058] Figure 4 is a structural schematic diagram of the electronic device provided by the application. DETAILED DESCRIPTION

[0059] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0060] The following is combined with Figures 1-4 This invention describes a method and apparatus for fine-grained target type recognition in remote sensing images, guided by domain knowledge.

[0061] This invention provides a domain knowledge-guided method for fine-grained target type identification in remote sensing images. Existing target identification methods, based on deep learning models, rely on features extracted from images for target recognition. However, due to the characteristics of remote sensing images—large imaging scenes, low target frequency, significant scale effects, large differences in observation angles, large intra-class differences, and high inter-class similarity—existing target identification methods have low accuracy in fine-grained target type identification when applied to remote sensing images, failing to meet the high-precision detection requirements for fine-grained types. The domain knowledge-guided method for fine-grained target type identification based on remote sensing images provided by this invention extracts feature information from different dimensions of the remote sensing image, uses these as input data for the model, and further utilizes prior target knowledge to confirm and correct the results after model output, thereby improving the accuracy and rationality of fine-grained target category identification.

[0062] Specifically, refer to Figure 1 , Figure 1 This is a flowchart illustrating a domain knowledge-guided method for fine-grained target type recognition in remote sensing images, as provided in this embodiment of the invention. Figure 1 The domain knowledge-guided method for fine-grained target type recognition in remote sensing images provided in this invention includes:

[0063] Step 100: Obtain the original remote sensing image to be identified, and perform multi-dimensional feature extraction on the original remote sensing image to obtain the edge feature image and texture feature image of the original remote sensing image;

[0064] In target recognition, the original remote sensing image to be identified is first acquired, and multidimensional feature extraction is performed on the acquired original remote sensing image to obtain edge feature image and texture feature image. The acquired original remote sensing image is a high-resolution image; the edge feature image contains the edge features of the original remote sensing image, and the texture feature image contains the texture features of the original remote sensing image. The edge features and texture features are obtained from different dimensions of the original remote sensing image based on different feature extraction methods.

[0065] Step 200, performing feature extraction and fusion of the original remote sensing image and the edge feature image and the texture feature image by a multi-stream convolutional neural network to obtain a fusion feature image of the original remote sensing image;

[0066] The fusion mode of the multi-stream convolutional neural network intermediate feature output layer of the original remote sensing image, the edge feature image and the texture feature image includes feature fusion based on an attention mechanism, that is, based on the attention mechanism, the network intermediate layer output features corresponding to the original remote sensing image, the edge feature image and the texture feature image are fused to obtain a fusion feature image corresponding to the original remote sensing image.

[0067] Step 300, identifying an initial class of the to-be-identified target in the original remote sensing image based on the fusion feature image;

[0068] The initial class of the to-be-identified target in the original remote sensing image is identified based on the fusion feature image, wherein the to-be-identified target in the remote sensing image is a target of interest, and the to-be-identified target includes one or more. When the to-be-identified target includes multiple targets, the classes of the multiple to-be-identified targets can be the same or different. Similarly, the initial class of the to-be-identified target includes one or more.

[0069] Step 400, obtaining a target knowledge template library corresponding to the initial class, and identifying a fine type of the to-be-identified target according to the target knowledge template library.

[0070] Based on the identified initial class of the to-be-identified target, a target knowledge template library corresponding to the initial class is obtained. The target knowledge template library is prior knowledge of the to-be-identified target, and contains information such as a fine class and a size of the to-be-identified target. According to the target knowledge template library, the initial class of the to-be-identified target can be further confirmed.

[0071] It can be understood that when classifying targets, the targets can be classified more finely according to the differences between different targets, and divided into different precision classes such as large classes and small classes. The more finely the target classes are divided, the fewer the distinguishing features between different classes of targets. Based on the prior knowledge in the target knowledge template library, the initial class of the to-be-identified target can be further confirmed to identify the fine type of the to-be-identified target and improve the classification accuracy of the to-be-identified target.

[0072] In the embodiment, edge feature images and texture feature images of the original remote sensing image are obtained by performing multi-dimensional feature extraction on the original remote sensing image; the original remote sensing image, the edge feature images and the texture feature images are subjected to feature extraction and fusion of a multi-stream convolutional neural network to obtain a fusion feature image of the original remote sensing image; an initial category of the to-be-identified target in the original remote sensing image is identified based on the fusion feature image; a target knowledge template library corresponding to the initial category is obtained, and the to-be-identified target is identified according to the obtained target knowledge template library. Through edge feature extraction and texture feature extraction on the original remote sensing image, feature learning and fusion of the multi-stream convolutional neural network, preliminary identification of the target based on the fusion feature, and refinement and re-confirmation of the preliminary identification result in combination with the prior knowledge of the knowledge template library, the accuracy of fine type identification of the target is improved, and the high-precision detection requirement for the fine type of the to-be-identified target is met.

[0073] Further, in step 300, the initial category of the to-be-identified target in the original remote sensing image is identified based on the fusion feature image, specifically including:

[0074] In step 301, the outer bounding rectangle of the to-be-identified target in the original remote sensing image is determined according to the fusion feature, and the long direction angle of the outer bounding rectangle is determined; the long direction angle is the angle between the long side of the outer bounding rectangle and the calibration direction;

[0075] In step 302, the image block covered by the outer bounding rectangle is subjected to rotation and clipping processing according to the long direction angle and the calibration direction to obtain a target image containing the to-be-identified target;

[0076] In step 303, the target image is input into the pre-trained target fine classification model to obtain the category prediction value output by the target fine classification model; the category prediction value is the probability value of the to-be-identified target in the original remote sensing image being each preset category;

[0077] In step 304, the category prediction value is sorted in descending order, and a preset number of categories are selected as the initial category of the to-be-identified target according to the sorting order.

[0078] In identifying the initial category of the to-be-identified target in the original remote sensing image, the outer bounding rectangle of the to-be-identified target in the original remote sensing image and the long direction angle of the outer bounding rectangle are determined according to the target position bounding box and category information detected and identified in the fusion feature image. The long direction angle of the outer bounding rectangle is the angle between the long side of the outer bounding rectangle and the calibration direction, and the calibration direction is a set geographical direction, such as the north direction. The long direction angle can be the clockwise angle between the long side of the outer bounding rectangle and the calibration direction, or the counterclockwise angle between the long side of the outer bounding rectangle and the calibration direction, which is not specifically limited here.

[0079] According to the angle between the long direction of the outer rectangle of the to-be-recognized target, the outer rectangle is normalized and cropped to obtain a target image containing the to-be-recognized target. Specifically, according to the angle between the long direction of the outer rectangle of the to-be-recognized target and the calibration direction, the outer rectangle is rotated to uniformly rotate the outer rectangle to the calibration direction, the direction of the outer rectangle is normalized, and the rotated outer rectangle is cropped to obtain the target image containing the to-be-recognized target.

[0080] The target image is input into the trained target fine classification model to obtain a category prediction value output by the target fine classification model, and the category prediction value is used to represent a probability value that the category of the to-be-recognized target is a preset category. That is, the probability that the to-be-recognized target is a preset category is predicted by the target fine classification model, the preset category includes multiple, and the probability value predicted by the target fine classification model includes the probability value that the to-be-recognized target is each preset category. The category prediction value of the to-be-recognized target is sorted in descending order, and a preset number of categories are selected as initial categories of the to-be-recognized target according to the sorting order, for example, the first K categories with the highest category prediction value are selected as the initial categories of the to-be-recognized target according to the sorting order. When there are multiple to-be-recognized targets, the first K categories are selected as the initial categories for the category prediction value of each to-be-recognized target, and the first K categories are the categories with the highest matching degree to the to-be-recognized target.

[0081] Further, based on the K initial categories of the to-be-recognized target, target prior knowledge in a target knowledge template library corresponding to the K initial categories is obtained, and in step 400, the fine type of the to-be-recognized target is identified according to the target knowledge template library, including:

[0082] In step 401, the prior target in the target knowledge template library is obtained.

[0083] In step 402, the size mean deviation of the to-be-recognized target and the prior target is determined according to the spatial resolution of the original remote sensing image.

[0084] In step 403, the initial category is screened according to the size mean deviation to determine the first target category of the to-be-recognized target.

[0085] In step 404, the target image is binarized to obtain a target binary image corresponding to the target image.

[0086] In step 405, the similarity between the to-be-recognized target and the prior target in the first template binary image is calculated based on the target binary image; the first template binary image is a template binary image corresponding to the first target category in the target knowledge template library.

[0087] Step 406, determining the fine type of the target to be identified according to the similarity.

[0088] In identifying the target to be identified, the fine category of the target to be identified needs to be identified. Specifically, first, according to the spatial resolution of the original remote sensing image and the prior target in the target knowledge template library, the prior target is the target prior knowledge under each initial category. According to the spatial resolution of the original remote sensing image, the size mean deviation of the target to be identified and the prior target is determined, and the initial category is filtered according to the size mean deviation, and the first target category of the target to be identified is determined. Specifically, based on the target knowledge template library, the category consistency of K categories and the corresponding category target prior knowledge in the target knowledge template library is confirmed in sequence. First, according to the prior knowledge description of the current category in the target knowledge template library, combined with the spatial resolution of the original remote sensing image, the size (including the perimeter, the length-width ratio, etc.) mean deviation of the target to be identified in the current target image and the prior target in the target knowledge template library corresponding to the current initial category is calculated. If the size mean deviation exceeds a preset threshold T1, it is considered that the current initial category is inconsistent with the prior knowledge under the category, and the next initial category is continued to be judged; such a cycle is repeated until the consistency judgment of the Kth initial category is completed; if the K initial categories are inconsistent, it is considered that the target to be identified in the target image does not belong to any of the initial categories, and the current recognition result of the original remote sensing image is deleted; if there is a certain initial category consistent with the prior knowledge under the category, the category can be determined as the first target category of the target to be identified.

[0089] After screening the K initial categories, if there is a first target category that meets the conditions, the target image containing the target to be identified is subjected to binaryzation processing to obtain a target binaryzation image corresponding to the target image.

[0090] The similarity between the to-be-recognized target and the prior target in the first template binary image is calculated based on the target binary image and the template binary image. The target binary image contains the binary image of the to-be-recognized target, and the template binary image contains the binary image of the prior target. The first template binary image is a target template binary image selected from a plurality of target knowledge template libraries corresponding to K initial categories based on the screening result of the K initial categories. Specifically, the similarity between the to-be-recognized target and the prior target is calculated based on the binary image of the to-be-recognized target in the target binary image and the binary image of the prior target in the template binary image. According to the similarity between the to-be-recognized target and the prior target, the similarity between the to-be-recognized target and the prior target in the first template binary image is matched, and the fine category of the to-be-recognized target is further determined. Specifically, if the similarity between the to-be-recognized target and the prior target in the first template binary image exceeds a preset similarity threshold, the category of the prior target, that is, the category corresponding to the first template binary image, is the fine category of the to-be-recognized target. Otherwise, if the similarity between the to-be-recognized target and the prior target in the first template binary image does not exceed the preset similarity threshold, the category of the to-be-recognized target does not match the category of the prior target in the first template binary image.

[0091] Further, in step 405, when calculating the similarity between the to-be-recognized target and the prior target in the first template binary image, the following steps are specifically included:

[0092] In step 4051, the target hash value of the target binary image and the template hash value of the template binary image are respectively calculated based on the difference hash algorithm.

[0093] In step 4052, the Hamming distance between the target hash value and the template hash value is calculated.

[0094] In step 4053, the similarity between the to-be-recognized target and the prior target in the first template binary image is determined according to the Hamming distance.

[0095] In calculating the similarity between the to-be-recognized target and the prior target in the first template binary image, first, the target hash value of the target binary image and the template hash value of the template binary image are respectively calculated based on the difference hash algorithm. Then, the Hamming distance between the target hash value and the template hash value is calculated according to the calculated target hash value and template hash value. Finally, the similarity between the to-be-recognized target and the prior target in the first template image is determined according to the calculated Hamming distance. Specifically, if the Hamming distance is less than a preset threshold T2 (T2 is a positive number), the similarity between the to-be-recognized target and the prior target in the first template binary image meets the condition, the current determination category result is correct, and the first target category is the fine category of the to-be-recognized target. Otherwise, the current recognition result is deleted.

[0096] In the embodiment, the initial category of the to-be-identified target is screened through the prior target in the knowledge template library, the to-be-identified target is matched with the prior target in similarity, and the category of the to-be-identified target is further confirmed according to the similarity between the to-be-identified target and the prior target, so that the recognition accuracy of the fine type of the to-be-identified target can be improved.

[0097] Referring to Figure 2 Another flowchart of the remote sensing image target fine type recognition method under the guidance of domain knowledge is shown in FIG. 8. In a preferred embodiment, the recognition of the initial category and the fine category of the to-be-identified target is respectively based on different detection models and a target knowledge template library constructed in advance. The target knowledge template library includes target template images and target template binary images under each category. Specifically, before the to-be-identified target is detected and recognized, a corresponding sample data set needs to be constructed for model training. In the embodiment, the position bounding box of the to-be-identified target is detected based on a pre-trained multi-stream convolutional neural network model, and the initial category of the to-be-identified target is recognized based on a pre-trained target fine classification model.

[0098] In step 200, the edge feature image and the texture feature image corresponding to the original remote sensing image each include multiple. The original remote sensing image, the edge feature image, and the texture feature image are subjected to feature extraction and fusion of a multi-stream convolutional neural network to obtain a fusion feature image of the original remote sensing image. The fusion feature image of the original remote sensing image specifically includes:

[0099] In step 201, the original remote sensing image is subjected to grid division processing with overlapping based on a sliding window of a preset size and a preset sliding step to obtain multiple grid images corresponding to the original remote sensing image. The edge feature image includes an edge feature sub-image corresponding to each of the grid images, and the texture feature image includes a texture feature sub-image corresponding to each of the grid images.

[0100] In step 202, a target grid image, an edge feature sub-image corresponding to the target grid image, and a texture feature sub-image corresponding to the target grid image are input into a pre-trained multi-stream convolutional neural network model. The target grid image, the edge feature sub-image corresponding to the target grid image, and the texture feature sub-image corresponding to the target grid image are subjected to feature extraction and fusion of a multi-stream convolutional neural network by using the multi-stream convolutional neural network model to obtain a fusion feature image of the original remote sensing image. The target grid image is any one of the multiple grid images.

[0101] The multi-stream convolutional neural network model includes multiple convolutional neural networks sharing weights and an attention mechanism. The multiple convolutional neural networks include a first convolutional neural network, a second convolutional neural network, and a third convolutional neural network.

[0102] wherein, the input of the first convolutional neural network is the grid image, the input of the second convolutional neural network is the edge feature subgraph, and the input of the third convolutional neural network is the texture feature subgraph; the attention mechanism is used for feature fusion of the outputs of the plurality of convolutional neural networks.

[0103] For the obtained high-resolution original remote sensing image, grid division is performed to obtain a plurality of grid images. The edge feature image of the original remote sensing image includes the edge feature subgraph corresponding to each grid image, and correspondingly, the texture feature image of the original remote sensing image includes the texture feature subgraph corresponding to each grid image. For the extraction of edge features and texture features, multi-dimensional feature extraction can be performed on the original remote sensing image before grid division, to obtain the edge feature image and the texture feature image of the original image. After the original remote sensing image is divided, the edge feature image and the texture feature image are divided into grid images in the same manner as the original remote sensing image, to obtain the edge feature subgraph and the texture feature subgraph corresponding to each grid image. Alternatively, after the original remote sensing image is divided into grid images, multi-dimensional feature extraction is performed on each grid image, to obtain the edge feature subgraph and the texture feature subgraph corresponding to each grid image. The order of multi-dimensional feature extraction and grid division is not limited.

[0104] In the method, Figure 2 , the original remote sensing image is first divided into grid images, and the edge feature and the texture feature are extracted for each grid image to obtain the edge feature subgraph and the texture feature subgraph corresponding to the grid image. Each grid image is taken as a target grid image, and the target grid image, the edge feature subgraph and the texture feature subgraph corresponding to the target grid image are input into the multi-stream convolutional neural network model, and the multi-stream convolutional neural network model is used for convolutional neural network feature self-learning of the target grid image, the edge feature subgraph and the texture feature subgraph. When the original remote sensing image is divided into grid images, the image is divided into grid images with overlapping according to a preset sliding window GxG and a preset sliding step g x g. The edge feature and the texture feature are extracted for each grid image, and each grid image, the corresponding edge feature image and the texture feature image are input into the trained multi-stream convolutional neural network model. The multi-stream convolutional neural network model is used for convolutional neural network feature self-learning and feature fusion of the grid image, the edge feature image and the texture feature image, and the fused feature image is used for prediction of the position bounding box and the category of the to-be-identified target.

[0105] Further, in the method, Figure 2In the multi-stream convolutional neural network model, a weight-sharing target detection backbone network and an attention mechanism are provided. The weight-sharing target detection backbone network is composed of multiple convolutional neural networks. Different convolutional neural networks have different input data and are used to process different data. Specifically, the multiple convolutional neural networks include a first convolutional neural network, a second convolutional neural network, and a third convolutional neural network. The input of the first convolutional neural network is a target grid image of an original remote sensing image. The input of the second convolutional neural network is an edge feature subgraph of the target grid image. The input of the third convolutional neural network is a texture feature subgraph of the target grid image. For any grid image obtained by grid division, the weight-sharing target detection backbone network is used to perform feature extraction on the grid image, an edge feature subgraph of the grid image, and a texture feature subgraph of the grid image, respectively, to obtain corresponding feature images 1, 2, and 3. Based on the attention mechanism, the feature images 1, 2, and 3 are fused to obtain a corresponding fused feature image.

[0106] Based on the fused feature image, the position bounding box and the category of the target to be recognized in the grid image are predicted. Based on the position bounding box of the target to be recognized, the position bounding box of the target to be recognized is normalized and cropped to obtain a target image containing the target to be recognized. The position bounding box is a rectangle that encloses the target to be recognized.

[0107] In Figure 2 As shown in the remote sensing image target fine type recognition process diagram guided by domain knowledge, a model training phase is included. Specifically, a sample data set and a target knowledge template library are first constructed, and the model is trained using the constructed sample data set. Before step 200, the following steps can also be included:

[0108] Step 001: generating a fine classification system according to each preset category; obtaining an initial remote sensing image under the fine classification system; the initial remote sensing image contains classification targets under each preset category;

[0109] Step 002: obtaining annotation information of the initial remote sensing image, and performing overlapping cropping on the initial remote sensing image based on the annotation information, and determining a sample image containing the classification target from the cropped image;

[0110] Step 003: constructing a first sample data set based on the sample image, and iteratively training a preset basic multi-stream convolutional neural network model using the first sample data set;

[0111] Step 004: constructing a target template image corresponding to the fine classification system in the target knowledge template library based on the image block corresponding to the classification target under each preset category in the sample image;

[0112] Step 005, the mean image of the image block corresponding to the classification target in the sample image under each of the preset categories is binarized to construct the target template binarization image in the target knowledge template library corresponding to the fine classification system; the target knowledge template library includes the target template image and the target template binarization image under each of the preset categories, and the attribute information and the global geospatial distribution range information of the classification target corresponding to the target template image and the target template binarization image;

[0113] Step 006, according to a preset angle interval, the target image block in the target template image is subjected to rotation amplification processing to obtain a plurality of angle images corresponding to the sample image; the target image block is an image block in the target template image, the classification recognition result of which is inconsistent with the true value obtained by the multi-flow convolutional neural network model;

[0114] Step 007, a second sample data set is constructed based on the target template image and the angle image;

[0115] Step 008, the second sample data set is used to iteratively train a preset basic fine classification model.

[0116] In constructing the sample data set, a fine classification system S is generated according to each of the preset categories of interest, and then an initial remote sensing image of each category under the fine classification system S is obtained, which is a high-resolution image of meter or sub-meter level, and contains classification targets under each of the preset categories, i.e. the target to be identified, i.e. the object to be detected and identified. Labeling information of the initial remote sensing image is obtained, which is used to label the category of the classification target in the initial remote sensing image, and the four corner coordinates of the classification target are recorded in order from the beginning to the end and from left to right, and the fine category is assigned. The labeling result is stored in a corresponding storage format required by model training. The initial remote sensing image is subjected to overlapping cutting processing based on the labeling information, and a sample image containing a classification target is determined from a plurality of images obtained by cutting, and a first sample data set is constructed based on the sample image. The first sample data set is used to iteratively train a preset basic multi-flow convolutional neural network model.

[0117] Further, based on the sample image, a target knowledge template library Z and a second sample data set, i.e., a sample data set TS for training the target fine classification model, are respectively established. In constructing the target knowledge template library Z, according to the fine classification system S, a target image block corresponding to each fine category and a corresponding binary image block are generated, and a target template image and a target template binary image under the fine category are constructed. In the binary image corresponding to each fine category, the pixel value of the region corresponding to the classification target is a first pixel value, for example, 0, and the pixel value of the background region other than the classification target is a second pixel value, for example, 255, wherein the original image corresponding to the binary image is the mean image of the image block of the region covered by the classification target in the sample image. For the target prior knowledge in the target knowledge template library corresponding to any fine category, the attribute information and the spatial distribution range information of the classification target under the fine category are included, and the attribute information of the classification target includes the name, length, width, etc. of the classification target.

[0118] For the sample data set TS, its constitution includes two parts: one part is the image block of the position boundary box covered region of the classification target in the sample image in the first sample data set (including the target fine category information), i.e., the target template image in the target template knowledge library Z, and the other part is the image block after the rotation transformation of the specific image block in the first sample data set, for which the classification recognition result of the target detection and recognition multi-stream convolutional neural network model M is inconsistent with the true value. That is, in constructing the sample data set TS, according to a certain angle interval δ (in this embodiment, δ takes the value of 5), only the target image block in the first sample data set, for which the classification recognition result of the target detection and recognition multi-stream convolutional neural network model M is inconsistent with the true value, is subjected to rotation transformation, a plurality of angle images corresponding thereto are obtained, and are expanded into the target template image in the target template knowledge library Z, to obtain the second sample data set, i.e., the sample data set TS. The true value of the target image block can be determined according to the labeling information of the sample image, and the target fine classification model is obtained by iteratively training the preset basic fine classification model using the sample data set TS.

[0119] Further, based on Figure 2 The target image is input into the pre-trained target fine classification model, and the top K initial categories with the highest matching degree based on the category prediction of the target image are output. The K initial categories are subjected to category consistency determination and spatial consistency determination based on the target template image in the target knowledge template library. The category consistency determination is used to screen the K initial categories, and the first target category of the to-be-identified target is screened out according to the size mean deviation of the to-be-identified target and the prior target. The spatial consistency determination is specifically to determine the geographic spatial position of the to-be-identified target according to the original remote sensing image, and further confirm the to-be-identified target based on the spatial distribution range information in the target knowledge template library. Specifically, after step 400, the method further includes:

[0120] Step 501: Convert the target cell coordinates in each of the grid images into cell coordinates in the original remote sensing image; the target cell coordinates are the cell coordinates of the target to be identified in the grid image.

[0121] Step 502: Obtain the projection parameter information of the original remote sensing image, and determine the geospatial location information of the target to be identified based on the projection parameter information and the pixel coordinates;

[0122] Step 503: Determine the global geospatial distribution range information of the target to be identified based on the target knowledge template library, and determine whether the geospatial location of the target to be identified is within the global geospatial distribution range based on the geospatial location information and the global geospatial distribution range information;

[0123] Step 504: If so, perform deduplication and stitching processing on each of the grid images based on the preset nonmaximum suppression algorithm.

[0124] It should be noted that any target image input into the target fine-grained classification model is obtained based on a target grid image. Based on the K initial categories identified in the target image corresponding to this grid image, and based on the target image and the binarized target template image, the fine-grained category of the target to be identified is determined, resulting in the identification result corresponding to the target grid image. The identification results of all grid images are then summarized to obtain the final identification result of the target to be identified in the original remote sensing image. During spatial consistency determination, the target pixel coordinates in each grid image are converted to pixel coordinates in the original remote sensing image, where the target pixel coordinates are the pixel coordinates of the target to be identified in the grid image. The process involves acquiring projection parameter information from the original remote sensing image, determining the geospatial location information of the target to be identified based on this projection parameter information, determining the spatial distribution range information of the target to be identified based on the target knowledge template library, and performing a spatial consistency judgment on the target to be identified based on the geospatial location information and the global geospatial distribution range information to determine whether the spatial location of the target to be identified is within its spatial distribution range. If so, a pre-set non-maximum suppression algorithm is used to perform deduplication and stitching processing on each grid image to remove overlapping areas during grid division, obtaining the final identification result of the target to be identified. This identification result includes the latitude and longitude coordinate set of the location boundary rectangle of the target to be identified and fine category information. After identifying the fine category of the target to be identified, the grid image is deduplicated and stitched to obtain the final identification result that can be used for output display.

[0125] Furthermore, coordinate transformation is performed on the target pixels, converting the target pixel coordinates in the grid image to the pixel coordinates in the original remote sensing image. Specifically, based on the detection and recognition results of all grid images, the pixel coordinates of the target to be identified in each grid image are uniformly converted to the pixel coordinates in the original remote sensing image according to their position coordinates in the original remote sensing image. The conversion formula is as follows:

[0126] X = Xstart + Xpos

[0127] Y = Ystart + Ypos

[0128] Where X and Y are the pixel coordinates in the converted original remote sensing image, Xstart and Ystart are the pixel coordinates of the origin of the current grid image in the original remote sensing image, and Xpos and Ypos are the pixel coordinates of the bounding box of any target to be identified in the current grid image. Based on the pixel coordinates of the bounding box of the target to be identified and the projection parameter information of the original remote sensing image, the geographic coordinates of the feature points of the target to be identified (the target center point used in this embodiment) are calculated to obtain the geospatial location information of the target to be identified, which includes the latitude and longitude.

[0129] In this embodiment, by performing grid division processing, target recognition is performed on each grid image, which solves the problem of large imaging scene in remote sensing images when recognizing targets. After identifying the initial category of the target to be recognized, category consistency is determined, and after identifying the fine category of the target to be recognized, spatial consistency is determined, which further improves the accuracy of target recognition, while ensuring the effectiveness and rationality of target recognition.

[0130] The following describes the domain knowledge-guided fine-type identification device for remote sensing image targets provided by the present invention. The domain knowledge-guided fine-type identification device for remote sensing image targets described below can be referred to in correspondence with the domain knowledge-guided fine-type identification method for remote sensing image targets described above.

[0131] Reference Figure 3 The domain knowledge-guided remote sensing image target fine-type recognition device provided in this embodiment of the invention includes:

[0132] The feature extraction module 10 is used to acquire the original remote sensing image to be identified and to perform multi-dimensional feature extraction on the original remote sensing image to obtain the edge feature image and texture feature image of the original remote sensing image;

[0133] The feature fusion module 20 is used to perform feature extraction and fusion of the original remote sensing image, the edge feature image and the texture feature image using a multi-stream convolutional neural network to obtain a fused feature image of the original remote sensing image;

[0134] The first identification module 30 is used to identify the initial category of the target to be identified in the original remote sensing image based on the fused feature image;

[0135] The second identification module 40 is used to obtain the target knowledge template library corresponding to the initial category, and to identify the fine type of the target to be identified based on the target knowledge template library.

[0136] In one embodiment, the feature fusion module 20 is further configured to:

[0137] The original remote sensing image is divided into overlapping grids according to a preset sliding window size and a preset sliding step size to obtain multiple grid images corresponding to the original remote sensing image; the edge feature image includes edge feature sub-images corresponding to each grid image, and the texture feature image includes texture feature sub-images corresponding to each grid image.

[0138] The target grid image, the corresponding edge feature sub-image, and the corresponding texture feature sub-image are input into a pre-trained multi-stream convolutional neural network model. The multi-stream convolutional neural network model is used to perform feature extraction and fusion on the target grid image, the corresponding edge feature sub-image, and the corresponding texture feature sub-image to obtain a fused feature image of the original remote sensing image. The target grid image is any one of the multiple grid images.

[0139] The multi-stream convolutional neural network model includes multiple convolutional neural networks with shared weights and an attention mechanism. The multiple convolutional neural networks include a first convolutional neural network, a second convolutional neural network, and a third convolutional neural network.

[0140] The input to the first convolutional neural network is the grid image, the input to the second convolutional neural network is the edge feature sub-image, and the input to the third convolutional neural network is the texture feature sub-image; the attention mechanism is used to perform feature fusion on the outputs of the multiple convolutional neural networks.

[0141] In one embodiment, the domain knowledge-guided remote sensing image target fine-type recognition device further includes a model training module, used for:

[0142] A refined classification system is generated based on each preset category; an initial remote sensing image under the refined classification system is obtained; the initial remote sensing image contains the classification targets under each preset category;

[0143] Obtain the annotation information of the initial remote sensing image, perform overlapping cropping processing on the initial remote sensing image based on the annotation information, and determine the sample image containing the classification target from the cropped image;

[0144] A first sample dataset is constructed based on the sample images, and the preset basic multi-stream convolutional neural network model is iteratively trained using the first sample dataset.

[0145] Based on the image patches corresponding to the classification targets under each preset category in the sample images, construct the target template images corresponding to the fine classification system in the target knowledge template library;

[0146] The mean image of the image patch corresponding to the classification target under each preset category in the sample image is binarized to construct the target template binarized image corresponding to the fine classification system in the target knowledge template library; the target knowledge template library includes the target template image and the target template binarized image under each preset category, as well as the attribute information and global geospatial distribution range information of the classification target corresponding to the target template image and the target template binarized image;

[0147] According to a preset angle interval, the target image block in the target template image is rotated and augmented to obtain multiple angle images corresponding to the target template image; the target image block is the image block in the target template image whose classification and recognition results obtained by the multi-stream convolutional neural network model are inconsistent with the true value.

[0148] A second sample dataset is constructed based on the target template image and the angle image;

[0149] The pre-defined basic fine classification model is iteratively trained using the second sample dataset.

[0150] In one embodiment, the domain knowledge-guided remote sensing image target fine-type recognition device further includes a third recognition module, used for:

[0151] The target pixels in each of the grid images are converted into pixel coordinates in the original remote sensing image; the target pixel coordinates are the pixel coordinates of the target to be identified in the grid image.

[0152] The projection parameter information of the original remote sensing image is obtained, and the geospatial location information of the target to be identified is determined based on the projection parameter information and the pixel coordinates.

[0153] Based on the target knowledge template library, the global geospatial distribution range of the target to be identified is determined, and based on the geospatial location information and the global geospatial distribution range, it is determined whether the geospatial location of the target to be identified is within the global geospatial distribution range;

[0154] If so, based on the preset nonmaximum suppression algorithm, the grid images are deduplicated and stitched together.

[0155] In one embodiment, the first identification module 30 is further configured to:

[0156] The bounding rectangle of the target to be identified in the original remote sensing image is determined based on the fused feature image, as well as the longitudinal angle of the bounding rectangle; the longitudinal angle is the angle between the long side of the bounding rectangle and the calibration direction.

[0157] Based on the longitudinal angle and the calibration direction, the image block covered by the outer rectangle is rotated and cropped to obtain a target image containing the target to be identified;

[0158] The target image is input into a pre-trained fine-grained target classification model to obtain the category prediction value output by the fine-grained target classification model; the category prediction value is the probability value of the target to be identified in the original remote sensing image as each preset category;

[0159] The predicted category values ​​are sorted in descending order, and a preset number of categories are selected as the initial categories of the target to be identified according to the sorting order.

[0160] In one embodiment, the initial category includes multiple categories, and the second identification module 40 is further configured to:

[0161] Obtain the prior target from the target knowledge template library;

[0162] The deviation of the mean size between the target to be identified and the prior target is determined based on the spatial resolution of the original remote sensing image.

[0163] The initial categories are filtered based on the mean deviation of the size to determine the first target category of the target to be identified;

[0164] The target image is binarized to obtain a target binarized image corresponding to the target image;

[0165] The similarity between the target to be identified and the prior target in the first template image is calculated based on the target binarized image; the first template binarized image is the template binarized image corresponding to the first target category in the target knowledge template library;

[0166] The fine type of the target to be identified is determined based on the similarity.

[0167] In one embodiment, the second identification module 40 is further configured to:

[0168] Based on the differential hashing algorithm, the target hash value of the target binarized image and the template hash value of the first template binarized image are calculated respectively.

[0169] Calculate the Hamming distance between the target hash value and the template hash value;

[0170] The similarity between the target to be identified and the prior target in the first template binarized image is determined based on the Hamming distance.

[0171] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4 As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other via the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute a domain knowledge-guided method for fine-grained target type recognition in remote sensing images, which includes:

[0172] The original remote sensing image to be identified is acquired, and multidimensional feature extraction is performed on the original remote sensing image to obtain the edge feature image and texture feature image of the original remote sensing image;

[0173] The original remote sensing image, the edge feature image, and the texture feature image are subjected to feature extraction and fusion by a multi-stream convolutional neural network to obtain a fused feature image of the original remote sensing image;

[0174] Based on the fused feature image, the initial category of the target to be identified in the original remote sensing image is determined;

[0175] Obtain the target knowledge template library corresponding to the initial category, and identify the fine type of the target to be identified based on the target knowledge template library.

[0176] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0177] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is capable of executing the domain knowledge-guided remote sensing image target fine type recognition method provided by the above methods, the method comprising:

[0178] The original remote sensing image to be identified is acquired, and multidimensional feature extraction is performed on the original remote sensing image to obtain the edge feature image and texture feature image of the original remote sensing image;

[0179] The original remote sensing image, the edge feature image, and the texture feature image are subjected to feature extraction and fusion by a multi-stream convolutional neural network to obtain a fused feature image of the original remote sensing image;

[0180] Based on the fused feature image, the initial category of the target to be identified in the original remote sensing image is determined;

[0181] Obtain the target knowledge template library corresponding to the initial category, and identify the fine type of the target to be identified based on the target knowledge template library.

[0182] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a domain knowledge-guided method for fine-grained target type recognition in remote sensing images, the method comprising:

[0183] The original remote sensing image to be identified is acquired, and multidimensional feature extraction is performed on the original remote sensing image to obtain the edge feature image and texture feature image of the original remote sensing image;

[0184] The original remote sensing image, the edge feature image, and the texture feature image are subjected to feature extraction and fusion by a multi-stream convolutional neural network to obtain a fused feature image of the original remote sensing image;

[0185] Based on the fused feature image, the initial category of the target to be identified in the original remote sensing image is determined;

[0186] Obtain the target knowledge template library corresponding to the initial category, and identify the fine type of the target to be identified based on the target knowledge template library.

[0187] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0188] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0189] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for fine type recognition of remote sensing image target under the guidance of domain knowledge, characterized in that, The method comprises the following steps: obtaining an original remote sensing image to be identified, and performing multi-dimensional feature extraction on the original remote sensing image to obtain an edge feature image and a texture feature image of the original remote sensing image; performing feature extraction and fusion on the original remote sensing image, the edge feature image and the texture feature image by using a multi-stream convolutional neural network to obtain a fusion feature image of the original remote sensing image; identifying an initial category of a target to be identified in the original remote sensing image based on the fusion feature image; obtaining a target knowledge template library corresponding to the initial category, and identifying a fine type of the target to be identified according to the target knowledge template library; Before the step of performing feature extraction and fusion on the original remote sensing image, the edge feature image and the texture feature image by using a multi-stream convolutional neural network to obtain a fusion feature image of the original remote sensing image, the method further comprises the following steps: generating a fine classification system according to each preset category; obtaining an initial remote sensing image under the fine classification system; the initial remote sensing image contains classification targets under each preset category; obtaining label information of the initial remote sensing image, and performing overlapping cropping processing on the initial remote sensing image based on the label information, and determining a sample image containing the classification target from the cropped image; constructing a first sample data set based on the sample image, and iteratively training a preset basic multi-stream convolutional neural network model by using the first sample data set; constructing a target template image corresponding to the fine classification system in the target knowledge template library based on the image block corresponding to the classification target under each preset category in the sample image; performing binaryzation processing on the mean image of the image block corresponding to the classification target under each preset category in the sample image to construct a target template binaryzation image corresponding to the fine classification system in the target knowledge template library; the target knowledge template library comprises target template images and target template binaryzation images under each preset category, and attribute information and global geospatial distribution range information of the classification target corresponding to the target template image and the target template binaryzation image; performing rotation expansion processing on a target image block in the target template image according to a preset angle interval to obtain a plurality of angle images corresponding to the target template image; the target image block is an image block in the target template image, and the classification recognition result obtained by performing classification recognition on the target image block by using the multi-stream convolutional neural network model is inconsistent with the true value; constructing a second sample data set based on the target template image and the angle image; iteratively training a preset basic fine classification model by using the second sample data set.

2. The method of claim 1, wherein the method further comprises: The step of performing feature extraction and fusion on the original remote sensing image, the edge feature image and the texture feature image by using a multi-stream convolutional neural network to obtain a fusion feature image of the original remote sensing image comprises the following steps: According to a preset size sliding window and a preset sliding step, the original remote sensing image is subjected to an overlapped grid division processing, so as to obtain a plurality of grid images corresponding to the original remote sensing image; the edge feature image comprises an edge feature sub-image corresponding to each of the grid images, and the texture feature image comprises a texture feature sub-image corresponding to each of the grid images; The target grid image, the edge feature sub-image corresponding to the target grid image, and the texture feature sub-image corresponding to the target grid image are input into a pre-trained multi-stream convolutional neural network model, and the multi-stream convolutional neural network model is used to perform feature extraction and fusion of the target grid image, the edge feature sub-image corresponding to the target grid image, and the texture feature sub-image corresponding to the target grid image, so as to obtain a fusion feature image of the original remote sensing image; the target grid image is any one of the plurality of grid images; The multi-stream convolutional neural network model comprises a plurality of convolutional neural networks sharing weights and an attention mechanism, and the plurality of convolutional neural networks comprise a first convolutional neural network, a second convolutional neural network, and a third convolutional neural network; The input of the first convolutional neural network is the grid image, the input of the second convolutional neural network is the edge feature sub-image, and the input of the third convolutional neural network is the texture feature sub-image; and the attention mechanism is used to perform feature fusion on the outputs of the plurality of convolutional neural networks.

3. The method of claim 2, wherein the method further comprises: After the fine type of the to-be-identified target is identified according to the target knowledge template library, the method further comprises: Converting a target pixel coordinate in each of the grid images into a pixel coordinate under the original remote sensing image; the target pixel coordinate is a pixel coordinate corresponding to the to-be-identified target in the grid image; Obtaining projection parameter information of the original remote sensing image, and determining geographical spatial position information of the to-be-identified target according to the projection parameter information and the pixel coordinate; Determining a global geographical spatial distribution range of the to-be-identified target based on the target knowledge template library, and determining whether the geographical spatial position information of the to-be-identified target is within the global geographical spatial distribution range according to the geographical spatial position information and the global geographical spatial distribution range; If yes, performing a de-duplication splicing processing on each of the grid images based on a preset non-maximum suppression algorithm.

4. The method of claim 1, wherein the method further comprises: The step of identifying an initial category of a to-be-identified target in the original remote sensing image based on the fusion feature image comprises: Determining an enclosing rectangle of the to-be-identified target in the original remote sensing image and a long direction included angle of the enclosing rectangle according to the fusion feature image; the long direction included angle is an included angle between a long side of the enclosing rectangle and a calibration direction; Performing a rotation and clipping processing on an image block covered by the enclosing rectangle according to the long direction included angle and the calibration direction, so as to obtain a target image containing the to-be-identified target; and inputting the target image into a pre-trained target fine classification model to obtain a category prediction value output by the target fine classification model; the category prediction value is a probability value that the to-be-identified target in the original remote sensing image is each preset category; performing descending order sorting on the category prediction value, and selecting a preset number of categories in the order of sorting as initial categories of the to-be-identified target.

5. The method of claim 4, wherein the method further comprises: The initial categories include multiple, and the step of identifying the fine type of the to-be-identified target according to the target knowledge template library includes: acquiring a prior target in the target knowledge template library; determining a size mean deviation of the to-be-identified target and the prior target according to a spatial resolution of the original remote sensing image; screening the initial categories according to the size mean deviation to determine a first target category of the to-be-identified target; performing binaryzation processing on the target image to obtain a target binaryzation image corresponding to the target image; calculating a similarity between the to-be-identified target and a prior target in a first template binaryzation image based on the target binaryzation image; the first template binaryzation image is a template binaryzation image corresponding to the first target category in the target knowledge template library; determining the fine type of the to-be-identified target according to the similarity.

6. The method of claim 5, wherein the method further comprises: The step of calculating the similarity between the to-be-identified target and the prior target in the first template binaryzation image based on the target binaryzation image includes: calculating a target hash value of the target binaryzation image and a template hash value of the first template binaryzation image respectively based on a difference hash algorithm; calculating a Hamming distance between the target hash value and the template hash value; determining the similarity between the to-be-identified target and the prior target in the first template binaryzation image according to the Hamming distance.

7. A device for fine type recognition of remote sensing image target under the guidance of domain knowledge, characterized in that, It includes: a feature extraction module configured to acquire an original remote sensing image to be identified and perform multi-dimensional feature extraction on the original remote sensing image to obtain an edge feature image and a texture feature image of the original remote sensing image; a feature fusion module configured to perform feature extraction and fusion on the original remote sensing image, the edge feature image and the texture feature image through a multi-stream convolutional neural network to obtain a fusion feature image of the original remote sensing image; a first identification module configured to identify initial categories of a to-be-identified target in the original remote sensing image based on the fusion feature image; a second identification module configured to acquire a target knowledge template library corresponding to the initial categories and identify a fine type of the to-be-identified target according to the target knowledge template library; a model training module configured to generate a fine classification system according to each preset category; acquire an initial remote sensing image under the fine classification system; the initial remote sensing image contains classification targets under each preset category; acquire labeling information of the initial remote sensing image, perform overlapping cropping processing on the initial remote sensing image based on the labeling information, and determine a sample image containing the classification target from the cropped image; constructing a first sample dataset based on the sample images, and iteratively training a preset basic multi-stream convolutional neural network model by using the first sample dataset; constructing a target template image corresponding to the fine classification system in the target knowledge template library based on an image block corresponding to a classification target under each preset category in the sample images; performing binaryzation processing on a mean image of the image block corresponding to the classification target under each preset category in the sample images, to construct a target template binaryzation image corresponding to the fine classification system in the target knowledge template library; the target knowledge template library comprises target template images and target template binaryzation images under each preset category, and attribute information and global geospatial distribution range information of classification targets corresponding to the target template images and the target template binaryzation images; target image blocks in the target template images are subjected to rotation expansion processing according to a preset angle interval, to obtain a plurality of angle images corresponding to the target template images; the target image blocks are image blocks in the target template images, for which a classification recognition result obtained by performing classification recognition by using the multi-stream convolutional neural network model is inconsistent with a true value; constructing a second sample dataset based on the target template images and the angle images; and iteratively training a preset basic fine classification model by using the second sample dataset.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the steps of the remote sensing image target fine type recognition method guided by the domain knowledge according to the program. 9.A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by the processor, implements the steps of the remote sensing image target fine type recognition method guided by the domain knowledge according to any one of claims 1 to 6.

Images