Visible light remote sensing image target fine granularity detection method based on hierarchical knowledge guidance

Abstract

The invention relates to a hierarchical knowledge guidance-based visible light remote sensing image target fine granularity detection method, which comprises the following steps of: acquiring visible light remote sensing image data, and constructing a multi-semantic hierarchical label system; extracting visible light remote sensing image features, and generating multi-scale region-of-interest features through an RPN network; extracting multi-level semantic features from the multi-scale features of the region of interest by adopting multiple branches; performing local-global feature fusion of adjacent levels on the multi-level semantic features to obtain enhanced features; supervising multi-level classification by using a plurality of level labels, and supervising regression at a first semantic level; the network structure is simplified in the reasoning stage, and the reasoning speed is increased. According to the method, for multiple types of targets in the visible light remote sensing image, hierarchical relation and information injection in the target fine-grained detection process is achieved, and extraction and learning of common features and fine-grained features of the targets by the network are enhanced in combination with multi-layer feature fusion.

Description

Technical Field

[0001] The present invention relates to the technical field of visible light remote sensing image target detection and recognition, and in particular to a visible light remote sensing image target fine-grained detection method based on hierarchical knowledge guidance. Background Art

[0002] The rise of deep learning has profoundly changed the traditional interpretation of remote sensing images. Compared with visual interpretation and traditional manual feature extraction methods, today's object detection methods based on rotating frames have achieved higher efficiency and better performance. With the continuous maturity and development of remote sensing technology, the spatial resolution of visible light remote sensing images has been continuously improved, making it possible to achieve fine-grained detection of targets in visible light remote sensing images. However, as the target classification level deepens, the similarities between different subclasses are greater. At the same time, due to issues such as scale, rotation, and lighting in the same subclass, it is extremely challenging to perform fine-grained detection tasks. Due to factors such as uneven distribution of objects, selection bias of data collection locations, and selection bias of target types, the number of targets in different categories may vary greatly, further increasing the difficulty of fine-grained detection tasks.

[0003] When detecting multiple fine-grained targets, current rotation target detection methods usually regard all target labels as labels of the same level, and assume that they have the same mutually exclusive relationship, which leads to the lack of hierarchical structure and information between targets, and to a certain extent deepens the problem of poor performance of the model in fine-grained detection tasks. Most typical targets in remote sensing images are man-made targets with a strict classification system, and the classification level and the hierarchical features of the target are closely related. Research on them can better perform differentiated extraction of target category features under weak supervision. At the same time, features at high semantic levels can be migrated to low semantic levels, which can alleviate the problem of insufficient feature extraction of some targets due to insufficient training samples to a certain extent.

[0004] Therefore, it is urgent to conduct research on fine-grained target detection methods guided by hierarchical prior knowledge to improve the performance of fine-grained target detection. Summary of the invention

[0005] In order to solve the technical problems existing in the above-mentioned prior art, the purpose of the present invention is to provide a fine-grained detection method for visible light remote sensing image targets based on hierarchical knowledge guidance, so as to realize the refined recognition of typical targets in visible light remote sensing images, such as aircraft, ships, etc.

[0006] To achieve the above-mentioned object of the invention, the present invention provides a method for fine-grained detection of targets in visible light remote sensing images based on hierarchical knowledge guidance, comprising the following steps: Step S1, obtaining visible light remote sensing image data and constructing a multi-semantic level label system; Step S2, extracting visible light remote sensing image features, and generating multi-scale region of interest features through the RPN network; Step S3, extracting multi-level semantic features of the multi-scale region of interest features using multiple branches; Step S4, performing local-global feature fusion of adjacent levels on multi-level semantic features to obtain enhanced features; Step S5, using multiple level labels to supervise multi-level classification and supervise regression at the first semantic level; Step S6: Streamline the network structure during the inference phase, retaining only the most detailed classification branches to increase the inference speed.

[0007] According to a technical solution of the present invention, step S1 specifically includes: Step S11, obtaining visible light remote sensing image data and fine-grained category annotations and rotation bounding box annotations of the targets therein; Step S12: construct a label tree based on the target classification hierarchy to generate a coarse-to-fine Hierarchical semantic labels.

[0008] According to a technical solution of the present invention, step S2 specifically includes: ResNet50+FPN is used to extract multi-scale feature maps; Based on the rotation characteristics of remote sensing targets, Oriented RPN is used to extract a fixed-size feature set of the region of interest. ,in .

[0009] According to a technical solution of the present invention, step S3 specifically includes: The features output by Oriented RPN Input to multiple parallel branches; The first branch is directly output to the standard head module, and the remaining branches are generated by the feature re-extraction module except the first level. Semantic features at each level ; Among them, the feature re-extraction module is implemented through the operations of 1×1 convolution dimensionality reduction, 3×3 convolution re-extraction, and 1×1 convolution dimensionality restoration.

[0010] According to a technical solution of the present invention, in step S4, the features of the adjacent levels are Perform feature fusion, including: Step S41: feature and Features Perform global pooling and splicing to generate global features ; Step S42: feature and Features Directly add and then generate local features through convolution operation ; Step S43: Global features With local features Add and generate weights through Sigmoid function ; Step S44: input the fused enhanced features.

[0011] According to a technical solution of the present invention, in step S44, the enhanced feature It is expressed as:

[0012] According to a technical solution of the present invention, step S5 specifically includes: Step S51: In the first semantic level, the classification branch is used to perform coarse-grained category prediction, and the regression branch is used to perform rotated bounding box coordinate regression; Step S52: In the remaining levels, the enhanced features are used for fine-grained classification; Step S53: balancing the multi-task loss weights based on the loss function balancing method.

[0013] According to a technical solution of the present invention, the classification branch uses multi-category cross entropy as a classification loss function, and the classification loss function calculation formula of the multi-category cross entropy is expressed as:

[0014] in, is an indicator variable, when =1, indicating that the predicted category is consistent with the actual category; =0, indicating that the predicted category is inconsistent with the actual category; Indicates The probability that a sample is correctly classified; Indicates the total number of ship categories.

[0015] According to a technical solution of the present invention, the regression branch uses As the regression loss function, the training of the supervised branch is then supervised. The loss calculation formula is expressed as:

[0016] in, Represents the difference between the predicted value and the true value.

[0017] According to a technical solution of the present invention, the loss function balancing method includes: The DWA dynamic weight allocation method is used to dynamically adjust the multi-task weights according to the loss ratio of adjacent training stages. Specifically: ,

[0018] in, is the temperature coefficient, which is used to adjust the degree of weight difference between different tasks; Indicates Tasks in the training phase loss ratio; and Respectively represent Tasks in the training phase and The loss value of Indicates Tasks in the training phase Dynamic weight.

[0019] Compared with the prior art, the present invention has the following beneficial effects: The present invention provides a method for fine-grained target detection in visible light remote sensing images based on hierarchical knowledge guidance. By constructing a multi-layer semantic feature extraction and fusion feature network supervised by hierarchical multi-labels, the problem of hierarchical relationship and information loss caused by flat labels is solved. The problem of insufficient feature learning of target categories with fewer samples caused by the difference in the number of samples in fine-grained detection tasks is alleviated to a certain extent through the interaction of features between upper and lower levels, thereby improving the performance of the model in fine-grained detection tasks. The network can be improved and implemented on the basis of multiple existing target detection algorithms, and can be used as a general network construction method for fine-grained target detection tasks under visible light remote sensing images. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 A schematic diagram schematically shows a process flow of a method for fine-grained target detection in visible light remote sensing images based on hierarchical knowledge guidance according to an embodiment of the present invention; Figure 2 A schematic diagram of a network of a method for fine-grained target detection in visible light remote sensing images guided by hierarchical knowledge according to an embodiment of the present invention is shown; Figure 3A schematic diagram of the structure of a feature re-extraction module according to an embodiment of the present invention is shown; Figure 4 The figure schematically shows a structural diagram of a feature fusion module according to an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] like Figure 1 As shown, the present invention is a method for fine-grained detection of targets in visible light remote sensing images based on hierarchical knowledge guidance. When obtaining visible light remote sensing images and their related annotations, the fine-grained level labels are used to construct hierarchical multi-labels from coarse to fine according to the existing hierarchical knowledge. At the same time, a multi-level feature extraction network is designed to realize the extraction of different semantic feature information supervised by labels of different semantic levels. Then, a feature fusion method of adjacent levels is designed to realize feature fusion from shallow to deep. On the basis of maintaining common features, the discriminative features are enhanced, and then the features of the fine-grained level are enhanced. Finally, the loss summation method of dynamic weight allocation is combined in the training process to make the model training more balanced. Compared with traditional methods, the present invention realizes the injection of hierarchical relationships and information in the process of fine-grained target detection, and through multi-layer feature fusion, it is more conducive to the network to extract and learn the common features and fine-grained features of the target, further improves the accuracy of the model, and realizes the dual drive of knowledge and data.

[0024] A method for fine-grained detection of targets in visible light remote sensing images based on hierarchical knowledge guidance of the present invention comprises the following steps: Step S1: Obtain visible light remote sensing image data and construct a multi-semantic level label system, specifically including: Step S11, obtaining visible light remote sensing image data and fine-grained category annotations and rotation bounding box annotations of the targets therein; Step S12: construct a label tree based on the target classification hierarchy to generate a coarse-to-fine Hierarchical semantic labeling; Step S2: extract visible light remote sensing image features and generate multi-scale region of interest features through the RPN network; Figure 2 As shown, the feature extraction process includes: The multi-scale feature extraction structure of ResNet50+FPN is used to obtain the multi-scale feature map of the image. Due to the rotation characteristics of the target in the remote sensing image, the Oriented RPN is used to extract the region of interest on the feature map to obtain the features of multiple regions of interest. ,in .

[0025] Step S3: extract multi-level semantic features from the multi-scale region of interest features using multiple branches, such as Figure 3 As shown, specifically including: After the preceding Oriented RPN structure, the output is sent to multiple parallel branches; The first branch takes the features obtained in the previous order Output directly to the standard header module; The remaining branches will feature Through a feature re-extraction module, semantic features of different levels are extracted to obtain features other than the first level. Semantic features at each level .

[0026] The process of extracting features at different semantic levels by the feature re-extraction module includes: The features obtained from the previous order Use a scale of The convolution module is used to perform dimensionality reduction operations, using Function performs network activation and obtains features .

[0027] The characteristics Use scale The convolution module performs feature re-extraction operations, and uses To keep the length and width of the feature consistent. Function to activate the network. Through this process, the features of the remaining layers are obtained

[0028] The re-extracted features are scaled as The convolution module performs dimensionality recovery operations, using Function performs network activation and obtains features .

[0029] Step S4: perform local-global feature fusion of adjacent levels of multi-level semantic features to obtain enhanced features, that is, the features of multiple levels obtained in the previous order , according to the local-global feature fusion method from shallow to deep, the feature fusion operation of adjacent levels is obtained Enhanced hierarchical features .

[0030] The feature fusion operations at adjacent levels include: For the features of the three adjacent levels , the implementation method of feature fusion is as follows:

[0031]

[0032] Among them Represents a specific feature fusion method.

[0033] like Figure 4 As shown, the local-global feature fusion method includes: The two features to be fused, such as , respectively, using global pooling to obtain two Features , and splice the two. Then the spliced ​​features are sequentially The convolution operation and Function activation to obtain the final global features .

[0034] The two features to be fused, such as , directly use the addition operation to get a Then, the feature is sequentially The convolution operation, Function activation, The convolution operation is performed to obtain the enhanced local features. .

[0035] The global features and local features are added, and then Function output feature fusion weight ,pass The final enhanced feature is obtained, that is, .

[0036] In summary, in step S4, the enhanced features It is expressed as:

[0037] Step S5, using multiple level labels to supervise multi-level classification, and supervise regression at the first semantic level, the step S5 specifically includes: Step S51: In the first semantic level, use the feature After input into the head module, a classification branch and a regression branch are constructed respectively. The classification branch is used to achieve the coarsest level of target classification; the regression branch is used to achieve the coordinate regression of the rotation bounding box of the target, using the five-point representation method. Respectively represent the horizontal and vertical coordinates of the target center point ( )、target width( )、Target height( ) and the target rotation angle ( ); Step S52: semantic level, using enhanced features After being input to the head module of each level, each level adopts a classification branch. The classification branch performs category classification of the target level of each level; Step S53: Use a loss function balancing method to balance the relationship between the losses of each branch, so that the training of each branch of the network is more balanced.

[0038] In step S51 and step S52, the classification branch uses multi-class cross entropy loss as the loss function, and the regression branch uses The loss is used as the loss function to supervise the training of the branch.

[0039] The classification loss function calculation formula of multi-category cross entropy is:

[0040] in, is an indicator variable, when =1, indicating that the predicted category is consistent with the actual category; =0, indicating that the predicted category is inconsistent with the actual category; Indicates The probability that a sample is correctly classified; Represents the total number of categories for categorical targets.

[0041] The loss calculation formula is:

[0042] in, Represents the difference between the predicted value and the true value. For each parameter represented by the five-point formula, its loss, and obtain the final regression loss value by weighted summation.

[0043] In one embodiment of the present invention, preferably, in step S53, in order to balance the loss functions among multiple branches, The loss summation method replaces the ordinary loss summation method, which uses the loss ratio of different tasks in adjacent training to measure their learning speed and then calculate the weight. For each loss, the weight calculation method is as follows: ,

[0044] in, is the temperature coefficient, which is used to adjust the degree of weight difference between different tasks; Indicates Tasks in the training phase loss ratio; and Respectively represent Tasks in the training phase and The loss value of Indicates Tasks in the training phase Dynamic weight.

[0045] Step S6: Streamline the network structure during the inference phase, retaining only the most detailed classification branches to increase the inference speed.

[0046] It should be noted that although the embodiments of the present invention described above are illustrative, they are not intended to limit the present invention, and therefore the present invention is not limited to the above specific embodiments. Without departing from the principles of the present invention, any other embodiments obtained by those skilled in the art under the guidance of the present invention are deemed to be within the protection of the present invention.

Claims

1. A method for fine-grained target detection in visible light remote sensing images based on hierarchical knowledge guidance, characterized in that: The following steps are involved: Step S1, obtaining visible light remote sensing image data and constructing a multi-semantic level label system; Step S2, extracting visible light remote sensing image features, and generating multi-scale region of interest features through the RPN network; Step S3, extracting multi-level semantic features of the multi-scale region of interest features using multiple branches; Step S4, performing local-global feature fusion of adjacent levels on multi-level semantic features to obtain enhanced features; Step S5, using multiple level labels to supervise multi-level classification and supervise regression at the first semantic level; Step S6: Streamline the network structure during the inference phase, retaining only the most detailed classification branches to increase the inference speed.

2. The method for fine-grained target detection in visible light remote sensing images based on hierarchical knowledge guidance according to claim 1 is characterized in that: The step S1 specifically includes: Step S11, obtaining visible light remote sensing image data and fine-grained category annotations and rotation bounding box annotations of the targets therein; Step S12: construct a label tree based on the target classification hierarchy to generate a coarse-to-fine Hierarchical semantic labels.

3. The method for fine-grained target detection in visible light remote sensing images based on hierarchical knowledge guidance according to claim 1 is characterized in that: The step S2 specifically includes: ResNet50+FPN is used to extract multi-scale feature maps; Based on the rotation characteristics of remote sensing targets, Oriented RPN is used to extract a fixed-size feature set of the region of interest. ,in .

4. The method for fine-grained target detection in visible light remote sensing images based on hierarchical knowledge guidance according to claim 1 is characterized in that: The step S3 specifically includes: The features output by Oriented RPN Input to multiple parallel branches; The first branch is directly output to the standard head module, and the remaining branches are generated by the feature re-extraction module except the first level. Semantic features at each level ; Among them, the feature re-extraction module is implemented through the operations of 1×1 convolution dimensionality reduction, 3×3 convolution re-extraction, and 1×1 convolution dimensionality restoration.

5. The method for fine-grained target detection in visible light remote sensing images based on hierarchical knowledge guidance according to claim 4 is characterized in that: In step S4, the features of the adjacent levels are Perform feature fusion, including: Step S41: feature and Features Perform global pooling and splicing to generate global features ; Step S42: feature and Features Directly add and then generate local features through convolution operation ; Step S43: Global features With local features Add and generate weights through Sigmoid function ; Step S44: input the fused enhanced features.

6. The method for fine-grained target detection in visible light remote sensing images based on hierarchical knowledge guidance according to claim 5 is characterized in that: In step S44, the enhanced feature It is expressed as: 。 7. The method for fine-grained target detection in visible light remote sensing images based on hierarchical knowledge guidance according to claim 1 is characterized in that: The step S5 specifically includes: Step S51: In the first semantic level, the classification branch is used to perform coarse-grained category prediction, and the regression branch is used to perform rotated bounding box coordinate regression; Step S52: In the remaining levels, the enhanced features are used for fine-grained classification; Step S53: balancing the multi-task loss weights based on the loss function balancing method.

8. The method for fine-grained target detection in visible light remote sensing images based on hierarchical knowledge guidance according to claim 7 is characterized in that: The classification branch uses multi-category cross entropy as the classification loss function. The classification loss function calculation formula of multi-category cross entropy is expressed as: in, is an indicator variable, when =1, indicating that the predicted category is consistent with the actual category; =0, indicating that the predicted category is inconsistent with the actual category; Indicates The probability that a sample is correctly classified; Represents the total number of categories for categorical targets.

9. The method for fine-grained target detection in visible light remote sensing images based on hierarchical knowledge guidance according to claim 7, characterized in that: Regression branch usage As the regression loss function, the supervised branch is trained. The loss calculation formula is expressed as: in, Represents the difference between the predicted value and the true value.

10. The method for fine-grained target detection in visible light remote sensing images based on hierarchical knowledge guidance according to claim 7, characterized in that: The loss function balancing method comprises: The DWA dynamic weight allocation method is used to dynamically adjust the multi-task weights according to the loss ratio of adjacent training stages. Specifically: , in, is the temperature coefficient, which is used to adjust the degree of weight difference between different tasks; Indicates Tasks in the training phase loss ratio; and Respectively represent Tasks in the training phase and The loss value of Indicates Tasks in the training phase Dynamic weight of .

Images