Fine-grained auxiliary visible light remote sensing image airplane target detection method and device

Abstract

The application provides a fine-grained auxiliary visible light remote sensing image airplane target detection method and device. The method comprises steps 1 to 7. The application can effectively utilize the fine-grained attributes of the airplane target to assist in classification, and the pre-training model obtained on the GF-RarePlanes dataset has good generalization ability, and can effectively improve the airplane target detection precision.

Description

TECHNICAL FIELD

[0001] The embodiment of the application relates to the technical field of image detection, in particular to a fine-grained auxiliary visible light remote sensing image airplane target detection method and equipment. BACKGROUND

[0002] With the rapid development of economy, airplanes have become an important means of transportation for people's daily travel. As an important strategic military target, accurately monitoring the dynamics of airplanes in remote sensing images can provide important intelligence for war. The airplane target detection task in remote sensing images can be divided into positioning and model-level classification. The former is relatively easy because the size of the airplane is large, the number of pixels occupied in the remote sensing image is large, and each airplane is parked in the airport with enough space between them, so there is no problem of dense and small targets as in the vehicle target detection task. The model-level classification of the airplane is a difficult problem of the detection task, and the key task is to solve the problem of small differences between sample classes. At present, there is no airplane target detection data set that can be directly used for classification considering fine-grained attributes, so that the detection task lacks sample support, and related methods often seek solutions from network structure, and rarely consider the prior knowledge that the data set itself can provide to the model. However, in actual remote sensing images, airplane targets are often very similar in shape and structure and only slightly different in some basic structures. Therefore, developing a fine-grained auxiliary visible light remote sensing image airplane target detection method and equipment can effectively overcome the defects in the related art, and has become a technical problem to be solved in the industry. SUMMARY

[0003] In view of the above problems existing in the prior art, the embodiment of the application provides a fine-grained auxiliary visible light remote sensing image airplane target detection method and equipment.

[0004] In a first aspect, embodiments of the present application provide a fine-grained assisted visible light remote sensing image aircraft target detection method, comprising: step 1, constructing a GF-RarePlanes joint dataset, uniformly labeling the GF dataset and the RarePlanes dataset, that is, each aircraft sample in the RarePlanes dataset has consistent rotated rectangular frame labeling as the GF dataset; step 2, reclassifying the GF dataset and the RarePlanes dataset, that is, selecting five attributes from the eleven fine-grained attributes of aircraft targets in the RarePlanes dataset, merging the sub-attributes under the five attributes, and dividing the aircraft samples in the GF dataset and the RarePlanes dataset into corresponding attribute combinations after combining the attributes; step 3, generating a joint dataset label file, uniformly converting the label files of the merged GF dataset and the RarePlanes dataset into.json format, extracting the labeling contents of the two datasets, one picture corresponding to one.txt file; then converting the.txt file to.xml file, which converts the four vertices of the rotated rectangular frame labeling to the format of the inclined text rectangular frame; finally, converting the.xml file to the.json file, thereby completing the construction of the GF-RarePlanes joint dataset; step 4, pre-training on the GF-RarePlanes joint dataset, wherein the feature extractor is G(·), and a pre-trained model Mp is obtained; step 5, using the pre-trained model Mp obtained in step 4, fine-tuning on the GF dataset, wherein the feature extractor is G(·), and a fine-tuned model Mf is obtained; step 6, testing the model Mf obtained in step 5 on the GF dataset to obtain the test results, that is, the average precision of each class and the average precision of all classes; step 7, in order to verify the generalization ability of the pre-trained model obtained in step 4 and the effectiveness of the fine-grained attribute assisted aircraft target detection method, the FAIR1M dataset is verified, and the average precision of each class and the average precision of all classes are obtained.

[0005] On the basis of the above method embodiment content, the fine-grained assisted visible light remote sensing image aircraft target detection method provided in the embodiments of the present application, the GF dataset, that is, the GaoFenPlanes dataset, is derived from the 2020 high-resolution earth observation automatic interpretation challenge, and the models include: Boeing737, Boeing747, Boeing777, Boeing787, Airbus A220, Airbus A321, Airbus A330, Airbus A350 and ARJ21, wherein each aircraft is labeled with a rotated rectangular frame.

[0006] On the basis of the above method embodiment content, the fine-grained assisted visible light remote sensing image aircraft target detection method provided in the embodiment of the application, the RarePlanes dataset is an open source aircraft target detection dataset, each aircraft sample in the dataset adopts a special diamond-shaped label, and each aircraft sample is pasted with a separate label according to a predetermined classification manner.

[0007] On the basis of the above method embodiment content, the fine-grained assisted visible light remote sensing image aircraft target detection method provided in the embodiment of the application, the feature extractor G(·) in steps 4 and 5 selects ResNet-50, and the initial value of the parameter adopts a ResNet-50 pre-training model provided in Pytorch.

[0008] In a second aspect, embodiments of the present application provide a fine-grained auxiliary visible light remote sensing image aircraft target detection device, comprising: a first main module for implementing step 1, constructing a GF-RarePlanes joint dataset, uniformly labeling the GF dataset and the RarePlanes dataset, that is, each aircraft sample in the RarePlanes dataset has consistent rotated rectangular frame labeling as the GF dataset; step 2, reclassifying the GF dataset and the RarePlanes dataset, that is, selecting five attributes from the fine-grained attributes of the eleven aircraft targets in the RarePlanes dataset, merging the sub-attributes under the five attributes, and dividing the aircraft samples in the GF dataset and the RarePlanes dataset into corresponding attribute combinations after combining the attributes; a second main module for implementing step 3, generating a joint dataset label file, uniformly labeling the label files of the merged GF dataset and the RarePlanes dataset as.json format, extracting the labeling contents of the two datasets, one picture corresponding to one.txt file; then converting the.txt file to.xml file, which converts the four vertices of the rotated rectangular frame labeling to the format of the inclined text rectangular frame; finally, converting the.xml file to the.json file, thereby completing the construction of the GF-RarePlanes joint dataset; a third main module for implementing step 4, pre-training on the GF-RarePlanes joint dataset, wherein the feature extractor is G(·), and a pre-training model Mp is obtained; step 5, using the pre-training model Mp obtained in step 4, fine-tuning on the GF dataset, wherein the feature extractor is G(·), and a fine-tuned model Mf is obtained; a fourth main module for implementing step 6, testing the model Mf obtained in step 5 on the GF dataset to obtain test results, that is, the average precision of each class and the average precision of all classes; step 7, in order to verify the generalization ability of the pre-training model obtained in step 4 and the effectiveness of the fine-grained attribute auxiliary aircraft target detection method, the FAIR1M dataset is verified, and the average precision of each class and the average precision of all classes are obtained.

[0009] In a third aspect, embodiments of the present application provide an electronic device, comprising:

[0010] at least one processor; and

[0011] at least one memory communicatively connected with the processor, wherein:

[0012] The memory stores program instructions executable by the processor, and the processor invoking the program instructions can execute the fine-grained auxiliary visible light remote sensing image aircraft target detection method provided in any of the various implementation manners of the first aspect.

[0013] In a fourth aspect, an embodiment of the present application provides a non-transitory computer-readable storage medium storing computer instructions, the computer instructions causing a computer to perform the method for fine-grained assisted visible light remote sensing image airplane target detection provided in any one of the various implementation manners of the first aspect.

[0014] The method and device for fine-grained assisted visible light remote sensing image airplane target detection provided by the embodiments of the present application can effectively utilize the fine-grained attributes of airplane targets to assist classification, and the pre-training model obtained on the GF-RarePlanes dataset has good generalization ability, which can effectively improve the airplane target detection accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0015] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0016] Figure 1 The flowchart of the method for fine-grained assisted visible light remote sensing image airplane target detection provided by the embodiments of the present application is shown in the figure.

[0017] Figure 2 The structural schematic diagram of the device for fine-grained assisted visible light remote sensing image airplane target detection provided by the embodiments of the present application is shown in the figure.

[0018] Figure 3 The structural schematic diagram of the electronic device provided by the embodiments of the present application is shown in the figure.

[0019] Figure 4 The schematic diagram of airplane target detection provided by the embodiments of the present application is shown in the figure.

[0020] Figure 5 The schematic diagram of the incomplete diamond annotation of airplane samples in the RarePlanes dataset used by the embodiments of the present application is shown in the figure.

[0021] Figure 6 The schematic diagram of converting the incomplete diamond annotation of airplane samples in the RarePlanes dataset into a rectangular annotation provided by the embodiments of the present application is shown in the figure.

[0022] Figure 7 The schematic diagram of converting the diamond annotation of airplane samples in the RarePlanes dataset into a rotated rectangular annotation provided by the embodiments of the present application is shown in the figure.

[0023] Figure 8A flowchart for generating a joint dataset annotation file is provided for the embodiment of the present application.

[0024] Figure 9 Part of the visual detection results of the fine-tuned model are provided for the embodiment of the present application. DETAILED DESCRIPTION

[0025] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application. In addition, the technical features in each embodiment or in a single embodiment provided by the present application can be combined with each other at will to form a feasible technical solution, and this combination is not restricted by the order of steps and / or structure mode, but must be based on the realization by those of ordinary skill in the art, and when the combination of technical solutions appears contradictory or cannot be realized, it should be considered that the combination of technical solutions does not exist and is not within the protection scope required by the present application.

[0026] The embodiment of the present application provides a fine-grained auxiliary visible light remote sensing image airplane target detection method, which is described with reference to Figure 1The method comprises the following steps: step 1, constructing a GF-RarePlanes joint dataset, uniformly labeling the GF dataset and the RarePlanes dataset, that is, each aircraft sample in the RarePlanes dataset has a consistent rotating rectangular frame label as in the GF dataset; step 2, reclassifying the GF dataset and the RarePlanes dataset, that is, selecting five attributes from the eleven aircraft target fine-grained attributes in the RarePlanes dataset, and merging the sub-attributes under the five attributes, and dividing the aircraft samples in the GF dataset and the RarePlanes dataset into corresponding attribute combinations after combining the attributes; step 3, generating a joint dataset label file, uniformly converting the label files of the merged GF dataset and the RarePlanes dataset into a.json format, extracting the label contents of the two datasets, one picture corresponding to one.txt file; then converting the.txt file into an.xml file, which converts the four vertices of the rotating rectangular frame label into the format of the inclined text rectangular frame; finally, converting the.xml file into a.json file, thereby completing the construction of the GF-RarePlanes joint dataset; step 4, pre-training on the GF-RarePlanes joint dataset, wherein the feature extractor is G(·), to obtain a pre-training model Mp; step 5, using the pre-training model Mp obtained in step 4, fine-tuning on the GF dataset, wherein the feature extractor is G(·), to obtain a fine-tuned model Mf; step 6, testing the model Mf obtained in step 5 on the GF dataset to obtain test results, that is, the average precision of each class and the average precision of all classes; step 7, in order to verify the generalization ability of the pre-training model obtained in step 4 and the effectiveness of the fine-grained assisted aircraft target detection method, the FAIR1M dataset is verified, and the average precision of each class and the average precision of all classes are obtained.

[0027] Based on the content of the above method embodiment, as an optional embodiment, the fine-grained assisted visible light remote sensing image aircraft target detection method provided in the embodiment of the application, the GF dataset is GaoFenPlanes dataset, which is derived from the 2020 High Resolution Earth Observation Automatic Interpretation Challenge, and the types include: Boeing737, Boeing747, Boeing777, Boeing787, Airbus A220, Airbus A321, Airbus A330, Airbus A350 and ARJ21, wherein each aircraft is labeled with a rotating rectangular frame.

[0028] Based on the content of the above method embodiment, as an optional embodiment, the fine-grained assisted visible light remote sensing image aircraft target detection method provided in the embodiment of the application is that the RarePlanes dataset is an open-source aircraft target detection dataset, each aircraft sample in the dataset adopts a special diamond-shaped label, and each aircraft sample is labeled with a separate label according to a predetermined classification manner.

[0029] Based on the content of the above method embodiment, as an optional embodiment, the fine-grained assisted visible light remote sensing image aircraft target detection method provided in the embodiment of the application is that the feature extractor G(·) in steps 4 and 5 selects ResNet-50, and the initial value of the parameter adopts a ResNet-50 pre-training model provided in Pytorch.

[0030] The fine-grained assisted visible light remote sensing image aircraft target detection method provided in the embodiment of the application can effectively utilize the fine-grained attributes of the aircraft target to assist classification, and the pre-training model obtained on the GF-RarePlanes dataset has good generalization ability, which can effectively improve the aircraft target detection precision.

[0031] In another embodiment, the application provides a fine-grained assisted visible light remote sensing image aircraft target detection method, as shown in Figure 4 , specifically comprising the following steps:

[0032] Step 1.1: First, unify the annotation formats of the GF dataset and the RarePlanes dataset, and need to convert the diamond-shaped labels in the RarePlanes dataset into rotated rectangular box labels. The annotation file of the RarePlanes dataset is used to visualize the annotation on the image, and it is found that due to the image cropping, the annotation of the aircraft sample located at the edge of the cropping is not a complete diamond-shaped label, but a triangular, pentagonal or hexagonal shape, as shown in Figure 5 .

[0033] Step 1.2: For the triangular, pentagonal or hexagonal label described in step 1.1, convert it into a horizontal bounding box (HBB) label, specifically: use xmin, xmax, ymin and ymax in the original diamond-shaped label to directly take a minimum circumscribed rectangle, and the conversion result is as shown in Figure 6 .

[0034] Step 1.3: For the complete diamond annotation described in step 1.1, convert it to an oriented bounding box (OBB) annotation, specifically: first draw the diagonals A and B of the diamond, and find the intersection point P0; then fix one diagonal A, and calculate the lengths L1 and L2 of the two parts of diagonal B divided by P0; then draw perpendicular segments of lengths L1 and L2 in two directions respectively with the two endpoints of diagonal A as the starting points, and finally connect the two pairs of endpoints of the perpendicular segments to form a closed OBB, as shown in Figure 7 .

[0035] Step 2.1: Re-classify the GF dataset and the RarePlanes dataset according to 5 carefully selected fine-grained attributes. First, select 5 attributes (number of engines, wing position, wing type, FAA wing span category, military / civilian) from the 11 fine-grained attributes of the RarePlanes dataset. In order to make the total number of classes obtained by the final combination not too large, the sub-attributes under the 5 attributes are also combined, and the combined attribute numbers are shown in Table 1.

[0036] Table 1: Combined 5 fine-grained attributes and sub-attribute numbers

[0037]

[0038] Step 2.2: According to the classification rules in Table 1, first re-classify the RarePlanes dataset, and finally obtain 27 fine-grained attribute-assisted classes. Each class number corresponds to a string, and each element of the string from left to right represents the sub-attribute of the fine-grained attribute in Table 1. For example, the string corresponding to '1' is "00200", which means that the aircraft samples meeting the characteristics of having less than or equal to two engines, mid-wing or lower-wing position, swept wing or variable swept wing, FAA wing span category number less than or equal to 3, and civilian aircraft will be classified into class '1'.

[0039] Step 2.3: According to the fine-grained attribute-assisted class numbers obtained in the classification rules in Table 1, classify the GF dataset. The classification results are shown in Table 2. Except for the other class, the aircraft of the remaining nine types can be divided into the existing 27 fine-grained attribute-assisted classes, so the 28th class is added to the joint dataset, and the other class is divided into this class, thus obtaining 28 fine-grained attribute-assisted classes.

[0040] Table 2: Classification results and sequence numbers of fine-grained attribute-assisted GF dataset

[0041]

[0042]

[0043] Step 3.1: On the basis of step 2, the annotation file of the joint dataset is generated, and the specific process is as shown in Figure 8 First, the annotations of the two datasets are extracted, and each picture corresponds to a.txt file, and each plane corresponds to a line of data, which contains the category number and the coordinates of the four vertices of the rotated rectangular frame (x1, y1, x2, y2, x3, y3, x4, y4).

[0044] Step 3.2: Then, the.txt file is converted into an.xml file. This process converts the coordinates of the four vertices of the rotated rectangular frame (x1, y1, x2, y2, x3, y3, x4, y4) into the format of the inclined text rectangular frame (cx, cy, w, h, ang).

[0045] Step 3.3: Finally, the.xml file is converted into a.json file (COCO format), and thus the construction of the GF-RarePlanes joint dataset is completed. This joint dataset contains 6481 training and validation set samples and 2824 test set samples, and a total of 28 fine-grained attribute-assisted categories.

[0046] Step 4: Pre-training is performed on the joint dataset obtained in step 3 using the Faster R-CNN, Mask R-CNN, Cascade R-CNN, and DetectoRS frameworks, respectively. The SGD optimizer is used, the learning rate is set to 0.005, and the training epoch is set to 24. The test accuracy of the pre-trained model is shown in Table 3.

[0047] Table 3 Test accuracy of pre-trained model (%)

[0048]

[0049] Step 5: Fine-tuning is performed on the pre-trained model obtained in step 4 using the Faster R-CNN, Mask R-CNN, Cascade R-CNN, and DetectoRS frameworks on the GF dataset. The training and validation set in the GF dataset is used, which contains a total of 803 pictures. The SGD optimizer is used, the learning rate is set to 0.005, and the training epoch is set to 24. The fine-tuned model Mf is obtained.

[0050] Step 6: Test the fine-tuned model Mf on the GF dataset, which uses the test set in the GF dataset, containing 197 images. The detection results are shown in Table 4, i.e., the average precision (AP) of each class and the mean average precision (mAP) of all classes. The visualization of the detection results is shown in FIG. 6. Figure 9 .

[0051] Table 4 Detection results of fine-grained attribute assisted aircraft target detection (%)

[0052]

[0053] Step 7: To further verify the generalization ability of the pre-trained model obtained in step 4 and the effectiveness of the fine-grained attribute assisted aircraft target detection method, the embodiment performs a verification experiment on the FAIR1M dataset. The dataset contains 901 training and validation set samples and 831 test set samples, and contains 11 aircraft target categories. Compared with the GF dataset, it only adds one C919 model aircraft. Through experiments on four target detection network frameworks, the test results are shown in Table 5.

[0054] Table 5 Test results of FAIR1M dataset (%)

[0055]

[0056] During the experiment, the fine-grained attribute assisted aircraft target detection method is used to detect aircraft targets in remote sensing images. The detection results on the GF dataset are shown in Table 4, including the detection results without using fine-grained attribute assistance and the detection results with fine-grained attribute assistance. Table 5 shows the test results of the pre-trained model trained on the GF-RarePlanes dataset and fine-tuned on the FAIR1M dataset. It can be seen that the detection accuracy is greatly improved. The above two experiments show the effectiveness of the fine-grained attribute assisted aircraft target detection method used in the embodiment and the generalization ability of the pre-trained model trained on the GF-RarePlanes dataset. That is, the method can effectively improve the accuracy of the high-resolution visible light remote sensing image aircraft target detection task.

[0057] The implementation basis of each embodiment of the present application is achieved by a programmed process through a device with processor function. Therefore, in engineering practice, the technical solutions and functions of each embodiment of the present application can be packaged into various modules. Based on this actual situation, on the basis of the above-mentioned embodiments, the embodiment of the present application provides a fine-grained assisted visible light remote sensing image aircraft target detection device, which is used to execute the fine-grained assisted visible light remote sensing image aircraft target detection method in the above-mentioned method embodiment. Referring to Figure 2 The device comprises: a first main module for implementing step 1, constructing a GF-RarePlanes joint dataset, and uniformly labeling the formats of the GF dataset and the RarePlanes dataset, that is, each aircraft sample in the RarePlanes dataset has consistent rotated rectangular frame labeling as the GF dataset; step 2, reclassifying the GF dataset and the RarePlanes dataset, that is, selecting five attributes from the eleven fine-grained attributes of aircraft targets in the RarePlanes dataset, and merging the sub-attributes under the five attributes, and dividing the aircraft samples in the GF dataset and the RarePlanes dataset into corresponding attribute combinations after combining the attributes; a second main module for implementing step 3, generating a joint dataset labeling file, uniformly labeling the labeling files of the merged GF dataset and RarePlanes dataset as.json format, extracting the labeling contents of the two datasets, one picture corresponding to one.txt file; then converting the.txt file to an.xml file, which converts the four vertices of the rotated rectangular frame labeling to the format of the inclined text rectangular frame; finally, converting the.xml file to a.json file, thereby completing the construction of the GF-RarePlanes joint dataset; a third main module for implementing step 4, pre-training on the GF-RarePlanes joint dataset, wherein the feature extractor is G(·), and a pre-training model Mp is obtained; step 5, using the pre-training model Mp obtained in step 4, fine-tuning on the GF dataset, wherein the feature extractor is G(·), and a fine-tuned model Mf is obtained; a fourth main module for implementing step 6, testing the model Mf obtained in step 5 on the GF dataset to obtain the test results, that is, the average precision of each class and the average precision of all classes; step 7, in order to verify the generalization ability of the pre-training model obtained in step 4 and the effectiveness of the fine-grained attribute assisted aircraft target detection method, the FAIR1M dataset is verified, and the average precision of each class and the average precision of all classes are obtained.

[0058] The fine-grained assisted visible light remote sensing image aircraft target detection device provided by the embodiment of the present application adopts Figure 2The several modules in the method can effectively utilize the fine-grained attributes of the aircraft target to assist classification, and the pre-trained model obtained on the GF-RarePlanes dataset has good generalization ability and can effectively improve the aircraft target detection accuracy.

[0059] It should be noted that the device in the device embodiment provided by the application can be used to implement the methods in the above method embodiments, and can also be used to implement the methods in other method embodiments provided by the application. The difference is only that the corresponding functional modules are set, and the principle is basically the same as that of the above device embodiments provided by the application. As long as the person skilled in the art improves the device in the above device embodiments by combining technical features and obtaining corresponding technical means and technical solutions composed of these technical means on the basis of the above device embodiments, and on the premise of ensuring the practicability of the technical solution, the corresponding device class embodiment can be obtained to implement the method in other method class embodiments. For example:

[0060] Based on the content of the above device embodiment, as an optional embodiment, the fine-grained assisted visible light remote sensing image aircraft target detection device provided in the embodiment of the application further comprises: a first sub-module for implementing the GF dataset, i.e., the GaoFenPlanes dataset, which is derived from the 2020 High Resolution Earth Observation Automatic Interpretation Challenge. The models include: Boeing737, Boeing747, Boeing777, Boeing787, Airbus A220, Airbus A321, Airbus A330, Airbus A350 and ARJ21, wherein each aircraft is labeled with a rotating rectangular frame.

[0061] Based on the content of the above device embodiment, as an optional embodiment, the fine-grained assisted visible light remote sensing image aircraft target detection device provided in the embodiment of the application further comprises: a second sub-module for implementing the RarePlanes dataset, which is an open source aircraft target detection dataset. Each aircraft sample in the dataset is labeled with a special diamond, and each aircraft sample is labeled with a separate label according to a predetermined classification method.

[0062] Based on the content of the above device embodiment, as an optional embodiment, the fine-grained assisted visible light remote sensing image aircraft target detection device provided in the embodiment of the application further comprises: a third sub-module for implementing the feature extractor G(·) in steps 4 and 5, which is ResNet-50, and the initial value of the parameter is the ResNet-50 pre-trained model provided in Pytorch.

[0063] The method of the embodiment of the present application is realized by relying on an electronic device, and therefore it is necessary to introduce the related electronic device. For this purpose, the embodiment of the present application provides an electronic device, as shown in the accompanying drawings, which comprises at least one processor, a communications interface, at least one memory and a communications bus, wherein the at least one processor, the communications interface and the at least one memory complete the communications among each other through the communications bus. The at least one processor can invoke the logic instructions in the at least one memory to execute all or part of the steps of the method provided by the foregoing various method embodiments. Figure 3

[0064] In addition, the logic instructions in the at least one memory described above can be realized in the form of a software functional unit and sold or used as an independent product, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various method embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0065] The device embodiments described above are only schematic, wherein the units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, that is, they can be located in one place, or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment of the present application according to actual needs. Those skilled in the art can understand and implement it without creative labor.

[0066] ​Those skilled in the art can clearly understand the implementation of the various embodiments by means of software and necessary universal hardware platforms through the above description of the embodiments, and the various embodiments can also be implemented by hardware. Based on such understanding, the above technical solutions can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, and the like, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments or some parts of the methods.

[0067] The flowcharts and block diagrams in the drawings show the possible implementation architecture, function and operation of the systems, methods and computer program products according to the various embodiments of the present application. Based on this understanding, each block in the flowchart or block diagram can represent a module, a program segment or a part of code, which includes one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in different orders from that shown in the drawings. For example, two consecutive blocks can actually be executed substantially in parallel, and sometimes in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0068] It should be noted that the terms "comprising", "including", or any other variant thereof are intended to cover non-exclusive inclusion, so that processes, methods, articles or devices that include a series of elements not only include those elements, but also include other elements not explicitly listed, or inherent to such processes, methods, articles or devices. Without more limitations, the elements defined by the statement "comprising" do not exclude the presence of additional identical elements in the processes, methods, articles or devices that include the elements.

[0069] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of the present application.

Claims

1. A fine-grained assisted method for aircraft target detection in visible light remote sensing images, characterized in that, include: Step 1: Construct a GF-RarePlanes joint dataset, unifying the annotation format for both the GF and RarePlanes datasets. This means each aircraft sample in the RarePlanes dataset has the same rotated bounding box annotation as in the GF dataset. Step 2: Reclassify the GF and RarePlanes datasets. This involves selecting five attributes from the eleven fine-grained attributes of aircraft targets in the RarePlanes dataset, merging the sub-attributes under these five attributes, and assigning the combined attributes to the corresponding attribute combinations for each aircraft sample in both datasets. Step 3: Generate the joint dataset annotation files. The annotation files for both the GF and RarePlanes datasets are unified into a .json format. The annotation content from both datasets is extracted, with one .txt file corresponding to each image. Then, the .txt files are converted to XML. The process involves converting the four vertices of the rotated rectangle annotation into a slanted text rectangle format; finally, converting the .xml file into a .json file, thus completing the construction of the GF-RarePlanes joint dataset; Step 4, pre-training on the GF-RarePlanes joint dataset, with the feature extractor being G(·), to obtain the pre-trained model Mp; Step 5, fine-tuning the pre-trained model Mp obtained in Step 4 on the GF dataset, with the feature extractor being G(·), to obtain the fine-tuned model Mf; Step 6, testing the model Mf obtained in Step 5 on the GF dataset to obtain the test results, namely the average accuracy of each class and the mean average accuracy of all classes; Step 7, to verify the generalization ability of the pre-trained model obtained in Step 4 and the effectiveness of this fine-grained attribute-assisted aircraft target detection method, validation is performed on the FAIR1M dataset to obtain the average accuracy of each class and the mean average accuracy of all classes.

2. The fine-grained assisted visible light remote sensing image aircraft target detection method according to claim 1, characterized in that, The GF dataset, or GaoFenPlanes dataset, comes from the 2020 High Resolution Earth Observation Automated Interpretation Challenge. The models include: Boeing 737, Boeing 747, Boeing 777, Boeing 787, Airbus A220, Airbus A321, Airbus A330, Airbus A350, and ARJ21. Each aircraft is labeled using a rotated rectangle.

3. The fine-grained assisted visible light remote sensing image aircraft target detection method according to claim 2, characterized in that, The RarePlanes dataset is an open-source aircraft target detection dataset. Each aircraft sample in the dataset uses a special diamond-shaped annotation, and each aircraft sample is labeled separately according to a predetermined classification method.

4. The fine-grained assisted visible light remote sensing image aircraft target detection method according to claim 3, characterized in that, In steps 4 and 5, the feature extractor G(·) is selected as ResNet-50, and the initial values ​​of the parameters are obtained from the ResNet-50 pre-trained model provided in PyTorch.

5. A fine-grained assisted visible light remote sensing image aircraft target detection device, characterized in that, include: The first main module is used to implement step 1, which involves constructing a GF-RarePlanes joint dataset and unifying the annotation format of the GF dataset and the RarePlanes dataset, that is, making each aircraft sample in the RarePlanes dataset have the same rotated rectangle annotation as in the GF dataset; step 2, which involves reclassifying the GF dataset and the RarePlanes dataset, that is, selecting 5 attributes from the eleven fine-grained attributes of aircraft targets in the RarePlanes dataset, merging the sub-attributes under the 5 attributes, and combining the attributes to classify the aircraft samples in the GF dataset and the RarePlanes dataset into the corresponding attribute combination. The second main module is used to implement step 3, generating the joint dataset annotation files. It unifies the annotation files of the merged GF dataset and RarePlanes dataset into .json format, extracting the annotation content from both datasets, with one .txt file corresponding to one image; then, it converts the .txt files to .xml files, converting the four vertices of the rotated bounding box annotations into slanted text bounding boxes; finally, it converts the .xml files to .json files, thus completing the construction of the GF-RarePlanes joint dataset. The third main module is used to implement step 4, performing pre-training on the GF-RarePlanes joint dataset. Step 4: The feature extractor is G(·), and a pre-trained model Mp is obtained. Step 5: The pre-trained model Mp obtained in Step 4 is fine-tuned on the GF dataset, where the feature extractor is G(·), and a fine-tuned model Mf is obtained. Step 6: The model Mf obtained in Step 5 is tested on the GF dataset to obtain the test results, namely the average accuracy of each category and the mean average accuracy of all categories. Step 7: In order to verify the generalization ability of the pre-trained model obtained in Step 4 and the effectiveness of the fine-grained attribute-assisted aircraft target detection method, the model is verified on the FAIR1M dataset to obtain the average accuracy of each category and the mean average accuracy of all categories.

6. An electronic device, characterized in that, include: At least one processor, at least one memory, and a communication interface; wherein, The processor, memory, and communication interface communicate with each other; The memory stores program instructions that can be executed by the processor, which invokes the program instructions to perform the method described in any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer instructions that cause the computer to perform the method described in any one of claims 1 to 4.

Images