Aircraft model identification method for remote sensing image

Abstract

The application provides a kind of airplane model identification method for remote sensing image, comprising: constructing aircraft key structure detector;The image of the aircraft to be identified is input into the aircraft key structure detector, and the outline data thereof is output;Acquire prior key structure information of aircraft, and calculate the structure characteristic value based on the outline data according to the prior key structure information of aircraft and the key structure information of the aircraft to be identified extracted in step 2, and arrange to obtain the key structure characteristic prior value set A of aircraft model i And the key structure characteristic value set A of the aircraft to be identified extracted;From set A i Screen out the key structure characteristic prior value set A i Of model that meets discrete value matching;Matching is carried out, and the prior aircraft model with the highest matching degree is taken as the final recognition result.The application can still maintain high available recognition accuracy when the sample amount of aircraft model is small, and avoids the problem that the existing deep learning method cannot accurately identify due to limited sample amount.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of automatic target recognition of remote sensing images, and particularly relates to a method for identifying the model of an airplane in a remote sensing image. BACKGROUND

[0002] Thanks to the development of aerial photogrammetry and remote sensing technology, people can obtain information of a target of interest through satellites and aerial images, and target detection and recognition technology plays an important role in this process and is developing rapidly. At present, the mainstream target recognition method based on deep learning requires a large amount of data for the recognition of small targets such as airplanes in images, and the sample size for a specific model of an airplane often cannot meet the requirements of such a method. A large number of experiments show that although such a deep learning method has high accuracy in identifying the type of an airplane, it is subject to the sample size and cannot guarantee the effect in identifying a specific model of an airplane. An airplane has fewer features in a remote sensing image, and when the number of layers of a deep convolutional neural network is large, the detailed information is easily lost, which affects the recognition accuracy. SUMMARY

[0003] The present application aims at the deficiencies of the prior art and provides a method for identifying the model of an airplane in a remote sensing image. The method starts from the detection of key structures, trains a deep learning model to extract specific components of an airplane target, and realizes the identification of the model of an airplane by comparing the similarity between the component structures. The method can maintain high accuracy when the sample size is small and has good versatility.

[0004] To solve the above technical problems, the present application adopts the following technical scheme:

[0005] A method for identifying the model of an airplane in a remote sensing image comprises the following steps:

[0006] Step 1: constructing an airplane key structure detector based on a deep convolutional neural network;

[0007] Step 2: inputting an airplane image to be identified into the airplane key structure detector constructed in step 1, outputting the key structure point coordinates of the airplane image, and connecting the key structure points in sequence to obtain the contour data of the key structure of the airplane to be identified;

[0008] Step 3: obtaining prior key structure information of an airplane, and calculating the structure characteristic values based on the contour data according to the prior key structure information of the airplane and the key structure information of the airplane to be identified extracted in step 2, and arranging to obtain a key structure characteristic prior value set A of a prior airplane model i and the key structure characteristic value set A of the airplane to be identified extracted;

[0009] Step 4: screening a key structure characteristic prior value set Ai' of a model that matches the discrete value from the set Ai;

[0010] Step 5, match the to-be-identified aircraft with the prior aircraft models in the set Ai', and take the prior aircraft model with the highest matching degree as the final identification result.

[0011] Further, in step 1, a lightweight deep convolutional neural network MobileNetV2 is used as the basis to construct an aircraft key structure detector, and the input of the aircraft key structure detector is an image, and the output is the horizontal and vertical coordinates of all key structure points.

[0012] Further, the aircraft key structure includes but is not limited to fuselage, wings, engines, and tail.

[0013] Further, the method for obtaining aircraft prior key structure information in step 3 is:

[0014] Using domain prior knowledge to divide the key structures of each type of aircraft, each type of aircraft is disassembled into structures including but not limited to fuselage, wings, engines, and tail, and an operation f n of the characteristic values of the structures and the characteristic values of the structures themselves is designed n1n2 to obtain the aircraft prior key structure information.

[0015] Further, in step 3, the structure characteristic values include but are not limited to area ratio, angle ratio, and length ratio representing relative relationships.

[0016] Further, in step 4, the method for screening the prior value set Ai' is:

[0017] Set the aircraft key structure characteristic value difference range set K, judge the difference between the prior aircraft model j and the to-be-identified aircraft on various characteristic values and K, if more than half of the characteristic values are in the set K, and some important discrete characteristic values are completely matched, then the model j is included in the candidate set, and the model key structure characteristic prior value set A i ′ that meets the discrete value matching is screened out from the set A i obtained in step 3.

[0018] Further, some important discrete characteristic values include engine quantity and engine arrangement matching.

[0019] Further, in step 4, for the case where the projection coordinate system to which the to-be-identified aircraft image belongs is known, the recognition accuracy is further improved by additionally designing an aircraft length and width tolerance T, that is, the model key structure characteristic prior value set A i ′ that meets the fuselage length and wingspan tolerance range is further screened out from the set A i ′, and the mathematical description is A i ′={a i |Abs(a il-a l ) <T&Abs(a iw -a w ) <T},a il a iw These respectively represent the fuselage length and wingspan of the prior model, a l a w These correspond to the fuselage length and wingspan of the aircraft to be identified, respectively, and Abs(·) represents the absolute value operation.

[0020] Furthermore, the method for matching the shape of the aircraft to be identified with the shape of the previous N prior aircraft models in step 5 is as follows:

[0021] Establish a structural template library for all common aircraft models, based on the key aircraft structure x obtained in step 2. n Location information, including each key structure of the aircraft to be identified. n A simplified aircraft model template is obtained by cropping from the input image, performing image preprocessing, correction, and binarization, and then combining this simplified aircraft model template with set A. i The structural templates of the prior model are matched, and the shape similarity between the two is calculated.

[0022] Furthermore, the shape similarity between the simplified aircraft model template and the structural template of the prior model is calculated using a correlation factor algorithm, and the formula is as follows:

[0023]

[0024]

[0025]

[0026] Among them, T x′y′ and T x″y″ These respectively represent the grayscale value of each pixel in the prior aircraft model template and the grayscale values ​​of all other pixels. x′y′ and I x″y″ These represent the grayscale value of each pixel in the preprocessed image of the aircraft model to be identified, and the grayscale values ​​of all other pixels, respectively. h and w represent the template size.

[0027] Compared with the prior art, the beneficial effects of the present application are that the automatic aircraft model identification method for remote sensing images proposed in the present application extracts aircraft component structures through a deep learning model, can find subtle differences between similar models, replaces the overall matching of traditional methods with structure similar matching, can still accurately match when part of the structure is blocked, and has strong versatility; at the same time, the advantages of aircraft field knowledge are fully utilized, and high usable identification accuracy can be maintained when the aircraft model sample size is small, and the problem that existing deep learning methods cannot accurately identify due to limited sample size is avoided. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 The flowchart of the aircraft model identification method for remote sensing images in the embodiment of the present application. DETAILED DESCRIPTION

[0029] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0030] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0031] The present application will be further described below with reference to specific embodiments, but is not limited to the present application.

[0032] As shown in the drawings, the embodiment of the present application provides an aircraft model identification method for remote sensing images, comprising the following steps: Figure 1

[0033] Step 1, constructing a key structure detector

[0034] In this embodiment, a lightweight deep convolutional neural network MobileNetV2 is used as the basis to construct an aircraft key structure detector, and the network structure of the aircraft key detector is shown in Table 1 as follows:

[0035] Table 1 is the network structure of the aircraft key detector

[0036]

[0037]

[0038] ​One line in Table 1 represents one layer of network repeated n times, c output channels, the step length s when the network is executed for the first time, and the expansion t times when the network is input. The input of the aircraft key detection network is an image, and the final output is the horizontal and vertical coordinates of all key structural points.

[0039] Step 2, extraction of key structures

[0040] The image to be recognized is input into the aircraft key structure detector constructed in step 1, and the coordinates of the key structural points on the aircraft image are output. The points constituting the corresponding aircraft structure are sequentially concatenated into an aircraft structure polygon to obtain the outline data of the aircraft key structure. The above process of obtaining these outline data is the extraction of key structures, which is mathematically described as {x0, x1, …, x2, x n} = f (I), where f is the key structure detection and outline concatenation operation, x n is the outline data of the nth key structure of the aircraft obtained by the operation f.

[0041] Step 3, key structure feature modeling

[0042] The key structures of each type of aircraft are divided using prior knowledge in the field, and each type of aircraft is disassembled into fuselage, wings, engines, tail, etc. The operations f n and f n1n2 of the characteristic values of the key structures and the characteristic values between structures are designed. n Examples are given for reference:

[0043]

[0044]

[0045]

[0046] In the formula, A w and A f represent the area of the single wing and the area of the fuselage, respectively, A e1 and A e2 represent the area of the first engine and the area of the second engine near the fuselage side of the wing, respectively, and the ratio on the left side of the inequality represents the operation of the characteristic value, f n belongs to the structure itself of the engine, f n1n2 belongs to the operation of the characteristic value between the fuselage and the wing, and the engine and the wing structure. T wf represents the wing-body area ratio threshold, T ew represents the engine-wing area ratio threshold, and T eeThese three thresholds represent the area ratio between engines. These thresholds need to be determined manually based on the specific type of aircraft being studied. All these areas can be obtained from the contour data extracted in step 2 using a vector product algorithm. The above example can be used as a reference when calculating other feature values ​​in custom design.

[0047] Based on the aforementioned prior key structural information of the aircraft and the key structural information of the aircraft to be identified extracted in step 2, structural feature values ​​representing relative relationships, such as area ratio, angle ratio, and length ratio, are calculated on the basis of the contour data. This results in a set A of prior values ​​for key structural features of common aircraft models. i The extracted set of key structural feature values ​​A of the aircraft to be identified is mathematically described as A i ={f n (x in ),f n1n2 (x in1 ,x in2 )},A={f n (x n ),f n1n2 (x n1 ,x n2 )}, where x in This represents the nth key structure of the i-th type of aircraft, defined by prior knowledge of the domain. n1 and n2 are two specific key structures for which the characteristic values ​​between structures need to be calculated. Common characteristic values ​​between structures include the position of the engine on the wing and the ratio of the fuselage width to the wingspan.

[0048] Step 4: Structural similarity calculation

[0049] Define a set K of key structural feature value differences for the aircraft. Determine the relationship between the differences in various feature values ​​between the prior aircraft model j and the aircraft to be identified and K. If more than half of the feature value differences are in set K, and they completely match on some important discrete feature values ​​(e.g., engine quantity and engine arrangement), then add model j to the candidate set. Similarly, from set A obtained in step 3... i Select the set of prior values ​​A of key structural features of the model that meet the discrete value matching criteria. i ′.

[0050] When the projected coordinate system of the aircraft image to be identified is known, additionally designing an aircraft length tolerance error T can help further improve the identification accuracy, in set A. i Further screening yielded a set A of prior values ​​for key structural features of the model that fell within the tolerance range of fuselage length and wingspan. i ′, mathematically described as A i ′={a i |Abs(a il -a l ) <T&Abs(a iw -aw il iw respectively correspond to the body length and wingspan of the prior model, a l w respectively correspond to the body length and wingspan of the to-be-identified aircraft, and Abs(·) represents an absolute value operation.

[0051] Step 5, identification based on similarity

[0052] On the basis of the set A i ' that meets the discrete value matching (if the projection coordinate system information is known, the prior value of the key structural feature also needs to meet the body length and wingspan tolerance), further structural similarity calculation is performed on the continuous feature value (i.e. the structural outer contour). Before the similarity calculation, a structural template library of all common aircraft models also needs to be established. According to the position information of the key structure x n extracted in step 2, each corresponding structure x n of the to-be-identified aircraft is cut from the input image, and image preprocessing, correction and binarization are performed to obtain a simple aircraft model template. Then, the simple aircraft model template is matched with the structural template of the prior aircraft model in the set A i ', and the shape similarity of the two is calculated by using a correlation factor algorithm, and the calculation formula is as follows:

[0053]

[0054]

[0055]

[0056] wherein T x′y′ and T x″y″ represent the gray value of each pixel in the prior aircraft model template and the gray value of all pixels except the pixel, I x′y′ and I x″y″ represent the gray value of each pixel in the preprocessed to-be-identified aircraft model image and the gray value of all pixels except the pixel, and h and w represent the template size.

[0057] Finally, the prior aircraft model corresponding to the maximum similarity value in the calculated similarity is selected as the aircraft model with the highest matching degree, and the identification result is output.

[0058] The above is only the preferred embodiment of the present application, and does not limit the implementation and protection scope of the present application. For those skilled in the art, it should be realized that any equivalent replacement and obvious changes made by using the content of the present application should be included in the protection scope of the present application.​​​

Claims

1. A method for aircraft model identification based on remote sensing imagery, characterized in that, Includes the following steps: Step 1: Construct a key aircraft structure detector based on a deep convolutional neural network; Step 2: Input the aircraft image to be identified into the aircraft key structure detector constructed in Step 1, output the coordinates of the key structure points of the aircraft image, and concatenate the key structure points in sequence to obtain the contour data of the key structure of the aircraft to be identified. Step 3: Obtain the prior key structural information of the aircraft. Based on the prior key structural information of the aircraft and the key structural information of the aircraft to be identified extracted in Step 2, calculate its structural feature values ​​on the basis of the contour data, and organize them to obtain the prior value set A of key structural features of the prior aircraft model. i The extracted set of key structural feature values ​​of the aircraft to be identified, A; Step 4, from set A i Select the set of prior values ​​A of key structural features of the model that meet the discrete value matching criteria. i ′; Step 5: Match the aircraft to be identified with set A i The prior aircraft model in the '' is matched, and the prior aircraft model with the highest matching degree is taken as the final recognition result; In step 4, the prior value set A is selected. i The method is as follows: Define a set K of key structural feature value differences for the aircraft. Determine the relationship between the differences between the prior aircraft model j and the aircraft to be identified in various feature values ​​and K. If more than half of the feature value differences are in set K, and they completely match in some important discrete feature values, then model j is added to the candidate set. This process is repeated for each subsequent step, starting with set A obtained in step 3. i Select the set of prior values ​​A of key structural features of the model that meet the discrete value matching criteria. i ′; In step 4, when the projected coordinate system of the aircraft image to be identified is known, the recognition accuracy is further improved by designing an additional allowable error T for the aircraft length tolerance. That is, in set A... i Further screening yielded a set A of prior values ​​for key structural features of the model that fell within the tolerance range of fuselage length and wingspan. i ′, mathematically described as A i ′={a i |Abs(a il -a l ) <T & Abs(a iw -a w ) <T},a il a iw These respectively represent the fuselage length and wingspan of the prior model, a l a w These correspond to the fuselage length and wingspan of the aircraft to be identified, respectively, and Abs(·) represents the absolute value operation.

2. The aircraft model identification method for remote sensing imagery according to claim 1, characterized in that, In step 1, a key aircraft structure detector is constructed based on the lightweight deep convolutional neural network MobileNetV2. The input of the key aircraft structure detector is an image, and the output is the horizontal and vertical coordinates of all key structure points.

3. The aircraft type identification method for remote sensing imagery according to claim 1, characterized in that, Key structures of an aircraft include, but are not limited to, the fuselage, wings, engines, and tail.

4. The aircraft model identification method for remote sensing imagery according to claim 1, characterized in that, Step 3 involves obtaining the prior critical structural information of the aircraft as follows: Using prior knowledge of the domain, the key structures of each aircraft model are identified, and each aircraft model is disassembled into structures including but not limited to the fuselage, wings, engines, and tail. The calculation of the key structures and their characteristic values ​​f is then designed. n Operations on eigenvalues ​​between structures f n1n2 This allows us to obtain prior key structural information about the aircraft.

5. The aircraft type identification method for remote sensing imagery according to claim 1, characterized in that, In step 3, structural feature values ​​include, but are not limited to, area ratios, angle ratios, and length ratios that represent relative relationships.

6. The aircraft model identification method for remote sensing imagery according to claim 1, characterized in that, Some important discrete features include the number of engines and the matching of engine layout.

7. The aircraft model identification method for remote sensing imagery according to claim 1, characterized in that, The method for matching the shape of the aircraft to be identified with the shape of the previous N prior aircraft models in step 5 is as follows: Establish a structural template library for all common aircraft models, based on the key aircraft structure x obtained in step 2. n Location information, including each key structure of the aircraft to be identified. n A simplified aircraft model template is obtained by cropping from the input image, performing image preprocessing, correction, and binarization, and then combining this simplified aircraft model template with set A. i The structural templates of the prior model are matched, and the shape similarity between the two is calculated.

8. The aircraft model identification method for remote sensing imagery according to claim 7, characterized in that, The shape similarity between the simplified aircraft model template and the structural template of the prior model is calculated using a correlation factor algorithm, and the formula is as follows: Among them, T x′y′ and T x″y″ These respectively represent the grayscale value of each pixel in the prior aircraft model template and the grayscale values ​​of all other pixels. x′y′ and I x″y″ These represent the grayscale value of each pixel in the preprocessed image of the aircraft model to be identified, and the grayscale values ​​of all other pixels, respectively. h and w represent the template size.

Images