Armored target and key part detection method based on deep learning and knowledge reasoning

Abstract

The application discloses an armored target and key part detection method based on deep learning and knowledge reasoning, and comprises the following steps: constructing a missile-borne image dataset; constructing a deep learning detection model and a key part auxiliary detection unit; wherein the deep learning detection model is used for detecting armored targets and key parts in the missile-borne image and outputting corresponding target detection frames; when the deep learning detection model cannot generate the target detection frame of the key part under a preset condition, the key part auxiliary detection unit is called to output the detection frame coordinates of the key part; the deep learning detection model is trained and tested by using the image dataset, and the trained deep learning detection model is saved; the missile-borne image of the armored target to be detected is acquired, the trained deep learning detection model and the key part auxiliary detection unit are used for detection, and the detection results of the armored target and the key part are outputted.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of smart processing of missile-borne images, and particularly relates to a method for detecting armored targets and key parts (key and important parts) of missile-borne images based on deep learning and knowledge reasoning. BACKGROUND

[0002] In the field of precision guidance, target detection of missile-borne images is a core key technology of information processing of optical imaging guided weapons. The technology comprehensively utilizes target and background characteristics, image information of missile-borne sensors and related prior knowledge / data, and completes the detection and identification of threat targets through target detection image processing algorithms, so as to provide important target information for the implementation of precision strike of weapon systems. Taking the anti-armor precision guidance application as an example, if the armored targets and their key parts on the battlefield can be accurately detected, it means "strike as soon as found", which is directly related to the precision strike effectiveness of anti-armor weapon systems in actual combat. Therefore, it is necessary to develop target detection algorithms of missile-borne images, and optimize and improve the application performance of the algorithms, so as to meet the development and field requirements of precision guidance technology.

[0003] The key of target detection algorithm lies in feature extraction. Traditional target detection algorithms such as feature matching method and template matching method mainly rely on manual design to extract pre-defined features. This way makes the algorithm itself have strong limitations, and it is difficult to adapt to different complex scenes. In recent years, intelligent detection algorithms based on deep learning models have become the mainstream research direction in the field of target detection. Its outstanding feature is to use deep neural networks to automatically / autonomously learn from different levels and scales of a large number of data samples to form efficient and adaptive feature representation capabilities, effectively eliminating the subjectivity and tediousness brought by manual design, making the target detection algorithm processing autonomous and intelligent, so as to more stably cope with different targets and complex scenes. For the task of target detection of missile-borne images, some classic and general deep learning model architectures and intelligent algorithms such as YOLO series and RCNN series have been developed and utilized. From the actual application point of view, the high dynamic scale, view angle and complex scene change characteristics of missile-borne images, as well as the fine recognition ability requirements for target key parts and background interference, information real-time processing, limited computing resources and other design constraints, all of these put forward higher requirements and new challenges to the intelligent target detection algorithm. SUMMARY

[0004] The purpose of the present application is to provide a method for detecting armored targets and key parts based on deep learning and knowledge reasoning. By combining the data-driven deep learning target detection model with the knowledge reasoning based on the imaging geometric relationship of armored targets and key parts, the fine detection of armored targets and key parts of missile-borne images is realized, thereby providing technical support for precision guidance information processing.

[0005] In order to achieve the above-mentioned tasks, the present application adopts the following technical solutions:

[0006] A method for detecting armored targets and critical parts based on deep learning and knowledge reasoning, comprising:

[0007] Obtain missile-borne images for armored targets through target scene simulation or flight test, label armored targets and critical parts in the missile-borne images to construct an image dataset;

[0008] Construct a deep learning detection model and a critical part auxiliary detection unit; wherein the deep learning detection model is used to detect armored targets and critical parts in the missile-borne images and output corresponding target detection boxes; when the deep learning detection model only detects target detection boxes for armored targets under a preset condition and cannot generate target detection boxes for critical parts, the critical part auxiliary detection unit is called to output detection box coordinates of critical parts through the unit;

[0009] Train and test the deep learning detection model using the image dataset, and save the trained deep learning detection model;

[0010] Obtain missile-borne images of armored targets to be detected, and use the trained deep learning detection model and critical part auxiliary detection unit for detection to output detection results of armored targets and critical parts.

[0011] Further, in the target scene simulation and flight test, when the missile-borne platform captures a ground armored target, the armored target side image sequence should be collected in a far-to-near manner;

[0012] Preprocess, clean and label armored targets and critical parts for the obtained missile-borne images to form a dataset with label information; the critical part refers to the key and important part on the armored target, which is set and labeled according to the detection requirements.

[0013] Further, divide the dataset into a training set and a test set according to a ratio of 3:1, and ensure that the missile-borne images of the two datasets come from different test or simulation scenes; first train the model using the training set, and test it using the test set to meet the requirements, and then save the trained model.

[0014] Further, the deep learning detection model is:

[0015] Replace the backbone network ResNet in the existing Faster RCNN with Res2Net with a multi-branch structure, and use the following distance to measure the position relationship of the target bounding box:

[0016]

[0017] wherein N a and N b are Gaussian distribution models describing two candidate target detection boxes A = [x a , y a , h a , w a ] and B = [x b , y b , h b , w b ], W(N a , N b ) and D(N a , N b ) represent the distance between the two Gaussian distribution models and its exponential normalized form, c is a control parameter less than 1, x a , y a represent the center point coordinates of the candidate target detection box A, w a , h a represent the width and height of the candidate target detection box A; ||·||2 represents the l2 norm;

[0018] In the detection process, when the deep learning detection model generates multiple confidence candidate target detection boxes for the armored target, for all candidate target detection boxes of each type, the distance D(N a , N b ) between each two candidate target detection boxes A and B is calculated, then all the calculated distances are sorted, the two candidate target detection boxes with the largest distance are selected, and the one with higher confidence is taken as the final target detection box of the type, so that the target detection box for the armored target and the critical part can be obtained.

[0019] Further, when the deep learning detection model generates the target detection box for the armored target but does not generate the target detection box for the critical part, first, the target detection box of the armored target is judged, and when the pixel points in the target detection box are not less than a preset number of pixel points, the critical part auxiliary detection unit is used to detect the critical part.

[0020] Further, the preset number of pixel points is 400.

[0021] Further, the modeling process of the critical part auxiliary detection unit is as follows:

[0022] Suppose the physical size of the armored target is l target × w target × h target , the height of the critical part is ρh target , and ρ is a height control parameter, 0 < ρ < 1; considering that the missile-borne platform approaches the armored target from far to near at a lateral angle, the size of the target detection box of the armored target is hbox xw box , the center coordinates of the target frame detection frame are (x box , y box ); according to the optical imaging geometric relationship, w box is relatively fixed and is proportional to l target , and h box can be regarded as the sum of the object plane projection of h target and w taxget ; assuming that the line-of-sight angle of the platform is θ, the ratio of w target to h target is b, then:

[0023] h box = κ (a w target sin θ + h target cos θ)

[0024] In the formula, k and a are proportional coefficients, k can be calculated by w box / l target ;

[0025] Therefore, the image center coordinates of the key part are

[0026] The target detection frame of the key part is generated based on the image center coordinates of the key part and is output.

[0027] Further, for different key parts, different values of ρ can be taken to locate the key parts at different positions on the armored target.

[0028] A terminal device comprises a processor, a memory and a computer program stored in the memory; when the processor executes the computer program, the armored target and key part detection method based on deep learning and knowledge reasoning is realized.

[0029] A computer readable storage medium, the medium stores a computer program; when the computer program is executed by the processor, the armored target and key part detection method based on deep learning and knowledge reasoning is realized.

[0030] Compared with the prior art, the present application has the following technical features:

[0031] 1. The present application is based on an improved Faster RCNN deep learning detection model, which realizes visual detection and recognition of armored targets and key parts; Res2Net backbone network and normalized distance measurement are adopted, which effectively enhances the detection ability of small targets at a long distance.

[0032] 2. The application constructs a target critical part auxiliary detection unit based on knowledge reasoning according to the optical imaging geometry of the target-part, further strengthening the detection reasoning ability of the model for the target critical part. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 It is a schematic diagram of the overall process of the method of the application. DETAILED DESCRIPTION

[0034] Referring to the drawings, the application provides an armored target and critical part detection method based on deep learning and knowledge reasoning, comprising:

[0035] Obtain the missile-borne image for the armored target through target scene simulation or flight test, label the armored target and critical part in the missile-borne image to construct an image data set;

[0036] Construct a deep learning detection model and a critical part auxiliary detection unit; wherein the deep learning detection model is used to detect the armored target and critical part in the missile-borne image and output the corresponding target detection frame; when the deep learning detection model only detects the detection frame for the armored target under the preset condition and cannot generate the target detection frame of the critical part, the critical part auxiliary detection unit is called to output the detection frame coordinates of the critical part to assist the deep learning detection model in reasoning detection when the critical part is not detected;

[0037] Train and test the deep learning detection model and the critical part auxiliary detection unit using the image data set, save the trained deep learning detection model and critical part auxiliary detection unit;

[0038] Obtain the missile-borne image of the armored target to be detected, use the trained deep learning detection model and critical part auxiliary detection unit for detection, and output the detection result of the armored target and critical part, i.e. the target detection frame and center coordinates.

[0039] The specific implementation process of the application is further described below.

[0040] 1. Establishment of image data set

[0041] In this scheme, the missile-borne image is obtained through target scene simulation, flight test, etc.; wherein in the target scene simulation and flight test, when the missile-borne platform captures the ground armored target, the armored target side image sequence should be collected in a far-to-near manner; that is, the scene collection is that the missile-borne platform is above the side of the armored target and approaches the armored target from far to near, and in order to make the features of the armored target in the image more abundant, the side image of the armored target should be collected.

[0042] The acquired missile-borne image is preprocessed, cleaned, and armored targets and critical parts are labeled to form a data set with label information; in the present scheme, the critical parts refer to key and important parts, which are set and labeled according to detection requirements.

[0043] To evaluate the detection effect of the model, the data set is divided into a training set and a test set according to a ratio of 3:1 in the present scheme, and it is ensured that the missile-borne images of the two data sets come from different test or simulation scenes; the model is trained using the training set, and the test set is used for testing to meet the requirements, and the trained model is saved.

[0044] 2. Deep learning detection model

[0045] In the present scheme, an improved Faster RCNN is used as a deep learning detection model, specifically:

[0046] On the basis of the existing Faster RCNN, the backbone network ResNet in the present scheme is replaced by Res2Net with a multi-branch structure to enhance the network's expression and extraction ability for multi-scale features and small targets; at the same time, since the original IOU measurement method of the model has the defect of instability in the small target detection process, the following distance is used to measure the position relationship of the target bounding box in the present scheme to improve the measurement accuracy and robustness of small target detection. The specific formula is as follows:

[0047]

[0048] In the formula, N a and N b are Gaussian distribution models describing two candidate target detection boxes A = [x a , y a , h a , w a ] and B = [x b , y b , h b , w b ], W(N a , N b ) and D(N a , N b ) represent the distance and exponential normalized form of the two Gaussian distribution models, c is a control parameter less than 1, x a , y a represent the center point coordinates of the candidate target detection box A, w a , h a represent the width and height of the candidate target detection box A; ||·||2 represents the l2 norm;

[0049] In the detection process, after the deep learning detection model generates multiple confidence candidate target detection boxes for the armored target, for all candidate target detection boxes of each type (in this embodiment, armored target, critical part 1, critical part 2, etc.), the distance D(N a , N b ) between each two candidate target detection boxes A and B is calculated, and then the distances are sorted, and the two candidate target detection boxes with the largest distance are selected, and the one with higher confidence is taken as the final target detection box of the type.

[0050] Through the above method, the target detection box for the armored target and the critical part can be obtained.

[0051] Since the armored target is relatively large as a whole, and the critical part is a certain part of the armored target, which is relatively small in size in the image; in actual application, since image acquisition for the armored target is continuous, when the distance from the armored target is far, the critical part cannot generate a corresponding target detection box due to too few pixel information corresponding to the target detection box of the armored target; as the missile-borne platform continuously approaches the armored target and continuously acquires images, when the target detection box for the armored target is not less than 400 pixel points, if the deep learning detection model cannot detect the critical part, the critical part auxiliary detection unit is used to assist the deep learning detection model to perform inference detection when the critical part is not detected.

[0052] 3. Critical part auxiliary detection unit

[0053] When the deep learning detection model generates a target detection box for the armored target but does not generate a target detection box for the critical part, first, the target detection box of the armored target is judged, and under the preset condition in the present scheme, it means that when the pixel points in the target detection box are not less than the preset pixel value 400 (i.e., the size of the detection box is not less than 20*20), according to the optical imaging geometric relationship of the target-part and the target frame length-width variation law, an armored target critical part image coordinate calculation model based on imaging geometry under a lateral viewing angle is constructed, and the critical part image coordinates are calculated from the detected target information. The specific modeling and calculation process is as follows:

[0054] Let the physical size of the armored target be l target ×w target ×h target (length x width x height), and the height of the critical part is ρh target , ρ is a height control parameter, 0<ρ<1; considering that the missile-borne platform approaches the armored target from far to near at a lateral viewing angle, the size of the target detection box of the armored target is h box ×w box (longitudinal height x lateral width), and the center coordinates of the target detection box are (xbox , y box ); then according to the optical imaging geometry, w box is proportional to l target , and h box can be regarded as the sum of the object plane projection of h target and w target ; let the line-of-sight angle of the platform be θ, and the ratio of w target to h target be b, then:

[0055] h box = κ (a w target sin θ + h target cos θ)

[0056] where k and a are proportional coefficients, k can be calculated by w box / l target , and a can be simplified to 1;

[0057] Therefore, the image center coordinates of the key joint are It should be noted that when θ is greater than a certain value θ th , the target key joint will disappear in the field of view.

[0058] The target detection frame of the key joint is generated based on the image center coordinates of the key joint and output.

[0059] In actual application, different values of p can be used to locate the key joint at different positions on the armored target for different key joints.

[0060] Embodiment:

[0061] (1) Experimental conditions and related settings

[0062] In this embodiment, 1670 images of armored targets and key joint datasets are constructed, of which 1206 images are used as training samples and 464 images are used as test samples. The training parameters are set as follows: the learning rate is 0.005, the optimizer is ADAM, the training step number is 2000, the training round number is 24, the batch training size is 4, and the classification model parameters of the backbone network Res2Net pre-trained on the imagenet dataset are used as the initialization parameters. The benchmark methods are YOLOv5 and FasterRCNN target detection models.

[0063] The processor used for simulation is Xeon(R)CPU E5-2630 v4@2.20GHzx40, memory is 64.00GB, GPU is 8G of GeForce RTX Titan, simulation platform is Ubuntu16.04 operating system, uses Pytorch deep learning framework, and is implemented by using Python language.

[0064] (2) Experimental results and comparative analysis

[0065] The average accuracy is used as the evaluation index of target detection, the simulation test is carried out on the armored target and the important part data set constructed by using the application and the two kinds of benchmark methods, and the evaluation index value results are compared and analyzed, as shown in Table 1.

[0066] Table 1: Comparison of average accuracy index results

[0067]

[0068] As can be seen from Table 1, the detection accuracy of the armored target and the important part of the application is obviously higher than that of the two kinds of benchmark methods, which shows that the application has relatively superior detection performance.

[0069] The above examples are only used to illustrate the technical solutions of the application, but not to limit them; although the application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that the technical solutions recorded in the foregoing examples can still be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the application, and should be included in the protection scope of the application.

Claims

1. An armored target and key point detection method based on deep learning and knowledge reasoning, characterized in that, The application relates to a method for detecting an armored target and a key part based on deep learning and knowledge reasoning. The method comprises the following steps: An image of an armored target is acquired through target scene simulation or flight test, and the armored target and the key part in the image are labeled to construct an image dataset; A deep learning detection model and a key part auxiliary detection unit are constructed; the deep learning detection model is used to detect the armored target and the key part in the image and output a corresponding target detection frame; when the deep learning detection model only detects a target detection frame of the armored target and cannot generate a target detection frame of the key part under a preset condition, the key part auxiliary detection unit is called to output the detection frame coordinates of the key part; The deep learning detection model and the key part auxiliary detection unit are trained and tested by using the image dataset, and the trained deep learning detection model and the key part auxiliary detection unit are saved; An image of an armored target to be detected is acquired, and the trained deep learning detection model and the key part auxiliary detection unit are used for detection to output the detection result of the armored target and the key part; In the target scene simulation and flight test, when a ground armored target is captured by a missile-borne platform, the armored target side image sequence should be collected in a far-to-near manner; The acquired missile-borne image is preprocessed and cleaned, and the armored target and the key part are labeled to form an image dataset with label information; the key part refers to a key and important part on the armored target, which is set and labeled according to detection requirements; When the deep learning detection model generates a target detection frame of the armored target but does not generate a target detection frame of the key part, the target detection frame of the armored target is first judged; when the pixel points in the target detection frame are not less than a preset pixel point number, the key part auxiliary detection unit is used for key part detection; The physical size of the armored target is The height of the important part of the armored target is , is a high-low control parameter, When the projectile-borne platform approaches the armored target from far to near at a lateral angle, the size of the target detection frame of the armored target is The center coordinates of the target frame detection frame are According to the optical imaging geometric relationship, is proportional to , and can be regarded as the sum of the object plane projections of and ; let the line-of-sight angle of the projectile-borne platform be , , and the ratio of is , then: wherein and is a proportionality factor, may be calculated from ; Thus, the image center coordinates of the critical site are ; The modeling process of the key part auxiliary detection unit is as follows: 2.The method of claim 1, wherein the method comprises: The image center coordinates of the key part are used to generate a target detection frame of the key part and output the target detection frame. 3.The method of claim 1, wherein the method further comprises: determining a target area of interest (AOI) based on the target armor target and the target AOI; and determining a target AOI type based on the target AOI. The image dataset is divided into a training set and a test set according to a ratio of 3:1, and it is ensured that the missile-borne images in the training set and the test set come from different test or simulation scenes; the deep learning detection model is trained by using the training set, and the test set is used for testing to meet the requirements, and the trained deep learning detection model is saved. The deep learning detection model is as follows: wherein, and are Gaussian distribution models describing two candidate target detection boxes and , and denote the distance between the two Gaussian distribution models and its exponential normalized form, is a control parameter less than 1, denote the center point coordinates of the candidate target detection boxes A , denote the width and height of the candidate target detection boxes A ; denote norms; In the detection process, after the deep learning detection model generates multiple confidence candidate target detection boxes for the armored target, for all candidate target detection boxes of each type, the distance between each two candidate target detection boxes is calculated A , B , then all the calculated distances are sorted, the two candidate target detection boxes with the largest distance are selected, and the one with higher confidence is taken as the final target detection box of the type, so that the target detection box for the armored target and the key part can be obtained.​ 4.The method of claim 1, wherein the method further comprises: determining a target area of interest (AOI) based on the target armor target and the target AOI; and determining a target AOI type based on the target AOI. On the basis of an existing Faster RCNN, a backbone network ResNet is replaced by a Res2Net with a multi-branch structure, and the following distance is used to measure the position relationship of the target boundary frame: 5.The method of claim 1, wherein the method further comprises: determining a target area of interest (AOI) based on the target armor target and the target AOI; and determining a target AOI type based on the target AOI. For different key parts, by the different values of high and low control parameters , to locate the different positions of the key parts on the armored target. 6.A terminal device, comprising a processor, a memory, and a computer program stored in the memory; characterized in that, The preset pixel point number is 400.

7. A computer readable storage medium having stored therein a computer program; characterized in that, When a computer program is executed by a processor, the method for detecting an armored target and a key part based on deep learning and knowledge reasoning according to any one of claims 1-5 is realized. When a computer program is executed by a processor, the method for detecting an armored target and a key part based on deep learning and knowledge reasoning according to any one of claims 1-5 is realized.

Images