Ground battlefield threat target intelligent sensing method based on deep multi-task network

Abstract

According to the ground battlefield threat target intelligent sensing method based on the deep multi-task network, multiple attributes of multiple threat targets are obtained through a passive image processing mode, and end-to-end multi-target detection and distance prediction are achieved. The method comprises the following steps: constructing a ground target comprehensive perception deep multi-task network model capable of realizing an image target detection task and a distance estimation task in parallel, and designing a target loss function fused with geometric priori knowledge on the basis of multiple tasks; the model adopts an anchor-frame-free prediction architecture based on an encoder-decoder, and comprises a basic feature extraction module and two parallel subtask modules. The target loss function comprises target detection loss, distance prediction loss and geometric priori loss, and in the target detection and distance estimation task, modeling is performed on scene geometric priori knowledge, the geometric priori knowledge is fused into the target loss function, and the reliability of a network prediction result is improved through constraint of priori geometric knowledge.

Description

Technical Field

[0001] The present invention relates to the field of intelligent perception of ground targets, and more particularly to a target perception method in a specific scenario in which computer vision is combined with deep learning. Background Art

[0002] The ground-based optoelectronic reconnaissance system monitors threat targets within its detection range around the clock and detects and senses threat targets to ensure its own safety. During the reconnaissance and detection process, it is necessary to obtain multiple attributes of threat targets, such as category, size, location, distance, etc., to assess and judge their threat level, thereby triggering response measures and means at different levels. The intelligent construction of ground-based optoelectronic reconnaissance systems is an important means to improve combat effectiveness. Therefore, intelligent and comprehensive perception of threat targets is an issue that needs to be urgently addressed.

[0003] In the comprehensive perception of threat targets, in addition to target detection and identification, the spatial attributes of the target (size, position, distance, etc.) are also key factors in estimating its threat level. At present, the acquisition of spatial attributes mainly relies on active perception. In target ranging, the laser pulse ranging method is mainly used. The laser ranging method has poor concealment and is easily interfered with. It can only achieve single-point ranging and cannot measure the distance of multiple threat targets in the field of view at the same time in real time. Target perception technology based on intelligent image processing can passively obtain multiple attributes of multiple targets in real time, realize intelligent detection of threat targets, and provide important support for the assessment of their threat level.

[0004] Target perception technology based on intelligent image processing involves multiple image processing and computer vision tasks, including image preprocessing, environmental perception, target detection, image ranging, etc. The rapid development of deep learning technology has promoted the leapfrog development of image processing and computer vision. As an important technical approach, it has been applied to more and more tasks such as autonomous identification of threat targets. However, deep learning consumes a lot of computing resources such as data and hardware. Its excellent performance depends on a large amount of manually labeled data. At the same time, due to the large amount of calculation, it has high requirements for hardware resources. For the application field of ground battlefield reconnaissance and perception, data sample collection is difficult, the computing resources of embedded platforms are limited, and real-time requirements are high. Under such constraints, it is impossible to process each task separately. Considering the processing performance, power consumption, efficiency and other issues of multiple tasks from the application system level, deep multi-task learning is a potential solution.

[0005] In the research on deep multi-task based object detection and distance estimation, given the location of the target in the image, Jing and Yi

[12] proposed an end-to-end multi-task network that predicts the target category and target distance through two branches respectively. During the training phase, a 3D target pose estimation task is added to the prediction branch as an auxiliary task to further enhance the network performance. Xuepeng et al.

[13] Based on Faster R-CNN, a 3D object detection network MonoRCNN was designed. MonoRCNN includes three branch tasks: 2D object detection, 3D object detection (physical length, width, height and yaw angle) and distance estimation. Yufeng et al.

[14] Based on R-CNN, a multi-task network for target detection and distance estimation is proposed. A parallel shallow network GeomNet is added to extract features from the input coordinate map, which is then integrated into the distance estimation branch to improve model performance. Distance prediction multi-task networks mostly use the R-CNN series network structure. Adding prediction branches to multiple anchor frames results in repeated calculations, and the network structure has poor scalability and cannot be directly extended to other tasks, such as image dehazing, dark light enhancement, and other image generation tasks.

[0006] In the field of ground equipment reconnaissance and perception, intelligent target perception is achieved through a deep learning-based target detection network, and distance acquisition relies on a passive laser rangefinder. Under current circumstances, target detection and distance acquisition have not been deeply combined. Intelligent target perception only obtains the position of the target in the image, and distance acquisition requires additional instruments and control processes. On the one hand, the potential and computing resources of deep learning algorithms have not been fully utilized, and on the other hand, the additional active distance measurement method has poor concealment and low efficiency. Summary of the invention

[0007] In view of this, the present invention proposes an intelligent perception method for threat targets on the ground battlefield based on a deep multi-task network, which acquires multiple attributes of multiple threat targets through passive image processing, and realizes end-to-end multi-target detection and distance prediction. The present invention provides important support for the judgment of the threat level of the target, overcomes the problems of poor concealment and low efficiency caused by the passive ranging method of ground equipment, and solves the problem that deep learning methods only realize one task of target detection, resulting in insufficient utilization of resources. The present invention fully improves the concealment and survivability of ground equipment, improves the resource utilization efficiency of existing optoelectronic systems, and thereby improves the comprehensive perception capability of threat targets of ground equipment.

[0008] The specific technical solutions are as follows:

[0009] An intelligent perception method for ground battlefield targets based on deep multi-task networks is proposed. A deep multi-task network model for comprehensive ground target perception is constructed, which can realize image target detection tasks and distance estimation tasks in parallel. On the basis of multi-tasks, a target loss function integrating geometric prior knowledge is designed to train the constructed multi-task network model. The trained deep multi-task network model for comprehensive ground target perception is used to detect and estimate the target pixel position and physical distance of the image to be detected.

[0010] Furthermore, the ground target comprehensive perception deep multi-task network model adopts an anchor-free prediction architecture based on an encoder-decoder, including a basic feature extraction module and two parallel subtask modules, the subtask modules are a target detection subtask module and a distance prediction subtask module; the basic feature extraction module extracts the features of the input image, and then inputs the extracted features into the two subtask modules respectively, and each subtask module outputs the target position and target distance in the form of a result graph. The anchor-free prediction architecture based on the encoder-decoder is compatible with target detection, multi-attribute estimation tasks, and image generation tasks, and has strong scalability.

[0011] Furthermore, the objective loss function L is calculated as follows:

[0012] L=L det +L dis +L reproj

[0013] in,

[0014] L det is the target detection loss, specifically the anchor-free target detection loss.

[0015] L dis Distance prediction loss, specifically using L1 loss,

[0016] L reproj It is the geometric prior loss. In the target detection and distance estimation tasks, by modeling the scene geometric prior knowledge, the geometric prior loss is designed, and the geometric prior knowledge is integrated into the target loss function. Through the constraints of prior geometric knowledge, the reliability of the network prediction results is improved.

[0017] Furthermore, the geometric prior loss L reproj The calculation process is as follows:

[0018] (1) Model the scene geometry prior knowledge to obtain a geometric prior model, as follows:

[0019]

[0020] Among them, H camis the camera height, θ is the camera's downward angle, f is the camera's focal length, H obj Target physical height, D Target physical distance, v t The image ordinate of the target upper endpoint, v b The image ordinate of the target lower endpoint, v c The ordinate of the principal point of the image;

[0021] The surveillance camera is installed at a fixed position, with a height of H from the ground. cam , the downward angle toward the ground is θ, the focal length of the camera is f, and the rotation angle is assumed to be zero, without considering the camera roll angle. The origin of the camera coordinate system is located at the center of the camera, the world coordinate system is located on the ground directly below the center of the camera, and the image coordinate system is located in the lower left corner; the camera position and posture and scene structure diagram are shown in Figure 3 shown.

[0022] (2) Based on the geometric prior model, the camera height H is calculated by the target distance D' predicted by the network cam '.

[0023] When the terrain is undulating, for each target, the reference of the camera coordinate system should be adjusted in the height direction so that it is on the same reference as the target ground contact point. cam It varies with the terrain where the target is located, so for each target, the camera height under its baseline is calculated first. At the same time, after identifying the target category, its physical height is known;

[0024] (3) The target distance D' and H predicted by the network are cam 'And the known height H of the target in the training set obj , bring in the geometric prior model to estimate the coordinate value vt' of the upper endpoint of the target in the image;

[0025] (4) Calculate the geometric prior loss. The specific formula is as follows:

[0026]

[0027] in, It is the predicted value of the target detection subtask module.

[0028] Furthermore, in the encoding stage, the basic feature extraction module obtains feature maps of different resolutions of the input image as the convolution proceeds; in the decoding stage, upsampling is performed on the feature map with the lowest resolution, and multi-scale fusion of the upsampled high-level features and the features extracted in the encoding stage is performed, and finally the basic features of the input image are obtained;

[0029] The process of obtaining the target position (i.e., target pixel position) and category on the image by the target detection subtask module includes: using an anchor-free framework, the output layer uses a center point or corner point prediction method, and the output form is a result map;

[0030] The process of the distance prediction subtask module obtaining the target distance information includes: the output layer of the distance prediction module is a single branch, and the output form is a result graph.

[0031] The beneficial effects of the present invention are as follows:

[0032] (1) In the method of intelligent perception of ground battlefield targets based on a deep multi-task network proposed in the present invention, first, an end-to-end target perception deep multi-task network model is established to realize target detection and distance prediction tasks; a loss function integrating geometric prior knowledge is designed to train the constructed multi-task network. At the application system level, the deep multi-task network architecture is used to perceive targets, and the positions and physical distance information of multiple targets on the image can be obtained simultaneously, realizing passive target perception, improving ranging efficiency, and thus enhancing the concealment of equipment;

[0033] (2) In the encoder-decoder-based anchor-free prediction architecture, multiple subtasks share the basic feature module, which is conducive to improving the efficiency of calculation and resource usage. At the same time, compared with directly adopting the R-CNN series architecture, this architecture avoids adding prediction branches on multiple anchor boxes and reduces repeated calculations;

[0034] (3) A loss function that integrates geometric prior knowledge is designed to incorporate geometric constraints into network training, using known knowledge to guide the training of the network model and improve the accuracy of target detection and distance prediction;

[0035] (4) This network architecture is highly scalable. It can integrate tasks such as image dehazing and image dark light enhancement, deeply integrate image preprocessing and target perception, and achieve mutual promotion of task performance during training.

[0036] (5) The present invention can simultaneously obtain the pixel positions and physical distances of multiple targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the implementation of the present invention or the existing technical solutions, the following briefly introduces the drawings required to be used in the embodiments or the description of the prior art.

[0038] Figure 1 It is the end-to-end threat target perception deep multi-task network model of the present invention.

[0039] Figure 2 This is the target loss calculation process of the present invention that integrates geometric prior knowledge.

[0040] Figure 3 It is a schematic diagram of the camera posture and scene structure of the present invention.

[0041] Figure 4 Schematic diagram of the multi-task network structure based on CenterNet-DLA34 DETAILED DESCRIPTION

[0042] The following is a clear and complete description of the technical solutions in the embodiments of the invention in conjunction with the drawings in the embodiments of the invention. Obviously, the described embodiments are only part of the embodiments of the invention, not all of the embodiments. Based on the embodiments of the invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the invention.

[0043] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0044] Considering that the existing intelligent perception of ground battlefield targets lacks a data set to achieve distance prediction tasks, this embodiment combines the optoelectronic reconnaissance perception equipment of existing ground equipment to collect threat target image data and distance information, and uses annotation software to annotate. Regarding distance acquisition, for long-distance field scenes, laser ranging is used to obtain target distance information, and the target detection, numbering management and distance association processes of the equipment are combined to obtain the distance information corresponding to the target; for urban scenes, laser radar is used to obtain target distance information, and the optical camera and laser radar need to be calibrated in this process.

[0045] The present invention discloses a method for intelligently sensing ground battlefield threat targets based on a deep multi-task network, comprising the following steps:

[0046] (1) Build a deep multi-task network model for ground target comprehensive perception that can perform image target detection and distance estimation tasks in parallel;

[0047] (2) Based on multi-task, a target loss function that integrates geometric prior knowledge is designed to train the constructed multi-task network model.

[0048] As a preferred embodiment of the present invention, refer to Figure 1In step (1), the threat target perception deep multi-task network model adopts an anchor-free prediction architecture based on an encoder-decoder, which includes a basic feature extraction module and two parallel target detection subtask modules and a distance prediction task module. The basic feature extraction module extracts the features of the input image and inputs them into the target detection module and the distance prediction module respectively. Each subtask module outputs the target position and target distance in the form of a result map. The anchor-free prediction architecture based on the encoder-decoder is compatible with multi-attribute estimation such as target detection and distance estimation, as well as image generation tasks, and has strong scalability.

[0049] As a preferred embodiment of the present invention, the process of extracting features of input image data by the feature extraction module includes:

[0050] In the encoding stage, the basic feature extraction module obtains feature maps of different resolutions of the input image as the convolution proceeds; in the decoding stage, upsampling is performed on the feature map with the lowest resolution, and the upsampled high-level features are fused with the features extracted in the encoding stage at multiple scales to finally obtain the basic features of the input image.

[0051] In the encoding stage, as the convolution proceeds, the feature map scale becomes smaller and the semantic abstraction increases. On the feature map with the lowest resolution, multiple upsampling operations are performed step by step, and fused with low-level features through skip connections until the resolution of the original input image is restored, or 1 / 2 or 1 / 4 of the original input image resolution, to obtain the basic features of the input image.

[0052] As a preferred embodiment of the present invention, the process of obtaining the target position and category on the image by the target detection module includes: adopting an anchor-free framework structure, the output layer adopts a center point or corner point prediction method, and the output form is a result map.

[0053] As a preferred embodiment of the present invention, the process of the distance prediction module acquiring the target distance information includes: the output layer of the distance prediction module is a single branch, and the output form is a result graph.

[0054] As a preferred embodiment of the present invention, in step (2), a target loss function integrating geometric prior knowledge is designed, which is the target detection loss L det , distance estimation loss L dis and geometric prior loss L reproj The sum of L = L det +L dis +L reproj , refer to Figure 2 . Object detection loss L det Adopt anchor-free target detection loss and distance prediction loss L disL1 loss is used. In the target detection and distance estimation tasks, the geometric prior knowledge of the scene is modeled, the geometric prior loss is designed, the geometric prior knowledge is integrated into the target loss function, and the reliability of the network prediction results is improved through the constraints of the prior geometric knowledge.

[0055] As a preferred embodiment of the present invention, refer to Figure 3 , construct the geometric prior loss L reproj The process is as follows:

[0056] (2-1) Model the scene geometry prior knowledge and construct a reprojection model:

[0057]

[0058] Among them, H cam Camera height; θ camera looking down angle; f camera focal length; H obj Target physical height; D Target physical distance; v b The image vertical coordinate of the lower endpoint of the target; v t The image ordinate of the target upper endpoint; v c The vertical coordinate of the image principal point.

[0059] The surveillance camera is installed at a fixed position, with a height of H from the ground. cam , the downward angle toward the ground is θ, the focal length of the camera is f, and the rotation angle is assumed to be zero, without considering the camera roll angle. The origin of the camera coordinate system is located at the center of the camera, the world coordinate system is located on the ground directly below the center of the camera, and the image coordinate system is located in the lower left corner. The camera position and posture and scene structure diagram are shown in Figure 3 shown.

[0060] (2-2) Based on the geometric prior model, the network predicts the target distance D and calculates the camera height H cam '.

[0061] When the terrain is undulating, for each target, the reference of the camera coordinate system should be adjusted in the height direction so that it is on the same reference as the target ground contact point, then H cam It varies with the terrain where the target is located, so the camera height under the baseline should be calculated for each target. At the same time, after identifying the target category, its physical height is known.

[0062] (2-3) Using the network to predict the target distance D, H cam 'And the target's prior height H obj , estimate the coordinates of the upper endpoint of the target in the image

[0063] (2-4) The geometric prior loss is the reprojection estimate and the network prediction value vt The difference between:

[0064]

[0065] By adding scene geometry prior constraints, the accuracy of the prediction results of the deep multi-task network can be further improved.

[0066] As a preferred embodiment of the present invention, in the loss calculation, the true value of the target position and size class is mapped to [0,1] using the maximum normalization method. By normalizing, each loss function is brought to a similar size, which can optimize the convergence speed and performance of network training.

[0067] The present invention also has multiple implementation forms, which are as follows:

[0068] The basic feature extraction module can use existing network models such as Resnet, DLA, and Hourglass. Figure 4 The basic feature extraction module uses the CenterNet-DLA34 network. The downsampling process of the network structure is divided into 5 stages n1 to n5, and each stage is downsampled by 2 times. The number of channels in stages n1 to n5 is 16, 32, 64, 128, and 256 respectively. Starting from stage n5, 3 upsamplings are performed. In this process, the low-level features are upsampled and added to the high-resolution features of the upper layer after deformable convolution, and multi-level and multi-scale feature fusion output is performed. The original image input is 3*M*N, the extracted feature map size is 1 / 4 of the original input, and the number of channels is 256, that is, 256*M / 4*N / 4.

[0069] The basic version of the target detection module is generally composed of 2-4 convolutional layers, and other carefully designed modules can also be inserted. The output layer can use the center point prediction of CenterNet, the corner point prediction method of CornerNet, or other key point prediction methods, such as the prediction method of combining extreme points and center points represented by ExtremeNet. This embodiment uses the center point prediction method of CenterNet. Figure 4 The input of the target detection module is the feature map output by the basic feature extraction module. After one layer of convolution, it enters three branches, namely the center point prediction branch, the height width prediction branch and the rounding error prediction branch. Each branch contains a layer of convolution and an output layer. The size of the center point prediction heat map is n*M / 4*N / 4, n is the target category, and the heat map value range is [0,1]; the output size of the height width prediction branch is 2*M / 4*N / 4, and the output size of the error branch is 2*M / 4*N / 4. Due to the 4-fold downsampling between the input and output, there will be some offset when the key points are rounded. Therefore, the two channels predict the horizontal and vertical offsets, that is, the rounding error.

[0070] The basic version of the distance prediction module is generally composed of 2-3 convolutional layers, and other carefully designed modules can also be inserted. The input of the distance prediction module is the feature map output by the basic feature extraction module, which includes convolutional layers and output layers. The size of the output layer is 1*M / 4*N / 4.

[0071] The loss of target detection without anchor boxes can directly refer to the existing CenterNet, CornerNet, and ExtremeNet target detection network losses. This embodiment adopts CenterNet target detection loss, including key point loss, height width, and rounding error loss. Among them, the loss of key points adopts Focal Loss, and the true value heat map is generated with a Gaussian kernel. If there is an overlap between the generated areas of two points, the larger response value is taken; in the calculation process of the height width and rounding error loss function, only the loss at the key points is considered, and L1 loss is adopted.

[0072] In the calculation process of the distance prediction loss function, only the loss at the key point is considered, and the L1 loss is adopted.

[0073] The position, height and width of the key points of target detection are normalized based on the image size. The quantization error itself is calculated in the range [0,1]. Distance normalization is based on the upper limit of the dataset distance.

[0074] In the inference stage, the key point is located by extracting the peak value of the response on the heat map, and then the height, width, quantization rounding error and distance of the target are extracted based on the key point position.

Claims

1. A ground battlefield target intelligent perception method based on deep multi-task network, characterized by: A ground target comprehensive perception deep multi-task network model is constructed, which can realize image target detection and distance estimation tasks in parallel. Based on the multi-task, a target loss function integrating geometric prior knowledge is designed to train the constructed multi-task network model. The trained ground target comprehensive perception deep multi-task network model is used to detect the target pixel position and physical distance of the image to be detected.

2. According to the method of intelligent perception of ground battlefield targets based on deep multi-task network described in right 1, it is characterized in that: The ground target comprehensive perception deep multi-task network model adopts an anchor-free box prediction architecture based on an encoder-decoder, including a basic feature extraction module and two parallel sub-task modules, wherein the sub-task modules are a target detection sub-task module and a distance prediction sub-task module; the basic feature extraction module extracts the features of the input image, and then inputs the extracted features into the two sub-task modules respectively, and each sub-task module outputs the target pixel position and the target physical distance in the form of a result graph.

3. A method for intelligently sensing ground battlefield targets based on a deep multi-task network according to claim 2, characterized in that: The objective loss function L is calculated as follows: L=L det +L dis +L reproj in, L det is the target detection loss, specifically the anchor-free target detection loss. L dis Distance prediction loss, specifically using L1 loss, L reproj It is the geometric prior loss. In the target detection and distance estimation tasks, by modeling the scene geometric prior knowledge, the geometric prior loss is designed, and the geometric prior knowledge is integrated into the target loss function. Through the constraints of prior geometric knowledge, the reliability of the network prediction results is improved.

4. A ground target comprehensive perception deep multi-task network model according to claim 3, characterized in that: Geometric prior loss L reproj The calculation process is as follows: (1) Model the scene geometry prior knowledge to obtain a geometric prior model, as follows: Among them, H cam is the camera height, θ is the camera's downward angle, f is the camera's focal length, H obj Target physical height, D Target physical distance, v t The image ordinate of the target upper endpoint, v b The image ordinate of the target lower endpoint, v c The ordinate of the principal point of the image; The surveillance camera is installed at a fixed position, with a height of H from the ground. cam , the viewing angle toward the ground is θ, the camera focal length is f, the camera rotation angle is zero, the camera roll angle is not considered, the origin of the camera coordinate system is located at the center of the camera, the world coordinate system is located on the ground directly below the camera center, and the image coordinate system is located in the lower left corner; (2) Based on the geometric prior model, the camera height H is calculated by the target distance D' predicted by the network cam '. When the terrain is undulating, for each target, the reference of the camera coordinate system should be adjusted in the height direction so that it is located at the same reference as the target ground contact point. cam It varies with the terrain where the target is located, so for each target, the camera height under its baseline is calculated first. At the same time, after identifying the target category, its physical height is known; (3) The target distance D' and H predicted by the network are cam 'And the known height H of the target in the training set obj , bring in the geometric prior model to estimate the coordinate value vt' of the upper endpoint of the target in the image; (4) Calculate the geometric prior loss. The specific formula is as follows: in, It is the predicted value of the target detection subtask module.

5. According to any one of the ground target comprehensive perception deep multi-task network models described in rights 1-4, it is characterized in that: In the encoding stage, the basic feature extraction module obtains feature maps of different resolutions of the input image as the convolution proceeds; in the decoding stage, upsampling is performed on the feature map with the lowest resolution, and multi-scale fusion of the upsampled high-level features and the features extracted in the encoding stage is performed, and finally the basic features of the input image are obtained; The process of obtaining the position and category of the target on the image by the target detection subtask module includes: using an anchor-free framework, the output layer uses the center point or corner point prediction method, and the output form is the result map; The process of the distance prediction subtask module obtaining the target distance information includes: the output layer of the distance prediction module is a single branch, and the output form is a result graph.

Images