Multi-granularity and multi-modal 3D object detection method and system based on weakly supervised learning

Abstract

The present invention proposes a multi-granularity and multi-modal 3D target detection method and system based on weakly supervised learning. Combining a multi-granularity position representation method and pre-established consistency constraints, the 3D target detection is decomposed into two progressive subtasks: coarse-grained 3D target detection and fine-grained 3D key point positioning, by utilizing the different accuracy of target position descriptions of different position representation methods. Fine-grained 3D key point positioning is further performed based on the coarse-grained detection results obtained by coarse-grained 3D target detection, the 3D key point coordinates on the local area of ​​the target are estimated, and the target 3D position and 3D key point information in the video are output. The above scheme solves the runway safety hazards caused by aircraft intrusion into the runway. Under the condition of limited annotated data, it is possible to automatically analyze runway area targets, including tasks such as accurate positioning and posture estimation of targets at the image and spatial levels, to achieve automatic detection of safety hazards such as runway intrusion.

Description

Technical Field

[0001] The present invention relates to the technical field of image target detection, and in particular to a multi-granularity and multi-modal 3D target detection method and system based on weakly supervised learning. Background Art

[0002] In recent years, with the increase in civil aviation transport volume year by year, air traffic has also shown an increasingly busy development trend. The high density of take-off and landing flight missions inside the airport has posed a great threat to runway safety. Therefore, effective monitoring of runway safety events is an important means to ensure civil aviation safety and improve flight operation efficiency. At present, automatic dependent surveillance-broadcast (ADS-B) and multi-lateration positioning (MLAT) are commonly used as monitoring sensor equipment, but the existing equipment still has the following problems when applied to runway safety monitoring: (1) Insufficient stability, mainly reflected in the large impact of electromagnetic interference; (2) Strong dependence, the onboard equipment needs to be equipped with a transponder, and the positioning depends on the satellite navigation system; (3) Insufficient accuracy: the aircraft is large in size and not compact in structure. The existing positioning method uses the entire target as a unit and cannot accurately describe the position of the local area (such as the nose, tail, etc.); (4) Low update rate: The update rate of the existing method is about 0.5Hz~2Hz, and the update rate is further reduced when the target is stationary.

[0003] In the prior art, video-based runway safety monitoring methods are almost unaffected by electromagnetic interference and can realize video target analysis under non-cooperative conditions. The prior art generally uses target detection methods to judge runway intrusion events, combined with marker point positioning methods to improve the accuracy of detection. In addition, the use of target posture estimation methods can further realize the prediction of runway deviation events. The above modules use deep learning-based methods to train target detection and marker point positioning models respectively, and a lightweight network can be used to achieve real-time (25Hz) update of target information.

[0004] However, visible light video itself is sensitive to external environments such as lighting and bad weather. Therefore, when encountering environmental changes such as low illumination, fog, and rain, the images captured by the visible light camera may have reduced clarity and increased noise interference, resulting in the runway monitoring system based on single-light video not being able to work properly. To solve this problem, the present invention considers introducing multimodal video data on the one hand to make up for the shortcomings of visible light video under harsh conditions, thereby improving the availability and robustness of the system; on the other hand, starting from the method level, the accuracy of target analysis of existing methods under conditions such as low illumination and environmental interference is improved. In existing video-based image analysis methods, deep models are usually trained separately for tasks such as target detection and marker location based on tasks. Due to the differences in task granularity between different tasks, this method not only ignores the correlation between different tasks, but also fails to characterize the high-order constraints between different granularities, thereby limiting the processing capabilities of the model under external interference conditions.

[0005] In addition, the existing technology has the following problems when obtaining the true value of training data: (1) 3D point cloud data is introduced, but for the scenario of this project, the perception range of the depth camera is limited, and the lidar may have problems such as relatively sparse point clouds; (2) A synthetic dataset is constructed using the target 3D model, but it is difficult to extend to uncontrollable outdoor scenes. Summary of the invention

[0006] In order to address the deficiencies in the prior art solutions, the present invention provides a multi-granularity and multi-modal 3D target detection method and system based on weakly supervised learning, which are used to solve the availability and stability problems of the anti-runway intrusion system based on video target analysis, reduce the system's dependence on the external environment, and improve the all-weather processing capability of the video-based runway safety monitoring system.

[0007] The purpose of the present invention is achieved by adopting the following technical solutions:

[0008] A multi-granularity and multi-modal 3D object detection method based on weakly supervised learning, the method comprising:

[0009] Input the visible light video and thermal infrared video taken at the same time as the 3D object detection input;

[0010] When performing 3D object detection on the 3D object detection input, based on a multi-granularity object 3D position representation method, the 3D object detection is decomposed into a coarse-grained 3D object detection task and a fine-grained 3D key point positioning task; wherein the coarse-grained 3D object detection task is used to estimate the 3D detection box position and detection box confidence information of the target; the fine-grained 3D key point positioning task is used to estimate the 3D key point coordinates on the local area of ​​the target based on the coarse-grained detection result;

[0011] By performing coarse-grained 3D object detection tasks and fine-grained 3D key point positioning tasks, combined with pre-established consistency constraints, a weak supervision loss for the 3D object detection is constructed, and the weak supervision loss is solved based on weak supervision learning to obtain the target 3D position and 3D key point coordinates in the video.

[0012] Preferably, the multi-granularity position representation method is to divide the target 3D position representation into a coarse-grained global target key point representation and a fine-grained local target key point representation;

[0013] The coarse-grained global target key point representation describes the 3D position of the overall target;

[0014] The fine-grained local target key point representation is to describe the local feature points on the target surface;

[0015] The consistency constraint condition is formulated based on the mutual constraint relationship between the coarse-grained and fine-grained target positions involved in the multi-grained position representation method;

[0016] The consistency constraints include: 3D space multi-granularity position constraints and 2D projection coordinate consistency constraints;

[0017] The 2D projection coordinate consistency constraint also includes: a fine-grained 2D projection coordinate consistency constraint and a coarse-grained 2D projection coordinate consistency constraint.

[0018] Preferably, the performing of the coarse-grained 3D object detection task and the fine-grained 3D key point positioning task comprises:

[0019] Construct an object detection framework for performing coarse-grained 3D object detection tasks and fine-grained 3D key point localization tasks; including a coarse-grained 3D object detection framework and a fine-grained 3D object key point localization framework;

[0020] The coarse-grained detection results output by the coarse-grained 3D target detection framework are input into the fine-grained 3D target key point positioning framework to output the 3D key point coordinates on the local area of ​​the target.

[0021] Furthermore, the coarse-grained 3D object detection framework includes: a backbone network and a coarse-grained prediction network;

[0022] The backbone network includes Darknet53 and a cross-stage local network added after each residual block of Darknet53;

[0023] The coarse-grained prediction network includes a feature pyramid and a path aggregation network.

[0024] Furthermore, the performing of the coarse-grained 3D object detection task includes:

[0025] Input the paired RGB-T images into the backbone network, obtain a 4-channel image through pixel-level fusion, and extract the features of the fused image;

[0026] The image features output by the backbone network are input into the coarse-grained prediction network, which outputs the target 3D detection box offset and the corresponding detection box confidence.

[0027] The confidence value range is [0, 1], which is used to indicate the probability that the target in the corresponding detection box belongs to the specified image feature.

[0028] Furthermore, the execution of the fine-grained 3D target key point positioning task includes: inputting the visible light image, the coarse-grained 3D detection frame and the corresponding detection frame confidence into a predefined image cropping module for image cropping operation to generate input data for fine-grained 3D target key point positioning; wherein the image cropping module includes a feature extraction network based on a ResNet50 structure and a fine-grained prediction network including a multi-layer perceptron;

[0029] When performing the task of fine-grained 3D target key point localization, the feature vector output by the feature extraction network is input into the fine-grained prediction network to output the target 3D key point offset.

[0030] Further, the generating input data for fine-grained 3D target key point positioning includes: projecting the target 3D detection frame onto the visible light image, and using the minimum 2D rectangular frame surrounding all the projected points as a cropping area;

[0031] If the confidence of the detection box exceeds a predefined cropping threshold, the corresponding cropping area is cropped, and the cropped image is input into the fine-grained 3D object key point positioning framework as input data for performing fine-grained 3D object key point positioning.

[0032] Preferably, the method of solving the weak supervision loss based on weak supervision learning to obtain the target 3D position and 3D key point coordinates in the video includes: encoding all output variables obtained by coarse and fine granularity 3D target detection into residual form, and calculating the final estimated value using the initial anchor frame, the initial key point and the regressed offset;

[0033] The estimation of the size of the 3D anchor frame includes: statistically analyzing the length, width and height distribution of common aircraft, and using the KMeans clustering algorithm to determine the initial size information of the target; and determining the initial depth of the target by measuring the distance from the camera to the runway.

[0034] The initial center coordinates of the 3D anchor frame are estimated based on the initial depth of the target. The center position of the 3D anchor frame is derived by combining the center coordinates of the 2D anchor frame using the pinhole camera imaging principle.

[0035] The initial key points are obtained by calculating the predefined aircraft 3D model: calculating the rigid transformation matrix from the aircraft 3D model to the initial anchor frame, applying the rigid transformation matrix to the model 3D feature points to obtain the initial key points;

[0036] The step of calculating the rigid transformation matrix from the aircraft 3D model to the initial anchor frame includes: estimating a scaling factor using a pre-established distance between two wings of the aircraft 3D model and a width of the initial anchor frame;

[0037] The rotation matrix is ​​estimated using the parallel constraint between the line connecting the two wings of the aircraft 3D model and the wide side of the initial anchor frame, and the parallel constraint between the nose direction and the long side;

[0038] Estimate the minimum detection box that completely encloses the aircraft 3D model, and calculate the translation vector between the center point of the detection box and the center point of the initial anchor box.

[0039] Furthermore, the weak supervision loss for constructing 3D object detection includes: coarse-grained 2D projection coordinate consistency loss, fine-grained 2D projection coordinate consistency loss, 3D space cross-granularity position loss and multi-view consistency loss;

[0040] The coarse-grained 2D projection coordinate consistency loss is used to project the estimated coarse-grained 3D bounding box onto the image, calculate a minimum 2D rectangular box using the projection points, and estimate the L2 distance between the 2D rectangular box and the annotated 2D detection box;

[0041] The fine-grained 2D projection coordinate consistency loss is used to project the estimated fine-grained 3D key points to the visible light image and calculate the Euclidean distance between the projected points and the annotated 2D key points; the 3D spatial cross-granularity position loss is used to assume that the feature points are tangent to the corresponding plane on the bounding box and calculate the Euclidean distance from the point to the corresponding plane;

[0042] The multi-view consistency loss is used to assume that the two cameras have been calibrated based on the different perspectives of the scene captured by the visible light image and the thermal infrared image. The estimated 3D key points are projected onto the thermal infrared image to obtain the corresponding 2D key points. The 2D key points projected on the thermal infrared image and the 2D key points marked on the visible light image are used to construct key point pairs at different perspectives. The key point pairs are reconstructed using the triangulation principle, and the L2 distance between the reconstructed 3D key points and the estimated 3D key points is calculated.

[0043] A multi-granularity and multi-modal 3D object detection system based on weakly supervised learning, comprising:

[0044] The image input module is used to collect visible light video and thermal infrared video taken at the same time and use them as 3D target detection input;

[0045] A task division module is used to decompose the 3D target detection into a coarse-grained 3D target detection task and a fine-grained 3D key point positioning task based on a multi-granularity target 3D position representation method when performing 3D target detection on the 3D target detection input; wherein the coarse-grained 3D target detection task is used to estimate the 3D detection box position and detection box confidence information of the target; the fine-grained 3D key point positioning task is used to estimate the 3D key point coordinates on the local area of ​​the target based on the coarse-grained detection result;

[0046] The target detection module is used to construct the weak supervision loss of the 3D target detection by performing the coarse-grained 3D target detection task and the fine-grained 3D key point positioning task in combination with the pre-established consistency constraint conditions, solve the weak supervision loss based on weak supervision learning, and obtain the target 3D position and 3D key point coordinates in the video.

[0047] The beneficial effects of the present invention are embodied in:

[0048] The present invention provides a multi-granularity and multi-modal 3D target detection method and system based on weakly supervised learning, which can automatically analyze runway area targets under the condition of limited annotated data, including tasks such as precise positioning and posture estimation of targets at the image level and space level; thereby realizing automatic detection of safety hazards such as runway intrusion.

[0049] The present invention is applied to the runway safety monitoring system, which can effectively warn / alert intrusive targets, greatly improve the airport runway safety protection capability, and enhance the level of smart air traffic control construction. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following is a brief introduction to the drawings required for the specific embodiments or the description of the prior art. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn according to the actual scale.

[0051] Figure 1 A flowchart of a multi-task and multi-modal 3D object detection method based on weakly supervised learning provided by the present invention;

[0052] Figure 2 A schematic diagram of the principle of 3D position representation of multi-granularity targets provided in Example 1 of the present invention;

[0053] Figure 3 A technical block diagram of a multi-task target 3D pose estimation method based on weakly supervised learning provided in Example 1 of the present invention. DETAILED DESCRIPTION

[0054] The following embodiments of the technical solution of the present invention are described in detail in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and are therefore only used as examples, and cannot be used to limit the protection scope of the present invention.

[0055] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in this application should have the common meanings understood by those skilled in the art to which the present invention belongs.

[0056] The present invention aims at the potential safety hazard of runway caused by aircraft intrusion into the runway. In the specific implementation of the present invention, a multi-granularity position representation method and constraint conditions are combined to propose an accurate and efficient 3D target detection solution - a multi-task multi-modal 3D target detection method and system based on weakly supervised learning. The system input is visible light and thermal infrared video, and the output is the target 3D position and 3D key point information in the video.

[0057] The multi-granularity position representation method is to divide the target 3D position representation into a coarse-grained global target key point representation and a fine-grained local target key point representation;

[0058] The coarse-grained global target key point representation describes the 3D position of the overall target;

[0059] The fine-grained local target key point representation is to describe the local feature points on the target surface;

[0060] The consistency constraint condition is formulated based on the mutual constraint relationship between the coarse-grained and fine-grained target positions involved in the multi-grained position representation method;

[0061] The consistency constraint conditions include: 3D space multi-granularity position constraints and 2D projection coordinate consistency constraints; the 2D projection coordinate consistency constraints also include: fine-grained 2D projection coordinate consistency constraints and coarse-grained 2D projection coordinate consistency constraints.

[0062] like Figure 1 As shown, in combination with the above-mentioned multi-granularity position representation method and constraint conditions, a multi-granularity multi-modal 3D target detection method based on weakly supervised learning proposed in a specific embodiment of the present invention specifically includes the following steps:

[0063] S1 collects visible light video and thermal infrared video taken at the same time and uses them as 3D object detection input;

[0064] S2 performs 3D target detection on the 3D target detection input, decomposing the 3D target detection into a coarse-grained 3D target detection task and a fine-grained 3D key point positioning task based on a multi-granularity target 3D position representation method; wherein the coarse-grained 3D target detection task is used to estimate the 3D detection frame position and detection frame confidence information of the target; the fine-grained 3D key point positioning task is used to estimate the 3D key point coordinates on the local area of ​​the target based on the coarse-grained detection result;

[0065] S3 constructs the weak supervision loss of the 3D object detection by performing the coarse-grained 3D object detection task and the fine-grained 3D key point positioning task, combined with the pre-established consistency constraint conditions, solves the weak supervision loss based on weak supervision learning, and obtains the target 3D position and 3D key point coordinates in the video.

[0066] In step S3, executing the coarse-grained 3D object detection task and the fine-grained 3D key point positioning task includes:

[0067] According to the multi-task decomposition of 3D target detection, a corresponding target detection framework is constructed; including a coarse-grained 3D target detection framework and a fine-grained 3D target key point positioning framework;

[0068] The coarse-grained 3D object detection framework is used to perform coarse-grained 3D object detection tasks; the coarse-grained 3D object detection framework includes: a backbone network and a coarse-grained prediction network; wherein the backbone network includes Darknet53, and a cross-stage local network added after each residual block of Darknet53; the coarse-grained prediction network includes a feature pyramid and a path aggregation network.

[0069] The fine-grained 3D object key point positioning framework is used to perform fine-grained 3D key point positioning tasks;

[0070] When performing 3D object detection on the 3D object detection input, after using the coarse-grained 3D object detection framework to perform the coarse-grained 3D object detection task, the coarse-grained detection result output by the coarse-grained 3D object detection framework is input into the fine-grained 3D object key point positioning framework, and the 3D key point coordinates on the local area of ​​the target are output.

[0071] 1) Executing coarse-grained 3D object detection tasks includes:

[0072] Input the paired RGB-T images into the backbone network, obtain a 4-channel image through pixel-level fusion, and extract the features of the fused image;

[0073] The image features output by the backbone network are input into the coarse-grained prediction network, which outputs the target 3D detection box offset and the corresponding detection box confidence.

[0074] The confidence value range is [0, 1], which is used to indicate the probability that the target in the corresponding detection box belongs to the specified image feature.

[0075] 2) Executing the fine-grained 3D target key point positioning task includes: inputting the visible light image, the coarse-grained 3D detection frame and the corresponding detection frame confidence into a predefined image cropping module for image cropping operation to generate input data for fine-grained 3D target key point positioning; wherein the image cropping module includes a feature extraction network based on the ResNet50 structure and a fine-grained prediction network including a multi-layer perceptron;

[0076] When performing the task of fine-grained 3D target key point localization, the feature vector output by the feature extraction network is input into the fine-grained prediction network to output the target 3D key point offset.

[0077] In step 2), generating input data for fine-grained 3D target key point positioning includes: projecting the target 3D detection frame onto the visible light image, and using the minimum 2D rectangular frame surrounding all projection points as the cropping area; if the confidence of the detection frame exceeds a pre-defined cropping threshold, the corresponding cropping area is cropped, and the cropped image is input into the fine-grained 3D target key point positioning framework as input data for performing fine-grained 3D target key point positioning.

[0078] In step S3, solving the weak supervision loss based on weak supervision learning to obtain the target 3D position and 3D key point coordinates in the video includes: encoding all output variables obtained by coarse and fine-grained 3D target detection into residual form, and using the initial anchor frame, initial key point and regressed offset to infer the final estimated value;

[0079] The estimation of the size of the 3D anchor box includes: statistically analyzing the length, width and height distribution of common aircraft, using the KMeans clustering algorithm to determine the initial size information of the target; determining the initial depth of the target by measuring the distance from the camera to the runway;

[0080] The initial center coordinates of the 3D anchor frame are estimated based on the initial depth of the target. The center position of the 3D anchor frame is derived by combining the center coordinates of the 2D anchor frame using the imaging principle of a pinhole camera.

[0081] The initial key points are obtained by calculating the predefined aircraft 3D model: calculating the rigid transformation matrix from the aircraft 3D model to the initial anchor frame, applying the rigid transformation matrix to the model 3D feature points to obtain the initial key points;

[0082] The calculating of the rigid transformation matrix from the aircraft 3D model to the initial anchor frame comprises: estimating a scaling factor using a pre-established distance between two wings of the aircraft 3D model and a width of the initial anchor frame;

[0083] The rotation matrix is ​​estimated using the parallel constraint between the line connecting the two wings of the aircraft 3D model and the wide side of the initial anchor frame, and the parallel constraint between the nose direction and the long side;

[0084] Estimate the minimum detection box that completely encloses the aircraft 3D model, and calculate the translation vector between the center point of the detection box and the center point of the initial anchor box.

[0085] The weak supervision loss for constructing 3D object detection in step S3 includes: coarse-grained 2D projection coordinate consistency loss, fine-grained 2D projection coordinate consistency loss, 3D space cross-granularity position loss and multi-view consistency loss;

[0086] The coarse-grained 2D projection coordinate consistency loss is used to project the estimated coarse-grained 3D bounding box onto the image, calculate a minimum 2D rectangular box using the projection points, and estimate the L2 distance between the 2D rectangular box and the annotated 2D detection box;

[0087] The fine-grained 2D projection coordinate consistency loss is used to project the estimated fine-grained 3D key points to the visible light image and calculate the Euclidean distance between the projected points and the annotated 2D key points; the 3D spatial cross-granularity position loss is used to assume that the feature points are tangent to the corresponding plane on the bounding box and calculate the Euclidean distance from the point to the corresponding plane;

[0088] The multi-view consistency loss is used to assume that the two cameras have been calibrated based on the different perspectives of the scene captured by the visible light image and the thermal infrared image. The estimated 3D key points are projected onto the thermal infrared image to obtain the corresponding 2D key points. The 2D key points projected on the thermal infrared image and the 2D key points marked on the visible light image are used to construct key point pairs at different perspectives. The key point pairs are reconstructed using the triangulation principle, and the L2 distance between the reconstructed 3D key points and the estimated 3D key points is calculated.

[0089] Embodiment 1: Embodiment 1 of the present invention proposes a multi-task 3D target detection method based on weakly supervised learning. The method is described based on two aspects: multi-granularity target 3D position representation and multi-task 3D target detection based on weakly supervised learning.

[0090] 1. Target 3D position representation based on multi-granularity:

[0091] Embodiment 1 of the present invention uses a method for representing the 3D position of a target based on multi-granularity representation. Figure 1 As shown, in Example 1 of the present invention, different position marking methods are used to describe the target position with different degrees of accuracy, and the target 3D position representation is divided into a coarse-grained global target key point representation and a fine-grained local target key point representation: (1) Coarse-grained global target key point representation: The 3D position of the entire target is described, assuming that the symbol p i =(x i ,y i ,z i ) represents a 3D point in space, {p1, p2, ..., p8} represents the vertex coordinates of the 3D bounding box surrounding the target, and the average of all vertex coordinates is used as the 3D coordinate point p0 of the target center;

[0092] (2) Fine-grained local target key point representation: local feature points with significant semantic features on the target surface are described. For an aircraft, the local key points defined in this invention include the nose, tail, left wing, and right wing, which are represented by p9, p 10 ,p 11 ,p12 express. Figure 2 The yellow part represents the coarse-grained global object keypoints, and the red part represents the fine-grained local object keypoints.

[0093] According to the above multi-granularity position characterization method, Embodiment 1 of the present invention utilizes the mutual constraint relationship between the coarse-grained and fine-grained target positions to propose the following consistency constraint conditions:

[0094] 3D spatial cross-granularity position constraint: Assume that the external bounding box formed by the global coarse-grained key points p1~p8 is the smallest box that completely encloses the target. In theory, the local fine-grained 3D key points should be tangent to a plane of the external bounding box. Based on the above assumption, define P i (i∈[9,12]) is the local key point p i Tangent planes, establish 3D space multi-granularity position constraints dist(p i ,P i )=0, where dist represents the Euclidean distance from the local key point to the corresponding tangent plane. Figure 2 As shown, the feature points and corresponding planes that meet the above position constraints include: p9 and p5p6p7p8, p 10 and p1p2p5p6, p 11 and p1p4p8p5, p 12 and p2p6p7p3.

[0095] 2D projection coordinate consistency constraint: When the target 3D feature point is projected onto the 2D image, it should be consistent with the 2D coordinate point marked in the image space. According to the distribution difference of coordinate projection data under different granularity conditions, the 2D projection coordinate consistency constraints at the coarse and fine granularity levels are designed in a targeted manner: (1) Fine-grained 2D projection coordinate consistency constraint: Assume that the fine-grained 3D key point p i The corresponding point (i∈[9,12]) projected onto the image is m i (i∈[9,12]), and the true value of the corresponding key point on the image is loc i (i∈[9,12]), the fine-grained 2D / 3D coordinate position consistency constraint can be expressed as ∑|m i -loc i |; (2) Coarse-grained 2D projection coordinate consistency constraint: Assume that the coarse-grained 3D key point p i The corresponding point (i∈[1,8]) projected onto the image is m i (i∈[1,8]), with the point set {m i ,i∈[1,8]}, find the i ,i∈[1,8]}, the smallest rectangular box of all points in the point set {m' i,i∈[1,4]} are connected sequentially. Assume that the corresponding detection box on the 2D image is composed of the point set {loc i ,i∈[1,4]}, then the 3D-2D coordinate position consistency constraint at the coarse-grained level can be expressed as ∑|m' i -loc i |.

[0096] 2. Multi-task and multi-modal 3D object detection method based on weakly supervised learning:

[0097] In combination with the above multi-granularity position representation method and constraint conditions, this embodiment proposes a multi-task multi-modal 3D object detection method based on weakly supervised learning. The visible light image and thermal infrared image taken at the same time are input, and the 3D object detection problem is decomposed into two progressive subtasks, and an end-to-end object detection framework is constructed based on this: (1) coarse-grained 3D object detection task; (2) fine-grained 3D key point positioning task.

[0098] Among them, the coarse-grained 3D target detection task is mainly used to estimate the target 3D bounding box position and bounding box confidence and other information; the fine-grained 3D key point positioning task is based on the coarse-grained detection results to further estimate the precise 3D key point coordinates in the local area of ​​the target. The specific technical block diagram is as follows Figure 3 shown.

[0099] 1. Multi-task 3D object detection

[0100] For the coarse-grained 3D target detection task, the present invention uses the structure of a backbone network plus a prediction network. First, a paired RGB-T image is input, and a 4-channel image is obtained by pixel-level fusion. Darknet53 is used as the backbone network to extract the fused image features, and a cross-stage local network is added after each residual block of Darknet53 to effectively enhance the learning ability of Darknet53. The features output by the backbone network are further input into the coarse-grained prediction network, which contains a feature pyramid and a path aggregation network to better utilize the underlying features and enhance the multi-scale prediction ability of the model. Finally, the coarse-grained target 3D bounding box and the corresponding confidence are output. Here, the confidence value range is [0,1], which is used to indicate the probability that the target in the corresponding detection box belongs to the aircraft.

[0101] For the task of localizing the key points of a fine-grained 3D target, the visible light image, the coarse-grained 3D detection box, and the corresponding confidence are first input into the image cropping module at the same time. This module is used to eliminate the large amount of background interference in the visible light image and provide more reliable input data for the localization of the 3D key points of the target. First, the 3D detection box is projected onto the image, and the minimum 2D rectangular box around all the projected points is calculated as the cropping area. A cropping threshold is defined in the image cropping module. When the estimated confidence exceeds the cropping threshold, the image cropping operation is performed on the corresponding cropping area. The cropped image output by the image cropping module will be used as the input of the fine-grained 3D target key point localization module, which includes a feature extraction network based on the ResNet50 structure and a fine-grained prediction network containing a multi-layer perceptron.

[0102] During the training process, the present invention does not directly output the vertex coordinates of the 3D detection frame, but indirectly obtains the vertex coordinates of the 3D detection frame by estimating the 3D coordinates of the center position and the length, width and height of the target. In order to reduce the training task and shorten the training time, the present invention encodes all output variables into residual form, and uses the initial anchor frame, initial key points and the regressed offset to infer the final estimated value.

[0103] 2. 3D object detection based on weakly supervised learning

[0104] The existing method for calculating the initial anchor box is to cluster the reference bounding boxes in the training data set. However, since the training data of the present invention lacks the real 3D position information of the target, the size and position of the initial 3D anchor box cannot be calculated using the above method.

[0105] In view of the particularity of the application scenarios and objects of the present invention, the size of the 3D anchor box is estimated by using the category prior. The initial size information of the target is obtained by statistically analyzing the length, width and height distribution of common aircraft using the KMeans clustering algorithm;

[0106] The initial depth information of the target is determined by estimating the distance from the camera to the runway, and the initial center coordinates of the 3D anchor frame are estimated based on the initial depth. The initial depth of the target is measured by manual interaction. The corresponding point of the runway area on the different modal images taken at the same time is clicked with the mouse, and the 3D coordinates of the point are estimated using the binocular reconstruction principle. The third dimension value of the 3D coordinate is the initial depth value of the corresponding position of the point. Knowing the initial depth value, the corresponding 3D anchor frame center position can be derived by using the pinhole camera imaging principle and combining the 2D anchor frame center coordinates.

[0107] The calculation of the initial key points is solved by introducing the 3D model of the aircraft. First, the rigid transformation matrix from the 3D model to the initial anchor frame is calculated: (1) Scaling: The distance between the two wings of the 3D model and the width of the initial anchor frame are used to estimate the scaling factor; (2) Rotation: The parallel constraint between the line between the two wings of the 3D model and the wide side of the initial anchor frame, as well as the parallel constraint between the nose direction and the long side are used to estimate the rotation matrix; (3) Translation: Estimate the minimum bounding box that completely encloses the 3D model, and calculate the translation vector between the center point of the bounding box and the center point of the initial anchor frame. The above transformation matrix is ​​applied to the 3D feature points of the model to obtain the initial key points.

[0108] In terms of loss function, we design a confidence loss based on cross entropy in combination with the estimated confidence. However, since the training data lacks labeled data for 3D detection boxes and 3D key points, it is impossible to use strong supervised learning to solve it.

[0109] To solve the above problems, the present invention uses consistency constraints of different spaces, different granularities, and different perspectives, and considers the following weak supervision losses: (1) Coarse-grained 2D projection coordinate consistency loss: the estimated coarse-grained 3D bounding box is projected onto the image, a minimum 2D rectangular box is calculated using the projection points, and the L2 distance between the 2D rectangular box and the annotated 2D detection box is estimated; (2) Fine-grained 2D projection coordinate consistency loss: the estimated fine-grained 3D key points are projected onto the visible light image, and the Euclidean distance between the projection points and the annotated 2D key points is calculated; (3) 3D spatial cross-granularity position loss Loss: Assume that the feature point is tangent to the corresponding plane on the bounding box, and calculate the Euclidean distance from the point to the corresponding plane. (4) Multi-view consistency loss: Since the visible light image and the thermal infrared image capture the scene from different perspectives, assuming that the two cameras have been calibrated, the estimated 3D key points are projected onto the thermal infrared image to obtain the corresponding 2D key points. The 2D key points projected on the thermal infrared image and the 2D key points marked on the visible light image are used to construct key point pairs at different perspectives. The key point pairs are reconstructed using the triangulation principle, and the L2 distance between the reconstructed 3D key points and the estimated 3D key points is calculated.

[0110] Embodiment 2: Embodiment 2 of the present invention is based on the same technical concept of the specific implementation method, and further provides a multi-task 3D object detection system based on weakly supervised learning, specifically including:

[0111] The image input module is used to collect visible light video and thermal infrared video taken at the same time and use them as 3D target detection input;

[0112] A task division module is used to decompose the 3D target detection into a coarse-grained 3D target detection task and a fine-grained 3D key point positioning task based on a multi-granularity target 3D position representation method when performing 3D target detection on the 3D target detection input; wherein the coarse-grained 3D target detection task is used to estimate the 3D detection box position and detection box confidence information of the target; the fine-grained 3D key point positioning task is used to estimate the 3D key point coordinates on the local area of ​​the target based on the coarse-grained detection result;

[0113] The target detection module is used to construct the weak supervision loss of the 3D target detection by performing the coarse-grained 3D target detection task and the fine-grained 3D key point positioning task in combination with the pre-established consistency constraint conditions, solve the weak supervision loss based on weak supervision learning, and obtain the target 3D position and 3D key point coordinates in the video.

[0114] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and specification of the present invention.

Claims

1. A multi-granularity and multi-modal 3D object detection method based on weakly supervised learning, characterized in that: The method comprises: Collect visible light video and thermal infrared video taken at the same time and use them as 3D object detection input; When performing 3D object detection on the 3D object detection input, based on a multi-granularity target 3D position representation method, the 3D object detection is decomposed into a coarse-grained 3D object detection task and a fine-grained 3D key point positioning task; wherein the coarse-grained 3D object detection task is used to estimate the target 3D detection box position and detection box confidence information; the fine-grained 3D key point positioning task is used to estimate the 3D key point coordinates on the local area of ​​the target based on the coarse-grained detection result; By performing a coarse-grained 3D object detection task and a fine-grained 3D key point positioning task, combined with a pre-defined consistency constraint condition, a weak supervision loss of the 3D object detection is constructed, and the weak supervision loss is solved based on weak supervision learning to obtain the target 3D position and 3D key point coordinates in the video; The weak supervision loss for constructing 3D object detection includes: coarse-grained 2D projection coordinate consistency loss, fine-grained 2D projection coordinate consistency loss, 3D space cross-granularity position loss and multi-view consistency loss; The multi-granularity position representation method is to divide the target 3D position representation into a coarse-grained global target key point representation and a fine-grained local target key point representation; The coarse-grained global target key point representation describes the 3D position of the overall target; The fine-grained local target key point representation is to describe the local feature points on the target surface; The consistency constraint condition is formulated based on the mutual constraint relationship between the coarse-grained and fine-grained target positions involved in the multi-grained position representation method; The consistency constraints include: 3D space multi-granularity position constraints and 2D projection coordinate consistency constraints; The 2D projection coordinate consistency constraint also includes: a fine-grained 2D projection coordinate consistency constraint and a coarse-grained 2D projection coordinate consistency constraint.

2. The method according to claim 1, characterized in that The execution of the coarse-grained 3D object detection task and the fine-grained 3D key point positioning task includes: Construct an object detection framework for performing coarse-grained 3D object detection tasks and fine-grained 3D key point localization tasks; including a coarse-grained 3D object detection framework and a fine-grained 3D object key point localization framework; The coarse-grained detection results output by the coarse-grained 3D target detection framework are input into the fine-grained 3D target key point positioning framework to output the 3D key point coordinates on the local area of ​​the target.

3. The method according to claim 2, characterized in that The coarse-grained 3D object detection framework includes: a backbone network and a coarse-grained prediction network; The backbone network includes Darknet53 and a cross-stage local network added after each residual block of Darknet53; The coarse-grained prediction network includes a feature pyramid and a path aggregation network.

4. The method according to claim 3, characterized in that The execution of the coarse-grained 3D object detection task includes: Input the paired RGB-T images into the backbone network, obtain a 4-channel image through pixel-level fusion, and extract the features of the fused image; The image features output by the backbone network are input into the coarse-grained prediction network, which outputs the target 3D detection box offset and the corresponding detection box confidence. The confidence value range is [0, 1], which is used to indicate the probability that the target in the corresponding detection box belongs to the specified image feature.

5. The method according to claim 4, characterized in that The execution of the fine-grained 3D target key point positioning task includes: inputting the visible light image, the coarse-grained 3D detection frame and the corresponding detection frame confidence into a predefined image cropping module for image cropping operation to generate input data for fine-grained 3D target key point positioning; wherein the image cropping module includes a feature extraction network based on a ResNet50 structure and a fine-grained prediction network including a multi-layer perceptron; When performing the task of fine-grained 3D target key point localization, the feature vector output by the feature extraction network is input into the fine-grained prediction network to output the target 3D key point offset.

6. The method according to claim 5, characterized in that The input data for generating fine-grained 3D target key point positioning includes: projecting the target 3D detection frame onto the visible light image, and using the minimum 2D rectangular frame surrounding all the projected points as the cropping area; If the confidence of the detection box exceeds a predefined cropping threshold, the corresponding cropping area is cropped, and the cropped image is input into the fine-grained 3D object key point positioning framework as input data for performing fine-grained 3D object key point positioning.

7. The method according to claim 1, characterized in that Solving the weak supervision loss based on weak supervision learning to obtain the target 3D position and 3D key point coordinates in the video includes: encoding all output variables obtained by coarse and fine-grained 3D target detection into residual form, and using the initial anchor frame, initial key point and regressed offset to infer the final estimated value; The estimation of the size of the 3D anchor frame includes: statistically analyzing the length, width and height distribution of common aircraft, and using the KMeans clustering algorithm to determine the initial size information of the target; and determining the initial depth of the target by measuring the distance from the camera to the runway. The initial center coordinates of the 3D anchor frame are estimated based on the initial depth of the target. The center position of the 3D anchor frame is derived by combining the center coordinates of the 2D anchor frame using the imaging principle of a pinhole camera. The initial key points are obtained by calculating the predefined aircraft 3D model: calculating the rigid transformation matrix from the aircraft 3D model to the initial anchor frame, applying the rigid transformation matrix to the model 3D feature points to obtain the initial key points; The step of calculating the rigid transformation matrix from the aircraft 3D model to the initial anchor frame includes: estimating a scaling factor using a pre-established distance between two wings of the aircraft 3D model and a width of the initial anchor frame; The rotation matrix is ​​estimated using the parallel constraint between the line connecting the two wings of the aircraft 3D model and the wide side of the initial anchor frame, and the parallel constraint between the nose direction and the long side; Estimate the minimum detection box that completely encloses the aircraft 3D model, and calculate the translation vector between the center point of the detection box and the center point of the initial anchor box.

8. The method according to claim 7, characterized in that The coarse-grained 2D projection coordinate consistency loss is used to project the estimated coarse-grained 3D bounding box onto the image, calculate a minimum 2D rectangular box using the projection points, and estimate the L2 distance between the 2D rectangular box and the annotated 2D detection box; The fine-grained 2D projection coordinate consistency loss is used to project the estimated fine-grained 3D key points to the visible light image and calculate the Euclidean distance between the projected points and the annotated 2D key points; The 3D spatial cross-granularity position loss is used to assume that the feature point is tangent to the corresponding plane on the bounding box and calculate the Euclidean distance from the point to the corresponding plane; The multi-view consistency loss is used to assume that the two cameras have been calibrated based on the different perspectives of the scene captured by the visible light image and the thermal infrared image. The estimated 3D key points are projected onto the thermal infrared image to obtain the corresponding 2D key points. The 2D key points projected on the thermal infrared image and the 2D key points marked on the visible light image are used to construct key point pairs at different perspectives. The key point pairs are reconstructed using the triangulation principle, and the L2 distance between the reconstructed 3D key points and the estimated 3D key points is calculated.

9. A multi-granularity and multi-modal 3D object detection system based on weakly supervised learning, characterized in that: include: The image input module is used to collect visible light video and thermal infrared video taken at the same time and use them as 3D target detection input; A task division module is used to decompose the 3D target detection into a coarse-grained 3D target detection task and a fine-grained 3D key point positioning task based on a multi-granularity target 3D position representation method when performing 3D target detection on the 3D target detection input; wherein the coarse-grained 3D target detection task is used to obtain the 3D detection box position and detection box confidence information of the target; the fine-grained 3D key point positioning task is used to estimate the 3D key point coordinates on the local area of ​​the target based on the coarse-grained detection result; The object detection module is used to construct a weak supervision loss for the 3D object detection by performing a coarse-grained 3D object detection task and a fine-grained 3D key point positioning task, combined with a pre-defined consistency constraint condition, solve the weak supervision loss based on weak supervision learning, and obtain the object 3D position and 3D key point coordinates in the video; The weak supervision loss for constructing 3D object detection includes: coarse-grained 2D projection coordinate consistency loss, fine-grained 2D projection coordinate consistency loss, 3D space cross-granularity position loss and multi-view consistency loss; The multi-granularity position representation method is to divide the target 3D position representation into a coarse-grained global target key point representation and a fine-grained local target key point representation; The coarse-grained global target key point representation describes the 3D position of the overall target; The fine-grained local target key point representation is to describe the local feature points on the target surface; The consistency constraint condition is formulated based on the mutual constraint relationship between the coarse-grained and fine-grained target positions involved in the multi-grained position representation method; The consistency constraints include: 3D space multi-granularity position constraints and 2D projection coordinate consistency constraints; The 2D projection coordinate consistency constraint also includes: a fine-grained 2D projection coordinate consistency constraint and a coarse-grained 2D projection coordinate consistency constraint.

Images