A multi-target pose real-time online estimation method, system, device and medium

Abstract

The application discloses a multi-target pose real-time online estimation method, system, device and medium, and aims at real-time aerial images. On one hand, the ground target pose is quickly estimated through block detection and head-to-tail interconnection. On the other hand, the ground target is tracked in real time online in combination with a data association thought. The problems of small target volume, large target quantity, easy to be shielded, dynamic non-cooperation and limited calculation capacity of embedded equipment in the aerial images are effectively overcome. The efficiency of aerial multi-target pose estimation is improved. The aerial ground target online real-time estimation in the airborne embedded environment is realized. The multi-target, multi-dimensional and strong real-time requirements of ground target detection and sensing in the fields of disaster rescue and military monitoring are met.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent visual perception of unmanned systems, in particular to a multi-target pose real-time online estimation method system, device and medium. BACKGROUND

[0002] In recent years, with the rapid development of unmanned aerial vehicle technology, due to its strong mobility, low cost, variable monitoring height and other advantages, unmanned aerial vehicles have been widely used in disaster relief, military monitoring and other fields. In view of the increasingly diverse unmanned aerial vehicle detection and perception tasks, combined with the small size, light weight, low power consumption and low cost of visual sensors, the unmanned perception system further puts forward the demand for multi-target, multi-dimensional and strong real-time capability to support the instant perception of various types and quantities of disaster relief / monitoring target position and attitude information. Real-time estimation of aerial multi-target pose has become an important basis for perfecting the perception and detection of target information, and is a key link to improve the intelligent level of the unmanned aerial vehicle perception system itself.

[0003] At present, due to the limited number of payloads and information processing capability of the unmanned aerial vehicle perception system, as well as the small size, large quantity, dynamic non-cooperation and easy to be shielded of the aerial target, the real-time online estimation of aerial multi-target pose based on embedded devices is difficult to achieve. SUMMARY

[0004] The present application aims to provide a multi-target pose real-time online estimation method to overcome the problem that the real-time online estimation of aerial multi-target pose based on embedded devices in the prior art is difficult to achieve.

[0005] To achieve the above-mentioned purpose, the present application provides the following technical scheme:

[0006] A multi-target pose real-time online estimation method, comprising the following steps:

[0007] S1: Collect aerial vehicle images from different heights and angles, and label the vehicle head and vehicle in the images to generate an aerial vehicle training set, and train the aerial vehicle training set to generate a target detection model;

[0008] S2: Lighten the target detection model, and deploy the lightened target detection model on a GPU embedded development board;

[0009] S3: Install the GPU embedded development board with a gimbal camera on an unmanned aerial vehicle, acquire aerial images, and input the acquired aerial images into the lightened target detection model to obtain a target detection result;

[0010] S4: Calculate the overlap rate of the vehicle head and vehicle detection frame according to the target detection result, realize the association of the vehicle and the corresponding vehicle head, then connect the centroids of the vehicle and the vehicle head, and calculate the attitude angle of the vehicle.

[0011] S5: According to the target detection result, the Euclidean distance is taken as the similarity measurement between adjacent frames of targets, a correlation matrix is established, a global optimal matching is solved, the optimal matching result is screened, a target tracking result is obtained, the target tracking result is matched with the vehicle attitude angle, and a real-time pose of the target is obtained.

[0012] Preferably, the YOLOv5s network is used in S1 to train the aerial vehicle training set.

[0013] Preferably, the target detection model takes CSPDarknet-53 added with a Focus structure as a backbone network.

[0014] Preferably, the overlap rate calculation formula of the vehicle head and the vehicle detection frame is:

[0015]

[0016] wherein, are the areas of the detection frames of the vehicle head i and the vehicle j respectively.

[0017] Preferably, the calculation formula of the vehicle attitude angle is:

[0018]

[0019] wherein, (x1, y1) and (x2, y2) are the center of mass coordinates of the vehicle head and the vehicle respectively.

[0020] Preferably, the correlation matrix is:

[0021]

[0022] wherein, denotes the i-th vehicle detected in the t-th frame, M and N denote the number of targets detected in the t-th frame and the t+1-th frame respectively, denotes the Euclidean distance between the vehicle .

[0023] Preferably, the global optimal matching is solved by using the Hungarian algorithm.

[0024] A multi-target pose real-time online estimation system based on the above-mentioned multi-target pose real-time online estimation method, comprising:

[0025] A training set module: used for collecting aerial vehicle images from different heights and angles, labeling the vehicle head and the vehicle in the images, generating an aerial vehicle training set, training the aerial vehicle training set, and generating a model;

[0026] A deployment module is configured to lighten the target detection model and deploy the lightened target detection model on a GPU embedded development board.

[0027] A detection module is configured to install the GPU embedded development board with the gimbal camera on the unmanned aerial vehicle, acquire aerial images, input the acquired aerial images into the lightened target detection model, and obtain target detection results.

[0028] A calculation module is configured to calculate the overlap rate of the vehicle detection frame and the vehicle detection frame according to the target detection results, associate the vehicle and the corresponding vehicle head, connect the center of mass of the vehicle and the vehicle head, and calculate the attitude angle of the vehicle.

[0029] A pose acquisition module is configured to establish a correlation matrix by taking the Euclidean distance as the similarity measurement between adjacent frames of targets according to the target detection results, solve the global optimal matching, filter the optimal matching results, obtain target tracking results, match the target tracking results with the vehicle attitude angle, and obtain the real-time pose of the target.

[0030] A computer device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of a multi-target pose real-time online estimation method when executing the computer program.

[0031] A computer readable storage medium stores a computer program, and the computer program implements the steps of a multi-target pose real-time online estimation method when executed by a processor.

[0032] Compared with the prior art, the present application has the following beneficial effects: the present application provides a multi-target pose real-time online estimation method, which is aimed at real-time aerial images. On the one hand, the method realizes fast estimation of the ground target pose through block detection and head-to-tail interconnection. On the other hand, the method combines the data association idea to track the ground target online, effectively overcomes the problems of small target volume, large number of targets, easy occlusion, dynamic non-cooperation, limited computing capacity of embedded devices, etc. in aerial images, improves the efficiency of aerial multi-target pose estimation, realizes online real-time estimation of aerial ground targets in an airborne embedded environment, and further meets the multi-target, multi-dimensional, and strong real-time requirements of ground target detection and perception in disaster rescue, military monitoring, and other fields.

[0033] The present application provides a multi-target pose real-time online estimation system, which can realize the above-mentioned method and improve the efficiency of aerial multi-target pose estimation. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 is a schematic diagram of an offline preprocessing process of the target detection model of the present application.

[0035] Figure 2 This is a flowchart of the real-time online multi-target pose estimation process of the present invention;

[0036] Figure 3 This is the result of the static target pose estimation of the present invention;

[0037] Figure 4 This is the result of the motion target pose estimation of the present invention. Detailed Implementation

[0038] The present invention will be further described in detail below with reference to specific embodiments. These descriptions are for explanation purposes only and are not intended to limit the scope of the invention.

[0039] This invention provides a real-time online method for estimating the pose of multiple targets.

[0040] like Figure 1 As shown, steps 1 and 2 are offline preprocessing processes; steps 3, 4, and 5 are real-time online multi-target pose estimation processes, and their flowcharts are as follows: Figure 2 As shown.

[0041] Step 1: To improve the accuracy of object detection, an aerial image dataset of vehicles was constructed. This dataset contains over 20,000 images, covering various complex scenes such as streets, parks, and highways. All images were captured by a rotary-wing drone equipped with a gimbal camera from different angles and flight altitudes, ranging from approximately 50 to 500 meters. Then, the vehicle fronts and other vehicle details in the aerial images of this self-built dataset were manually labeled. Finally, based on this self-built training dataset, a YOLOv5s network was selected for training, and weights were generated.

[0042] Step 2: After the object detection model is trained, in order to accelerate the inference process, the model generated in Step 1 is compressed and optimized, and then deployed on the GPU embedded development platform.

[0043] Step 3: Real-time aerial images are acquired using a drone equipped with a gimbal camera and input into the target detection model. This network uses a CSPDarknet-53 network with an added focus structure as its backbone. Simultaneously, it combines a feature pyramid network and a path aggregation network for multi-scale feature fusion, improving the feature extraction capability of the neural network and obtaining feature maps at three different scales. These three feature maps are used for prediction, and a non-maximum constraint method is used for filtering. Untrusted targets are then eliminated based on the confidence level of the detected targets.

[0044] Step 4: Calculate the overlap rate between the front of the vehicle and the detection frame of each vehicle.

[0045]

[0046] in, Let i be the area of ​​the detection bounding box for vehicle i and vehicle j, respectively. Then, find the vehicle with the maximum overlap with that vehicle i.

[0047]

[0048] Here, N represents the number of detected vehicles. The overlap rate is checked against a set threshold (0.6 in this paper). If it does, the vehicle's front end is identified as the front end of the current vehicle, thus establishing the association between the vehicle and its corresponding front end. Finally, the centroids of the front end and the vehicle are connected, and the vehicle's attitude angle θ is calculated.

[0049]

[0050] Where (x1,y1) and (x2,y2) are the centroid coordinates of the front of the vehicle and the vehicle itself, respectively.

[0051] Step 5: Combining the target detection results from Step 3, and using Euclidean distance as the similarity measure between targets in adjacent frames, establish the correlation matrix S.

[0052]

[0053] in, Let M represent the i-th vehicle detected in frame t, and let M and N represent the number of targets detected in frame t and frame (t+1), respectively.

[0054] Indicates vehicle The Euclidean distance between them is defined as follows:

[0055]

[0056] In the above formula, and Let i and j be the centroid coordinates of the detection boxes corresponding to targets i and j.

[0057] After obtaining the correlation matrix, it is input into the Hungarian algorithm to obtain the global optimal match of vehicles between adjacent frames. The matching results are then filtered to remove false matches, and the target tracking results are obtained. The target tracking results are then matched with the vehicle attitude angles to obtain the real-time pose of the target.

[0058] A multi-target pose real-time online estimation system includes:

[0059] Training set module: Used to acquire aerial vehicle images from different heights and angles, label the front of the vehicle and the vehicle in the images, generate an aerial vehicle training set, and train the aerial vehicle training set to generate a model;

[0060] A deployment module is configured to compress and optimize the model to generate a target detection model and deploy the target detection model on a GPU embedded development board.

[0061] A detection module is configured to install the GPU embedded development board with the gimbal camera on a UAV, acquire aerial images, input the acquired aerial images into the target detection model, and obtain a target detection result.

[0062] A calculation module is configured to calculate the overlap rate of the vehicle detection frame and the vehicle head according to the target detection result, associate the vehicle and the corresponding vehicle head, connect the center of mass of the vehicle and the vehicle head, and calculate the attitude angle of the vehicle.

[0063] A pose acquisition module is configured to establish a correlation matrix by taking the Euclidean distance as the similarity measure between adjacent frames of targets according to the target detection result, solve the global optimal matching, filter the optimal matching result, obtain a target tracking result, match the target tracking result with the vehicle attitude angle, and obtain the real-time pose of the target.

[0064] In another embodiment of the present application, a computer device is provided, which comprises a processor and a memory, the memory is configured to store a computer program, the computer program comprises program instructions, and the processor is configured to execute the program instructions stored in the computer storage medium. The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more instructions in the computer storage medium to realize the corresponding method flow or corresponding function. The processor in the embodiment of the present application can be used for the operation of the glass channel welding type thermocouple temperature compensation method.

[0065] The application further provides a storage medium, specifically a computer readable storage medium (Memory), which is a memory device in a computer device and is used for storing programs and data. It can be understood that the computer readable storage medium herein can include a built-in storage medium in the computer device, and of course can also include an extended storage medium supported by the computer device. The computer readable storage medium provides a storage space, which stores an operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and the instructions can be one or more computer programs (including program codes). It should be noted that the computer readable storage medium herein can be a high-speed RAM memory or a non-volatile memory, for example, at least one disk memory. One or more instructions stored in the computer readable storage medium can be loaded and executed by the processor to realize the corresponding steps of the glass channel welding type thermocouple temperature compensation method in the above embodiments.

[0066] Those skilled in the art will appreciate that embodiments of the application can be supplied as a method, a system, or a computer program product. Therefore, the application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage media, etc.) containing computer-usable program code.

[0067] The application is described with reference to flowcharts and / or block diagrams of the method, device (system), and computer program product according to embodiments of the application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device that implements the flowcharts and / or block diagrams. Figure 1 The function specified in one or more flows and / or blocks Figure 1 The device that implements the function specified in one or more flows or blocks.

[0068] These computer program instructions can also be stored in a computer readable memory that can guide the computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer readable memory produce a product including instruction devices that implement the flowcharts and / or block diagrams. Figure 1 The function specified in one or more flows and / or blocks Figure 1the function specified in the one or more blocks.

[0069] These computer program instructions can also be loaded into computer or other programmable data processing devices, so that a series of operation steps are performed on the computer or other programmable data processing devices to generate computer-implemented processes, so that the instructions executed on the computer or other programmable data processing devices provide processes for implementing the flows Figure 1 the flows or the plurality of flows and / or blocks Figure 1 the steps of the function specified in the one or more blocks.

[0070] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A multi-target pose real-time online estimation method, characterized in that, The method comprises the following steps: S1: Collect aerial vehicle images from different heights and angles, label the vehicle head and vehicle in the images, generate an aerial vehicle training set, train the aerial vehicle training set, and generate a target detection model; S2: Lighten the target detection model, and deploy the lightened target detection model on a GPU embedded development board; S3: Install the GPU embedded development board with a gimbal camera on a UAV, obtain aerial images, input the obtained aerial images into the lightened target detection model, and obtain a target detection result; S4: Calculate the overlap rate of the vehicle head and each vehicle detection box according to the target detection result, calculate the vehicle with the maximum overlap rate with the vehicle head, and determine whether the maximum overlap rate reaches a set threshold value, if yes, connect the centroids of the vehicle and the vehicle head, and calculate the attitude angle of the vehicle; S5: According to the target detection result, take the Euclidean distance as the similarity measurement between adjacent frames of targets, establish a correlation matrix, solve the global optimal matching, filter the optimal matching result, obtain a target tracking result, match the target tracking result with the vehicle attitude angle, and obtain the real-time pose of the target.

2. The multi-target pose real-time online estimation method according to claim 1, characterized in that, In S1, the YOLOv5s network is used to train the aerial vehicle training set.

3. The method of claim 1, wherein, The lightening of the target detection model is a compression and optimization of the target detection model.

4. The multi-target pose real-time online estimation method according to claim 1, characterized in that, The overlap rate calculation formula of the vehicle head and the vehicle detection box is: wherein, , are the areas of the bounding boxes of the vehicle head i and vehicle j, respectively.

5. The method of claim 1, wherein, The calculation formula of the vehicle attitude angle is: wherein, and are the vehicle head and center of mass coordinates, respectively.

6. The method of claim 1, wherein, The correlation matrix is: wherein, denotes the i-th vehicle detected in the t-th frame, M and N denote the number of targets detected in the t-th and t+1-th frames, respectively, denotes the Euclidean distance between vehicles .

7. The method of claim 1, wherein, The global optimal matching is solved by using the Hungarian algorithm.

8. A multi-target pose real-time online estimation system based on any one of the multi-target pose real-time online estimation methods of claims 1-7. It comprises: The training set module is used to collect aerial vehicle images from different heights and angles, label the vehicle head and vehicle in the images, generate an aerial vehicle training set, train the aerial vehicle training set, and generate a model; The deployment module is used to lighten the target detection model, and deploy the lightened target detection model on a GPU embedded development board; The detection module is used to install the GPU embedded development board with a gimbal camera on a UAV, obtain aerial images, input the obtained aerial images into the lightened target detection model, and obtain a target detection result; The calculation module is used to calculate the overlap rate of the vehicle head and each vehicle detection box according to the target detection result, realize the correlation between the vehicle and the corresponding vehicle head, then connect the centroids of the vehicle and the vehicle head, and calculate the attitude angle of the vehicle; The pose acquisition module is used to take the Euclidean distance as the similarity measurement between adjacent frames of targets according to the target detection result, establish a correlation matrix, solve the global optimal matching, filter the optimal matching result, obtain a target tracking result, match the target tracking result with the vehicle attitude angle, and obtain the real-time pose of the target.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to realize the steps of the multi-target pose real-time online estimation method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9. The computer program is executed by the processor to realize the steps of the multi-target pose real-time online estimation method according to any one of claims 1 to 7.

Images