A method and system for target detection and three-dimensional positioning based on unmanned aerial vehicle

Abstract

A method for target detection and real-time three-dimensional positioning based on drones, including: recording multiple current images through drones, extracting targets based on the images, and sorting and classifying the targets; calculating the center position and category of a certain target in the image based on the effective output obtained; calculating the accurate position of the image center in the world coordinate system with the help of the three-dimensional coordinate system of the three-dimensional world by knowing the flight altitude and the drone gimbal posture; obtaining the three-dimensional coordinates of the target in the camera coordinate system according to the zoom ratio of the camera itself and the physical parameters of the camera corresponding to the image, and correcting the data; obtaining the latitude and longitude of each target of interest in the real world coordinate system through the geographical location of the drone and the posture of the drone relative to the camera, so as to achieve the purpose of three-dimensional positioning of the target by the high-altitude drone. The method solves the problem that long-distance target positioning and stable control of drones cannot be achieved in the prior art.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a target detection and three-dimensional positioning method and system based on an unmanned aerial vehicle. Background Art

[0002] In computer vision, object detection and depth estimation of objects are both fundamental problems. Object detection has achieved fruitful results from traditional methods to neural network-based methods. At present, deep learning models have been widely used in the entire field of computer vision, including general object detection and domain-specific object detection. Most of the most advanced object detectors use deep learning networks as their backbone and detection networks to extract features from input images (or videos) for classification and positioning, respectively. As an important part of scene understanding, object detection is widely used in many fields of modern life, such as security, military, transportation, medical and life. However, for small drones, the lack of system resources makes the network run slowly, and completing real-time, fast and effective object detection is a test of this task.

[0003] For the task of target depth estimation. In traditional methods, depth estimation based on stereo vision is solved by matching manually extracted features on multiple images. Despite extensive research, these traditional methods are still affected by complex texture areas, large indistinguishable areas, and occlusion. It is a long-standing ill-posed problem that has applications in many fields, such as robotics, autonomous driving, object recognition, scene understanding, 3D modeling and animation, augmented reality, industrial control, and medical diagnosis. This problem has been widely studied for decades. Among them, stereo matching is traditionally the most studied one because it is closely related to human binocular vision. Due to the success of using deep learning to solve the problem of depth estimation based on stereo vision in solving various 2D and 3D vision problems, the method of using deep learning to solve the problem of depth estimation based on stereo vision has received widespread attention. Between 2014 and 2019, more than 150 papers were published in this field. However, due to the huge resource consumption, the use of deep learning cannot be effectively applied in mobile devices. In traditional binocular positioning algorithms, due to the baseline, it is impossible to achieve long-distance target positioning. Secondly, there is a stability and control effect during the operation of drones. Therefore, the three-dimensional positioning of the drone's target must work normally under the interference of factors such as the drone's stability and resources, which poses great technical challenges. Summary of the invention

[0004] In view of the above problems, the present invention is proposed to provide a method and system for target detection and three-dimensional positioning based on a drone, which overcomes the above problems or at least partially solves the above problems.

[0005] In order to solve the above technical problems, the embodiments of the present application disclose the following technical solutions:

[0006] A method for target detection and real-time three-dimensional positioning based on a drone, comprising:

[0007] S100. Preset the tilt angle of the drone gimbal, and under the condition of determining the latitude and longitude position and altitude, record the current multiple images through the drone, extract the target object according to the obtained image, and sort and classify the target;

[0008] S200. Calculate the center position and category of a certain target in the image according to the valid output obtained in S100, including the danger category and the pixel position of the target;

[0009] S300. By knowing the flight altitude and the gimbal posture of the drone, the accurate position of the image center in the world coordinate system is calculated with the help of the three-dimensional coordinate system of the three-dimensional world;

[0010] S400. According to the zoom ratio of the camera itself and the physical parameters of the camera corresponding to the image, the three-dimensional coordinates of the target in S300 in the camera coordinate system are obtained, and the data is corrected using polynomial optimization;

[0011] S500. The latitude and longitude of each target of interest in the real world coordinate system are finally obtained through the geographical location of the drone and the position of the drone relative to the camera, so as to achieve the three-dimensional positioning purpose of the high-altitude drone for the target.

[0012] Furthermore, in S100, the drone records the current multiple images, extracts the target objects according to the obtained images, and sorts and classifies the targets. Specifically, the image is collected in real time by the drone camera and transmitted back to the CNN network running in the background in real time.

[0013] Furthermore, in S100, effective detection of the image is performed, and the stable output of multiple frames of images is used as the judgment standard. If a stable image cannot be obtained, return to S100, zoom the camera, and reacquire the image. After the continuous multi-frame target is determined, the multiple output results are used as the effective output.

[0014] Further, in S300, the method for calculating the accurate position of the image center in the world coordinate system is: using the external parameter matrix (R pw ,T pw ), the drone coordinate system and the onboard camera coordinate system (R cp ,T cp ) to perform the corresponding transformation, and the exact position transformation formula of the image center in the world coordinate system is as follows:

[0015] P w =R pw P p +T pw

[0016] P p =R cp P c +T cp

[0017] Among them, P w is the world coordinate system, P p is the UAV coordinate system, P c is the onboard camera coordinate system, R pw is the rotation matrix between the world coordinate system and the drone coordinate system, T pw is the translation matrix between the world coordinate system and the drone coordinate system, R cp T is the rotation matrix between the airborne camera coordinate system and the drone coordinate system; cp It is the translation matrix between the airborne camera coordinate system and the drone coordinate system.

[0018] Furthermore, the translation matrix T between the world coordinate system and the drone coordinate system is pw =(0,0,0), the rotation matrix R between the world coordinate system and the drone coordinate system pw The calculation method is: rotate the image in the order of ZYX axis, the three-axis rotation angles are γ, β, α, and define the sine and cosine values ​​corresponding to the three-axis rotation angles as s α ,s β ,s γ ,c α ,c β ,c γ , then the rotation matrix R between the world coordinate system and the drone coordinate system pw for:

[0019]

[0020] Furthermore, in S300, it is also necessary to obtain the distance between the center of the image pixel and the origin of the airborne camera. The distance between the center of the image pixel and the origin of the airborne camera is calculated using the formula:

[0021]

[0022] Among them, h plane is the altitude of the drone, h ground is the ground altitude, s β is the cosine of the rotation angle about the y-axis.

[0023] Furthermore, the implementation method of S400 is as follows: the preset image center coordinates are (u0, v0), the actual target center point coordinates are (u, v), and the corresponding coordinates in the airborne camera coordinate system are (x0, y0) and (x, y). The actual positions of the target pixel and the center pixel on the imaging plane can be calculated according to the pixel size, and the calculation is as follows:

[0024] dx=pixel×(u-u0)

[0025] dy=pixel×(v-v0)

[0026] Where dx and dy represent the actual physical distance between the target pixel and the center pixel.

[0027] Furthermore, after calculating the actual position, the position of the target pixel in the onboard camera coordinate system can be obtained according to the camera's intrinsic parameter matrix, including: After obtaining the azimuth and pitch angle of the target pixel, the rotation matrix R' from the onboard camera coordinate system to the drone coordinate system can be recalculated cp This rotation matrix is ​​equivalent to the rotation matrix when the camera positions the target at the center of the image when shooting. The actual distance of the target pixel relative to the camera origin can be obtained. The coordinate system coordinate point can be obtained according to the angle c After obtaining the valid coordinates, use the rotation matrix R' cp and the invariant translation matrix T cp =(x0, y0, z0) can obtain the actual coordinate Tar of the target pixel in the drone coordinate system, which is calculated as follows:

[0028] Tar=R' cp *point c +T cp

[0029] Among them, Tar is the actual coordinate of the target pixel in the drone coordinate system, R' cp The recalculated rotation matrix from the airborne camera coordinate system to the drone coordinate system, point c To obtain the coordinate system coordinates based on the angle, T cp is the translation matrix.

[0030] Furthermore, the specific method of S500 is: after the world coordinates of the target object in the world coordinate system defined in S300, the Z-axis coordinates in the world coordinates are removed, the actual distance is obtained through the x-axis and y-axis in the world coordinate system, and the positioning of the target object in the real world can be obtained by converting the distance and longitude and latitude.

[0031] The present invention also discloses a target detection and real-time three-dimensional positioning system based on a drone, comprising: a target extraction module, an image center point coordinate acquisition module, a world coordinate system coordinate conversion module, a camera coordinate system conversion module, and a three-dimensional positioning module; wherein:

[0032] The target extraction module is used to record multiple images through the drone, extract the target objects according to the obtained images, and sort and classify the targets;

[0033] The image center point coordinate acquisition module is used to calculate the center position and category of a certain target in the image based on the effective output obtained by the target extraction module, including the danger category and the pixel position of the target;

[0034] The world coordinate system coordinate conversion module is used to calculate the exact position of the image center in the world coordinate system by using the three-dimensional coordinate system of the three-dimensional world through the known flight altitude and the gimbal posture of the drone;

[0035] The camera coordinate system conversion module is used to obtain the three-dimensional coordinates of the target in the camera coordinate system according to the zoom ratio of the camera itself and the physical parameters of the camera corresponding to the image, and to correct the data using polynomial optimization;

[0036] The three-dimensional positioning module is used to obtain the latitude and longitude of each target of interest in the real-world coordinate system through the geographical location of the drone and the position of the drone relative to the camera, so as to achieve the three-dimensional positioning of the target by the high-altitude drone.

[0037] The beneficial effects of the above technical solution provided by the embodiment of the present invention include at least:

[0038] The present invention discloses a method for target detection based on monocular unmanned aerial vehicle images and positioning a specific target in a three-dimensional world. In the target detection part, a fast and efficient method is used to complete the detection of specific targets. While avoiding global depth estimation of the image, the present invention only performs depth estimation on the specific target. Since the height difference of a small part of the ground target relative to the unmanned aerial vehicle is not significantly affected when the unmanned aerial vehicle is flying at high altitudes. Secondly, there is a significant difference between the viewing angle facing the ground and the viewing angle under the offset of the unmanned aerial vehicle gimbal. The present invention adjusts the image through the attitude position of the unmanned aerial vehicle. And the accurate position of the specific point in the two-dimensional image in the three-dimensional world is obtained according to the three-dimensional geometric solution of the real world. It solves the problem in the prior art that the traditional binocular positioning algorithm is limited by the baseline and cannot achieve long-distance target positioning and the problem that the unmanned aerial vehicle is affected by stability and control during operation.

[0039] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0041] Figure 1 This is a flow chart of a target detection and three-dimensional positioning method based on a drone in Example 1 of the present invention;

[0042] Figure 2 A flow chart of generating a pre-trained neural network model in Embodiment 1 of the present invention;

[0043] Figure 3 This is a schematic diagram of the conversion between drone coordinates and world coordinates in Example 2 of the present invention;

[0044] Figure 4 A schematic diagram of the rotation order and angle of the coordinate system in Example 2 of the present invention;

[0045] Figure 5 This is a schematic diagram of target distance determination based on the camera-world rotation matrix in Example 2 of the present invention;

[0046] Figure 6 is the direction x from the camera optical center to the center pixel in Embodiment 2 of the present invention. c and camera optical center - target pixel direction x' c Schematic diagram of the geometric relationship. DETAILED DESCRIPTION

[0047] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0048] In order to solve the problems existing in the prior art, an embodiment of the present invention provides a target detection and three-dimensional positioning method and system based on a drone.

[0049] Example 1

[0050] This embodiment discloses a method for target detection and real-time three-dimensional positioning based on a drone. Figure 1 ,include:

[0051] S100. Preset the tilt angle of the drone gimbal, and under the condition of determining the latitude and longitude position and altitude, record the current multiple images through the drone, extract the target object based on the obtained image, and sort and classify the target. In this embodiment, S100 specifically includes: through the real-time acquisition work of the drone camera, the image is transmitted back to the CNN network running in the background in real time. This embodiment will also effectively detect the image, using the stable output of multiple frames of images as the judgment standard. If a stable image cannot be obtained, return to S100, zoom the camera, and reacquire the image. After the continuous multi-frame target is determined, the multiple output results are used as the effective output.

[0052] Specifically, during the normal flight of the UAV, its flight speed v is set, the three axes of the gimbal operate periodically at a fixed angular velocity ωrad / s, and the target detection neural network is started at the same time. The target detection part uses the yoloV4 algorithm in the one-stage method as a benchmark. Using CSPDarknet53 as the backbone network benchmark, SPP as an additional module of Neck, and PANet as the feature fusion module of Neck, the speed is extremely fast. However, due to the high flight altitude of the UAV and the small target object, discontinuity may occur during the detection process, and it will also be affected by lighting. In this case, the target detection task needs to seek higher-definition input. The present invention uses the intermittent detection results that appear in the early detection task as the suspicious target ST i Perform precision detection and obtain the ST of suspicious targets i After the image position is determined, the suspicious target is rotated according to the difference in the center point of the pixel position of the suspicious target to locate it in the center of the image, and the camera is magnified 1.5x, and the suspicious target is confirmed in this iteration. If the target Target is determined in multiple consecutive frames, the specific position of the target object is further collected. Since yoloV4 is fast and can meet the real-time requirements, the accurate pixel position of the target object and the danger category of the target are confirmed by averaging multiple consecutive frames of images, and refer to Figure 1 Jump to the next step. If the suspicious target is not further confirmed, return to the previous step of the task, the camera is re-zoomed back to 1x, the gimbal continues to operate normally, and further confirms the next suspicious target. In the target detection part, we focus on the accuracy and precision of the position in the target detection, and need to use an iterative method to perform multiple stable and effective detections.

[0053] S200. According to the valid output obtained in S100, including the danger category and the pixel position of the target, calculate the center position and category of a certain determined danger target in the image; specifically, according to the valid output obtained in S100, including the danger category and the pixel position (x, y, width, height) of the target, where (x, y) is the pixel coordinate of the upper left corner of the target, and (width, height) is the length and width of the target, calculate the center position and category of a certain determined danger target in the image, and set it as label (x center ,y center ,k), where k is the category, preparing for the next step of calculation.

[0054] S300. By knowing the flight altitude and the gimbal position of the drone, the exact position of the image center in the world coordinate system is calculated with the help of the three-dimensional coordinate system of the three-dimensional world.

[0055] Specifically, when calculating the position of the target object in the world coordinate system, some prior information is needed as a guarantee. We define the following known information: the world coordinate system P w , UAV coordinate system P p , the airborne camera coordinate system P c , the heading angle of the drone plane yaw , pitch angle plane pitch , roll angle plane roll , the pitch angle of the airborne gimbal Camera pitch , heading angleCamera yaw Since the camera will not actively spin, the roll angle information can be ignored. For the camera, the distortion parameters and internal parameters need to be known. The altitude of the drone is h plane and ground altitude h ground , because it is necessary to maintain stability when performing precise target detection, the influence of natural wind is ignored here, and the drone is assumed to remain stationary and stable. The main task of this step is to complete the actual position of the center point of the image in the world coordinate system, that is, to use the effective conversion in three different coordinate systems, see Figure 2 and Figure 3 , by measuring and utilizing the onboard camera's own information, distortion parameters and internal parameters, including the pitch angle, heading angle and height above the ground relative to the horizon, the position of the target in the camera coordinate system is calculated. However, in actual practice, it is impossible to directly obtain the position of an ordinary camera relative to the world coordinate system. However, on the UAV system, the specific position of the UAV in the world coordinate system can be effectively obtained. The relative position of the gimbal and the UAV is fixed and can be effectively converted. The world coordinate system coordinates of the center point can be obtained through the conversion of three different coordinate systems.

[0056] In this embodiment, the specific conversion method is as follows: using the external parameter matrix (R pw ,T pw ), the drone coordinate system and the onboard camera coordinate system (R cp ,T cp ) to perform corresponding conversions. For specific models, see Figure 1 and Figure 2 , the conversion formula is as follows:

[0057]

[0058] World Coordinate System P w It should correspond to the division of the world's longitude and latitude lines. For the setting of the world coordinate system, see Figure 1 According to the right-hand rule, the positive directions of the three axes are defined as follows: north along the meridian is x w The positive direction of the axis is y, which is eastward along the latitude direction. w The positive direction of the axis is z, with the horizon pointing upwards w The positive direction of the three axes of the drone is defined as follows: the forward direction of the drone during flight is x p The positive direction of the axis is the same as the x p The right direction of the vertical axis is y p The positive direction of the axis is p The downward direction perpendicular to the axis is z p The positive direction of the three axes of the camera is defined in Figure 2 , with the camera shooting direction as x c The positive direction of the axis is the same as the x c The right direction of the vertical axis is y c The positive direction of the axis is the same as the x c The axis is vertically downwards, z c The positive direction of the axis, where the shooting direction of the airborne camera is defined as a vector passing through the optical center of the camera and the center pixel of the image.

[0059] Since the world coordinate system is artificially defined, the present invention makes the world coordinate system coincide with the origin of the drone coordinate system, thereby simplifying the conversion between the drone coordinate system and the world coordinate system, that is, the translation matrix T pw =(0,0,0), rotation matrix R pw See the calculation of Figure 2 In addition, since the origins coincide, the rotation is three-axis and can be distinguished in steps, refer to Figure 4 The rotation mode is in the order of ZYX axis, and the angles of rotation of the three axes are γ, β, α respectively. The rotation here is defined as positive rotation, that is, the angles are greater than 0, and the sine and cosine values ​​corresponding to the rotation angles of the three axes are defined as s α ,s β ,s γ ,cα ,c β , c γ , the rotation matrix can be calculated as follows:

[0060]

[0061] At this point, the effective conversion between the world coordinate system and the drone coordinate system is completed. When discussing the position of the drone in the world, the longitude and latitude correspondence has been added to the position calculation, and the three axes (x and y) of the drone relative to the world coordinate system w ,y w ,z w )’s rotation angle α w ,β w ,γ w Represent the heading (yaw) angle, pitch angle and roll angle of the drone respectively. The three-axis rotation matrix is ​​multiplied to complete the transformation matrix R pw When discussing the position of the airborne camera relative to the drone, the influence of the translation matrix on the coordinate system cannot be ignored. However, since the drone is relatively fixed to the desired position, the three-axis translation matrix of the origin of the camera coordinate system and the origin of the drone coordinate system can be manually measured. No specific analysis is done here, and only the translation matrix is ​​set to T cp =(x0,y0,z0). For the calculation of the rotation matrix, we can get the rotation matrix around the drone coordinate system (x p ,y p , z p ) Rotation angle α p ,β p ,γ p They represent the azimuth, pitch and roll angles of the camera’s shooting direction. In actual practice, most of the images captured during flight are of ground targets. In order to maintain the stability of the drone’s flight, the camera does not need a roll angle to ensure the clarity and stability of the image. Therefore, the roll angle γ p Can be set to 0° to ensure y c The axis must be in y p Axis and x p The rotation matrix between the onboard camera and the drone is calculated by the default result R. cp , after obtaining the external parameter matrices between the three different coordinate systems, we can use formula 1 to obtain the transformation from the camera coordinate system to the world coordinate system. The transformation formula is as follows:

[0062] P w =R pw R cp P c =R cw P c#(3)

[0063] Where R cw is the conversion from the airborne camera coordinate system to the world coordinate system. Since the origin of the drone coordinate system coincides with the origin of the world coordinate system, the translation matrix T from the camera to the drone coordinate system is recorded. cp That is the translation matrix T from the airborne camera to the world coordinate system cw , the present invention first needs to understand the actual physical relationship between the pixel center of the image and the coordinate system of the onboard camera. To simplify the model, since the pitch angle and azimuth angle are tilts in the horizontal and vertical directions respectively, and are independent of each other and do not interfere with each other, we use one of the tilt modes, the pitch angle, to explain the definition of this model, which can be defined here as α p = 0. Due to the flight altitude, the pitch angle is generally downward. For details, see Figure 5 , where distance is the distance from the center of the image to the origin of the camera coordinates. It is easy to see that, except for the rotation of the onboard camera itself, the distance is only related to the height of the drone relative to the ground. From formula 2, it can be obtained that when the camera rotates only in the vertical direction, only r 31 =-s β The following formula can be used to obtain the distance between the center of the image pixel and the origin of the airborne camera.

[0064]

[0065] Since there is no negative value for distance, the sine value needs to be processed as an absolute value. The drone itself records the height relative to sea level, and the actual height of the flight relative to the ground needs to be recorded in advance by the height of the ground relative to sea level. Figure 5 , we can know that the distance here is x in the camera coordinate system c The actual coordinates of the direction, which is important for the subsequent calculation of the coordinates of any pixel in the camera coordinate system.

[0066] S400. According to the zoom ratio of the camera itself and the physical parameters of the camera corresponding to the image, the three-dimensional coordinates of the target in S300 in the camera coordinate system are obtained, and the data is corrected using polynomial optimization.

[0067] In actual practice, the target will not appear exactly in the center. In this case, it is necessary to calculate the actual coordinates of any position in the image in the onboard camera coordinate system to obtain the actual three-dimensional world coordinates of the target, that is, the longitude and latitude values. In order to effectively calculate the position of the target at any pixel in the image in the camera coordinate system, some actual content needs to be known. The information required in this step includes the intrinsic parameters when the camera is not zoomed in (f x ,fy ,c x ,c y ), where (f x ,f y ) is the focal length of the camera in two directions, which corresponds to the actual focal length f of the camera and the image resolution. The actual focal length f is used uniformly here. (c x , c y ) is the offset, which is good in the physical structure of the camera, that is, when there is no radial distortion, it is generally half of the length and width of the pixel, the pixel size is pixel, which is the actual physical distance corresponding to each pixel, and the camera magnification mag. In step 1, in order to obtain the effective target output, the camera performs a zoom operation, which will linearly affect the camera's f, which is a change caused by the camera's own structure. The above parameters can help obtain the position of any pixel to the airborne camera coordinate system, thereby obtaining the position of any pixel in the world coordinate system, that is, longitude and latitude.

[0068] See also Figure 6 , the imaging plane here is not the actual ground, but a virtual plane parallel to the actual imaging plane based on the distance. This virtual plane is perpendicular to the normal line of the optical center, and the angle with the actual ground is exactly the pitch angle (90°-β). The ground needs to be imaged and projected to obtain the correspondence between the pixels in the imaging plane and the positions actually represented by the ground. Assuming that the coordinates of the image center are (u0, v0), the coordinates of the actual target center point are (u, v), and the coordinates corresponding to the coordinates in the airborne camera coordinate system are (x0, y0) and (x, y). The actual positions of the target pixel and the center pixel on the imaging plane can be calculated based on the pixel size, as shown in the following formula:

[0069]

[0070] Where dx, dy refer to Figure 6 The content on the right represents the actual physical distance between the target pixel and the center pixel. After calculation, the target pixel in the onboard camera coordinate system can be obtained according to the camera's intrinsic parameter matrix. The specific method is as follows:

[0071] Since the center pixel of the image in the imaging plane has no rotation angle relative to the camera on the three axes, the position of the target that may appear at any position on the imaging plane has a pitch angle and azimuth offset relative to the optical center of the camera. We define the azimuth angle of the target pixel relative to the center pixel in the drone coordinate system as Δα P , in order to simplify the model in step 3, we set the azimuth angle to 0°, but it does not affect the calculation of the pixel azimuth angle offset when there is azimuth rotation. The overall azimuth angle offset calculation is shown in Equation 6

[0072] α' P =αP +Δα P #(6)

[0073] where Δα P The calculation is as follows:

[0074]

[0075] Similarly, the new azimuth angle β' from the camera optical center to the target pixel to the drone coordinate system P As shown in formula 8:

[0076]

[0077] After obtaining the azimuth and pitch angles of the target pixel, the rotation matrix R' from the onboard camera coordinate system to the drone coordinate system can be recalculated. cp This rotation matrix is ​​equivalent to the rotation matrix when the camera positions the target at the center of the image. Reusing formula 4 can obtain the actual distance of the target pixel relative to the camera origin. According to the angle, a more specific coordinate system coordinate point can be obtained. c After obtaining the valid coordinates, use the rotation matrix R' cp and the invariant translation matrix T cp =(x0, y0, z0) can obtain the actual coordinates of the target pixel in the drone coordinate system, which can be calculated as follows

[0078] Tar=R' cp *point c +T cp #(9)

[0079] Before proceeding to the next step of conversion, this paper needs to further optimize the existing coordinates. In the world coordinate system set in this paper, the xoy plane is parallel to the ground, that is, the coordinate on the z-axis is the actual flight height of the drone. After the above effective conversion, the actual flight height of the drone and the calculation results are compared multiple times, and then optimized according to the differences. In multiple experiments, this paper introduced polynomial optimization, which effectively improved the accuracy of the coordinate values.

[0080] S500. Obtain the latitude and longitude of each target of interest in the real world coordinate system through the geographical location of the drone and the posture of the drone relative to the camera, so as to achieve the three-dimensional positioning of the target by the high-altitude drone. After obtaining the world coordinates of the target object in the world coordinate system defined in step 3, since the longitude and latitude only represent the actual position in a two-dimensional plane, the z-axis data here can be ignored and only represents the flight altitude of the drone, because the x-axis and y-axis in the world coordinate system are parallel to the longitude and latitude. The coordinate value is the actual distance, and the location of the target object in the real world can be obtained by converting the distance and the longitude and latitude. At this point, the overall process of the present invention is completed.

[0081] This embodiment also discloses a target detection and real-time three-dimensional positioning system based on a drone, including: a target extraction module, an image center point coordinate acquisition module, a world coordinate system coordinate conversion module, a camera coordinate system conversion module, and a three-dimensional positioning module; wherein:

[0082] The target extraction module is used to record multiple images through the drone, extract the target objects according to the obtained images, and sort and classify the targets;

[0083] The image center point coordinate acquisition module is used to calculate the center position and category of a certain target in the image based on the effective output obtained by the target extraction module, including the danger category and the pixel position of the target;

[0084] The world coordinate system coordinate conversion module is used to calculate the exact position of the image center in the world coordinate system by using the three-dimensional coordinate system of the three-dimensional world through the known flight altitude and the gimbal posture of the drone;

[0085] The camera coordinate system conversion module is used to obtain the three-dimensional coordinates of the target in the camera coordinate system according to the zoom ratio of the camera itself and the physical parameters of the camera corresponding to the image, and to correct the data using polynomial optimization;

[0086] The three-dimensional positioning module is used to obtain the latitude and longitude of each target of interest in the real-world coordinate system through the geographical location of the drone and the position of the drone relative to the camera, so as to achieve the three-dimensional positioning of the target by the high-altitude drone.

[0087] This embodiment discloses a method and system for target detection and real-time three-dimensional positioning based on a drone. In the target detection part, a fast and efficient way is used to complete the detection of specific targets. While avoiding the global depth estimation of the image, the present invention only performs depth estimation on the specific target. Since the drone is flying in the sky, the height difference of a small part of the ground target relative to the drone does not have a significant impact. Secondly, there is a big difference between the perspective of shooting directly facing the ground and the perspective under the offset of the drone gimbal. The present invention adjusts the image through the posture position of the drone. And the accurate position of the specific point in the two-dimensional image in the three-dimensional world is obtained according to the three-dimensional geometric solution of the real world. It solves the problem in the prior art that the traditional binocular positioning algorithm is limited by the baseline and cannot achieve long-distance target positioning and the problem that the drone is affected by stability and control during operation.

[0088] It should be understood that the specific order or hierarchy of steps in the disclosed process is an example of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process can be rearranged without departing from the scope of protection of the present disclosure. The attached method claims present the elements of the various steps in an exemplary order and are not intended to be limited to the specific order or hierarchy described.

[0089] In the above detailed description, various features are grouped together in a single embodiment to simplify the disclosure. This method of disclosure should not be interpreted as reflecting an intention that the embodiments of the claimed subject matter require more features than are clearly stated in each claim. On the contrary, as reflected in the appended claims, the invention is in a state of having less than all the features of the disclosed individual embodiments. Therefore, the appended claims are hereby expressly incorporated into the detailed description, with each claim standing on its own as a separate preferred embodiment of the invention.

[0090] Those skilled in the art will also appreciate that the various illustrative logic blocks, modules, circuits, and algorithmic steps described in conjunction with the embodiments herein can all be implemented as electronic hardware, computer software, or a combination thereof. In order to clearly illustrate the interchangeability between hardware and software, various illustrative components, blocks, modules, circuits, and steps are generally described above around their functions. Whether such functions are implemented as hardware or software depends on specific applications and the design constraints imposed on the entire system. A skilled person can implement the described functions in an alternative manner for each specific application, but such implementation decisions should not be interpreted as departing from the scope of protection of the present disclosure.

[0091] The steps of the method or algorithm described in conjunction with the embodiments herein may be directly embodied as hardware, a software module executed by a processor, or a combination thereof. The software module may be located in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a mobile disk, a CD-ROM, or any other form of storage medium well known in the art. An exemplary storage medium is connected to the processor so that the processor can read information from the storage medium and can write information to the storage medium. Of course, the storage medium may also be an integral part of the processor. The processor and the storage medium may be located in an ASIC. The ASIC may be located in a user terminal. Of course, the processor and the storage medium may also be present in a user terminal as discrete components.

[0092] For software implementation, the techniques described in this application can be implemented with modules (e.g., procedures, functions, etc.) that perform the functions described in this application. These software codes can be stored in a memory unit and executed by a processor. The memory unit can be implemented within the processor or outside the processor. In the latter case, it is coupled to the processor in a communication manner via various means, which are well known in the art.

[0093] The above description includes examples of one or more embodiments. Of course, it is impossible to describe all possible combinations of components or methods for the purpose of describing the above embodiments, but it should be recognized by those skilled in the art that the various embodiments may be further combined and arranged. Therefore, the embodiments described herein are intended to cover all such changes, modifications and variations that fall within the scope of protection of the appended claims. In addition, with respect to the term "comprising" used in the specification or claims, the word is covered in a manner similar to the term "including", just as "including," is explained as a transitional word in the claims. In addition, any term "or" used in the specification of the claims is intended to mean "non-exclusive or".

Claims

1. A method for target detection and real-time three-dimensional positioning based on drones, characterized in that: include: S100. Preset the tilt angle of the drone gimbal, and under the condition of determining the latitude and longitude position and altitude, record the current multiple images through the drone, extract the target object according to the obtained image, and sort and classify the target; S200. Calculate the center position and category of a certain target in the image according to the valid output obtained in S100, including the danger category and the pixel position of the target; S300. By knowing the flight altitude and the gimbal posture of the drone, the accurate position of the image center in the world coordinate system is calculated with the help of the three-dimensional coordinate system of the three-dimensional world; S400. According to the zoom ratio of the camera itself and the physical parameters of the camera corresponding to the image, the three-dimensional coordinates of the target in S300 in the camera coordinate system are obtained, and the data is corrected using polynomial optimization; S500. The latitude and longitude of each target of interest in the real world coordinate system are finally obtained through the geographical location of the drone and the position of the drone relative to the camera, so as to achieve the three-dimensional positioning purpose of the high-altitude drone for the target; In S300, the method for calculating the accurate position of the image center in the world coordinate system is: using the external parameter matrix (R pw ,T pw ), the drone coordinate system and the onboard camera coordinate system (R cp ,T cp ) to perform the corresponding transformation, and the exact position transformation formula of the image center in the world coordinate system is as follows: P w =R pw P p +T pw P p =R cp P c +T cp Among them, P w is the world coordinate system, P p is the UAV coordinate system, P c is the onboard camera coordinate system, R pw is the rotation matrix between the world coordinate system and the drone coordinate system, T pw is the translation matrix between the world coordinate system and the drone coordinate system, R cp T is the rotation matrix between the airborne camera coordinate system and the drone coordinate system; cp The translation matrix between the airborne camera coordinate system and the drone coordinate system; Translation matrix T between world coordinate system and drone coordinate system pw =(0,0,0), the rotation matrix R between the world coordinate system and the drone coordinate system pw The calculation method is: rotate the image in the order of ZYX axis, the three-axis rotation angles are γ, β, α, and define the sine and cosine values ​​corresponding to the three-axis rotation angles as s α ,s β ,s γ ,c α ,c β ,c γ , then the rotation matrix R between the world coordinate system and the drone coordinate system pw for: In S300, it is also necessary to obtain the distance between the center of the image pixel and the origin of the airborne camera. The distance between the center of the image pixel and the origin of the airborne camera is calculated using the formula: Among them, h plane is the altitude of the drone, h ground is the ground altitude, s β is the cosine of the rotation angle about the y-axis.

2. The method for target detection and real-time three-dimensional positioning based on an unmanned aerial vehicle according to claim 1, characterized in that: In S100, the drone records the current multiple images, extracts the target objects according to the obtained images, and sorts and classifies the targets. Specifically, the image is collected in real time by the drone camera and transmitted back to the CNN network running in the background in real time.

3. The method for target detection and real-time three-dimensional positioning based on a drone as claimed in claim 1, characterized in that: In S100, the image is also effectively detected, and the stable output of multiple frames of images is used as the judgment standard. If a stable image cannot be obtained, it returns to S100, zooms the camera, and reacquires the image. After the continuous multi-frame target is determined, the multiple output results are used as the effective output.

4. The method for target detection and real-time three-dimensional positioning based on a drone as claimed in claim 1, characterized in that: The implementation method of S400 is as follows: the preset image center coordinates are (u0, v0), the actual target center coordinates are (u, v), and the corresponding coordinates in the airborne camera coordinate system are (x0, y0) and (x, y). The actual positions of the target pixel and the center pixel on the imaging plane can be calculated according to the pixel size, as shown in the following formula: dx=pixel×(u-u0) dy=pixel×(v-v0) Where dx and dy represent the actual physical distance between the target pixel and the center pixel.

5. The method for target detection and real-time three-dimensional positioning based on an unmanned aerial vehicle as claimed in claim 4, characterized in that: After calculating the actual position, the position of the target pixel in the onboard camera coordinate system can be obtained according to the camera's intrinsic parameter matrix, including: After obtaining the azimuth and pitch angle of the target pixel, the rotation matrix R' from the onboard camera coordinate system to the drone coordinate system can be recalculated. cp This rotation matrix is ​​equivalent to the rotation matrix when the camera positions the target at the center of the image when shooting. The actual distance of the target pixel relative to the camera origin can be obtained. The coordinate system coordinate point can be obtained according to the angle c ; After obtaining the valid coordinates, use the rotation matrix R' cp and the invariant translation matrix T cp =(x0, y0, z0) can obtain the actual coordinate Tar of the target pixel in the drone coordinate system, which is calculated as follows: Tar=R' cp *point c +T cp Among them, Tar is the actual coordinate of the target pixel in the drone coordinate system, R' cp The rotation matrix from the airborne camera coordinate system to the drone coordinate system is recalculated, point c To obtain the coordinate system coordinates based on the angle, T cp is the translation matrix.

6. The method for target detection and real-time three-dimensional positioning based on an unmanned aerial vehicle according to claim 1, characterized in that: The specific method of S500 is: after the world coordinates of the target object in the world coordinate system defined in S300, the Z-axis coordinates in the world coordinates are removed, the actual distance is obtained through the x-axis and y-axis in the world coordinate system, and the location of the target object in the real world can be obtained by converting the distance and the longitude and latitude.

7. A target detection and real-time three-dimensional positioning system based on drones, characterized in that: include: Target extraction module, image center point coordinate acquisition module, world coordinate system coordinate conversion module, camera coordinate system conversion module, three-dimensional positioning module; among which: The target extraction module is used to record multiple images through the drone, extract the target objects according to the obtained images, and sort and classify the targets; The image center point coordinate acquisition module is used to calculate the center position and category of a certain target in the image based on the effective output obtained by the target extraction module, including the danger category and the pixel position of the target; The world coordinate system coordinate conversion module is used to calculate the exact position of the image center in the world coordinate system by using the three-dimensional coordinate system of the three-dimensional world through the known flight altitude and the gimbal posture of the drone; the specific methods include: Using the external parameter matrix (R pw ,T pw ), the drone coordinate system and the onboard camera coordinate system (R cp ,T cp ) to perform the corresponding transformation, and the exact position transformation formula of the image center in the world coordinate system is as follows: P w =R pw P p +T pw P p =R cp P c +T cp Among them, P w is the world coordinate system, P p is the UAV coordinate system, P c is the onboard camera coordinate system, R pw is the rotation matrix between the world coordinate system and the drone coordinate system, T pw is the translation matrix between the world coordinate system and the drone coordinate system, R cp T is the rotation matrix between the airborne camera coordinate system and the drone coordinate system; cp The translation matrix between the airborne camera coordinate system and the drone coordinate system; Translation matrix T between world coordinate system and drone coordinate system pw =(0,0,0), the rotation matrix R between the world coordinate system and the drone coordinate system pw The calculation method is: rotate the image in the order of ZYX axis, the three-axis rotation angles are γ, β, α, and define the sine and cosine values ​​corresponding to the three-axis rotation angles as s α ,s β ,s γ ,c α ,c β ,c γ , then the rotation matrix R between the world coordinate system and the drone coordinate system pw for: It is also necessary to obtain the distance between the center of the image pixel and the origin of the airborne camera. The formula for calculating the distance between the center of the image pixel and the origin of the airborne camera is: Among them, h plane is the altitude of the drone, h ground is the ground altitude, s β is the cosine value of the y-axis rotation angle; The camera coordinate system conversion module is used to obtain the three-dimensional coordinates of the target in the camera coordinate system according to the zoom ratio of the camera itself and the physical parameters of the camera corresponding to the image, and to correct the data using polynomial optimization; The three-dimensional positioning module is used to obtain the latitude and longitude of each target of interest in the real-world coordinate system through the geographical location of the drone and the position of the drone relative to the camera, so as to achieve the three-dimensional positioning of the target by the high-altitude drone.

Images