Camera labeling method and device for real-time video of unmanned aerial vehicle

Abstract

The invention discloses a camera labeling method and device for real-time videos of an unmanned aerial vehicle. The method comprises the following steps: constructing a POI database, wherein at least one POI data item is stored in the POI database; the visual radius and the horizontal visual angle of the camera of the unmanned aerial vehicle are obtained, the height difference, the pitch angle compensation value and the yaw angle compensation value are obtained, the height difference is the height difference of the take-off position of the unmanned aerial vehicle relative to the position of the POI label, and the pitch angle compensation value is the pitch angle compensation value of the camera of the unmanned aerial vehicle; the yaw angle compensation value is the yaw angle compensation value of the unmanned aerial vehicle camera; and receiving the unmanned aerial vehicle video stream and the flight telemetry data, calculating the picture point location according to the flight telemetry data to obtain a display point location, integrating the display point location with the unmanned aerial vehicle video stream, and playing a picture. Through the method and the device, the problem that POI information cannot be superposed in real time in a live broadcast picture of an unmanned aerial vehicle in related technologies is solved.

Description

Technical Field

[0001] This invention relates to the field of geographic information technology, and more particularly to a method and apparatus for camera annotation of real-time video from unmanned aerial vehicles (UAVs). Background Technology

[0002] With the popularization of drone technology, its applications in surveying, security monitoring, emergency rescue, and live streaming are becoming increasingly widespread. Operators understand the situation on-site through real-time video footage transmitted by drones. However, video footage alone lacks supplementary contextual information. For example, in urban security patrols, operators may not be able to quickly and accurately identify a specific monitoring pole, fire hydrant, or building exit based solely on video images; in security for large-scale events, organizers may find it difficult to know the location of fire-fighting facilities or emergency medical resources in the footage.

[0003] Existing technologies include methods for overlaying tags onto static maps or recorded videos, but these methods cannot meet the specific requirements of drone live streaming scenarios: 1. Complex spatial calculations: It is necessary to accurately map POI points with real-world latitude, longitude, and altitude coordinates to the constantly changing coordinate system of the two-dimensional video screen, resulting in complex calculation models; 2. Dynamic visibility management: Drones have limited field of view, so it is necessary to intelligently determine which POI points are currently within the camera's field of view and render only these points to avoid the screen being too crowded.

[0004] In the existing technology, there is no effective solution to the problem of being unable to overlay POI information in real time, accurately and stably in the live broadcast footage of high-speed moving drones with changing postures. Summary of the Invention

[0005] The main objective of this application is to provide a camera annotation method and apparatus for real-time video of drones, so as to at least solve the technical problem that POI information cannot be superimposed in real time on drone live broadcast footage in related technologies.

[0006] To achieve the above objectives, according to one aspect of this application, a camera annotation method for real-time video from a drone is provided. The method includes: constructing a POI database, which stores at least one POI data item, wherein each POI data item includes at least POI tag name, POI tag type, POI tag longitude, POI tag latitude, and POI tag altitude information, with one POI data item corresponding to one POI tag; acquiring the visible radius and horizontal visible angle of the drone camera, and acquiring the altitude difference, pitch angle compensation value, and yaw angle compensation value, wherein the altitude difference is the altitude difference between the drone's takeoff position and the POI tag position, the pitch angle compensation value is the drone camera's pitch angle compensation value, and the yaw angle compensation value is the drone camera's yaw angle compensation value; receiving the drone video stream and flight telemetry data, calculating the above-image point position based on the flight telemetry data to obtain the display point position, integrating the display point position with the drone video stream, and broadcasting the image, wherein the flight telemetry data includes the drone's real-time latitude and longitude, the drone's real-time flight altitude, the drone camera's pitch angle, and the drone camera's yaw angle.

[0007] Optionally, the vertical line-of-sight angle is calculated based on the computer's gauge spacing, where the gauge spacing is the physical distance between the drone's position and the POI tag's position; the azimuth angle is calculated based on the eastward and northward distances, and the horizontal line-of-sight angle is obtained based on the azimuth angle, where the eastward distance is the physical distance calculated by using the drone's latitude as the POI tag's latitude, and the northward distance is the physical distance calculated by using the drone's longitude as the POI tag's longitude, and the azimuth angle is the angle between the POI tag in the drone's line of sight and the X-axis in three-dimensional space; the equivalent focal length is obtained based on the pixel value of the image width, and the offset angle is calculated based on the equivalent focal length, where the offset angle is the horizontal offset angle of the POI tag relative to the drone's camera, and the offset angle is in radians; the two-dimensional X-axis coordinate value and the two-dimensional Y-axis coordinate value are calculated based on the offset angle, where the two-dimensional X-axis coordinate value and the two-dimensional Y-axis coordinate value are the position coordinates of the output image.

[0008] Optionally, the computer-generated gauge spacing is calculated using the following formula: , ,in, As the first auxiliary parameter, This is the second auxiliary parameter. For the spacing between machine marks, The difference in latitude between the drone's location and the POI's location. The longitude difference between the drone's location and the POI tag's location, The latitude of the drone's location. The latitude and longitude of the POI tag location. The average radius of the Earth, , , as well as It is measured in radians; when the logo spacing is greater than the visible radius, the POI label corresponding to the logo spacing will not be displayed in the image; calculate the vertical radian using the following formula: ,in, It is the arctangent function. For the spacing between machine marks, The difference in height; convert the vertical curvature to the vertical line-of-sight angle using the following formula: ,in, The vertical line-of-sight angle is 0; when the height difference is 0, the value of the vertical line-of-sight angle is directly 0.

[0009] Optionally, the eastward distance can be calculated by using the latitude of the drone as the latitude of the POI tag to calculate the physical distance, which is the eastward distance. The calculation formula is as follows:

[0010] ,

[0011] ,in, This is the first eastward auxiliary parameter. This is the second eastward auxiliary parameter. The distance is eastward. The difference in longitude between the drone's location and the POI tag's location. The latitude of the drone's location. The radius of the Earth is denoted by , and the distance between the drone and the point of interest (POI) is the physical distance between the drone's location and the POI's location. as well as The distance is measured in radians. When the longitude of the POI tag is less than the longitude of the drone, the eastward distance becomes negative. To calculate the northward distance, the drone's longitude is used as the longitude of the POI tag to calculate the physical distance, which is the northward distance. The calculation formula is as follows: , ,in, This is the first northward auxiliary parameter. This is the second northward auxiliary parameter. This is the distance to the north. The difference in latitude between the drone's location and the POI's location. The radius of the Earth is denoted by , and the distance between the drone and the point of interest (POI) is the physical distance between the drone's location and the POI's location. The distance is measured in radians. When the latitude of the POI tag is less than the latitude of the drone, the northward distance becomes a negative value.

[0012] Optionally, the azimuth angle is calculated using the following formula: ,in, It is the azimuth angle. It is the arctangent function in the fourth quadrant. The distance is eastward. This represents the distance to the north.

[0013] Optionally, when both the eastward and northward distances are positive, the POI tag is positioned northeast of the drone. The horizontal line-of-sight angle is calculated using the following formula: ,in, The horizontal line of sight. This is the azimuth angle; when the eastward distance is negative and the northward distance is positive, the POI tag is in the northwest direction of the drone. Calculate the horizontal line-of-sight angle using the following formula: When both the eastward and northward distances are negative, the POI tag is located southwest of the drone. The horizontal line-of-sight angle is calculated using the following formula: When the eastward distance is positive and the northward distance is negative, the POI tag is located northwest of the drone. The horizontal line-of-sight angle is calculated using the following formula: .

[0014] Optionally, the equivalent focal length is calculated using the following formula: ,in, For equivalent focal length, The pixel value representing the image width. The horizontal viewing angle is given; the offset angle is calculated using the following formula: ,in, For offset angle, This refers to the horizontal line of sight angle.

[0015] Optionally, the Y-axis coordinate value of the two-dimensional coordinate system is calculated using the following formula:

[0016] ,in, This refers to the drone's pitch angle, which is a fixed parameter of the drone. The angle is perpendicular to the line of sight. Indicates radians, For offset angle, To find the equivalent focal length; calculate the X-axis coordinate value in two-dimensional coordinates using the following formula:

[0017] .

[0018] According to another aspect of this application, a camera annotation device for real-time video of a drone is provided. The device includes: a construction unit for constructing a POI database, the POI database storing at least one POI data item, wherein each POI data item includes at least POI tag name, POI tag type, POI tag longitude, POI tag latitude, and POI tag altitude information, and one POI data item corresponds to one POI tag; an acquisition unit for acquiring the visible radius and horizontal visible angle of the drone camera, acquiring the altitude difference, pitch angle compensation value, and yaw angle compensation value, wherein the altitude difference is the altitude difference between the drone's takeoff position and the POI tag position, the pitch angle compensation value is the drone camera's pitch angle compensation value, and the yaw angle compensation value is the drone camera's yaw angle compensation value; and an integration unit for receiving the drone's video stream and flight telemetry data, calculating the above-image point position based on the flight telemetry data to obtain the display point position, integrating the display point position with the drone's video stream, and broadcasting the image, wherein the flight telemetry data includes the drone's real-time latitude and longitude, the drone's real-time flight altitude, the drone camera's pitch angle, and the drone camera's yaw angle.

[0019] To achieve the above objectives, according to another aspect of this application, a computer-readable storage medium is provided, the computer-readable storage medium including a stored program, wherein the program executes a camera annotation method for real-time video of a drone according to any one of the above claims.

[0020] According to another aspect of this application, an electronic device is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include a camera annotation method for performing any one of the following: a real-time video of a drone.

[0021] This application employs the following steps: Constructing a POI database, which stores at least one POI data item. Each POI data item includes at least the POI tag name, POI tag type, POI tag longitude, POI tag latitude, and POI tag altitude information; one POI data item corresponds to one POI tag; obtaining the visible radius and horizontal visible angle of the drone camera, and obtaining the altitude difference, pitch angle compensation value, and yaw angle compensation value. The altitude difference is the altitude difference between the drone's takeoff position and the POI tag position, and the pitch angle compensation value is the drone camera's pitch angle. The compensation value, specifically the yaw angle compensation value, is the yaw angle compensation value of the drone camera. It receives the drone video stream and flight telemetry data, calculates the points shown in the image above based on the flight telemetry data, integrates the displayed points with the drone video stream, and broadcasts the footage. The flight telemetry data includes the drone's real-time latitude and longitude, real-time flight altitude, drone camera pitch angle, and drone camera yaw angle. This solves the problem in related technologies where POI information cannot be overlaid in real-time on drone live stream footage, thus achieving the effect of accurately and timely displaying POI tags on the live stream footage during the live stream. Attached Figure Description

[0022] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0023] Figure 1 This is a flowchart of a camera annotation method for real-time video of a drone provided according to an embodiment of this application;

[0024] Figure 2 This is a diagram illustrating the distance between the drone's location and the POI tag's location;

[0025] Figure 3 This is a schematic diagram of the POI tag position relative to the drone position in a two-dimensional coordinate system;

[0026] Figure 4 This is a structural block diagram of a camera annotation device for real-time video of a drone, provided according to an embodiment of this application. Detailed Implementation

[0027] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0028] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0029] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0030] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0031] This embodiment provides a camera annotation method for real-time video of a drone that runs on a mobile terminal, computer terminal or similar computing device. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Also, although the logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0032] Figure 1 This is a flowchart of a camera annotation method for real-time video from a drone according to an embodiment of this application. Figure 1 As shown, the method includes the following steps:

[0033] Step S101: Construct a POI database. The POI database stores at least one POI data item. Each POI data item includes at least the POI tag name, POI tag type, POI tag longitude, POI tag latitude, and POI tag altitude information. One POI data item corresponds to one POI tag.

[0034] Specifically, the POI database needs to be built in advance, containing all POI tags that need to be displayed in the drone live video stream. The POI tag information is displayed on the live stream screen according to user needs. Users can retrieve POI data items through query operations.

[0035] Step S102: Obtain the visible radius and horizontal visible angle of the drone camera, and obtain the height difference, pitch angle compensation value and yaw angle compensation value. The height difference is the height difference between the drone's take-off position and the POI tag position, the pitch angle compensation value is the drone camera's pitch angle compensation value, and the yaw angle compensation value is the drone camera's yaw angle compensation value.

[0036] Specifically, the height difference between the drone's takeoff location and the POI tag location is pre-built. If the drone's takeoff location and the POI tag are on the same horizontal plane, the height difference is 0. If the drone takes off from a rooftop, the height difference is equal to the height of the building. If the drone takes off from a mountainous area, the data in the POI tag are not on the same plane, and the POI tag and the drone's takeoff location are not on the same plane either. In this case, the height difference needs to be dynamically calculated. The calculation method is to subtract the altitudes and take the absolute value. The altitude of the drone's takeoff point and the altitude of the POI tag need to be pre-built.

[0037] Step S103: Receive the UAV video stream and flight telemetry data, calculate the display point based on the flight telemetry data, integrate the display point with the UAV video stream, and broadcast the image. The flight telemetry data includes the UAV's real-time latitude and longitude, the UAV's real-time flight altitude, the UAV camera's pitch angle, and the UAV camera's yaw angle.

[0038] Specifically, pre-setting the pitch angle compensation value and yaw angle compensation value for the drone camera is to handle abnormal situations. For example, if the drone camera gimbal cannot be calibrated, the yaw angle should be 0 when the gimbal is due north. If the actual value is not 0, the compensation value can be used to make up for it. The drone can push real-time flight telemetry data to the cloud service, or it can push telemetry data to a messaging medium in real time, and then the cloud service can pull the data from the messaging medium.

[0039] In addition, UAV flight telemetry data generally uses the WGS84 coordinate system, while POI tag points can use any coordinate system. Before calculation, both need to be converted to the same coordinate system, for example, both can be converted to the bd09 coordinate system.

[0040] In one optional embodiment, the vertical line-of-sight angle is calculated based on the computer gauge spacing, where the gauge spacing is the physical distance between the drone's position and the POI tag's position; the azimuth angle is calculated based on the eastward and northward distances, and the horizontal line-of-sight angle is obtained based on the azimuth angle, where the eastward distance is the physical distance calculated by using the drone's latitude as the POI tag's latitude, and the northward distance is the physical distance calculated by using the drone's longitude as the POI tag's longitude, and the azimuth angle is the angle between the POI tag in the drone's line of sight and the X-axis in three-dimensional space; the equivalent focal length is obtained based on the pixel value of the image width, and the offset angle is calculated based on the equivalent focal length, where the offset angle is the horizontal offset angle of the POI tag relative to the drone camera, and the offset angle is in radians; the two-dimensional X-axis coordinate value and the two-dimensional Y-axis coordinate value are calculated based on the offset angle, where the two-dimensional X-axis coordinate value and the two-dimensional Y-axis coordinate value are the position coordinates of the output image.

[0041] In one optional embodiment, the computer gauge spacing is calculated using the following formula:

[0042] ,

[0043] ,in, As the first auxiliary parameter, This is the second auxiliary parameter. For the spacing between machine marks, The difference in latitude between the drone's location and the POI's location. The longitude difference between the drone's location and the POI tag's location, The latitude of the drone's location. The latitude and longitude of the POI tag location. The average radius of the Earth, , , as well as It is measured in radians; when the logo spacing is greater than the visible radius, the POI label corresponding to the logo spacing will not be displayed in the image; calculate the vertical radian using the following formula: ,in, It is the arctangent function. For the spacing between machine marks, The difference in height; convert the vertical curvature to the vertical line-of-sight angle using the following formula: ,in, The vertical line-of-sight angle is 0; when the height difference is 0, the value of the vertical line-of-sight angle is directly 0.

[0044] Specifically, physical distance is calculated using the Havesing formula, requiring the latitude and longitude of both the drone and the POI tag to be converted to radians. If the distance between the drone and the tag exceeds the drone's camera's field of view, the POI is considered unable to be superimposed onto the drone's view. When the drone is above the POI tag, the vertical line-of-sight angle is positive; when the drone's altitude is equal to the POI tag's altitude, the vertical line-of-sight angle is 0.

[0045] When the drone is above the POI tag, the vertical line-of-sight angle is positive; when the drone's altitude is equal to the POI's altitude, the vertical line-of-sight angle is 0.

[0046] Reference Figure 2 Assuming the drone's height is 10 meters, the POI tag's height is 5 meters, and the distance from the drone to the POI tag is 8.66 meters, the calculation process is as follows:

[0047] Height difference: 10–5 = 5 meters, radian value: atan(8.66 / 5)≈1.047, angle value: 60°, vertical line of sight angle: 90°-60°=30°.

[0048] In one optional embodiment, the eastward distance is calculated by using the latitude of the drone as the latitude of the POI tag to calculate the physical distance, which is the eastward distance. The calculation formula is as follows:

[0049] ,

[0050] ,in, This is the first eastward auxiliary parameter. This is the second eastward auxiliary parameter. The distance is eastward. The difference in longitude between the drone's location and the POI tag's location. The latitude of the drone's location. The radius of the Earth is denoted by , and the distance between the drone and the point of interest (POI) is the physical distance between the drone's location and the POI's location. as well as The distance is measured in radians. When the longitude of the POI tag is less than the longitude of the drone, the eastward distance becomes negative. To calculate the northward distance, the drone's longitude is used as the longitude of the POI tag to calculate the physical distance, which is the northward distance. The calculation formula is as follows: , ,in, This is the first northward auxiliary parameter. This is the second northward auxiliary parameter. This is the distance to the north. The difference in latitude between the drone's location and the POI's location. The radius of the Earth is denoted by , and the distance between the drone and the point of interest (POI) is the physical distance between the drone's location and the POI's location. The distance is measured in radians. When the latitude of the POI tag is less than the latitude of the drone, the northward distance becomes a negative value.

[0051] Specifically, the eastward distance is calculated by using a fixed latitude (i.e., using the drone's latitude as the POI tag's latitude) and then calculating the distance corresponding to the longitude difference. If the POI tag's longitude is less than the drone's longitude, then... The value is negative. The northward distance is calculated by using a fixed longitude (i.e., using the drone's longitude) and the distance corresponding to the latitude difference. If the POI tag's latitude is less than the drone's latitude, then... It is a negative value.

[0052] In one optional embodiment, the azimuth angle is calculated using the following formula: ,in, It is the azimuth angle. It is the arctangent function in the fourth quadrant. The distance is eastward. This represents the distance to the north.

[0053] Specifically, The arctangent function for the four quadrants is an extension of the arctangent function in trigonometric functions in mathematics, and it can be used to handle the relationship between rectangular and polar coordinates. Every point in the rectangular coordinate system can be represented using this function. Represented by coordinates.

[0054] In one optional embodiment, when both the eastward and northward distances are positive, the POI tag is located northeast of the drone. The horizontal line-of-sight angle is calculated using the following formula: ,in, The horizontal line of sight. This is the azimuth angle; when the eastward distance is negative and the northward distance is positive, the POI tag is in the northwest direction of the drone. Calculate the horizontal line-of-sight angle using the following formula: When both the eastward and northward distances are negative, the POI tag is located southwest of the drone. The horizontal line-of-sight angle is calculated using the following formula: When the eastward distance is positive and the northward distance is negative, the POI tag is located northwest of the drone. The horizontal line-of-sight angle is calculated using the following formula: .

[0055] Specifically, refer to Figure 3 , Figure 3 This is a schematic diagram of the POI tag position relative to the drone's position in a two-dimensional coordinate system. The azimuth angle is converted to an angle from the north direction (positive Y-axis), and the clockwise rotation angle ranges from 0° to 360°, resulting in a total of four cases. Therefore, four different calculation methods are used for different cases.

[0056] In one optional embodiment, the equivalent focal length is calculated using the following formula: ,in, For equivalent focal length, The pixel value representing the image width. The horizontal viewing angle is given; the offset angle is calculated using the following formula: ,in, For offset angle, This refers to the horizontal line of sight angle.

[0057] Specifically, the equivalent focal length can be calculated based on the pinhole imaging principle combined with trigonometric functions in mathematics. When calculating the offset angle, the drone camera may not be calibrated correctly, resulting in deviations. This is where the yaw angle compensation value comes in. When calibration is required, the calculation formula is as follows: ,in, This is the yaw angle compensation value.

[0058] In one optional embodiment, the two-dimensional Y-axis coordinate value is calculated using the following formula:

[0059] ,in, This refers to the drone's pitch angle, which is a fixed parameter of the drone. The angle is perpendicular to the line of sight. Indicates radians, For offset angle, Equivalent focal length; The angle in radians is the vertical line of sight. This refers to the drone's pitch angle in radians. If the drone's camera gimbal has calibration issues, a pitch angle compensation value is introduced. The drone's pitch angle needs to be added to the pitch angle compensation value before calculation.

[0060] Specifically, this formula is based on the pinhole camera model and the principle of perspective projection, projecting points in three-dimensional space onto a two-dimensional image plane. In detail:

[0061] Equivalent to focal length f, used to control the scaling of the projection; pinhole model formula:

[0062] Let (X,Y,Z): the three-dimensional coordinates of a point on an object in space (in the camera coordinate system, Z is the depth in front, and X and Y are the horizontal and vertical coordinates);

[0063] f: Focal length (distance from the pinhole to the imaging plane);

[0064] (x,y): The coordinates of the point on the imaging plane.

[0065] The imaging geometry (similar triangles) is given as follows: , ;

[0066] and Together, they determine the vertical direction of the POI point in the camera coordinate system;

[0067] It is the angular offset of the POI point in the horizontal direction relative to the camera's optical axis;

[0068] The numerator and denominator are designed to handle the pitch of the drone camera. The vertical line-of-sight angle from the drone camera to the POI (Point of Interest) in degrees. The influence of ) ensures that the projected y-coordinates correctly reflect the vertical position of the point in the image.

[0069] Let α = globalX (horizontal tilt angle between the drone camera and the POI); θ = tiltAngle (vertical line-of-sight angle between the drone camera and the POI); β = pitch (drone camera pitch angle); f = raduis (equivalent focal length); y is the vertical coordinate of the imaging plane (pixel coordinate system), i.e., the Y-axis coordinate.

[0070] Based on the pinhole imaging model and using trigonometric functions in mathematics, we can obtain:

[0071] Horizontal direction: = tan α;

[0072] Vertical section: ;

[0073] This leads to equation A: ;

[0074] Considering the effect of pitch angle changes in the UAV camera, when pitching around the X-axis by β, Y and Z are linearly combined only in the (Y,Z) plane, resulting in equation B:

[0075] = cosβZ – sinβY;

[0076] = sinβZ + cosβY;

[0077] Substituting equation A into equation B, we get:

[0078] Molecular part C formula:

[0079] ;

[0080] Denominator part D:

[0081] ;

[0082] If we explicitly write out the "horizontal reference" of Y in terms of α, and then substitute it without changing the structure, it will appear in the numerator as cosα and in the denominator as sinα. Then the common factor Y in equations C and D can be replaced as follows:

[0083] Replacing molecule Y with cosα, we obtain equation E:

[0084] = cosα(cosβ tanθ tanα - sinβ )= cosβ tanθ tanα cosα - sinβ cosα;

[0085] Replacing the denominator Y with sinα, we obtain equation F:

[0086] = sinα(sinβ tanθ tanα + cosβ)= cosβ sinα + sinβ tanθ tanα sinα;

[0087] After perspective division, we can obtain:

[0088] .

[0089] In one optional embodiment, the two-dimensional X-axis coordinate value is calculated using the following formula: If the drone camera gimbal has calibration issues, a pitch angle compensation value is introduced. The drone's pitch angle needs to be added to the pitch angle compensation value before calculation.

[0090] Specifically, based on the pinhole camera model: this model assumes that all light rays are projected onto the image plane through a single point (optical center), forming a perspective projection. In this model: radius is equivalent to the focal length f, controlling the image scaling ratio; globalX is the horizontal angular offset of the POI point relative to the optical axis; pitch is the tilt angle of the drone camera.

[0091] Calculation of horizontal coordinates in perspective projection: In perspective projection, the projected coordinates (x, y) of a 3D point (X, Y, Z) on the image plane can be expressed as: x = f * (X / Z); y = f * (Y / Z). However, in this application, due to the pitch and yaw of the drone camera, it is necessary to first transform the points in the world coordinate system to the camera coordinate system before projection.

[0092] Considering the correction for the pitch angle: in the formula In reality, it involves rotating the image plane to compensate for image tilt caused by camera pitch. This part can be understood as:

[0093] The projection of the equivalent focal length onto the image plane;

[0094] : Horizontal offset correction of the Y coordinate caused by the pitch angle.

[0095] This invention overlays Point of Interest (POI) data onto the real-time flight footage of a drone. The calculation process involves a coordinate system of latitude and longitude. This invention does not rely on any specific coordinate system; theoretically, any coordinate system can be used. However, it is necessary to ensure that the latitude and longitude of the drone and the POI are converted to the same coordinate system during calculation. Furthermore, it achieves the transformation of the relationship between two three-dimensional latitude and longitude points in the physical world into the relationship between pixel points in the virtual two-dimensional world.

[0096] This application also provides a camera annotation device for real-time drone video. It should be noted that this camera annotation device for real-time drone video can be used to execute the camera annotation method for real-time drone video provided in this application. This device is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0097] The following describes a camera annotation device for real-time video of a drone provided in an embodiment of this application.

[0098] Figure 4 This is a structural block diagram of a camera annotation device for real-time video from a drone, according to an embodiment of this application. Figure 4As shown, the device includes: a construction unit 401, used to construct a POI database, which stores at least one POI data item, wherein the POI data item includes at least the POI tag name, POI tag type, POI tag longitude, POI tag latitude, and POI tag altitude information, and one POI data item corresponds to one POI tag; an acquisition unit 402, used to acquire the visible radius and horizontal visible angle of the UAV camera, and acquire the altitude difference, pitch angle compensation value, and yaw angle compensation value, wherein the altitude difference is the altitude difference between the UAV's takeoff position and the POI tag position, the pitch angle compensation value is the UAV camera's pitch angle compensation value, and the yaw angle compensation value is the UAV camera's yaw angle compensation value; and an integration unit 403, used to receive the UAV's video stream and flight telemetry data, calculate the above-image point position based on the flight telemetry data to obtain the display point position, integrate the display point position with the UAV's video stream, and broadcast the image, wherein the flight telemetry data includes the UAV's real-time latitude and longitude, the UAV's real-time flight altitude, the UAV camera's pitch angle, and the UAV camera's yaw angle.

[0099] In an optional embodiment, the integration unit 403 includes: a first calculation subunit, used to calculate the distance between the machine and the markers, and to calculate the vertical line-of-sight angle based on the distance between the machine and the markers, wherein the distance between the machine and the markers is the physical distance between the drone's position and the POI tag's position; an acquisition subunit, used to calculate the azimuth angle based on the eastward distance and the northward distance, and to obtain the horizontal line-of-sight angle based on the azimuth angle, wherein the eastward distance is the physical distance calculated by using the latitude of the drone as the latitude of the POI tag, the northward distance is the physical distance calculated by using the longitude of the drone as the longitude of the POI tag, and the azimuth angle is the angle between the POI tag in the drone's line of sight and the X-axis in three-dimensional space; a second calculation subunit, used to obtain the equivalent focal length based on the pixel value of the image width, and to calculate the offset angle based on the equivalent focal length, wherein the offset angle is the horizontal offset angle of the POI tag relative to the drone camera, and the offset angle is in radians; and a third calculation subunit, used to calculate the two-dimensional X-axis coordinate value and the two-dimensional Y-axis coordinate value based on the offset angle, wherein the two-dimensional X-axis coordinate value and the two-dimensional Y-axis coordinate value are the position coordinates of the output image.

[0100] In an optional embodiment, the first calculation subunit includes: a first calculation module for calculating the gauge spacing, the calculation formula being: , ,in, As the first auxiliary parameter, This is the second auxiliary parameter. For the spacing between machine marks, The difference in latitude between the drone's location and the POI's location. The longitude difference between the drone's location and the POI tag's location, The latitude of the drone's location. The latitude and longitude of the POI tag location. The average radius of the Earth, , , as well as It is measured in radians; when the logo spacing is greater than the visible radius, the POI label corresponding to the logo spacing cannot be displayed on the screen; the second calculation module is used to calculate the vertical radian, and the calculation formula is as follows: ,in, It is the arctangent function. For the spacing between machine marks, The height difference is used to calculate the vertical line-of-sight angle. When the height difference is 0, the vertical line-of-sight angle is directly calculated as 0. The conversion module converts the vertical radian into the vertical line-of-sight angle, and the calculation formula is as follows: ,in, The angle is perpendicular to the line of sight.

[0101] In one optional embodiment, the acquisition subunit includes: a third calculation module, used to calculate the eastward distance, which is the physical distance calculated by using the latitude of the UAV as the latitude of the POI tag, and the calculation formula is as follows: , ,in, This is the first eastward auxiliary parameter. This is the second eastward auxiliary parameter. The distance is eastward. The difference in longitude between the drone's location and the POI tag's location. The latitude of the drone's location. The radius of the Earth is denoted by , and the distance between the drone and the point of interest (POI) is the physical distance between the drone's location and the POI's location. as well as The distance is measured in radians. When the longitude of the POI tag is less than the longitude of the drone, the eastward distance becomes negative. The fourth calculation module is used to calculate the northward distance. The drone's longitude is used as the longitude of the POI tag to calculate the physical distance, which is the northward distance. The calculation formula is as follows: , ,in, This is the first northward auxiliary parameter. This is the second northward auxiliary parameter. This is the distance to the north. The difference in latitude between the drone's location and the POI's location. The radius of the Earth is denoted by , and the distance between the drone and the point of interest (POI) is the physical distance between the drone's location and the POI's location. The distance is measured in radians. When the latitude of the POI tag is less than the latitude of the drone, the northward distance becomes a negative value.

[0102] In one optional embodiment, the acquisition subunit includes: a fifth calculation module, used to calculate the azimuth angle, the calculation formula being as follows: ,in, It is the azimuth angle. It is the arctangent function in the fourth quadrant. The distance is eastward. This represents the distance to the north.

[0103] In an optional embodiment, the acquisition subunit includes: a sixth calculation module, used to calculate the horizontal line-of-sight angle when both the eastward and northward distances are positive, and the POI tag is in the northeast direction of the UAV, using the following formula: ,in, The horizontal line of sight. The azimuth angle; the seventh calculation module is used to calculate the horizontal line-of-sight angle when the eastward distance is negative and the northward distance is positive, and the POI tag is in the northwest direction of the drone. The calculation formula is: The eighth calculation module is used to calculate the horizontal line-of-sight angle when both the eastward and northward distances are negative, and the POI tag is in the southwest direction of the UAV. The calculation formula is as follows: The ninth calculation module is used to calculate the horizontal line-of-sight angle when the eastward distance is positive and the northward distance is negative, and the POI tag is in the northwest direction of the drone. The calculation formula is as follows: .

[0104] In an optional embodiment, the second calculation subunit includes: a tenth calculation submodule, used to calculate the equivalent focal length, the calculation formula being... ,in, For equivalent focal length, The pixel value representing the image width. The horizontal viewing angle; the eleventh calculation submodule is used to calculate the offset angle, and the calculation formula is as follows: ,in, For offset angle, This refers to the horizontal line of sight angle.

[0105] In an optional embodiment, the third calculation subunit includes: a twelfth calculation submodule, used to calculate the two-dimensional Y-axis coordinate value, the calculation formula being:

[0106] ,in, This refers to the drone's pitch angle, which is a fixed parameter of the drone. The angle is perpendicular to the line of sight. Indicates radians, For offset angle, The equivalent focal length; the thirteenth calculation submodule is used to calculate the X-axis coordinate value in two-dimensional coordinates. The calculation formula is: .

[0107] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0108] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0109] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0110] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0111] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0112] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0113] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0114] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0115] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for camera annotation of real-time video of a UAV, characterized in that, The application relates to a method for displaying a POI (Point of Interest) label on a screen, comprising the following steps: constructing a POI database, wherein at least one POI data item is stored in the POI database, and the POI data item at least comprises POI label name, POI label type, POI label longitude, POI label latitude and POI label height information, and one POI data item corresponds to one POI label; acquiring the visual radius and horizontal visual angle of a UAV camera, acquiring the height difference, the pitch angle compensation value and the yaw angle compensation value, wherein the height difference is the height difference between the UAV take-off position and the POI label position, the pitch angle compensation value is the UAV camera pitch angle compensation value, and the yaw angle compensation value is the UAV camera yaw angle compensation value; receiving the UAV video stream and flight telemetry data, calculating the display point according to the flight telemetry data, integrating the display point and the UAV video stream, and playing the picture, wherein the flight telemetry data comprises the real-time longitude and latitude of the UAV, the real-time flight height of the UAV, the UAV camera pitch angle and the UAV camera yaw angle.

2. The method of claim 1, wherein, The method for displaying a POI (Point of Interest) label on a screen, receiving the UAV video stream and flight telemetry data, calculating the display point according to the flight telemetry data, integrating the display point and the UAV video stream, and playing the picture, comprises the following steps: calculating the machine label distance, and calculating the vertical visual angle according to the machine label distance, wherein the machine label distance is the physical distance between the UAV position and the POI label position; calculating the azimuth angle according to the eastward distance and the northward distance, and obtaining the horizontal visual angle according to the azimuth angle, wherein the eastward distance is the physical distance calculated by taking the latitude of the UAV as the latitude of the POI label, the northward distance is the physical distance calculated by taking the longitude of the UAV as the longitude of the POI label, and the azimuth angle is the included angle between the POI label in the UAV visual line and the X axis in the three-dimensional space; acquiring the equivalent focal length according to the pixel value of the image width, and calculating the offset angle according to the equivalent focal length, wherein the offset angle is the horizontal offset angle of the POI label relative to the UAV camera, and the offset angle is in radian; calculating the two-dimensional coordinate X axis coordinate value and the two-dimensional coordinate Y axis coordinate value according to the offset angle, wherein the two-dimensional coordinate X axis coordinate value and the two-dimensional coordinate Y axis coordinate value are the position coordinates of the output picture.

3. The method of claim 2, wherein, calculating the machine label distance, and calculating the vertical visual angle according to the machine label distance, comprising the following steps: the calculation formula of the machine label distance is: ， wherein, is a first auxiliary parameter, is a second auxiliary parameter, is the machine-to-marker distance, is the latitude difference between the drone position and the POI position, is the longitude difference between the drone position and the POI marker position, is the latitude of the drone position, is the latitude of the POI marker position, is the average radius of the Earth, , , and is in radians; when the machine label distance is greater than the visual radius, the POI label corresponding to the machine label distance cannot be displayed in the picture; The vertical radian is calculated, and the calculation formula is wherein, is an inverse tangent function, is the machine mark distance, is the height difference; The vertical radian is converted into a vertical line of sight angle, and the calculation formula is: wherein, is the vertical line of sight angle. when the height difference is 0, the value of the vertical visual angle is directly obtained as 0.

4. The method of claim 2, wherein, calculating the azimuth angle according to the eastward distance and the northward distance, and obtaining the horizontal visual angle according to the azimuth angle, comprising the following steps: calculating the eastward distance, wherein the physical distance calculated by taking the latitude of the UAV as the latitude of the POI label is the eastward distance, and the calculation formula is: ， wherein, is a first east auxiliary parameter, is a second east auxiliary parameter, is the east distance, is a longitude difference between the UAV position and the POI tag position, is a latitude of the UAV position, is an average radius of the earth, the tag-UAV distance is a physical distance between the UAV position and the POI position, and is in radian, the east distance becomes negative when the longitude of the POI tag is less than the longitude of the UAV. To calculate the northward distance, the longitude of the UAV is used as the longitude of the POI tag to calculate the physical distance, which is the northward distance. The calculation formula is as follows: , ,in, This is the first northward auxiliary parameter. This is the second northward auxiliary parameter. The northward distance is... The difference in latitude between the drone's location and the POI's location. The average radius of the Earth is given, and the distance between the drone's location and the point of interest (POI) location is defined as the physical distance between the drone's location and the POI's location. The distance is measured in radians. When the latitude of the POI tag is less than the latitude of the UAV, the northward distance becomes a negative value.

5. The method of claim 2, wherein, The azimuth is calculated according to the eastward distance and the northward distance, and the horizontal line-of-sight angle is obtained according to the azimuth, comprising: The azimuth angle is calculated with the formula wherein is the azimuth angle, is the four-quadrant arctangent function, is the eastward distance, is the northward distance.

6. The method of claim 2, wherein, The azimuth is calculated according to the eastward distance and the northward distance, and the horizontal line-of-sight angle is obtained according to the azimuth, comprising: When both the eastward distance and the northward distance are positive numbers, the POI label is in the northeast direction of the UAV, the horizontal line-of-sight angle is calculated, and the calculation formula is wherein, the horizontal line-of-sight angle is the azimuth angle; When the eastward distance is negative and the northward distance is positive, the POI tag is in the northwest of the UAV, the horizontal line-of-sight angle is calculated, and the calculation formula is ; When both the eastward distance and the northward distance are negative, the POI tag is in the southwest direction of the UAV, the horizontal line-of-sight angle is calculated according to the following formula ; When the eastward distance is positive and the northward distance is negative, the POI label is in the northwest direction of the UAV, the horizontal line-of-sight angle is calculated, and the calculation formula is .

7. The method of claim 2, wherein, The equivalent focal length is obtained according to the pixel value of the image width, and the offset angle is calculated according to the equivalent focal length, comprising: The equivalent focal length is calculated, and the calculation formula is wherein, is the equivalent focal length, is the pixel value of the image width, is the horizontal visual angle; The offset angle is calculated, with the calculation formula being wherein, is the offset angle, is the horizontal line-of-sight angle.

8. The method of claim 2, wherein, The two-dimensional coordinate X-axis coordinate value and the two-dimensional coordinate Y-axis coordinate value are calculated according to the offset angle, comprising: The two-dimensional coordinate Y-axis coordinate value is calculated, and the calculation formula is: ， wherein, is a UAV pitch angle, the UAV pitch angle being a UAV fixed parameter, is the vertical angle of view, denotes a radian, is the offset angle, is the equivalent focal length; The two-dimensional coordinate X-axis coordinate value is calculated, and the calculation formula is: 。 9.A camera labeling device for real-time video of a drone, characterized in that, Comprising: The POI database is constructed, and at least one POI data item is stored in the POI database, wherein the POI data item at least includes a POI label name, a POI label type, a POI label longitude, a POI label latitude, and POI label height information, and one POI data item corresponds to one POI label; The visual radius and the horizontal visual angle of the unmanned aerial vehicle camera are obtained, and the height difference, the pitch angle compensation value, and the yaw angle compensation value are obtained, wherein the height difference is the height difference between the unmanned aerial vehicle take-off position and the POI label position, the pitch angle compensation value is the unmanned aerial vehicle camera pitch angle compensation value, and the yaw angle compensation value is the unmanned aerial vehicle camera yaw angle compensation value; The video stream and the flight telemetry data of the unmanned aerial vehicle are received, the display point is obtained according to the flight telemetry data, the display point and the unmanned aerial vehicle video stream are integrated, and the picture is played out, wherein the flight telemetry data includes the real-time longitude and latitude of the unmanned aerial vehicle, the real-time flight height of the unmanned aerial vehicle, the unmanned aerial vehicle camera pitch angle, and the unmanned aerial vehicle camera yaw angle.

Images