Method and device for displaying target point information in real-time video of unmanned aerial vehicle and medium

Abstract

The invention provides a method and device for displaying target point information in a real-time video of an unmanned aerial vehicle and a medium. The method comprises the steps of obtaining unmanned aerial vehicle telemetry information and geographic position information of a target point needing to add to-be-displayed information; calculating the position of the target point in the view cone of the unmanned aerial vehicle camera by using the unmanned aerial vehicle telemetry information and the geographic position information of the target point, and calculating the distance from the unmanned aerial vehicle camera to the target point; to-be-displayed information related to the target point is configured for the information mask in advance; and configuring an information mask for the target point in the real-time video, determining the position of the information mask in the real-time video according to the position of the target point in the view cone of the unmanned aerial vehicle camera and the distance from the unmanned aerial vehicle camera to the target point, and controlling the display time of the information mask. According to the method, the user can directly display the to-be-displayed information of the customized position, namely the target point, in the real-time video of the unmanned aerial vehicle, so that the real-time video content of the unmanned aerial vehicle is better combined with the business, and the application of the unmanned aerial vehicle in the business is expanded.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicle (UAV) video processing, and particularly to a method for displaying target point information in real-time UAV video. Background Art

[0002] In recent years, with the development of UAV technology, various industries have successively introduced UAVs to assist in work. By using UAVs to collect videos of targets in real-time, the inspection of targets can be easily completed. The human visual system does not treat each object in the picture equally. It will focus on the area of attention in the picture and ignore the information that is not the focus of attention. Simply using the video of the target for inspection cannot better determine the focus of attention. In order to complete the inspection task of the target through the real-time video of the target with a limited duration, it is necessary to scan the entire picture for the attention points, which is time-consuming and laborious. Therefore, a method for displaying target point information in real-time UAV video is needed, which can display information in real-time UAV video. The displayed information not only describes the target, providing clear, vivid and specific information for users, but also gives the attention points, enabling the inspection personnel to pay attention to the important positions in the real-time video picture faster, and combining with the actual scene inspection requirements. Summary of the Invention

[0003] To solve the above technical problems or at least partially solve the above technical problems, the present invention provides a method, device and medium for displaying target point information in real-time UAV video.

[0004] In a first aspect, the present invention provides a method for displaying target point information in real-time UAV video, including: Obtaining the telemetry information of the UAV and the geographical location information of the target point where the information to be displayed needs to be added, where the UAV telemetry information includes the longitude and latitude position information of the UAV camera, and the target point geographical location information includes the longitude and latitude position information of the target point; Calculating the position of the target point in the viewing cone of the UAV camera by using the UAV telemetry information and the target point geographical location information; calculating the distance from the UAV camera to the target point by using the positions of the target point and the UAV camera in the Cartesian coordinate system; Pre-configuring the information to be displayed related to the target point for the information mask; Configuring an information mask for the target point where the information to be displayed needs to be added in the real-time video, where the position of the information mask in the real-time video is determined according to the position of the target point in the viewing cone of the UAV camera and the distance from the UAV camera to the target point, and the display time of the information mask is controlled.

[0005] Furthermore, the telemetry information of the UAV is obtained through the WebSocket long connection method.

[0006] Further, calculating the position of the target point in the field of view cone of the UAV camera using the UAV telemetry information and the geographical location information of the target point includes the following steps: Obtain the longitude and latitude position information of the UAV and the target point from the UAV telemetry information and the geographical location information of the target point, and convert the longitude and latitude into a Cartesian coordinate system; Calculate the azimuth angle and pitch angle of the target point in the Cartesian coordinate system. The calculation methods are as follows: , , where is the calculated azimuth angle of the target point, is the calculated pitch angle of the target point, is the x-axis coordinate of the target point in the Cartesian coordinate system, is the y-axis coordinate of the target point in the Cartesian coordinate system, is the z-axis coordinate of the target point in the Cartesian coordinate system, is the x-axis coordinate of the camera in the Cartesian coordinate system, is the y-axis coordinate of the camera in the Cartesian coordinate system, is the z-axis coordinate of the camera in the Cartesian coordinate system; Calculate the pitch angle and azimuth angle of the target point relative to the camera using the azimuth angle, pitch angle and pan-tilt attitude of the target point in the Cartesian coordinate system. The calculation methods are as follows: , , where and are the pitch angle and azimuth angle of the target point relative to the camera respectively, and are the yaw angle and pitch angle of the pan-tilt attitude; Calculate the horizontal angle and vertical angle of the target point relative to the camera field of view cone using the pitch angle, azimuth angle and pan-tilt attitude of the target point relative to the camera. The calculation methods are as follows: , , and are the horizontal angle and vertical angle of the target point relative to the camera field of view cone; Calculate the horizontal and vertical offset rates of the target point relative to the camera field of view cone using the normalized horizontal angle and vertical angle of the target point relative to the camera field of view cone. The calculation methods are as follows: , , Wherein, is the angle of the rolling angle of the pan-tilt, and are the horizontal and vertical offset rates of the target point with respect to the viewing cone, and are the horizontal angle and vertical angle of the normalized target point with respect to the camera viewing cone; Determine the position of the target point in the viewing cone using the offset rate: , , Wherein, and are the position information of the target point within the viewing cone.

[0007] Furthermore, the calculation method of the horizontal angle and vertical angle of the normalized target point with respect to the camera viewing cone is as follows: , , Wherein, and are the wide angle and height angle of the camera viewing cone.

[0008] Furthermore, calculate the distance from the drone to the target point using the positions of the target point and the drone camera in the Cartesian coordinate system: , is the distance from the camera to the target point.

[0009] Furthermore, when the horizontal offset rate of the target point with respect to the viewing cone , the vertical offset rate of the target point with respect to the viewing cone , and the distance of the target point from the camera , display an information mask in the real-time video of the drone.

[0010] Furthermore, the information mask is a rectangle centered on the pixel of the target point in the video, and the size of the information mask is adapted to the information to be displayed configured for it.

[0011] Furthermore, when pre-configuring the information to be displayed related to the target point for the information mask, associate and store the information to be displayed with the geographical information position of the target point, and establish a connection between the information to be displayed and the information mask based on the association between the information mask and the geographical location information.

[0012] In a second aspect, the present invention provides a device for displaying target point information in real-time video of an unmanned aerial vehicle (UAV), comprising: at least one processing unit, which connects the processing unit, a storage unit, a display unit, and a communication unit through a bus unit. The communication unit communicates with the UAV. The storage unit serves as a computer-readable storage medium and can be used to store software programs, computer-executable programs, and modules. The processing unit realizes the method for displaying target point information in the real-time video of the UAV by running the software programs, computer-executable programs, and modules stored in the storage unit.

[0013] In a third aspect, the present invention provides a computer-readable storage medium storing a computer program, which, when executed, realizes the method for displaying target point information in the real-time video of the UAV.

[0014] The above technical solutions provided by the embodiments of the present invention have the following advantages compared with the prior art: This application obtains the telemetry information of the UAV and the geographical location information of the target points where the information to be displayed needs to be added. The UAV telemetry information includes the longitude and latitude position information of the UAV camera, and the geographical location information of the target points includes the longitude and latitude position information of the target points. It calculates the position of the target points in the viewing cone of the UAV camera using the UAV telemetry information and the geographical location information of the target points. It calculates the distance from the UAV camera to the target points using the positions of the target points and the UAV camera in the Cartesian coordinate system. It pre-configures the information to be displayed related to the target points for the information mask. It configures the information mask for the target points where the information to be displayed needs to be added in the real-time video. Among them, the position of the information mask in the real-time video is determined according to the position of the target points in the viewing cone of the UAV camera and the distance from the UAV camera to the target points, and the display time of the information mask is controlled. It provides a simple and intuitive method, enabling users to directly display the information to be displayed at the customized positions, i.e., the target points, in the real-time video of the UAV, making the real-time video content of the UAV better integrated with the business, and expanding the application of the UAV in the business. This application provides an attention point through the information mask, helping the inspection personnel to determine the location of the important information in the picture and saving the time for the inspection personnel to check each picture of the real-time video. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings herein are incorporated into the specification and form a part of this specification, showing embodiments consistent with the present invention and, together with the specification, are used to explain the principles of the present invention.

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the accompanying drawings required for use in the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, other drawings can also be obtained based on these drawings without creative efforts.

[0017] Figure 1 Flow chart of the method for displaying target point information in the real-time video of the unmanned aerial vehicle provided by the embodiment of the present invention; Figure 2 Flow chart of calculating the position of a target point in the viewing cone of the unmanned aerial vehicle camera by using the telemetry information of the unmanned aerial vehicle and the geographical location information of the target point provided by the embodiment of the present invention; Figure 3 Schematic diagram of the real-time video and information mask in the inspection task of the dam facility by the present application provided by the embodiment of the present invention; Figure 4 Schematic diagram of the device for displaying target point information in the real-time video of the unmanned aerial vehicle provided by the embodiment of the present invention. Detailed implementation manners

[0018] To make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some but not all of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0019] It should be noted that in this document, the terms "include", "comprise" or any other variant thereof are intended to cover a non-exclusive inclusion, such that a process, method, article or device including a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the phrase "including a..." does not exclude the presence of additional identical elements in the process, method, article or device including the element.

[0020] Embodiment 1 The objective of the present invention is to provide a method for displaying target point information in the real-time video of an unmanned aerial vehicle, which enables a user to view the information to be displayed at the actual geographical location of the corresponding target point through the real-time video of the unmanned aerial vehicle.

[0021] As Figure 1 shown, the specific implementation steps are as follows: S1: Obtain the telemetry information of the unmanned aerial vehicle and the geographical location information of the target point where the information to be displayed needs to be added; the telemetry information of the unmanned aerial vehicle includes the longitude and latitude position information of the unmanned aerial vehicle camera, and the geographical location information of the target point includes the longitude and latitude position information of the target point. In the specific implementation process, the telemetry information of the unmanned aerial vehicle is obtained through the Websocket long connection method.

[0022] S2: Calculate the position of the target point in the viewing cone of the UAV camera using the UAV telemetry information and the geographical location information of the target point, as Figure 2 shown, including the following steps: S21: Obtain the longitude and latitude position information of the UAV and the target point from the UAV telemetry information and the geographical location information of the target point, and convert the longitude and latitude into the Cartesian coordinate system; S22: Calculate the azimuth angle and elevation angle of the target point in the Cartesian coordinate system. The calculation method is as follows: , , where is the calculated azimuth angle of the target point, is the calculated elevation angle of the target point, is the x-axis coordinate of the target point in the Cartesian coordinate system, is the y-axis coordinate of the target point in the Cartesian coordinate system, is the z-axis coordinate of the target point in the Cartesian coordinate system, is the x-axis coordinate of the camera in the Cartesian coordinate system, is the y-axis coordinate of the camera in the Cartesian coordinate system, is the z-axis coordinate of the camera in the Cartesian coordinate system; S23: Calculate the elevation angle and azimuth angle of the target point relative to the camera using the azimuth angle, elevation angle of the target point in the Cartesian coordinate system and the pan-tilt attitude. The calculation method is as follows: , , where and are the elevation angle and azimuth angle of the target point relative to the camera respectively, and are the yaw angle and elevation angle of the pan-tilt attitude; S24: Calculate the horizontal angle and vertical angle of the target point relative to the viewing cone of the camera using the elevation angle, azimuth angle of the target point relative to the camera and the pan-tilt attitude. The calculation method is as follows: , , and are the horizontal angle and vertical angle of the target point relative to the viewing cone of the camera.

[0023] S25: Normalize the horizontal angle and vertical angle of the target point relative to the viewing cone of the camera. The calculation method is as follows: , , where and are the horizontal and vertical angles of the normalized target point relative to the camera cone, and are the wide angle and height angle of the camera cone; S26: Calculate the horizontal and vertical offset rates of the target point relative to the camera cone using the horizontal and vertical angles of the normalized target point relative to the camera cone. The calculation method is as follows: , , where is the angle of the pan tilt, and are the horizontal and vertical offset rates of the target point relative to the cone, S27: Determine the position of the target point in the cone using the offset rate: , , where and are the position information of the target point in the cone.

[0024] S3: Calculate the distance from the UAV to the target point using the positions of the target point and the UAV camera in the Cartesian coordinate system: , is the distance from the camera to the target point.

[0025] S4: Pre-configure the information mask with the information to be displayed related to the target point. When pre-configuring the information mask with the information to be displayed related to the target point, store the information to be displayed in association with the geographical information position of the target point, and establish a connection between the information to be displayed and the information mask based on the association between the information mask and the geographical position information.

[0026] S5: Configure an information mask for the target point that needs to add the information to be displayed in the real-time video. Among them, determine the position of the information mask in the real-time video according to the position of the target point in the UAV camera cone and the distance from the UAV camera to the target point, and control the display time of the information mask.

[0027] When the horizontal offset rate of the target point relative to the cone , the vertical offset rate of the target point relative to the cone , and the distance from the target point to the camera , display the information mask in the real-time video of the UAV; Among them, are respectively the lower and upper limits of the horizontal offset rate of the target point with respect to the viewing frustum, are respectively the lower and upper limits of the vertical offset rate of the target point with respect to the viewing frustum; is the distance threshold.

[0028] The information mask is a rectangle centered on the pixel of the target point in the video, and the size of the information mask adapts to the information to be displayed it configures.

[0029] As Figure 3 shown, in a drone inspection flight mission of a reservoir dam, dam facility information is added. When the drone flies near the dam, according to the real-time telemetry data of the drone, the attitude information of the pan-tilt head, and the geographical location information of the dam facilities, it is determined whether the dam facilities are within the viewing frustum of the drone pan-tilt head, and an information mask of the dam facilities is added to the real-time video of the drone, which is combined with the real-time video to vividly display the information of the dam facilities.

[0030] Embodiment 2 Refer to Figure 4 shown, an embodiment of the present invention provides a device for displaying target point information in a drone real-time video, including: at least one processing unit, which connects the processing unit, the storage unit, the display unit, and the communication unit through a bus unit. The communication unit communicates with the drone. The storage unit is a computer-readable storage medium and can be used to store software programs, computer-executable programs, and modules, such as the software programs, computer-executable programs, and modules corresponding to a method for displaying target point information in a drone real-time video in an embodiment of the present invention. The processing unit realizes the above-mentioned method for displaying target point information in a drone real-time video by running the software programs, computer-executable programs, and modules stored in the storage unit, including: Obtain the drone telemetry information and the geographical location information of the target point where the information to be displayed needs to be added. The drone telemetry information includes the longitude and latitude position information of the drone camera, and the target point geographical location information includes the longitude and latitude position information of the target point; Calculate the position of the target point in the viewing frustum of the drone camera by using the drone telemetry information and the target point geographical location information; calculate the distance from the drone camera to the target point by using the positions of the target point and the drone camera in the Cartesian coordinate system; Pre-configure the information to be displayed related to the target point for the information mask; Configure an information mask for the target point where the information to be displayed needs to be added in the real-time video. Among them, determine the position of the information mask in the real-time video and control the display time of the information mask according to the position of the target point in the viewing frustum of the drone camera and the distance from the drone camera to the target point.

[0031] Certainly, the storage unit in the device for displaying target point information in the real-time video of an unmanned aerial vehicle provided by the embodiments of the present invention stores a computer program that is not limited to the method operations described above, and can also execute relevant operations in the method for displaying target point information in the real-time video of an unmanned aerial vehicle provided by any embodiment of the present invention.

[0032] Embodiment 3 The embodiments of the present invention provide a computer-readable storage medium that stores a computer program, and when the computer program is executed, it implements the method for displaying target point information in the real-time video of an unmanned aerial vehicle, including: Obtaining the telemetry information of the unmanned aerial vehicle and the geographical location information of the target point where the information to be displayed needs to be added, where the telemetry information of the unmanned aerial vehicle includes the longitude and latitude position information of the unmanned aerial vehicle camera, and the geographical location information of the target point includes the longitude and latitude position information of the target point; Calculating the position of the target point in the visual cone of the unmanned aerial vehicle camera using the telemetry information of the unmanned aerial vehicle and the geographical location information of the target point; calculating the distance from the unmanned aerial vehicle camera to the target point using the positions of the target point and the unmanned aerial vehicle camera in the Cartesian coordinate system; Pre-configuring the information to be displayed related to the target point for the information mask; Configuring an information mask for the target point where the information to be displayed needs to be added in the real-time video, where the position of the information mask in the real-time video is determined according to the position of the target point in the visual cone of the unmanned aerial vehicle camera and the distance from the unmanned aerial vehicle camera to the target point, and the display time of the information mask is controlled.

[0033] For the computer-readable storage medium provided by the embodiments of the present invention, the computer program stored therein is not limited to the method operations described above, and can also execute relevant operations in the method for displaying target point information in the real-time video of an unmanned aerial vehicle provided by any embodiment of the present invention.

[0034] In the embodiments provided by the present invention, it should be understood that the disclosed structure and method can be implemented in other ways. For example, the structural embodiments described above are merely illustrative. For example, the division of the units is only a logical function division, and there can be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the coupling or direct coupling or communication connection shown or discussed with each other can be through some interfaces, and the indirect coupling or communication connection of the structure or unit can be in an electrical, mechanical or other forms.

[0035] The unit described as a separation component may or may not be physically separated. The component shown as a unit may or may not be a physical unit, that is, it may be located in one place or may be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0036] In addition, each functional unit in various embodiments of the present invention may be integrated in a processing unit, may also exist separately as individual physical units, or two or more units may be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0037] The above are only specific embodiments of the present invention, enabling those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be obvious to those skilled in the art. The general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to these embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for displaying target point information in real-time video of an unmanned aerial vehicle, characterized in that, Including: Obtain the telemetry information of the drone and the geographical location information of the target point where the information to be displayed needs to be added. The drone telemetry information includes the longitude and latitude position information of the drone camera, and the geographical location information of the target point includes the longitude and latitude position information of the target point; Calculate the position of the target point in the field of view cone of the drone camera using the drone telemetry information and the geographical location information of the target point; calculate the distance from the drone camera to the target point using the positions of the target point and the drone camera in the Cartesian coordinate system; Pre-configure the information to be displayed related to the target point for the information mask; Configure an information mask for the target point that needs to add the information to be displayed in the real-time video. Among them, determine the position of the information mask in the real-time video according to the position of the target point in the field of view cone of the drone camera and the distance from the drone camera to the target point, and control the display time of the information mask.

2. The method for displaying target point information in the real-time video of the unmanned aerial vehicle according to claim 1, characterized in that, Obtain the telemetry information of the drone through the way of Websocket long connection.

3. The method for displaying target point information in the real-time video of the unmanned aerial vehicle according to claim 1, wherein Calculating the position of the target point in the field of view cone of the drone camera using the drone telemetry information and the geographical location information of the target point includes the following steps: Obtain the longitude and latitude position information of the drone and the target point from the drone telemetry information and the geographical location information of the target point, and convert the longitude and latitude into the Cartesian coordinate system; Calculate the azimuth angle and pitch angle of the target point in the Cartesian coordinate system. The calculation method is as follows: ， ， wherein is the calculated azimuth angle of the target point, is the calculated elevation angle of the target point, is the x-axis coordinate of the target point in the Cartesian coordinate system, is the y-axis coordinate of the target point in the Cartesian coordinate system, is the z-axis coordinate of the target point in the Cartesian coordinate system, is the x-axis coordinate of the camera in the Cartesian coordinate system, is the y-axis coordinate of the camera in the Cartesian coordinate system, is the z-axis coordinate of the camera in the Cartesian coordinate system; Calculate the pitch angle and azimuth angle of the target point relative to the camera using the azimuth angle, pitch angle and pan-tilt attitude of the target point in the Cartesian coordinate system. The calculation method is as follows: ， ， where and are respectively the pitch angle and azimuth angle of the target point relative to the camera, and are the yaw angle and pitch angle of the pan-tilt attitude; Calculate the horizontal angle and vertical angle of the target point relative to the camera field of view cone using the pitch angle, azimuth angle of the target point relative to the camera and the pan-tilt attitude. The calculation method is as follows: ， ， and are the horizontal and vertical angles of the target point relative to the camera's visual cone; Calculate the horizontal and vertical offset rates of the target point relative to the camera field of view cone using the normalized horizontal angle and vertical angle of the target point relative to the camera field of view cone. The calculation method is as follows: ， ， Among them, is the angle of the rolling angle of the pan-tilt, and are the horizontal and vertical offset rates of the target point with respect to the viewing cone, and are the horizontal and vertical angles of the normalized target point with respect to the camera viewing cone; Determine the position of the target point in the field of view cone using the offset rate: ， ， Among them, and are the position information of the target point within the viewing cone.

4. The method for displaying target point information in the real-time video of the unmanned aerial vehicle according to claim 3, wherein The calculation method for normalizing the horizontal angle and vertical angle of the target point relative to the camera field of view cone is as follows: ， ， Among them, and are the wide angle and height angle of the camera's viewing cone.

5. The method for displaying target point information in the real-time video of an unmanned aerial vehicle according to claim 3, characterized in that, Calculate the distance from the drone to the target point using the positions of the target point and the drone camera in the Cartesian coordinate system: ， is the distance from the camera to the target point.

6. The method for displaying target point information in the real-time video of the drone according to claim 3 or 5, characterized in that When the horizontal offset rate of the target point with respect to the frustum , the vertical offset rate of the target point with respect to the frustum , and the distance from the target point to the camera , display an information mask in the real-time video of the drone; wherein, are respectively the lower and upper limits of the horizontal offset rate of the target point with respect to the viewing frustum, are respectively the lower and upper limits of the vertical offset rate of the target point with respect to the viewing frustum; is the distance threshold.

7. The method for displaying target point information in the real-time video of an unmanned aerial vehicle according to claim 1, wherein The information mask is a rectangle centered on the pixel of the target point in the video, and the size of the information mask is adapted to the information to be displayed configured for it.

8. The method for displaying target point information in the real-time video of the unmanned aerial vehicle according to claim 1, wherein When pre-configuring the information to be displayed related to the target point for the information mask, associate and store the information to be displayed with the geographical information position of the target point, and establish the connection between the information to be displayed and the information mask based on the association between the information mask and the geographical location information.

9. A device for displaying target point information in real-time video of an unmanned aerial vehicle, characterized in that, Including: At least one processing unit, which connects the processing unit, storage unit, display unit and communication unit through the bus unit. The communication unit communicates with the drone. The storage unit is used as a computer-readable storage medium and can be used to store software programs, computer-executable programs and modules. The processing unit realizes the method for displaying target point information in the real-time video of the drone as described in any one of claims 1-8 by running the software programs, computer-executable programs and modules stored in the storage unit.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed, it implements the method for displaying target point information in the real-time video of the unmanned aerial vehicle as described in any one of claims 1-8.

Images