Water area shoreline element projection display method based on unmanned aerial vehicle virtual reality enhancement technology

Abstract

The invention provides a water area shoreline element projection display method based on an unmanned aerial vehicle virtual reality enhancement technology. The method comprises the steps of obtaining water area shoreline geographic element data; obtaining unmanned aerial vehicle position information, pan-tilt pitching angle information and camera lens parameters; obtaining the offset of the projection position of the geographic element in the screen coordinate system relative to the imaging center point of the camera; and superposing the geographic element projection positions on a real-time aerial picture of the unmanned aerial vehicle to realize real-time projection display of the water area shoreline geographic elements. According to the method, geographic element information can be projected and displayed on a terminal display screen of the unmanned aerial vehicle in real time through the mapping relation between the geographic elements of the water area shoreline and the geometric positions and different coordinates of the unmanned aerial vehicle camera, so that the purpose of reducing the working steps of manual identification and later-stage encirclement is achieved; therefore, the application capability of the unmanned aerial vehicle in water shoreline inspection and water resource protection tasks is improved.

Description

Technical Field

[0001] The present invention relates to the field of information technology - artificial intelligence - computer vision technology, and particularly relates to a method for projecting and displaying geographical elements on a drone side based on AR virtual reality enhancement. Background Art

[0002] Waterfront management is a complex and important task, involving the collection, processing, and analysis of a large amount of geographical information data. Traditional waterfront monitoring methods often rely on manual on-site inspections or static geographical information systems, and it is difficult to reflect the dynamic changes of the waterfront in a timely manner.

[0003] With the development of drone technology, real-time aerial photography provides a new means for waterfront monitoring. Drone aerial photography plays an important role in the supervision of water conservancy river and lake shorelines, and it has the characteristics of flexible use and high-altitude perspective.

[0004] In traditional drone inspection and review tasks, remote sensing points are not clearly indicated in aerial photos, and manual identification and post-labeling of aerial photos are required, which limits the application ability of drones in waterfront inspection and water resource protection tasks.

[0005] In the process of real-time drone aerial photography, how to achieve the precise positioning and real-time labeling of waterfront geographical elements is an urgent problem to be solved. Summary of the Invention

[0006] The purpose of the present invention is to address the above problems and propose a method for projecting and displaying waterfront elements based on drone virtual reality enhancement technology.

[0007] The technical solution of the present invention is as follows:

[0008] The present invention provides a method for projecting and displaying waterfront elements based on drone virtual reality enhancement technology, including:

[0009] S1. Obtain geographical element data of the waterfront;

[0010] S2. Obtain the drone position information, the pan-tilt angle information of the gimbal, and the camera lens parameters;

[0011] S3. According to the geographical element data of the waterfront, the drone position information, the pan-tilt angle information of the gimbal, and the camera lens parameters, obtain the offset of the projected position of the geographical element in the screen coordinate system relative to the camera imaging center point;

[0012] S4. Superimpose the projected position of the geographical element on the real-time drone aerial photography image to achieve real-time projection display of the waterfront geographical elements.

[0013] Further, the obtaining of the geographical element data of the water area shoreline in S1 includes: reading the spatial coordinate information of the geographical elements of the water area shoreline from the geographical information database, where the spatial coordinate information includes the longitude, latitude, and elevation information of the geographical element contour.

[0014] Further, the obtaining of the UAV position information, the pan-tilt angle information of the pan-tilt head, and the camera lens parameters in S2 includes:

[0015] Obtaining the real-time spatial position coordinates, yaw angle β, and flight altitude h of the UAV through the UAV flight control system;

[0016] Obtaining the real-time pitch angle α of the pan-tilt head through the UAV pan-tilt head;

[0017] Obtaining the lens focal length f through the camera.

[0018] Further, the obtaining of the offset of the projected position of the geographical element in the screen coordinate system relative to the camera imaging center point in S3 includes:

[0019] Constructing the geometric relationship of the imaging of the camera imaging center point P and the actual position T of the geographical element in the world coordinate system on the screen coordinate system through the camera position O;

[0020] In the vertical direction, the following formula is used to obtain the longitudinal offset T′P′ of the geographical element point in the screen coordinate system y ;

[0021]

[0022] where: f represents the camera lens focal length; h is the UAV flight altitude; α represents the real-time pitch angle of the pan-tilt head; O′T represents the distance between the projection position O′(X O′ , Y O′ ) of the camera on the ground and the spatial coordinate point T(X T , Y T ) of the geographical element,

[0023] In the horizontal direction, the following formula is used to obtain the lateral offset T′P′ of the geographical element point in the screen coordinate system x ;

[0024]

[0025] where: β represents the real-time yaw angle of the UAV, OP represents the distance between the camera position O and the camera imaging center point P,

[0026] Further, superimposing the projected position of the geographical element on the real-time aerial photography image of the UAV includes:

[0027] Obtain real-time aerial images of the drone;

[0028] According to the offset and the spatial coordinate information of the elements in the geographic information database, obtain the projection positions of all contour points of the geographic elements in the screen coordinate system;

[0029] Draw the contour points of the geographic elements at the corresponding positions in the real-time aerial image, and connect all the contour points to form a labeled box of the geographic elements.

[0030] Furthermore, the method further includes:

[0031] According to the flight state of the drone, update the projection position of the geographic elements in real time, so as to realize the synchronous change of the labeled box of the geographic elements and the aerial image of the drone.

[0032] Furthermore, the method further includes:

[0033] Perform contour extraction processing on the real-time aerial image of the drone, and extract the real-time contours of the geographic elements in the image;

[0034] Compare the extracted real-time contours of the geographic elements with the labeled box, calculate the deviation degree between the two; determine whether the calculated deviation degree exceeds a preset threshold;

[0035] If it exceeds the threshold, trigger an alarm prompt to remind the user that there is a large deviation between the labeled box and the actual contour of the geographic element;

[0036] Continuously obtain subsequent real-time aerial images of the drone, and perform continuous tracking annotation and deviation alarm on the geographic elements.

[0037] Furthermore, comparing the extracted real-time contours of the geographic elements with the labeled box and calculating the deviation degree between the two specifically means calculating the overlap degree Q between the real-time contour and the labeled box using the following formula;

[0038]

[0039] Where: Q represents the overlap degree, R represents the real-time contour area, and S represents the labeled box area.

[0040] Advantages of the present invention:

[0041] The method for projecting and displaying geographic elements of the water area shoreline based on the virtual reality enhancement technology of the drone of the present invention, by calculating the geometric position and the mapping relationship of different coordinates between the geographic elements of the water area shoreline and the drone camera, enables the geographic element information to be projected and displayed on the drone terminal display screen in real time, achieving the purpose of reducing the work steps of manual identification and later marking, thereby improving the application ability of the drone in the tasks of water area shoreline inspection and water resource protection.

[0042] The present invention discloses a method for projecting and displaying water area shoreline elements based on UAV virtual reality enhancement technology, which can dynamically annotate water area shoreline geographical elements during real-time UAV aerial photography. This method calculates the projection positions of geographical elements in the screen coordinate system by acquiring water area shoreline geographical element data, UAV position information, pan-tilt angle of the gimbal, and camera parameters, and superimposes them on the real-time aerial photography images.

[0043] The present invention also introduces a real-time contour extraction and deviation detection mechanism. By calculating the intersection-over-union ratio of the annotation box and the contour of the actual geographical element, it determines whether there is a significant deviation and triggers an alarm. The combination of this dynamic annotation and deviation detection not only achieves precise positioning and real-time tracking of geographical elements but also can timely detect annotation errors, improving the accuracy and reliability of water area shoreline monitoring.

[0044] The present invention has important application values in the fields of water conservancy projects, environmental monitoring, etc., providing a new solution for the application of UAV aerial photography technology in geographic information collection and analysis.

[0045] Other features and advantages of the present invention will be described in detail in the following specific implementation section. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] By describing the exemplary embodiments of the present invention in more detail in conjunction with the accompanying drawings, the above and other objects, features, and advantages of the present invention will become more apparent. Among them, in the exemplary embodiments of the present invention, the same reference numerals generally represent the same components.

[0047] Figure 1 Shows a flowchart of a method for projecting and displaying water area shoreline elements based on UAV virtual reality enhancement technology.

[0048] Figure 2 Shows a transformation principle diagram of the longitudinal offset of the projection position of a geographical element in the screen coordinate system relative to the camera imaging center point according to an embodiment of the present invention.

[0049] Figure 3 Shows a transformation principle diagram of the lateral offset of the projection position of a geographical element in the screen coordinate system relative to the camera imaging center point according to an embodiment of the present invention.

[0050] Figure 4 Shows a schematic diagram of the offset of the target point position in the X direction and Y direction relative to the camera center according to an embodiment of the present invention.

[0051] Figure 5 Shows a schematic diagram of the relative position of the aircraft and the camera shooting range according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0052] The preferred embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the preferred embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein.

[0053] As Figure 1 shown, the present invention provides a method for projecting and displaying water area shoreline elements based on UAV virtual reality enhancement technology, specifically including:

[0054] S1. Obtain water area shoreline geographic element data; including reading the spatial coordinate information of water area shoreline geographic elements from a geographic information database, and the spatial coordinate information includes longitude, latitude, and elevation information of the geographic element contour.

[0055] In one example, the specific implementation method for obtaining water area shoreline geographic element data is: obtaining the vector line position coordinates of the water area shoreline and the attribute information of the shoreline elements, such as shoreline name, type, length, etc. through a geographic information database and remote sensing image data. These geographic element data are usually stored and expressed in vector formats such as shp, kml, etc., which are common formats of GIS.

[0056] Perform simplification and thinning processing on the obtained vector shoreline element data, extract its feature points, reduce the data volume, and improve the calculation efficiency of subsequent projection transformation and display rendering. By obtaining authoritative and accurate geographic element data and performing necessary preprocessing, a data foundation is laid for subsequent real-time superposition of geographic elements and UAV aerial video.

[0057] S2. Obtain UAV position information, gimbal pitch angle information, and camera lens parameters; including obtaining the real-time spatial position coordinates, yaw angle β, and flight altitude h of the UAV through the UAV flight control system; obtaining the real-time pitch angle α of the gimbal through the UAV gimbal; and obtaining the lens focal length f through the camera.

[0058] In one example, the GPS position coordinates, altitude, yaw angle, and other attitude information of the UAV are obtained in real time through the UAV flight control interface, which characterizes the precise position and orientation of the UAV in space. At the same time, the pitch angle information of the gimbal is obtained through the interface of the UAV gimbal, and the internal parameter information such as the focal length, field of view angle, and imaging size of the camera is obtained through the interface of the camera. By using the gimbal angle information and the internal parameter information of the camera, the geometric relationship between the UAV camera shooting image and the ground target can be established. By combining the spatial position information, attitude information of the UAV with the parameter information of the camera, the mapping relationship between the geographic coordinate system, the UAV body coordinate system, and the camera coordinate system can be established.

[0059] When obtaining the information of the drone and the camera, it should be noted that the information needs to be obtained in real time at a certain frequency to facilitate capturing the dynamic changes of the drone platform and the camera attitude. At the same time, the obtained information should be filtered and smoothed to remove the jitter and noise interference of the GPS position. By accurately obtaining the state information of the drone platform and the camera, it provides parameter support for the subsequent projection transformation of geographical elements.

[0060] S3. According to the geographical element data of the water area shoreline, the drone position information, the pan-tilt angle information of the pan-tilt head, and the camera lens parameters, obtain the offset of the projection position of the geographical element in the screen coordinate system relative to the camera imaging center point.

[0061] In one example, construct the geometric relationship of the camera imaging center point P and the actual position T of the geographical element in the world coordinate system imaging at the camera position O in the screen coordinate system;

[0062] In the vertical direction, as Figure 2 、 4 、5 shows, use the following formula to obtain the longitudinal offset T′P′ of the geographical element point in the screen coordinate system y ;

[0063]

[0064] where: f represents the camera lens focal length; h is the flight altitude of the drone; α represents the real-time pan-tilt angle of the pan-tilt head; O′T represents the distance between the projection position O′(X O′ , Y O′ ) of the camera on the ground and the spatial coordinate point T(X T , Y T ) of the geographical element,

[0065] In the horizontal direction, as Figure 3 、 4 、5 shows, use the following formula to obtain the lateral offset T′P′ of the geographical element point in the screen coordinate system x ;

[0066]

[0067] where: β represents the real-time yaw angle of the drone, OP represents the distance between the camera position O and the camera imaging center point P,

[0068] In this embodiment, after obtaining the information, according to the mapping relationship among the geographic coordinates, the drone, the pan-tilt head, the camera, and the screen, the spatial position of the geographic feature is first converted into the coordinate system of the drone body, and the measurement error of the drone GPS position is considered for correction. Then, according to the pitch angle of the pan-tilt head, the position of the geographic feature is converted from the coordinate system of the drone body to the coordinate system of the pan-tilt head, and further converted to the coordinate system of the camera. Finally, using the camera imaging model and the principle of pinhole imaging, the three-dimensional coordinates of the camera are projected and converted into the two-dimensional screen pixel coordinate system through the internal parameters.

[0069] S4. Superimpose the projected position of the geographic feature on the real-time aerial photography image of the drone to realize the real-time projection display of the geographic features of the water area shoreline, including: obtaining the real-time aerial photography image of the drone; according to the offset and the spatial coordinate information of the features in the geographic information database, obtaining the projected positions of all the contour points of the geographic feature in the screen coordinate system; drawing the contour points of the geographic feature at the corresponding positions in the real-time aerial photography image, and connecting all the contour points to form the annotation box of the geographic feature.

[0070] In one example, the converted pixel coordinates of the geographic features of the water area shoreline are superimposed and rendered on the real-time aerial photography video stream image of the drone to form a virtual reality scene.

[0071] In this embodiment, when the drone moves or the camera attitude changes, the superimposed position of the geographic feature can be updated in real time and always maintain spatial registration with the aerial photography image. On the control display screen of the drone ground control station, the fusion display of the superimposed contour of the geographic features of the water area shoreline and the aerial photography image can be clearly seen. This method based on the virtual reality enhancement technology directly presents the geographic feature information and reduces the workload of manual interpretation and annotation.

[0072] This method realizes the real-time intercommunication and sharing between the geographic information system data and the drone system, and improves the informatization level of the water area shoreline inspection. Compared with the traditional in-house modeling processing method, this method has good real-time performance and visualization effect, and provides an intuitive reference for front-line law enforcement officers to quickly identify the target area.

[0073] Furthermore, this method further includes:

[0074] Performing contour extraction processing on the real-time aerial photography image of the drone to extract the real-time contour of the geographic feature in the image; comparing the extracted real-time contour of the geographic feature with the annotation box, and calculating the deviation degree between the two; determining whether the calculated deviation degree exceeds a preset threshold; if it exceeds the threshold, triggering an alarm prompt to remind the user that there is a large deviation between the annotation box and the actual contour of the geographic feature; continuously obtaining subsequent real-time aerial photography images of the drone, and performing continuous tracking annotation and deviation alarm on the geographic feature;

[0075] The intersection over union (IoU) between the real-time contour and the annotation box is calculated using the following formula: Q = (A(R ∩ S)) / (A(R ∪ S)); where: Q represents the overlap degree, R represents the real-time contour region, and S represents the annotation box region.

[0076] In this embodiment, contour extraction processing is performed on the real-time aerial images of the drone, and an image segmentation algorithm based on edge detection is used. First, the image is preprocessed, including filtering for noise reduction, contrast enhancement, etc., to highlight the contour features of geographical elements. Then, edge detection operators are used to calculate the gradients and directions of pixel points in the image to obtain candidate edge points. Finally, through thresholding and edge connection steps, the complete contours of geographical elements are extracted.

[0077] After extracting the real-time contours of geographical elements, they are compared with the pre-annotated rectangular boxes, and the areas of the overlapping and non-overlapping regions between the two are calculated. Based on this, the IoU is obtained as a quantitative indicator of the deviation degree. The higher the IoU, the closer the real-time contour is to the position of the annotation box, and the smaller the deviation.

[0078] When the deviation degrees of consecutive multiple frames of images all exceed the threshold, an alarm prompt is issued to attract the user's attention; there may be illegal construction, etc. here. Avoiding single-frame judgment can reduce false alarms. The alarm method can be selected according to requirements, such as voice, pop-up window, vibration, etc., to strive to remind the user in the first time.

[0079] Specific implementation:

[0080] The method for projecting and displaying water area shoreline element based on the drone virtual reality enhancement technology of the present invention; includes S1, obtaining water area shoreline geographical element data; S2, obtaining the drone position information, the pan-tilt angle information of the gimbal, and the camera lens parameters; S3, according to the water area shoreline geographical element data, the drone position information, the pan-tilt angle information of the gimbal, and the camera lens parameters, obtaining the offset of the geographical element projection position in the screen coordinate system relative to the camera imaging center point; S4, superimposing the geographical element projection position on the real-time aerial image of the drone to realize the real-time projection display of the water area shoreline geographical elements.

[0081] Under flight conditions, the conversion from the world coordinate system to the pixel coordinate system of the key points of geographical elements is decomposed into the vertical direction Y and the horizontal direction X:

[0082] In the vertical direction Y, the transformation calculation in the vertical direction of the drone is as Figure 2 、 4 shown:

[0083] Among them, point O is the position of the UAV, i.e., the camera position, the plane T’P’ is the virtual plane of camera imaging, OP is the straight line along the center of the lens optical axis, and T is the projection of the center point of the geographical feature in the Y direction. Solve for the offset value T’P’ of the target point in the Y direction in the screen image coordinate system. Among them, the angle α is measured by the tilt sensor of the UAV gimbal, and the camera imaging focal length f of OP’ is an internal parameter of the camera.

[0084]

[0085] In summary, it can be obtained that

[0086]

[0087] In the horizontal direction X, for the geometric transformation in the horizontal direction of the UAV, such as Figure 3 , 4 as shown in:

[0088] Among them, point O is the UAV position, the plane T’P’ is the virtual plane of camera imaging, the camera imaging focal length f of OP’, OP is the straight line along the center of the lens optical axis, T is the projection of the center point of the geographical feature in the X direction, and β represents the real-time yaw angle of the UAV, which is obtained by the UAV sensor. Solve for the included angle ω between the straight line in the due front of the fuselage and the target point, and obtain the offset value T’P’ of the target point in the X direction in the screen image coordinate system.

[0089]

[0090] ω = θ - β

[0091] TP = O’T · sinω

[0092]

[0093] In summary, it can be obtained that

[0094]

[0095] By reading the coordinate point data, obtaining the UAV position and the gimbal pitch angle information, and using the on-board frame annotation calculation module, the relative geometric relationship between the geographical feature, the UAV position, the gimbal pitch, and the lens parameters is solved, and the position of the geographical feature is projected and displayed in real time on the control screen. Its advantage is that it does not rely on traditional processing steps such as laser rangefinders, ground marking, 3D modeling, and post-annotation, realizes the functions of interacting and annotating with existing geographical features using ordinary consumer-grade UAVs with gimbal functions, and provides real-time reference for the review and decision-making of water resource protection actions.

[0096] Select the Hongze Lake area in Jiangsu Province as the study area, and implement the unapproved construction of a sports field in this area, as Figure 5 shown in:

[0097] Step 1: Formulate a flight plan, select a research area, import the kml file of suspected illegal points based on satellite remote sensing into the drone's on-board data module to complete the preliminary preparation for data collection;

[0098] Step 2: Select a DJI Mavic 3-E drone on-site and use the route flight mode to take pictures of the suspected illegal points; obtain the drone's position information, the pan-tilt angle information of the gimbal, and the camera lens parameters;

[0099] Step 3: According to the water area shoreline geographic feature data, the drone's position information, the pan-tilt angle information of the gimbal, and the camera lens parameters, obtain the offsets of the projected positions of the geographic features in the screen coordinate system relative to the camera imaging center point in the vertical direction Y and the horizontal direction X; for a picture with a screen coordinate of 4000*3000 pixels, the coordinate origin is at the lower left corner, and the offset is shown as Figure 4 shown as:

[0100] Step 4: Superimpose the projected positions of the geographic features on the real-time aerial photography image of the drone to achieve real-time projection display of the water area shoreline geographic features.

[0101] The embodiments of the present invention have been described above. The above description is exemplary and not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations are obvious to those of ordinary skill in the art in the technical field without departing from the scope and spirit of the described embodiments.

Claims

1. A method for projecting and displaying water shoreline elements based on UAV virtual reality enhancement technology, characterized in that: include: S1. Obtain geographical element data of water area and coastline; S2, obtain the drone position information, gimbal pitch angle information and camera lens parameters; S3, according to the water shoreline geographical element data, the drone position information, the gimbal pitch angle information and the camera lens parameters, obtaining the offset of the projection position of the geographical element in the screen coordinate system relative to the camera imaging center point; S4. Overlaying the projection position of the geographic element on the real-time aerial photography picture of the UAV to realize the real-time projection display of the geographical elements of the water area and coastline.

2. The method for projecting and displaying water shoreline elements based on UAV virtual reality augmented technology according to claim 1 is characterized in that In S1, the acquisition of geographical element data of water area and coastline includes: The spatial coordinate information of the water shoreline geographical elements is read from the geographical information database, wherein the spatial coordinate information includes the longitude, latitude and elevation information of the geographical element contour.

3. The method for projecting and displaying water shoreline elements based on UAV virtual reality augmented technology according to claim 1 is characterized in that In S2, the acquisition of drone position information, gimbal pitch angle information, and camera lens parameters includes: The real-time spatial position coordinates, yaw angle β and flight altitude h of the UAV are obtained through the UAV flight control system; Obtain the real-time pitch angle α of the gimbal through the drone gimbal; Get the lens focal length f through the camera.

4. The method for projecting and displaying water shoreline elements based on UAV virtual reality enhancement technology according to claim 1 is characterized in that In S3, obtaining the offset of the projection position of the geographic element in the screen coordinate system relative to the center point of the camera imaging includes: Construct the geometric relationship between the camera imaging center point P in the world coordinate system and the actual position T of the geographic element imaged through the camera position O in the screen coordinate system; In the vertical direction, the following formula is used to obtain the longitudinal offset of the geographic feature point in the screen coordinate system. ; ； Where: f is the focal length of the camera lens; h is the flight altitude of the drone; α is the real-time pitch angle of the gimbal; O´T is the projection position of the camera on the ground O´ ( ) and the geographic feature spatial coordinate point T ( ), ; In the horizontal direction, the following formula is used to obtain the lateral offset of the geographic feature point in the screen coordinate system. ; ； Where: β represents the real-time yaw angle of the drone, OP represents the distance between the camera position O and the camera imaging center point P, .

5. The method for projecting and displaying water shoreline elements based on UAV virtual reality augmented technology according to claim 1 is characterized in that Superimposing the projection position of the geographic element on the real-time aerial photography picture of the drone includes: Get real-time aerial images from drones; According to the offset and the spatial coordinate information of the element in the geographic information database, the projection positions of all contour points of the geographic element in the screen coordinate system are obtained; Geographical element contour points are drawn at corresponding positions of the real-time aerial image, and all contour points are connected to form a labeling frame of the geographic element.

6. The method for projecting and displaying water shoreline elements based on UAV virtual reality enhancement technology according to claim 5 is characterized in that The method further comprises: The projection position of geographic elements is updated in real time according to the flight status of the UAV, so that the geographic element annotation frame and the UAV aerial photography picture can be changed synchronously.

7. The method for projecting and displaying water shoreline elements based on UAV virtual reality enhancement technology according to claim 6 is characterized in that The method further includes: Perform contour extraction on the real-time aerial images taken by UAVs to extract the real-time contours of geographic elements in the images; Compare the extracted real-time outline of the geographic element with the annotation frame, and calculate the deviation between the two; determine whether the calculated deviation exceeds a preset threshold; If the threshold is exceeded, an alarm will be triggered to remind the user that there is a large deviation between the annotation box and the actual geographic feature outline; Continuously acquire subsequent real-time drone aerial images, continuously track and annotate geographic features, and issue deviation alarms.

8. The method for projecting and displaying water shoreline elements based on UAV virtual reality enhancement technology according to claim 7 is characterized in that The extracted real-time contour of the geographic element is compared with the annotation frame, and the deviation between the two is calculated. Specifically, the overlap Q between the real-time contour and the annotation frame is calculated using the following formula; ； Where: Q represents the overlap, R represents the real-time contour area, and S represents the annotation box area.

Images