River channel orthoimage rapid generation method based on aerial survey of unmanned aerial vehicle

Abstract

The invention provides a riverway orthoimage rapid generation method based on aerial survey of an unmanned aerial vehicle, and the method comprises the steps: determining a key frame: obtaining video data and inertial navigation data collected by the aerial survey unmanned aerial vehicle along a route, independently calculating pose change data between two frames based on the inertial navigation data, and extracting the key frame in the video data based on a key frame extraction strategy; collecting effective pictures, adjusting the size of a sampling frame according to a sampling frame strategy, and extracting pictures in the sampling frame to generate the effective pictures; and splicing effective pictures, obtaining the effective pictures, standardizing the sizes of the effective pictures, splicing the effective pictures according to the arrangement sequence of the key frames, and generating an orthoimage. According to the invention, the orthoimage can be quickly and accurately generated.

Description

Technical Field

[0001] The invention relates to the technical field of unmanned aerial vehicle (UAV) aerial survey, and in particular to a method for quickly generating orthophotos of a river channel based on unmanned aerial vehicle (UAV) aerial survey. Background Art

[0002] With the development of science and technology, environmental protection issues have gradually become prominent. Illegal water extraction, illegal sewage discharge, illegal occupation of river banks, dumping of garbage and other non-compliant behaviors will lead to river pollution and affect river water ecology. In order to avoid the deterioration of river ecology, it is necessary to monitor the river environment and the building environment on both sides of the river at regular intervals to detect non-compliant behaviors in a timely manner.

[0003] With the popularity of drones, the use of drones for patrolling has become common. Drones are used to take aerial photos of rivers and both sides of the river, generate video images and orthophotos of the river, quickly determine the location of non-compliant behavior based on the orthophotos, and determine the river environment based on the video images.

[0004] Normally, orthophotos require the planning of an arcuate route in advance, with sampling point coordinates set on the route. The drone flies along the route, hovers and collects orthophotos when passing through sampling points. Feature points in orthophotos are determined through intelligent recognition or manual annotation, and orthophotos are stitched together in the form of overlapping feature points to generate orthophotos. However, in the existing orthophoto generation process, the distribution density of the arcuate route is high, the time required to collect all orthophotos is long, and the amount of calculation required for image stitching is large, which greatly affects the efficiency of orthophoto generation. Summary of the invention

[0005] In view of this, the problem to be solved by the present invention is to provide a method for quickly generating orthophotos of a river channel based on drone aerial survey, which can quickly and accurately generate orthophotos.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is: A method for quickly generating orthophotos of a river channel based on drone aerial survey, comprising determining key frames: obtaining video data and inertial navigation data collected by the drone along a route, calculating the position change data between two frames based on the inertial navigation data, and extracting key frames in the video data based on a key frame extraction strategy; Collect valid images, adjust the sampling frame size according to the sampling frame strategy, and extract the images in the sampling frame to generate valid images; Stitching effective images, obtaining effective images and standardizing their sizes, stitching effective images according to the arrangement order of key frames, and generating orthophotos.

[0007] Furthermore, the key frame extraction strategy includes: initialization, obtaining the time when each frame of the video data is generated, defining the frame time, and setting the current key frame; Obtain inertial navigation data, obtain the inertial navigation data generated between the frame time of the current key frame and the nth adjacent frame, where n represents a number and is a positive integer greater than 0; Determine the moving distance, calculate the position change data of the aerial survey UAV between two frame times based on the inertial navigation data, and determine the moving distance of the picture within the frame; Determine the current key frame and judge whether the picture in the frame is moved out of the sampling frame. If yes, the nth adjacent frame is defined as the current key frame, n=1, and execute the step of obtaining inertial navigation data. If no, n+1=n, and execute the step of obtaining inertial navigation data.

[0008] Furthermore, the routes include straight routes and circular routes, and the circular routes are tangent to the connected straight routes; The sampling frame corresponding to the straight route is a rectangle, and the sampling frame corresponding to the arc route is a fan-shaped frame. The center angle of the corresponding fan-shaped frame is set according to the center angle of the arc route.

[0009] Furthermore, the sampling frame strategy includes: obtaining the current altitude of the aerial survey UAV between adjacent key frames, comparing the current altitude with a preset navigation altitude, and scaling the sampling frame size in proportion to the comparison result.

[0010] Furthermore, the river channel is divided into several river sections with altitude fluctuations below a threshold, and the navigation altitude of the aerial survey UAV is set according to the average altitude of the river sections.

[0011] Furthermore, the aerial survey drone is provided with a barometric altimeter for monitoring the altitude of the drone.

[0012] Furthermore, the size standardization includes: scaling the effective picture so that the size conforms to the sampling frame size set when the aerial survey UAV is at the navigation altitude.

[0013] Furthermore, the aerial survey drone includes a first lens, a second lens, and a third lens for respectively collecting videos of the oblique left side, directly below, and oblique right side of the aerial survey drone, the video data collected by the first lens, the second lens, and the third lens are respectively the first video, the second video, and the third video, the first video, the second video, and the third video correspond to the first sampling frame, the second sampling frame, and the third sampling frame, respectively, and the left and right sides of the second sampling frame partially overlap with the first sampling frame and the third sampling frame, respectively; A first orthophoto, a second orthophoto and a third orthophoto generated by the first video, the second video and the third video are respectively obtained, the first orthophoto and the third orthophoto are planarized by a planarization strategy, and then the first orthophoto, the second orthophoto and the third orthophoto are spliced ​​to generate an orthophoto of the entire river channel.

[0014] Further, the planarization strategy includes: obtaining a first overlapping image on the right side of the first orthophoto, a second left image on the left side of the second orthophoto overlapping with the first overlapping image, identifying the first overlapping image and the second left image to determine a first feature point and a left feature point, wherein the first feature point and the left feature point correspond to each other one by one; The first feature point and the left feature point are calculated using a spatial triangulation method to output a distortion parameter of the first feature point, and the first orthophoto is planarized based on the distortion parameter.

[0015] Furthermore, the third orthophoto is planarized using the same method.

[0016] The advantages and positive effects of the present invention are: Conventional video includes 24 or 30 frames per second, and the sampling frequency of the inertial sensor is between 100Hz and 1000Hz. By installing an inertial navigation module in the drone, the acquired inertial navigation data can accurately calculate the drone's posture change data between two adjacent key frames. Since the inertial navigation accumulation time is only the time between two frames or the time between adjacent key frames, the accumulation error can be ignored due to the extremely short time. Based on the drone's posture change data between the two key frames, the moving direction and distance of the picture within the frame are determined to determine the sampling area, ensuring that the collected positive picture, left picture, and right picture are Homan continuous and the pictures just do not overlap. Orthophotos can be generated by directly splicing the positive picture, left picture, and right picture, saving the calculation amount of picture splicing.

[0017] By setting cameras facing downward, tilted to the left, and tilted to the right on the drone, the drone can simultaneously obtain oblique left video, oblique right video, and downward video. When sampling, the drone only needs to fly directly above the river once to form the river channel frequency images and orthophotos required for aerial survey operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings: Figure 1 It is an overall flow chart of a method for quickly generating orthophotos of a river channel based on drone aerial surveying of the present invention; Figure 2 It is a schematic diagram of the lens imaging and aerial survey drone poster height of a method for quickly generating river channel orthophotos based on drone aerial survey of the present invention; Figure 3 It is a route schematic diagram of a method for quickly generating orthophotos of a river channel based on drone aerial surveying of the present invention; Figure 4It is a schematic diagram of a sampling frame for a straight route in a method for quickly generating a river orthophoto based on drone aerial surveying of the present invention; Figure 5 It is a schematic diagram of a sampling frame for an arc route in a method for quickly generating a river orthophoto based on drone aerial surveying of the present invention; In the figure: 1, first imaging area; 101, first sampling frame; 2, second imaging area; 201, second sampling frame; 3, third imaging area; 301, third sampling frame. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0021] The present invention provides a method for quickly generating orthophotos based on unmanned aerial vehicle (UAV) aerial survey, comprising an unmanned aerial vehicle (UAV). The unmanned aerial vehicle (UAV) is provided with an IMU / GNSS combined navigation module. The IMU / GNSS combined navigation module acquires inertial navigation data collected by an inertial navigation sensor and satellite data collected by a GPS receiver, and fuses the inertial navigation data and satellite data through a Kalman filtering algorithm, so as to realize long-distance accurate positioning of the unmanned aerial vehicle (UAV).

[0022] The process of generating orthophotos based on aerial survey UAVs: the aerial survey UAVs collect video data, accurately locate the posture change data of the aerial survey UAV between two frames based on the inertial navigation data, determine the key frames based on the posture change data, extract the valid images that just do not overlap in the key frames, and sequentially stitch the valid images to generate orthophotos.

[0023] In the prior art, the key frame image stitching process is: intelligently identify the feature points in the key frames, overlap the feature points in adjacent key frames to achieve image stitching, and generate orthophotos. Due to the irregular shape of the river channel and the topography, the feature point intelligent recognition process is prone to inaccurate feature point recognition. At this time, it is necessary to manually mark the feature points in two adjacent key frames, which increases the labor cost.

[0024] like Figure 1As shown, the method for generating orthophotos includes: determining key frames: obtaining video data collected by an aerial survey drone, and extracting key frames in the video data based on a key frame extraction strategy.

[0025] One embodiment of the present application is: 1s of video data contains 30 frames, the sampling frequency of the inertial navigation sensor in the aerial survey drone is 1000Hz, and the data output by the inertial navigation sensor is inertial navigation data. The sampling frame size is set according to the navigation speed of the drone. The greater the navigation speed of the drone, the greater the width of the sampling frame, and the lower the clarity of the generated orthophoto; the smaller the navigation speed of the drone, the smaller the width of the sampling frame, and the higher the clarity of the generated orthophoto.

[0026] The key frame extraction strategy includes: initialization, obtaining the time when each frame in the video data is generated, defining the frame time, and setting the current key frame.

[0027] An embodiment of the present application is: assuming that the video data is 10 minutes long and consists of 300 frames, the staff sets the first current key frame by themselves according to the picture clarity and picture angle, and assumes that the second frame is the current key frame.

[0028] Get inertial navigation data, get the inertial navigation data generated between the frame time of the current key frame and the nth adjacent frame, where n represents a number and is a positive integer greater than 0.

[0029] One embodiment of the present application is that the second frame is the current key frame, when n is 1, the nth adjacent frame is the third frame, and all inertial navigation data collected between the second frame and the third frame are obtained.

[0030] Determine the moving distance, calculate the position change data of the aerial survey UAV between two frame times based on the inertial navigation data, and determine the moving distance of the picture within the frame.

[0031] One embodiment of the present application is: taking the position at the second frame as the starting address, calculating the posture change data of the aerial survey UAV between the second frame and the third frame.

[0032] Determine the current key frame and judge whether the picture in the frame is moved out of the sampling frame. If yes, the nth adjacent frame is defined as the current key frame, n=1, and execute the step of obtaining inertial navigation data. If no, n+1=n, and execute the step of obtaining inertial navigation data.

[0033] An embodiment of the present application is: the posture change data corresponding to the second frame and the third frame determine that the moving distance of the drone is equal to the width of the sampling frame. If yes, the third frame is the current key frame, the nth adjacent frame is the fourth frame, and the step of obtaining the inertial navigation data is jumped; if no, the second frame is the current key frame, the nth adjacent frame is the fourth frame, and the step of obtaining the inertial navigation data is jumped.

[0034] Collect valid images and obtain the pose change data between two adjacent key frames. The sampling frame strategy sets the sampling frame size and position according to the pose change data, and extracts the images within the sampling frame to generate valid images.

[0035] The aerial survey drone is equipped with a barometric altimeter that collects air pressure data. The altitude at which the aerial survey drone is sailing is determined based on the relationship between air pressure and altitude, so that the drone can sail at a fixed altitude, making it convenient to determine the moving distance of the picture within the frame.

[0036] The southern region has a large altitude difference, and the river forms a waterfall with a large drop. The distance between the river below the waterfall and the aerial survey drone increases, which easily leads to the pixel value of the picture not meeting the requirements. One embodiment of the present application is to divide the river into several sections according to the altitude of the river and the altitude difference threshold, and set a separate route and navigation altitude for each section of the river.

[0037] When the aerial survey UAV is at a fixed altitude and flying horizontally, the navigation speed and distance of the UAV are the same as the moving speed and distance of the frame. Based on this principle, the sampling frame strategy includes: obtaining the altitude change data and posture change data of the aerial survey UAV between adjacent key frames; The moving distance and moving direction of the picture in two adjacent key frames are determined based on the posture change data, and the sampling frame position is determined based on the moving distance and moving direction of the picture. Figure 2 As shown in the figure, x1 represents the focal length, P1 and P2 both represent the altitude. When the aerial survey drone collects video data, the camera focal length remains unchanged. When the altitude of the aerial survey drone is lower than the navigation altitude, the graphics in the frame are proportionally enlarged, and the sampling frame is also proportionally enlarged. When the altitude of the aerial survey drone is higher than the navigation altitude, the graphics in the frame are proportionally reduced, and the sampling frame is also proportionally reduced to avoid overlapping of adjacent valid images and affect the stitching accuracy of orthophotos.

[0038] Stitch the effective images, obtain the effective images of all key frames, standardize the sizes, stitch the effective images according to the arrangement order of the key frames, and generate orthophotos.

[0039] Since the effective images are not the same in size, they are scaled to a standard size when stitching them together. The standard size is the sampling frame size set by the aerial survey drone at the flight altitude.

[0040] like Figure 3 As shown in the figure, the route of the aerial survey UAV is set manually according to the shape of the river. The route consists of a straight route and an arc route. The arc route is set tangent to the connected straight route. Among them, the a0-a1 section, a2-a3 section, a4-a5 section, a6-a7 section, and a8-a9 section are straight routes, and the rest are arc routes. The straight route is set first according to the shape of the river, and the arc route tangent to the straight route is set in the form of a chamfer.

[0041] The sampling frame corresponding to the straight route is a rectangular structure. The width of the rectangle is set according to the navigation speed of the UAV, the large radius and small radius of the fan ring are set according to the arc radius of the arc route, and the center angle of the fan ring is set according to the route speed of the UAV and the center angle of the arc route. An embodiment of the present application is: the radius of the arc route is 100m, the small radius of the sampling area corresponding to the fan ring is 70m, and the large radius is 130m.

[0042] The aerial survey drone includes a first lens, a second lens, and a third lens. The first lens is installed tilted to the left, the second lens is installed vertically downward, and the third lens is installed tilted to the right, respectively collecting the first video on the oblique left side of the aerial survey drone, the second video directly below, and the third video on the oblique right side. The aerial survey drone can generate an orthophoto of the river environment and the environment on both sides of the river with only one voyage.

[0043] like Figure 4 and Figure 5 As shown, the first lens, the second lens and the third lens correspond to the first camera area, the second camera area and the third camera area respectively, and the three camera areas are arranged in order from left to right and overlap in sequence. Since the river includes several river sections, the average altitude of the river section is obtained, and the navigation altitude of the aerial survey UAV is set according to the average altitude. An embodiment of the present application is: the right side of the first camera area overlaps with the left side of the second camera area, and the image overlap in the length direction of the camera area is 20%; the right side of the second camera area overlaps with the left side of the third camera area, and the image overlap in the length direction of the camera area is 20%.

[0044] The video data collected by the first lens, the second lens and the third lens are the first video, the second video and the third video respectively. The first video corresponds to the first sampling frame, the second video corresponds to the second sampling frame, and the third video corresponds to the third sampling frame. The left and right sides of the second sampling frame overlap with the first sampling frame and the third sampling frame respectively, and the overlap rate is set manually. When the route is an arc route, the center angles of the first sampling frame, the second sampling frame and the third sampling frame of the fan ring structure overlap with the center angle of the arc route.

[0045] Orthophoto stitching process: obtain the first video, the second video and the third video collected at the same time, and generate the first orthophoto, the second orthophoto and the third orthophoto respectively based on the orthophoto rapid generation method.

[0046] When the aerial survey unmanned aerial vehicle is flying, the first lens, the second lens and the third lens are opened at the same time to collect the first video, the second video and the third video, the altitude of the local river section and the aerial survey altitude of the aerial survey unmanned aerial vehicle, and the selection area and size of the first sampling frame, the second sampling frame and the third sampling frame are manually set so that the left and right sides of the second sampling frame overlap with the first sampling frame and the third sampling frame respectively. The right side of the generated first orthophoto overlaps with the left side of the second orthophoto, and the right side of the second orthophoto overlaps with the left side of the third orthophoto.

[0047] Planarize the orthophoto, obtain the first overlapping image on the right side of the first orthophoto, and the second left image on the left side of the second orthophoto that overlaps with the first overlapping image, identify the first overlapping image and the second left image to determine the overlapping first feature points and left feature points, calculate the distortion parameters of the first feature points based on the first feature points and the left feature points using a spatial triangulation method, planarize the first orthophoto based on the distortion parameters, and planarize the third orthophoto using the same method.

[0048] Based on experience, the staff set the partial area on the right side of the first orthophoto as the first overlapping image, and the partial area on the left side of the second orthophoto as the second left image, and used intelligent recognition of image feature points or manual labeling to determine the first feature points and the left feature points, and the first feature points and the left feature points correspond one to one.

[0049] Since the first orthophoto and the second orthophoto are equivalent to a large-size picture, the image range contained in the first overlapping image and the second left image is sufficient, and artificial intelligence can identify enough first feature points and left feature points. Orthophotos can be generated without manual annotation. If you want to improve the performance of orthophotos, you can correct the one-to-one correspondence between the first feature points and the left feature points through manual detection.

[0050] Spatial triangulation is to use aerial photographs with a certain degree of overlap to establish a route model or regional network model corresponding to the actual situation based on a small number of field control points using photogrammetry methods, so as to obtain the plane coordinates and elevations of the encrypted points.

[0051] Based on the spatial triangulation method, the first feature point and the left feature point that correspond to each other are calculated to obtain the distortion parameters of the first orthophoto. Since the relative positions of the first lens and the second lens on the drone are fixed and the camera parameters are fixed, the distortion parameters of the first orthophoto calculated based on the second orthophoto are also fixed, so the first orthophoto can be uniformly planarized. The same method is used to calculate the distortion parameters of the third orthophoto, and the third orthophoto is uniformly planarized.

[0052] Obtain one-to-one corresponding overlapping feature points between the planarized first orthophoto, the second orthophoto, and the planarized third orthophoto, and stitch the first orthophoto, the second orthophoto, and the third orthophoto according to the overlapping feature points.

[0053] The embodiments of the present invention are described in detail above, but the contents described are only preferred embodiments of the present invention and cannot be considered to limit the scope of implementation of the present invention. All equivalent changes and improvements made within the scope of the present invention should still fall within the scope of this application.

Claims

1. A method for quickly generating orthophotos of a river channel based on drone aerial survey, characterized in that: Including determining key frames: obtaining video data and inertial navigation data collected by the aerial survey UAV along the route, calculating the posture change data between two frames based on the inertial navigation data, and extracting key frames in the video data based on the key frame extraction strategy; Collect valid images, adjust the sampling frame size according to the sampling frame strategy, and extract the images in the sampling frame to generate valid images; Stitching effective images, obtaining effective images and standardizing their sizes, stitching effective images according to the arrangement order of key frames, and generating orthophotos.

2. The method for quickly generating orthophotos of a river channel based on drone aerial survey according to claim 1 is characterized in that: The key frame extraction strategy includes: initialization, obtaining the time when each frame of the video data is generated, defining the frame time, and setting the current key frame; Obtain inertial navigation data, obtain the inertial navigation data generated between the frame time of the current key frame and the nth adjacent frame, where n represents a number and is a positive integer greater than 0; Determine the moving distance, calculate the position change data of the aerial survey UAV between two frame times based on the inertial navigation data, and determine the moving distance of the picture within the frame; Determine the current key frame and judge whether the picture in the frame is moved out of the sampling frame. If yes, the nth adjacent frame is defined as the current key frame, n=1, and execute the step of obtaining inertial navigation data. If no, n+1=n, and execute the step of obtaining inertial navigation data.

3. The method for quickly generating orthophotos of a river channel based on drone aerial survey according to claim 1 is characterized in that: The routes include straight routes and circular routes, and the circular routes are tangent to the connected straight routes; The sampling frame corresponding to the straight route is a rectangle, and the sampling frame corresponding to the arc route is a fan-shaped frame. The center angle of the corresponding fan-shaped frame is set according to the center angle of the arc route.

4. The method for quickly generating orthophotos of a river channel based on drone aerial survey according to claim 3 is characterized in that: The sampling frame strategy includes: obtaining the current altitude of the aerial survey UAV between adjacent key frames, comparing the current altitude with the preset navigation altitude, and scaling the sampling frame size in proportion to the comparison result.

5. The method for quickly generating orthophotos of a river channel based on drone aerial survey according to claim 4 is characterized in that: The river channel is divided into a number of river sections with altitude fluctuations below a threshold, and the navigation altitude of the aerial survey UAV is set according to the average altitude of the river section.

6. The method for quickly generating orthophotos of a river channel based on drone aerial survey according to claim 4 is characterized in that: The aerial survey UAV is provided with a barometric altimeter for monitoring the altitude of the UAV.

7. The method for quickly generating orthophotos of a river channel based on drone aerial survey according to claim 1 is characterized in that: The size standardization includes: scaling the effective picture so that the size conforms to the sampling frame size set when the aerial survey UAV is at the navigation altitude.

8. The method for quickly generating orthophotos of a river channel based on drone aerial survey according to claim 1 is characterized in that: The aerial survey drone comprises a first lens, a second lens and a third lens respectively capturing videos of the oblique left side, directly below and oblique right side of the aerial survey drone, the video data captured by the first lens, the second lens and the third lens are respectively the first video, the second video and the third video, the first video, the second video and the third video correspond to the first sampling frame, the second sampling frame and the third sampling frame respectively, and the left and right sides of the second sampling frame partially overlap with the first sampling frame and the third sampling frame respectively; A first orthophoto, a second orthophoto and a third orthophoto generated by the first video, the second video and the third video are respectively obtained, the first orthophoto and the third orthophoto are planarized by a planarization strategy, and then the first orthophoto, the second orthophoto and the third orthophoto are spliced ​​to generate an orthophoto of the entire river channel.

9. The method for quickly generating orthophotos of a river channel based on drone aerial survey according to claim 8, characterized in that: The planarization strategy includes: obtaining a first overlapping image on the right side of the first orthophoto, a second left image on the left side of the second orthophoto and overlapping the first overlapping image, identifying the first overlapping image and the second left image to determine a first feature point and a left feature point, wherein the first feature point and the left feature point correspond to each other one by one; The first feature point and the left feature point are calculated using a spatial triangulation method to output a distortion parameter of the first feature point, and the first orthophoto is planarized based on the distortion parameter.

10. The method for quickly generating orthophotos of a river channel based on drone aerial survey according to claim 9, characterized in that: Use the same method to flatten the third orthophoto.

Images