Unmanned aerial vehicle flight video superposition and vector file real-time fusion method

Abstract

The invention relates to the field of photogrammetric video and vector file fusion, in particular to an unmanned aerial vehicle flight video superposition and vector file real-time fusion method, which comprises the following steps of: 1, acquiring data; 2, resampling a vector range line; 3, decoding the aerial photography video stream; 4, comparing the coordinates; step 5, intersection extraction; step 6, image rendering; 7, for each frame of image of the video stream, the step 4, the step 5 and the step 6 are repeated in sequence; and step 8, encoding the image into a video stream, and realizing fusion display of the vector range line and the video. According to the scheme, on the basis of oblique photogrammetry and computer video, the method for fusing the real-time video and the vector range line in aerial photography of the unmanned aerial vehicle is designed, and the video and the vector range line are fused in a real sense.

Description

Technical Field

[0001] The present invention relates to the field of photogrammetry video and vector file fusion, and in particular to a method for real-time fusion of unmanned aerial vehicle flight video overlay and vector file. Background Art

[0002] During the drone video monitoring process, although the backstage command screen can display the drone video in real time to assist managers in real-time supervision, there are still technical difficulties in directly displaying the vector range of the monitoring area in the video. Specifically, the current monitoring video, as an independent individual, cannot display the boundaries of the monitoring area and the location of the monitoring target in real time on the video screen, which poses a certain challenge to the management's supervision work.

[0003] A common method for fusing videos shot by drones with vector files is to project the videos shot by drones and vector range lines respectively, and then display them through the system platform in the form of superimposed video layers and vector range line layers. Specifically, the drone acquires video in real time and connects it to the system, and displays the video screen in the system as a layer in the form of projection according to the coordinate system used by the system, the precise positioning data of the drone when the video is acquired, the load parameters carried by the drone, and the corresponding projection method. Then the monitoring range vector file is added to the system as another layer, and the vector file layer is placed on the video layer to achieve the fusion display of the video and the vector file. However, this method is not a true fusion of the video and the vector file, but a visual superposition display. This method has the problem of projection deformation in practical application. In addition, due to the undulating terrain, when the video elevation is inconsistent with the elevation information of the vector file, it will also cause the problem that the vector file is not displayed or floats on the video. The above problems bring inconvenience to the supervisory work of the manager and cause misjudgment of the analysis. In addition, the above method depends on whether the system platform supports the superposition display of layers. Based on this, the present invention proposes a real-time fusion method for drone flight video superposition and vector file. Summary of the invention

[0004] In order to solve the problem of projection deformation of video and vector range line, non-display or floating of vector range line, and reduce the dependence on system platform. The purpose of the present invention is to propose a method for real-time fusion of drone flight video overlay and vector file that can be played on any video player, which can truly fuse vector range line into real-time video, solve the problem of projection deformation, non-display or floating of vector line, and get rid of the dependence on system platform.

[0005] To achieve the above object, the technical solution adopted by the present invention is: a method for real-time fusion of drone flight video overlay and vector file, comprising the following steps: Step 1: Data acquisition, real-time acquisition of drone aerial video stream, and acquisition of parameter information during drone video shooting; Step 2: Vector range line resampling, including sampling of point monitoring objects, line monitoring objects, and surface monitoring objects. For point monitoring objects, directly organize the monitoring object point coordinates into a point set C ; For linear monitoring objects, resample to a set of sequentially connected points C ; For surface monitoring objects, the sampling is a set of sequentially connected points C , and the initial point and the end point are the same point. In this case, the surface is replaced by a closed line, and the middle is not filled; Step 3: Decode the aerial video stream, and decode the aerial real-time video stream obtained in step 1 into images frame by frame according to time intervals; Step 4: Coordinate comparison, for t The image is taken at the same time, and the shooting time and the corresponding drone photography center position information are obtained. According to the principle of photogrammetry collinear equation, the relationship between the image space coordinate system and the object space coordinate system is established. The range covered by each frame of the image in the object space coordinate system is calculated through the relationship, and the image is obtained. , , , ; Step 5: Intersection extraction, traversing the vector range line point set C ,Will t Within the image range of the moment C Intersection points are organized into point sets M , M Any fixed point X , Y The following conditions must be met: and , for linear or planar monitoring objects, the point set M The connection order and point set of points in C in agreement; Step 6: Image rendering, based on the coordinates of each point in the point set M, the corresponding pixel coordinates are calculated according to the collinear equation , and define the pixel color value at the coordinate to obtain the rendered image; Step 7: Repeat steps 4, 5, and 6 for each frame of the video stream; Step 8: Encode the image into a video stream, re-encode each frame of the rendered image into a video stream, obtain a video fused with the vector range line, and realize the fusion display of the vector range line and the video.

[0006] In the above-mentioned method for real-time fusion of drone flight video overlay and vector file, in step 1, the parameter information includes the attitude angle element, focal length f, gimbal angle, etc. during the drone video shooting process. , relative to the ground height information h, the attitude angle element includes the heading inclination , lateral inclination , declination angle .

[0007] In the above-mentioned method for real-time fusion of drone flight video overlay and vector file, in step 4, the object space coordinate system O-XYZ is the coordinate system in the actual living environment, and the image space coordinate system S-xyz is the coordinate system established between the drone photography center and the frame image.

[0008] In the above-mentioned method for real-time fusion of drone flight video overlay and vector file, the relationship between the image space coordinate system and the object space coordinate system is established in step 4, including: Step 4-1: For each frame of image, obtain its shooting time as t, and obtain the drone position according to the drone RTK module; Step 4-2: Get the coordinates of the photography center in the object coordinate system ; Step 4-3: Get the coordinates of the photography center in the image space coordinate system ; Step 4-4: Each object coordinate system point , there is a corresponding point in the image space coordinate system .

[0009] In the above-mentioned method for real-time fusion of drone flight video overlay and vector file, an image space auxiliary coordinate system S-XYZ is set between the image space coordinate system S-xyz and the object space coordinate system O-XYZ, and the origin of the image space auxiliary coordinate system S-XYZ coincides with the image space coordinate system S-xyz, and both are projection centers; the coordinate axes of the image space auxiliary coordinate system S-XYZ are respectively consistent with the axis directions of the object space coordinate system O-XYZ.

[0010] In the above-mentioned method for real-time fusion of drone flight video overlay and vector file, the range covered by each frame of image in the object coordinate system is calculated in step 4, including: Step a: Derivation of the midpoint of the image space coordinate system based on the collinear equation in photogrammetry d The midpoint of the object coordinate system D The relationship between ,in, , R is the rotation matrix; Step b: Rotate the three attitude angle elements of the image space coordinate system to obtain the heading angle , lateral inclination , declination angle , get the image space auxiliary coordinate system, calculate The value of , Indicates the distance from the center of photography to the center point; Step c: Pass get ; Step d: Substitute the rotation matrix R and Substitute this into the relationship formula between the midpoint m in the image space coordinate system and the midpoint M in the object space coordinate system to obtain the object image calculation formula: ; Step e: By using the object-image calculation formula, the coordinate value in the object coordinate system is calculated according to the pixel coordinates. The maximum value of the pixel coordinates of each frame of the image is used to calculate the maximum value of the corresponding object coordinate system. According to the image space coordinate system, the relationship formula between the photographic center and the maximum value of the pixel coordinates is obtained as follows: ; Step f: Substitute the relationship formula between the photographic center and the maximum pixel coordinate into the object-image calculation formula to calculate the coordinate range of the frame image in the object coordinate system ( ), the expression is: .

[0011] In the above-mentioned method for real-time fusion of drone flight video overlay and vector file, in step 5, for point-like monitoring objects, the coordinates of each point in the point set M are inversely calculated in the image space coordinate system, and the color value of the pixel at the coordinate is defined; for linear and planar monitoring objects, the coordinates of each point in the point set M are inversely calculated in the image space coordinate system, and the color value of the pixel at the coordinate is defined, and the coordinates of each point in M ​​are connected in sequence, and the color value of the pixel in the line passing through the image is defined.

[0012] In the above-mentioned method for real-time fusion of drone flight video overlay and vector file, in step 6, the corresponding pixel coordinates are calculated inversely according to the collinear equation. , including: for collections For each point in the image, the corresponding It can be seen that: , , in, The value of is calculated by the formula of the rotation matrix R. The coordinates of the photography center in the object coordinate system are .

[0013] The beneficial effects of the method for real-time fusion of unmanned aerial vehicle flight video overlay and vector file of the present invention are as follows: based on oblique photogrammetry and computer video, a method for fusion of real-time unmanned aerial vehicle video and vector range lines is designed, and the real fusion of video and vector range lines is realized. The video is decoded into frames of images, and then the intersection of each frame of image and range line is calculated by collinear equation, and the image pixel is back-calculated according to the intersection to change the color value of the image pixel. In the calculation process, the comparison of vector range line and video projection elevation is effectively avoided, and the problems of projection deformation, missing and suspension of vector range line are solved, which provides a feasible technical reference for the fusion of video and vector range line. By using the image space auxiliary coordinate system, a relatively unified coordinate system is established, and the image space coordinate system of each film is unified, so that the calculation is more convenient. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 A flowchart of the steps for implementing the present invention; Figure 2 Schematic diagram for calculating the coverage of each frame of image; Figure 3 Rendering diagram for each frame of image; Figure 4 This is a diagram showing the relationship between the image space coordinate system, the object space coordinate system, and the image auxiliary coordinate system. DETAILED DESCRIPTION

[0015] In order to enable those skilled in the art to better understand the technical solution of the present invention, the technical solution of the present invention is described below in conjunction with specific implementation methods and drawings.

[0016] Example 1 A method for real-time fusion of unmanned aerial vehicle flight video overlay and vector file includes at least the following steps.

[0017] 1. Real-time data acquisition: Real-time acquisition of drone aerial video streams, as well as information such as attitude angle elements, focal length, gimbal angle, and height relative to the ground during drone video shooting.

[0018] 2. Vector range line resampling. According to the characteristics of the monitored object, it is divided into three categories: point, line, and surface. For point-shaped monitoring objects, the coordinates of the monitored object points are directly sorted into a point set C; for line-shaped monitoring objects, they are resampled into a sequentially connected point set C; for surface-shaped monitoring objects, they are sampled into a sequentially connected point set C, and the initial point and the end point are the same point. At this time, the surface is replaced by a closed line, and the middle is not filled.

[0019] 3. Decoding of aerial video stream: Decode the real-time aerial video stream into frames at time intervals.

[0020] Fourth, calculate the range of the object coordinate system covered by each frame of the image, where the object coordinate system O-XYZ is the coordinate system in the actual living environment. For the image at time t, obtain the shooting time and the corresponding drone photography center position information, and establish the relationship between the image space coordinate system S-xyz and the object coordinate system O-XYZ based on the principle of photogrammetry collinearity equation, where the image space coordinate system is the coordinate system established between the drone photography center and the frame of the image. Through this relationship, calculate the range covered by each frame of the image in the object coordinate system, that is, obtain , , , .

[0021] 5. Intersection extraction. Traverse the vector range line point set C and intersect the image range at time t with the point set C The intersection points are organized into a point set M, that is, the X and Y of any fixed point in M ​​satisfy the following conditions: and .

[0022] For a linear or planar monitoring object, the connection order of the points in the point set M is consistent with that in the point set C.

[0023] 6. Image rendering. According to the coordinates of each point in the point set M, the corresponding pixel coordinates are calculated according to the collinear equation. , and the pixel color value at the coordinate is RGB (255,0,0) (customizable). For point-like monitoring objects, according to the coordinates of each point in M, the corresponding coordinates in the image space coordinate system are calculated inversely, and the color value of the pixel at the coordinate is modified to RGB (255,0,0); for linear and planar monitoring objects, according to the coordinates of each point in M, the corresponding coordinates in the image space coordinate system are calculated inversely, and the color value of the pixel at the coordinate is modified to RGB (255,0,0), and the coordinates of each point in M ​​are connected in sequence, and the color value of the pixel in the line passing through the image between the pixels is also modified to RGB (255,0,0). The rendered image is obtained through the above operations.

[0024] 7. Repeat steps (4), (5), and (6) for each frame of the video stream.

[0025] 8. Encoding the image into a video stream. Re-encode each frame of the rendered image into a video stream, and obtain a video fused with the vector range line, thereby realizing the fusion display of the vector range line and the video.

[0026] Example 2 like Figure 1-4 As shown, a method for real-time fusion of drone flight video overlay and vector file includes the following steps.

[0027] 1. Real-time data acquisition: real-time acquisition of drone aerial video stream, and the focal length f of the drone video. The three angular elements are heading inclination , lateral inclination , declination angle , the height of the drone relative to the ground when shooting the video is h, and the angle of the gimbal when shooting the video is , the above values ​​are all known values.

[0028] 2. Resample the vector range line to a point set, that is, a coordinate set in the object coordinate system. When the monitored object is a discrete point, organize the monitored object into a point set ; When the monitoring object is a linear area, it is resampled into a set of sequentially connected points , that is, the monitoring area consists of points Connect the dots , connected to the points in sequence Constitute a line point set; when the monitoring object is a closed area, resample it into a sequential connection point set , that is, by the point Connect the dots , connected to the points in sequence , and then continue to connect until the point A set of points that form a closed region.

[0029] 3. Decode the aerial video stream and obtain the real-time aerial video stream at time intervals Decoded into frames of images, the resolution of the obtained images is , is the width of the image, is the height of the image in pixels.

[0030] 4. Calculate the coordinate range of the object space coordinate system covered by each frame of the image. Figure 2 As shown in the figure, for each frame of image, the shooting time is obtained as t, the position of the drone is obtained according to the drone RTK module, and the coordinates of the photography center in the object coordinate system are obtained as , and can obtain the coordinates of the photography center in the image space coordinate system , each object coordinate point , there is a corresponding point in the image space coordinate system ;It should be noted that the object space coordinate system is the real space projection coordinate system and the same as the coordinate system of the vector line; the image space coordinate system is the position of the image point in the image space, such as Figure 4As shown in the figure, the image space auxiliary coordinate system S-XYZ, the image space coordinates of the image point can be directly obtained from the coordinates of the film plane, but since the image space coordinate systems of each film are not unified, it brings difficulties to the calculation. Therefore, it is necessary to establish a relatively unified coordinate system, called the image space auxiliary coordinate system. The origin of the image space auxiliary coordinate system coincides with the image space coordinate system and is also the projection center, see Figure 4 As shown; the coordinate axes are consistent with the directions of the object coordinate system axes, see Figure 2 shown.

[0031] According to the collinear equation in photogrammetry, the relationship between the midpoint d in the image space coordinate system and the midpoint D in the object space coordinate system can be deduced as follows: ①.

[0032] in: ②.

[0033] R is the rotation matrix, and the image space auxiliary coordinate system is composed of the image space coordinate system according to the three angular elements , , Rotate to get , , are known values ​​and can be calculated separately The value of Figure 4 As shown, .

[0034] Figure 4 In the figure, the blue coordinate system represents the image space auxiliary coordinate system, the pink coordinate system represents the image space coordinate system, and the black coordinate system represents the object space coordinate system.

[0035] like Figure 2 As shown, represents the distance from the center of the photograph to the center point, and f is the focal length when shooting the video, which is a known value. According to the trigonometric formula, , h is the height from the ground to the center of the camera when shooting the video, are the gimbal angles when shooting video, all of which are known values.

[0036] According to the above formula, we can get ③.

[0037] Substituting formulas ③ and ② into formula ①, we can obtain ④.

[0038] From formula ④, we can know that the coordinate value in the object coordinate system can be calculated according to the pixel coordinates. Then, we can know that the maximum value of the pixel coordinates of each frame can be used to calculate the maximum value of the corresponding object coordinate system. According to the image space coordinate system, the relationship between the photography center and the maximum value of the pixel coordinates is: ⑤.

[0039] Substituting formula ⑤ into formula ④, the coordinate range of the frame image in the object coordinate system can be calculated ( ),Right now , .

[0040] In the prior art, drone videos and vector range lines are displayed in a superimposed manner. Due to the influence of video projection, there are errors in the edge area of ​​the video, causing projection deformation. In addition, the video projection has height information, which needs to be matched with the elevation information of the vector range line. If the vector line elevation is inconsistent with the video projection elevation, when the video projection elevation is higher than the vector line elevation, the video covers the vector line; when the video projection elevation is less than the vector line elevation, the vector line floats above the video.

[0041] This technical solution decodes the video into frames, calculates the intersection of each frame of the image and the range line through the collinear equation, and reversely calculates the image pixels based on the intersection to change the color value of the image pixels. During the calculation process, the comparison between the vector range line and the video projection elevation is effectively avoided, and the projection deformation and the missing and suspended vector range line problems are avoided.

[0042] 5. Intersection extraction. Traverse the vector range line point set C , the coordinate range covered by the image at time t With point collection C The intersection points are organized into a point set M, that is, the X and Y of any fixed point in M ​​satisfy the following conditions: and , For linear or planar monitoring objects, the connection order of the points in the point set M is the same as that of the point set C Consistent in.

[0043] 6. Image space coordinate inversion. For the set For each point in the image, the corresponding It can be seen that: , , in, The value of is calculated by formula ②, f is the focal length, and the coordinates of the photography center in the object coordinate system are .

[0044] 7. Image rendering: According to each point in the point set m, modify the color value of the pixel with the corresponding coordinate value in the image space coordinate system to RGB (255, 0, 0). For linear and planar monitoring objects, modify the color value of the pixel at the coordinate to RGB (255, 0, 0) according to the coordinate of each point in m, and connect the coordinates of each point in m in sequence, and modify the color value of the pixel in the line passing through the image to RGB (255, 0, 0). The rendered image is obtained through the above operations.

[0045] 8. Repeat steps 4, 5, 6, and 7 for each frame image in the video stream to obtain a set of rendered images sorted by time.

[0046] 9. Encode the image into a video stream. Re-encode each frame of the rendered image into a video stream, and obtain a video fused with the vector range line, thereby realizing the fusion display of the vector range line and the video.

[0047] The above embodiments are only for illustrating the inventive concept and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made based on the essence of the content of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for real-time fusion of drone flight video overlay and vector file, characterized in that: The following steps are involved: Step 1: Data acquisition, real-time acquisition of drone aerial video stream, and acquisition of parameter information during drone video shooting; Step 2: Vector range line resampling, including sampling of point monitoring objects, line monitoring objects, and surface monitoring objects. For point monitoring objects, directly organize the monitoring object point coordinates into a point set C ; For linear monitoring objects, resample to a set of sequentially connected points C ; For surface monitoring objects, the sampling is a set of sequentially connected points C , and the initial point and the end point are the same point. In this case, the surface is replaced by a closed line, and the middle is not filled; Step 3: Decode the aerial video stream, and decode the aerial real-time video stream obtained in step 1 into images frame by frame according to time intervals; Step 4: Coordinate comparison, for t The image is taken at the same time, and the shooting time and the corresponding drone photography center position information are obtained. According to the principle of photogrammetry collinear equation, the relationship between the image space coordinate system and the object space coordinate system is established. The range covered by each frame of the image in the object space coordinate system is calculated through the relationship, and the image is obtained. , , , ; Step 5: Intersection extraction, traversing the vector range line point set C ,Will t Within the image range of the moment C Intersection points are organized into point sets M , M Any fixed point X , Y The following conditions must be met: and , for linear or planar monitoring objects, the point set M The connection order and point set of points in C in agreement; Step 6: Image rendering, based on the coordinates of each point in the point set M, the corresponding pixel coordinates are calculated according to the collinear equation , and define the pixel color value at the coordinate to obtain the rendered image; Step 7: Repeat steps 4, 5, and 6 for each frame of the video stream; Step 8: Encode the image into a video stream, re-encode each frame of the rendered image into a video stream, obtain a video fused with the vector range line, and realize the fusion display of the vector range line and the video.

2. The method for real-time fusion of drone flight video overlay and vector file according to claim 1 is characterized in that: In step 1, the parameter information includes the attitude angle element, focal length f, gimbal angle, etc. , relative to the ground height information h, the attitude angle element includes the heading inclination , lateral inclination , declination angle .

3. The method for real-time fusion of drone flight video overlay and vector file according to claim 1 is characterized in that: In step 4, the object space coordinate system O-XYZ is a coordinate system in the actual living environment, and the image space coordinate system S-xyz is a coordinate system established between the drone photography center and the frame image.

4. The method for real-time fusion of drone flight video overlay and vector file according to claim 3 is characterized in that: The relationship between the image space coordinate system and the object space coordinate system is established in step 4, including: Step 4-1: For each frame of image, obtain its shooting time as t, and obtain the drone position according to the drone RTK module; Step 4-2: Get the coordinates of the photography center in the object coordinate system ; Step 4-3: Get the coordinates of the photography center in the image space coordinate system ; Step 4-4: Each object coordinate system point , there is a corresponding point in the image space coordinate system .

5. The method for real-time fusion of drone flight video overlay and vector file according to claim 4 is characterized in that: An image space auxiliary coordinate system S-XYZ is arranged between the image space coordinate system S-xyz and the object space coordinate system O-XYZ. The origin of the image space auxiliary coordinate system S-XYZ coincides with the image space coordinate system S-xyz and both are projection centers. The coordinate axes of the image space auxiliary coordinate system S-XYZ are respectively consistent with the axis directions of the object space coordinate system O-XYZ.

6. The method for real-time fusion of drone flight video overlay and vector file according to claim 5 is characterized in that: In step 4, the range covered by each frame of image in the object coordinate system is calculated as follows: Step a: Derivation of the midpoint of the image space coordinate system based on the collinear equation in photogrammetry d The midpoint of the object coordinate system D The relationship between ,in, , R is the rotation matrix; Step b: Rotate the three attitude angle elements of the image space coordinate system to obtain the heading angle , lateral inclination , declination angle , get the image space auxiliary coordinate system, calculate The value of , Indicates the distance from the center of photography to the center point; Step c: Pass get ; Step d: Substitute the rotation matrix R and Substitute this into the relationship formula between the midpoint m in the image space coordinate system and the midpoint M in the object space coordinate system to obtain the object image calculation formula: ； Step e: By using the object-image calculation formula, the coordinate value in the object coordinate system is calculated according to the pixel coordinates. The maximum value of the pixel coordinates of each frame of the image is calculated to obtain the maximum value of the corresponding object coordinate system. According to the image space coordinate system, the relationship formula between the photographic center and the maximum value of the pixel coordinates is obtained as follows: ; Step f: Substitute the relationship formula between the photographic center and the maximum pixel coordinate into the object-image calculation formula to calculate the coordinate range of the frame image in the object coordinate system ( ), the expression is: 。 7. The method for real-time fusion of drone flight video overlay and vector file according to claim 1 is characterized in that: In step 5, for point-shaped monitoring objects, according to the coordinates of each point in the point set M, the coordinates in the corresponding image space coordinate system are back-calculated, and the color value of the pixel at the coordinate is defined; for linear and surface monitoring objects, according to the coordinates of each point in the point set M, the coordinates in the corresponding image space coordinate system are back-calculated, and the color value of the pixel at the coordinate is defined. The coordinates of each point in M ​​are connected in sequence, and the color value of the pixel in the line passing through the image between the pixels is defined.

8. The method for real-time fusion of drone flight video overlay and vector file according to claim 1 is characterized in that: In step 6, the corresponding pixel coordinates are calculated based on the collinear equation. , including: for collections For each point in the image, the corresponding It can be seen that: ， ， in, The value of is calculated by the formula of the rotation matrix R. The coordinates of the photography center in the object coordinate system are .

Images