Unmanned aerial vehicle aerial video enhancement method and system, storage medium and electronic device

Abstract

This invention relates to the field of video enhancement technology, specifically disclosing a method, system, storage medium, and electronic device for enhancing drone aerial video. The method includes: acquiring any video frame during a drone's aerial photography of a target area; extracting an image of the target area corresponding to the video frame from an engineering drawing of the target area; and fusing the target area image with the video frame to obtain an enhanced video frame corresponding to the video frame. This invention is unaffected by area boundaries and drone flight paths, features a simple and efficient calculation process, requires no complex modeling, and facilitates high-frequency, long-distance inspection work.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of video enhancement, in particular to a UAV aerial video enhancement method and system, a storage medium and an electronic device. BACKGROUND

[0002] The existing UAV aerial video enhancement method includes the following:

[0003] 1) By listening to the change of the actual camera shooting parameter information, a virtual camera and an actual camera motion synchronization association relationship is established, so as to determine the scale relationship between the virtual three-dimensional scene and the actual scene, and finally according to the determined scale relationship, the object position in the actual scene is calibrated in the spatial coordinates of the virtual three-dimensional scene. This method is applied to video monitoring, so that the existing video can be processed in real time in the three-dimensional virtual scene.

[0004] 2) The UAV uses a camera to collect a strip video, obtains a key frame image, interpolates and calculates the initial exterior orientation element value of the key frame image according to the flight trajectory and attitude data recorded by the flight control or POS system, establishes a single strip oblique image space three project, collects three-dimensional coordinates of image control points, the image control points are distributed along the strip, performs dense matching of the image to obtain a digital surface model in the object field, establishes a three-dimensional geographic scene in a 3DGIS system, imports the digital surface model, sets the background in the three-dimensional scene to black, and only displays the objects or layers that need to be superimposed in the video, creates a viewpoint in the three-dimensional scene frame by frame, renders the viewpoints one by one in the three-dimensional scene to obtain a sequence frame information image, adds a transparent channel to the sequence frame information image, sets the black part to transparent, performs lens distortion transformation to obtain a new sequence frame information image, and directly superimposes the new sequence frame information image on the original video to obtain an augmented reality video result with text notes, graphic annotations and three-dimensional models.

[0005] 3) Obtain an engineering site distribution model and a UAV equipped with an AR device inspection path planning, the UAV flies according to the planned inspection path, adjusts the flight route to fly to at least one engineering site, obtains current engineering data of the at least one engineering site, processes the engineering data to form an inspection image, establishes an engineering quality intelligent evaluation model, controls the UAV to fly to the engineering site to be inspected to obtain the inspection image, and substitutes the inspection image into the model to obtain an evaluation prediction result of the current engineering quality, and forms an engineering quality evaluation result through the obtained engineering site distribution model.

[0006] In the above-mentioned several schemes, three-dimensional scene modeling is a common basic method, the spatial position is easy to locate, and it has good effect for smaller space. However, for large space, undeveloped unmanned area environment, the workload is large, the camera range is limited, and it is difficult to implement.

[0007] According to the flight strip flight and the tilt photography unmanned aerial vehicle aerial video enhancement method, the unmanned aerial vehicle flight strip flight is not supported, the self flight is not supported, the modeling, the video synthesis process and the technical complexity are higher, and the cost is higher, and there is certain technical threshold, and the application is not easy to popularize.

[0008] The unmanned aerial vehicle flies according to the planned inspection path, and the method for obtaining the inspection image provides a daily management tool for observation of a specific target object. However, this scheme has weak video enhancement effect, has size requirement on the engineering site range, and needs to obtain the engineering site panorama in a certain point field to perform engineering evaluation.

[0009] Therefore, there is an urgent need to provide a technical solution to solve the above problems. SUMMARY

[0010] To solve the above technical problems, the present application provides an unmanned aerial vehicle aerial video enhancement method, system, storage medium and electronic device.

[0011] In a first aspect, the present application provides an unmanned aerial vehicle aerial video enhancement method, and the technical scheme of the method is as follows:

[0012] Obtaining any video frame in the aerial photography process of a target unmanned aerial vehicle on a target region;

[0013] From the engineering drawing of the target region, the target region image corresponding to the any video frame is intercepted;

[0014] Fusing the target region image and the any video frame to obtain an enhanced video frame corresponding to the any video frame.

[0015] The beneficial effects of the unmanned aerial vehicle aerial video enhancement method of the present application are as follows:

[0016] The method of the present application is not affected by the region boundary and the unmanned aerial vehicle flight route, the calculation process is simple and efficient, complex modeling is not needed, and the implementation of long-distance and high-frequency inspection work is facilitated.

[0017] On the basis of the above-mentioned scheme, the unmanned aerial vehicle aerial video enhancement method of the present application can be further improved as follows.

[0018] In an optional manner, the step of intercepting the target region image corresponding to the any video frame from the engineering drawing of the target region comprises:

[0019] According to the view angle parameter and the flight parameter when the target unmanned aerial vehicle shoots the any video frame, the geographical coordinate information of the field of view corresponding to the any video frame is determined;

[0020] According to the geographic coordinate information of the field of view corresponding to the any video frame, the target region image corresponding to the any video frame is intercepted from the engineering drawing of the target region.

[0021] In an alternative way, according to the angle of view parameter and the flight parameter of the target UAV when shooting the any video frame, the step of determining the geographic coordinate information of the field of view corresponding to the any video frame comprises:

[0022] According to the angle of view parameter of the target UAV when shooting the any video frame, a plurality of target points in the field of view corresponding to the any video frame are determined.

[0023] According to the flight parameter of the target UAV when shooting the any video frame, the geographic coordinate information of each target point in the field of view corresponding to the any video frame is calculated.

[0024] In an alternative way, according to the geographic coordinate information of the field of view corresponding to the any video frame, the step of intercepting the target region image corresponding to the any video frame from the engineering drawing of the target region comprises:

[0025] The engineering drawing of the target region is converted in coordinate system to obtain a target engineering drawing represented by pixel value.

[0026] According to the geographic coordinate information of each target point in the field of view corresponding to the any video frame, the target region image corresponding to the any video frame is intercepted from the target engineering drawing.

[0027] In an alternative way, the step of fusing the target region image and the any video frame to obtain an enhanced video frame corresponding to the any video frame comprises:

[0028] The target region image and the any video frame are superimposed by using perspective conversion technology to obtain an enhanced video frame corresponding to the any video frame.

[0029] In a second aspect, the application provides an unmanned aerial vehicle aerial video enhancement system, and the technical scheme of the system is as follows:

[0030] The system comprises an acquisition module, a processing module and an enhancement module.

[0031] The acquisition module is configured to acquire any video frame in the process of aerial photography of a target region by a target unmanned aerial vehicle.

[0032] The processing module is configured to intercept a target region image corresponding to the any video frame from an engineering drawing of the target region.

[0033] The enhancement module is configured to fuse the target area image and the any video frame to obtain an enhanced video frame corresponding to the any video frame.

[0034] The unmanned aerial vehicle aerial video enhancement system has the following advantages:

[0035] The system is not affected by the region boundary and the flight route of the unmanned aerial vehicle, the calculation process is simple and efficient, complex modeling is not required, and the system can facilitate implementation of long-distance and high-frequency inspection work.

[0036] Based on the above scheme, the unmanned aerial vehicle aerial video enhancement system can be further improved as follows.

[0037] In an optional manner, the processing module is specifically configured to:

[0038] According to the view angle parameter and the flight parameter of the target unmanned aerial vehicle when the any video frame is captured, geographical coordinate information of a corresponding field of view of the any video frame is determined.

[0039] According to the geographical coordinate information of the corresponding field of view of the any video frame, a target area image corresponding to the any video frame is intercepted from engineering drawings of the target area.

[0040] In an optional manner, the processing module is specifically configured to:

[0041] According to the view angle parameter of the target unmanned aerial vehicle when the any video frame is captured, a plurality of target points in the corresponding field of view of the any video frame are determined.

[0042] According to the flight parameter of the target unmanned aerial vehicle when the any video frame is captured, geographical coordinate information of each target point in the corresponding field of view of the any video frame is calculated.

[0043] In a third aspect, the present application provides a storage medium technical solution as follows:

[0044] The storage medium stores instructions, and when a computer reads the instructions, the computer executes steps of a method for enhancing aerial video of an unmanned aerial vehicle.

[0045] In a fourth aspect, the present application provides an electronic device technical solution as follows:

[0046] The electronic device includes a memory, a processor, and a program stored in the memory and running on the processor, and the processor executes the program to implement steps of a method for enhancing aerial video of an unmanned aerial vehicle.

[0047] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0048] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0049] Figure 1 This is a flowchart illustrating an embodiment of a drone aerial video enhancement method according to the present invention.

[0050] Figure 2 This is a schematic diagram illustrating the principle of calculating target points based on viewpoint parameters;

[0051] Figure 3 This is a schematic diagram illustrating the principle of calculating the geographic coordinates of a target point based on flight parameters.

[0052] Figure 4 This is a schematic diagram illustrating the principle of coordinate system transformation.

[0053] Figure 5 This is a schematic diagram of an embodiment of a drone aerial video enhancement system according to the present invention. Detailed Implementation

[0054] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0055] Figure 1 The diagram illustrates a flowchart of an embodiment of a drone aerial video enhancement method provided by the present invention. Figure 1 As shown, it includes the following steps:

[0056] S1. Acquire any video frame during the aerial photography of the target area by the target drone.

[0057] The target drone can be any type of drone, with no restrictions.

[0058] The target area is the area where the drone will take aerial photos. It defaults to the construction site, but can be adjusted according to actual needs. There are no restrictions here.

[0059] The aerial photography process includes multiple consecutive video frames, each containing all or part of the scene of the target area.

[0060] S2. From the engineering drawing of the target area, extract the target area image corresponding to any video frame.

[0061] Among them, engineering drawings are pre-made engineering drawings.

[0062] The target area image is the image in the engineering drawing corresponding to the field of view of the video frame.

[0063] S3. Merge the target region image with any video frame to obtain the enhanced video frame corresponding to any video frame.

[0064] Among them, the enhanced video frame is the video frame obtained by superimposing and fusing the engineering drawing with the original video frame.

[0065] Preferably, S2 includes:

[0066] S21. Based on the perspective parameters and flight parameters when the target UAV captures any video frame, determine the geographic coordinate information of the field of view corresponding to any video frame.

[0067] The viewing angle parameter refers to the viewing angle parameter of the aerial camera on the drone. Specifically, it includes:

[0068] The flight parameters include: equivalent focal length, latitude and longitude, flight altitude, aircraft yaw / pitch / roll attitude parameters, and gimbal yaw / pitch / roll attitude parameters.

[0069] The field of view is the area that the aerial camera can capture.

[0070] S22. Based on the geographic coordinate information of the field of view corresponding to any video frame, extract the target area image corresponding to any video frame from the engineering drawing of the target area.

[0071] Preferably, S21 includes:

[0072] S211. Based on the viewing angle parameters when the target UAV captures any video frame, determine multiple target points within the field of view corresponding to any video frame.

[0073] Among them, multiple target points include: each endpoint, each edge midpoint, and the center point within the field of view.

[0074] Specifically, the angle relationships between each endpoint, the midpoint of the edge line, and the center point within the field of view are calculated using the viewing angle parameters of the target drone's aerial camera. For example... Figure 2 As shown. In Figure 2In this diagram, y represents the distance from the focal point to the center of the lens; b represents the distance from the focal point to the midpoint of the bottom edge of the lens; w represents the lens width; h represents the lens height; z = 0.5h; g represents the angle between b and y; j represents the angle between y and the line connecting the focal point to the midpoint of the right edge of the lens; k represents the angle between b and the line connecting the focal point to the left endpoint of the bottom edge of the lens; and x represents the line segment from the center of the lens to the left endpoint of the bottom edge of the lens.

[0075] S212. Based on the flight parameters of the target UAV when capturing any video frame, calculate the geographic coordinate information of each target point within the field of view corresponding to any video frame.

[0076] Specifically, the geographic coordinates of each target point within the field of view are calculated based on the flight parameters of the target UAV. For example... Figure 3 As shown, y′ represents the distance from the center of the lens to the ground plane; b′ represents the distance from the center of the bottom edge of the lens to the ground plane; z′ represents the distance between the endpoints of y′ and b′ on the ground plane; w′ represents the distance between the point on the left edge of the lens and the point on the right edge of the lens; crc represents the midpoint of w′; rel_alt represents the vertical distance from the focus to the ground plane; and dv represents the distance between rel_alt and the endpoint of b′ on the ground plane.

[0077] Preferably, S22 includes:

[0078] S221. Perform coordinate system transformation on the engineering drawing of the target area to obtain the target engineering drawing represented by pixel values.

[0079] The coordinate system transformation process involves converting the mapping coordinate system to pixel values ​​to obtain the target engineering drawing represented by pixel values.

[0080] It should be noted that, as Figure 4 As shown, the mapping coordinate system is transformed into pixel values ​​to obtain the actual distance value represented by pixel distance. Then, the distance range of the main ground areas within the field of view is calculated using the geographic coordinate information within the field of view, and converted into pixel distance values. Figure 4 In the diagram, clt represents the upper left corner of the field of view; clb represents the lower left corner of the field of view; crt represents the upper right corner of the field of view; and crb represents the lower right corner of the field of view.

[0081] S222. Based on the geographic coordinate information of each target point within the field of view corresponding to any video frame, extract the target area image corresponding to any video frame from the target engineering drawing.

[0082] Each point in the engineering drawing corresponds to a geographic coordinate. Therefore, the corresponding target area image can be extracted from the engineering drawing based on the geographic coordinates of each target point within the field of view of the video frame.

[0083] Preferably, S3 includes:

[0084] Using perspective transformation technology, the image of the target region is superimposed on any video frame to obtain an enhanced video frame corresponding to any video frame.

[0085] The perspective conversion technology is an existing technology, and its specific principles will not be elaborated on here.

[0086] The technical solution of this embodiment is not affected by regional boundaries and UAV flight paths. The calculation process is simple and efficient, requiring no complex modeling, and can facilitate the implementation of high-frequency inspection work over long distances.

[0087] Figure 5 A schematic diagram of an embodiment of a drone aerial video enhancement system 200 provided by the present invention is shown. Figure 5 As shown, the system 200 includes: an acquisition module 210, a processing module 220, and an enhancement module 230;

[0088] The acquisition module 210 is used to: acquire any video frame during the aerial photography process of the target drone over the target area;

[0089] The processing module 220 is used to: extract the target area image corresponding to any video frame from the engineering drawing of the target area;

[0090] The enhancement module 230 is used to: fuse the target region image with any video frame to obtain an enhanced video frame corresponding to any video frame.

[0091] Preferably, the processing module 220 is specifically used for:

[0092] Based on the perspective parameters and flight parameters of the target UAV when capturing any video frame, determine the geographic coordinate information of the field of view corresponding to any video frame;

[0093] Based on the geographic coordinates of the field of view corresponding to any video frame, the target area image corresponding to any video frame is extracted from the engineering drawing of the target area.

[0094] Preferably, the processing module 220 is specifically used for:

[0095] Based on the viewing angle parameters when the target drone captures any video frame, determine multiple target points within the field of view corresponding to any video frame;

[0096] Based on the flight parameters of the target UAV when capturing any video frame, calculate the geographic coordinate information of each target point within the field of view corresponding to any video frame.

[0097] Preferably, the processing module 220 is specifically used for:

[0098] The engineering drawing of the target area is transformed into a coordinate system to obtain the target engineering drawing represented by pixel values;

[0099] Based on the geographic coordinates of each target point within the field of view corresponding to any video frame, the target area image corresponding to any video frame is extracted from the target engineering drawing.

[0100] Preferably, the enhancement module 230 is specifically used for:

[0101] Using perspective transformation technology, the image of the target region is superimposed on any video frame to obtain an enhanced video frame corresponding to any video frame.

[0102] The technical solution of this embodiment is not affected by regional boundaries and UAV flight paths. The calculation process is simple and efficient, requiring no complex modeling, and can facilitate the implementation of high-frequency inspection work over long distances.

[0103] The parameters and steps for implementing the corresponding functions of each module in the UAV aerial video enhancement system 200 of this embodiment can be referred to the parameters and steps in the embodiments of the UAV aerial video enhancement method above, and will not be repeated here.

[0104] An embodiment of the present invention provides a storage medium, comprising: the storage medium storing instructions, which, when read by a computer, cause the computer to execute steps such as a drone aerial video enhancement method. For details, please refer to the parameters and steps in the embodiments of the drone aerial video enhancement method described above, which will not be repeated here.

[0105] Computer storage media, such as USB flash drives and external hard drives.

[0106] An electronic device provided by an embodiment of the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it causes the computer to perform steps such as the drone aerial video enhancement method. For details, please refer to the parameters and steps in the embodiments of the drone aerial video enhancement method described above, which will not be repeated here.

[0107] Those skilled in the art will know that the present invention can be implemented as a method, system, storage medium, and electronic device.

[0108] Therefore, the present invention can be implemented in the following forms: it can be entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software, generally referred to herein as a "circuit," "module," or "system." Furthermore, in some embodiments, the present invention can also be implemented as a computer program product contained in one or more computer-readable media, which contains computer-readable program code. Any combination of one or more computer-readable media can be used. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. Although embodiments of the invention have been shown and described above, it is to be understood that these embodiments are exemplary and should not be construed as limiting the invention. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the invention.

Claims

1. A method for enhancing drone aerial video, characterized in that, include: Acquire any video frame during the aerial photography of the target area by the target drone; Based on the viewing angle parameters when the target drone captures any video frame, multiple target points within the field of view corresponding to any video frame are determined; wherein, the viewing angle parameters are: the viewing angle parameters of the aerial camera on the drone; the multiple target points include: each endpoint, each edge midpoint, and the center point within the field of view; Based on the flight parameters captured by the target UAV when capturing any video frame, calculate the geographic coordinate information of each target point within the field of view corresponding to any video frame; wherein, the flight parameters include: equivalent focal length, latitude and longitude, flight altitude, aircraft yaw / pitch / roll attitude parameters, and gimbal yaw / pitch / roll attitude parameters. The engineering map of the target area is transformed into a coordinate system to obtain a target engineering map represented by pixel values; wherein, the mapping coordinate system is transformed into pixel values ​​to obtain the actual distance value represented by pixel distance, and then the distance range of the main ground area within the field of view is calculated using the geographic coordinate information within the field of view and converted into pixel distance values. Based on the geographic coordinate information of each target point within the field of view corresponding to any video frame, the target area image corresponding to any video frame is extracted from the target engineering drawing; Using perspective transformation technology, the image of the target region is superimposed on any video frame to obtain an enhanced video frame corresponding to any video frame.

2. A drone aerial video enhancement system, characterized in that, include: Acquisition module, processing module, and enhancement module; The acquisition module is used to: acquire any video frame during the aerial photography process of the target drone over the target area; The processing module is used to: determine multiple target points within the field of view corresponding to any video frame based on the viewing angle parameters captured by the target drone; calculate the geographic coordinate information of each target point within the field of view corresponding to any video frame based on the flight parameters captured by the target drone; perform coordinate system transformation on the engineering map of the target area to obtain a target engineering map represented by pixel values; and extract the target area image corresponding to any video frame from the target engineering map based on the geographic coordinate information of each target point within the field of view corresponding to any video frame. Among them, the viewing angle parameters are: the viewing angle parameters of the aerial camera on the drone; multiple target points include: each endpoint, each edge midpoint and center point within the field of view; flight parameters include: equivalent focal length, latitude and longitude, flight altitude, aircraft yaw / pitch / roll attitude parameters, and gimbal yaw / pitch / roll attitude parameters. The process involves converting the cartographic coordinate system to pixel values ​​to obtain the actual distance value represented by pixel distance, and then calculating the distance range of the main ground areas within the field of view using geographic coordinate information within the field of view, which is then converted into pixel distance values. The enhancement module is used to: superimpose the target region image with any video frame using perspective transformation technology to obtain an enhanced video frame corresponding to any video frame.

3. A storage medium, characterized in that, The storage medium stores instructions that, when read by a computer, cause the computer to execute the drone aerial video enhancement method as described in claim 1.

4. An electronic device comprising a memory, a processor, and a program stored in the memory and running on the processor, characterized in that, When the processor executes the program, it implements the steps of the drone aerial video enhancement method as described in claim 1.

Images