Method for fusing aerial video and real scene three-dimensional scene and method for fusing and displaying

Abstract

This invention discloses a method for fusing aerial video and real-world 3D scenes, and a method for displaying the fused images. The method includes: First, acquiring a real-world 3D model corresponding to a target area and multiple sequences of aerial video, as well as device control parameters recorded by a drone during aerial video acquisition. Then, setting a roaming path in the real-world 3D model; next, selecting corresponding aerial video based on the roaming path, and fusing the aerial video with the real-world 3D model according to the corresponding device control parameters. Next, determining corresponding transition frames through scene-switching frames in the aerial video and their adjacent video frames, along with the spatial and perspective information corresponding to the transition frames. Finally, fusing each transition frame with the real-world 3D model based on the corresponding spatial and perspective information, resulting in a smooth transition between the aerial video and the real-world 3D model, thereby reducing abruptness and jumpiness during the transition and maintaining a smooth visual experience.

Description

Technical Field

[0001] This invention relates to the field of video image processing technology, specifically to a method for fusing flight video and real-world 3D scene, and a method for displaying the fused video. Background Technology

[0002] Different ways of expressing real-world scenes have different effects. Aerial flight videos are high-definition videos shot along predetermined flight trajectories, which can highly restore real-world scenes and enhance the perception of reality. However, the perspective is relatively singular and the amount of content expressed is relatively limited. There is a sense of disjointedness and spatiotemporal dispersion between aerial flight videos. Real-world 3D scenes reconstruct spatial entities through high-resolution oblique photography images, reconstructing the real world from multiple angles and expanding spatial characteristics.

[0003] However, the modeling process suffers from occlusion and texture loss, failing to realistically recreate the real space and lacking temporal dynamics. Therefore, integrating aerial video with real-world 3D scenes is essential for better representing the real-world environment. This enhances the spatial depth and scene realism, imbues the scene with temporal sequence characteristics, and enriches the realism of the virtual environment while enhancing the spatiotemporal characteristics of the real scene. This combination of virtual and real elements allows for a more efficient and realistic representation of the real-world scene.

[0004] Patent CN111225191B discloses a method, apparatus, and electronic device for three-dimensional video fusion. By using device information and location information, it determines the real-scene three-dimensional video fusion parameters of the monitoring device, and based on these parameters, it fuses the real-time video data of the monitoring device with the real-scene three-dimensional model determined by a preset position on the monitoring device in real time.

[0005] Patent application CN113033412A discloses a traffic situation monitoring method, device, equipment and medium based on real scene fusion. It generates an initial real scene fusion map by fusing multiple acquired on-site videos to be monitored with a three-dimensional real scene model, and inputs the initial real scene fusion map into a preset operating system so that the preset operating system generates a target real scene fusion map.

[0006] While existing technologies have achieved the fusion of video and real-world 3D models, there are certain differences between the visual effects of video and real-world 3D models. If a real-world 3D model is directly switched to a video, or vice versa, it will produce certain abruptness and judder in the visuals, resulting in a less smooth viewing experience. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention proposes a method for fusing aerial flight videos and real-world 3D scenes, as well as a method for displaying the fused images. This reduces the abruptness and jarring transitions between video and real-world 3D models, maintaining a smooth visual experience. The specific technical solution is as follows:

[0008] Firstly, a method for fusing flight video and real-world 3D scenes is provided, including:

[0009] Acquire the real-world 3D model and flight video of the target area, as well as the equipment control parameters recorded when the drone collects the flight video;

[0010] The flight video is fused with the real-scene 3D model according to the corresponding equipment control parameters;

[0011] The corresponding transition frames, spatial information, and viewpoint information corresponding to the transition frames are determined by the switching frames involving scene changes and their adjacent video frames in the flight video.

[0012] Each transition frame is fused with the real-world 3D model based on the corresponding spatial and perspective information.

[0013] In conjunction with the first aspect, in the first possible implementation of the first aspect, an oblique photographic image corresponding to the target area is obtained, and the real-scene 3D model is constructed using the oblique photographic image.

[0014] In conjunction with the first aspect, in the second possible implementation of the first aspect, obtaining the flight video includes:

[0015] The flight path and shooting angle are preset in the real-world 3D model, and a pre-rehearsal is performed according to the preset flight path and shooting angle;

[0016] Adjust the flight trajectory and shooting angle based on the pre-rehearsal effect until the video shooting effect reaches the best, and determine the best flight trajectory and best shooting angle.

[0017] Based on the optimal flight trajectory and the optimal shooting angle, the drone is controlled to shoot the target area and acquire the flight video.

[0018] In conjunction with the first aspect, the third possible implementation of the first aspect further includes: performing noise reduction processing and / or color correction on the flight video.

[0019] In conjunction with the first aspect, in the fourth possible implementation of the first aspect, the fusion of the flight video with the real-world 3D model includes:

[0020] The spatial and perspective information corresponding to the switching frames involving scene switching in the flight video are determined by the corresponding equipment control parameters.

[0021] Linearly match the video frame image of the switching frame with the scene image at the corresponding position in the real-world 3D model;

[0022] The spatial and viewpoint information corresponding to the switching frame is corrected based on the linear matching results.

[0023] In conjunction with the first aspect, in the fifth possible implementation of the first aspect, a video frame interpolation algorithm is used to calculate the corresponding transition frame based on the switching frame and its adjacent video frames.

[0024] In conjunction with the first aspect, in a sixth possible implementation of the first aspect, determining the transition frame includes:

[0025] Based on the switching frame and its adjacent video frames, the initial transition frame, as well as the spatial and viewpoint information corresponding to the initial transition frame, are calculated using a video frame interpolation algorithm.

[0026] The scene image at the corresponding location in the real-world 3D model is determined by using the spatial and perspective information corresponding to the initial transition frame;

[0027] The transition frame is obtained by fusing the video frame image corresponding to the transition frame with the scene image using a weighted average method.

[0028] In conjunction with the sixth possible implementation of the first aspect, in the seventh possible implementation of the first aspect, the weights corresponding to the video frame image and the scene image are determined based on the relative positional relationship between the positions corresponding to the switching frame and the scene image and the positions corresponding to the initial transition frame.

[0029] Secondly, a method for fusing and displaying real-scene 3D model videos is provided, including:

[0030] Acquire the real-world 3D model of the target area and multiple sequences of flight videos, as well as the equipment control parameters recorded when the UAV collects flight videos;

[0031] Set the roaming path in the real-world 3D model;

[0032] Select the corresponding flight video according to the roaming path, and fuse the flight video with the real-scene 3D model according to the corresponding equipment control parameters;

[0033] The corresponding transition frames, spatial information, and viewpoint information corresponding to the transition frames are determined by the switching frames involving scene changes and their adjacent video frames in the flight video.

[0034] The transition frames are fused with the real-world 3D model based on the corresponding spatial and perspective information.

[0035] In conjunction with the second aspect, the first possible implementation of the second aspect further includes: performing a linear transformation on the set roaming path based on the UAV flight trajectory corresponding to the selected flight video.

[0036] Beneficial Effects: The flight video and real-world 3D scene fusion method and fusion display method of this invention can calculate the transition frames corresponding to the transformation between the flight video and the real-world 3D model by using the switching frames involving scene transitions in the flight video and the video frames adjacent to the switching frames. By fusing the transition frames with the real-world 3D model, the flight video can smoothly transition to the real-world 3D model, or vice versa, thereby reducing the abruptness and jumpiness when switching between the video and the real-world 3D model, and maintaining a smooth visual experience. Attached Figure Description

[0037] To more clearly illustrate the specific embodiments of the present invention, the accompanying drawings used in the specific embodiments will be briefly described below. In all the drawings, the elements or parts are not necessarily drawn to scale.

[0038] Figure 1 A flowchart of a method for fusing flight video and real-world 3D scene according to an embodiment of the present invention;

[0039] Figure 2 This is a flowchart of a flight video acquisition process provided in an embodiment of the present invention;

[0040] Figure 3 This is a flowchart illustrating the correction process for spatial and viewpoint information of a video frame according to an embodiment of the present invention.

[0041] Figure 4 This is a flowchart of a scene switching transition frame provided in an embodiment of the present invention;

[0042] Figure 5 A flowchart of a method for fusing and displaying real-scene 3D model videos according to an embodiment of the present invention;

[0043] Figure 6 This is a schematic diagram of a video preprocessing process provided in an embodiment of the present invention;

[0044] Figure 7 A schematic diagram illustrating linear matching between a video frame image and a scene image in a switching frame, provided in an embodiment of the present invention;

[0045] Figure 8 A schematic diagram illustrating the principle of calculating the initial transition frame according to an embodiment of the present invention;

[0046] Figure 9 This is a schematic diagram illustrating the principle of fusing an initial transition frame with a real-world 3D model image, as provided in an embodiment of the present invention.

[0047] Figure 10 This is a schematic diagram of a roaming path provided in an embodiment of the present invention;

[0048] Figure 11This is a schematic diagram of a roaming path after linear processing, provided in an embodiment of the present invention. Detailed Implementation

[0049] The embodiments of the technical solution of the present invention will now be described in detail with reference to the accompanying drawings. These embodiments are merely illustrative of the technical solution of the present invention and are therefore intended to limit the scope of protection of the present invention.

[0050] Example 1

[0051] like Figure 1 The flowchart shown illustrates a method for fusing flight video and real-world 3D scenes. This method includes:

[0052] Step 1: Obtain the real-world 3D model and multiple sequences of flight videos corresponding to the target area, as well as the equipment control parameters recorded by the UAV when collecting flight videos;

[0053] Step 2: Fuse the flight video with the real-scene 3D model according to the corresponding equipment control parameters;

[0054] Step 3: Determine the corresponding transition frames, spatial information, and viewpoint information of the transition frames by using the switching frames involving scene changes and their adjacent video frames in the flight video.

[0055] Step 4: Fuse each transition frame with the real-world 3D model based on the corresponding spatial and perspective information.

[0056] Specifically, firstly, a realistic 3D model of the target area can be built using existing methods for constructing realistic 3D models, and then aerial video of the target area can be collected using drones. During the aerial video collection, the equipment control parameters can be recorded in real time.

[0057] It should be understood that, in this embodiment, the device control parameters include parameters such as the flight trajectory, viewing angle, heading, and camera tilt of the UAV during flight.

[0058] Then, the spatial and perspective information of each video frame in each sequence of flight videos is determined by the recorded equipment control parameters. Based on the corresponding spatial and perspective information, each video frame is fused with the real-world 3D model, thereby achieving the purpose of fusion between flight videos and real-world 3D models.

[0059] Subsequently, transition frames used for scene transitions, along with their corresponding spatial and perspective information, can be calculated from the switching frames in the flight video and adjacent video frames. Finally, based on the corresponding spatial and perspective information, the transition frames are fused with the real-world 3D model, allowing for a smooth transition from flight video to the real-world 3D model, or vice versa. This reduces the abruptness and jumpiness of the transition between video and the real-world 3D model, maintaining a smooth visual experience.

[0060] In this embodiment, optionally, in step 1, the oblique photographic image corresponding to the target area can be obtained, and the real-scene 3D model can be constructed using the oblique photographic image.

[0061] Specifically, high-resolution, high-overlapping oblique photography images can be acquired using multi-lens photography sensors mounted on drones. Based on these oblique photography images, a real-world 3D model can be constructed. The model can then be repaired, textured, and cropped to create an effective stereoscopic field of view, making the 3D model more three-dimensional and intuitive.

[0062] In this embodiment, optionally, such as Figure 2 As shown, in step 1, acquiring the flight video includes:

[0063] Step 1-1: Preset the flight trajectory and shooting angle in the real-world 3D model, and conduct a pre-play according to the preset flight trajectory and shooting angle;

[0064] Steps 1-2: Adjust the flight trajectory and shooting angle based on the pre-rehearsal effect until the video shooting effect reaches the best, and determine the best flight trajectory and the best shooting angle;

[0065] Steps 1-3: Based on the optimal flight trajectory and optimal shooting angle, control the drone to shoot the target area and obtain the flight video.

[0066] Specifically, in this embodiment, a linear shooting method can be used to acquire flight video. Specifically, the model display effect can be previewed in the established realistic 3D model, marking the areas in the model that need to be specifically displayed in the video. A flight trajectory is preset in the 3D scene, and the scene viewpoint browses the model along the preset trajectory. The trajectory and camera angle are repeatedly adjusted to achieve a better browsing effect. During this process, the flight video shooting effect can be pre-rehearsed until the video shooting effect is optimal, ensuring that the video shooting focuses on the area that needs to be displayed, and finally deriving the optimal flight trajectory and optimal shooting angle.

[0067] When shooting aerial video, the drone's optimal flight path is used to determine its flight trajectory, and the optimal shooting angle is used to determine the drone's gimbal pitch, roll, and yaw angles. The drone flies along the flight path, and aerial video is collected in real time during the flight. During the acquisition of aerial video, the gimbal pitch, roll, and yaw angles are adjusted in real time according to the optimal shooting angle.

[0068] Because actual flight may deviate from the pre-set parameters, it's crucial to accurately record flight parameters such as trajectory, shooting angle, and heading during the actual flight process while acquiring video. In the subsequent fusion of the flight video and the real-world 3D model, these recorded flight parameters can be used for fusion.

[0069] It should be understood that this embodiment only uses linear shooting as an example for illustration; fixed-point shooting can also be used to collect flight video. For fixed-point shooting, the optimal spatial position and viewpoint can be selected in the real-world 3D model based on the shooting area, and the camera's spatial position and viewpoint information can be recorded. During shooting, the drone can move to the camera's spatial position and collect video according to the viewpoint information corresponding to that spatial position.

[0070] In this embodiment, optionally, the method further includes: noise reduction processing and / or color correction of the flight video.

[0071] Specifically, due to factors such as weather, flight conditions, and shooting environment, the acquired flight videos often suffer from problems such as multiple noise points and poor contrast, resulting in a certain degree of distortion. Therefore, preprocessing is necessary for the acquired flight videos, such as... Figure 6 As shown, the preprocessing operations include noise reduction and color correction.

[0072] Noise reduction processing addresses significant and coarse noise points in flight videos using a combination of single-frame and multi-frame noise reduction methods. Single-frame noise reduction primarily involves filtering the video frame, smoothing it by weighted averaging of surrounding pixels. Multi-frame noise reduction aligns and merges multiple video frames, or even the entire flight video, based on the single-frame noise reduction results, using weighted averaging.

[0073] Color adjustment involves modifying information such as brightness and sharpness in flight videos. During filming, factors like lighting angles and environmental changes can cause videos to be too dark, too bright, or lack color richness. Therefore, color correction is necessary to adjust parameters such as color saturation, brightness, contrast, and sharpness to improve visual effects and enhance clarity, resulting in clear and visually appealing flight videos. Simultaneously, the color space of the real-world 3D scene can be incorporated into the color adjustments to ensure a closer match between the two color spaces, thereby enhancing the overall integration.

[0074] In this embodiment, the switching frames include the start frame and end frame of the flight video, as well as the video frames corresponding to the entry and exit points within the flight video. Before fusing the flight video with the real-world 3D model, the spatial location information of the switching frames can be determined based on the recorded UAV flight trajectory. The fusion points of the flight video are then marked in the real-world 3D scene based on the spatial location information of the switching frames, serving as the initial registration for embedding the flight video into the real-world 3D scene. This facilitates subsequent linear matching of the video frames of the flight video with the scene images of the real-world 3D model.

[0075] In this embodiment, optionally, such as Figure 3 As shown, in step 2, fusing the flight video with the real-world 3D model includes:

[0076] Step 2-1: Determine the spatial and perspective information corresponding to the switching frames involving scene switching in the flight video using the relevant equipment control parameters;

[0077] Step 2-2: Perform linear matching between the video frame image of the switched frame and the scene image at the corresponding position in the real-world 3D model;

[0078] Steps 2-3: Correct the spatial and viewpoint information corresponding to the switching frame based on the linear matching results.

[0079] Specifically, aerial video is essentially photographic images with a higher overlap rate than the real-world 3D model, which is essentially a collection of high-resolution, full-view photographic images. The content of each frame in the aerial video is highly similar to the real-world 3D model. Therefore, by closely matching the video frames in the aerial video with the content of the real-world 3D scene, a high degree of fusion between the aerial video and the real-world 3D model can be achieved. The fusion points marked in the real-world 3D model are the key locations for the fitting of the aerial video and the real-world 3D scene content.

[0080] Because the flight trajectories and video frame images corresponding to each video frame in the flight video exhibit a high degree of continuity, the spatial and perspective information corresponding to each video frame in the flight video can be calculated by combining the recorded flight trajectory, shooting angle, and coordinate information. This allows for the accurate positioning of the video frame's spatial location and orientation within the real-world 3D scene.

[0081] Among them, the start and end frames of the flight video are key to embedding the flight video into the real-world 3D model. The image content expressed during the conversion process between the flight video and the real-world 3D model is consistent, and the video frame images of the start and end frames are highly continuous with the real-world 3D scene images.

[0082] like Figure 7 As shown, the video frame images of the start and end frames can be linearly matched by combining high-resolution images in the real-world 3D scene. Based on the matching results, the spatial and viewpoint information of the start and end frames can be corrected to obtain more accurate spatial and viewpoint information corresponding to the start and end frames. Finally, based on the spatial and viewpoint information corresponding to the start and end frames, a high degree of fit between the video frame images with the spatial and viewpoint information and the real-world 3D scene images can be achieved.

[0083] In this embodiment, optionally, in step 3, the corresponding transition frame can be calculated using a video frame interpolation algorithm based on the switching frame and its adjacent video frames.

[0084] Specifically, such as Figure 8 As shown, flight video is essentially a highly continuous image. By combining the switching frame and the motion direction of pixel 1 in the adjacent video frame within the flight video, existing video frame interpolation algorithms can be used to calculate the position of pixel 1 in the transition frame, and then the video frame content of the transition frame can be calculated. Combining the spatial and viewpoint information of the known frames, the spatial and viewpoint information corresponding to the transition frame can be calculated. The specific steps of the video frame interpolation algorithm are common techniques in this field and will not be elaborated here.

[0085] In this embodiment, optionally, in step 4, the spatial position of the transition frame in the real-world 3D model can be determined using the corresponding spatial information. The orientation of the transition frame in the real-world 3D model can also be determined using viewpoint information, thus achieving the fusion of the transition frame and the real-world 3D model.

[0086] Example 2

[0087] Example 2 is largely the same as Example 1, with the main difference being: Figure 4 As shown, in step 3, determining the transition frame includes:

[0088] Step 3-1: Based on the switching frame and its adjacent video frames, use a video frame interpolation algorithm to calculate the initial transition frame, as well as the spatial and viewpoint information corresponding to the initial transition frame;

[0089] Step 3-3: Determine the scene image at the corresponding location in the real-world 3D model using the spatial and perspective information corresponding to the initial transition frame;

[0090] Step 3-3: Use a weighted average method to fuse the video frame image corresponding to the initial transition frame with the scene image to obtain the transition frame.

[0091] Specifically, such as Figure 9 As shown, firstly, by combining the switching frame and the video frames adjacent to the switching frame within the flight video, the video frame content of the transition frame can be deduced using existing video frame interpolation algorithms. Combining the spatial and viewpoint information of the known frames above, the spatial and viewpoint information corresponding to the transition frame can be deduced.

[0092] Then, based on the spatial information corresponding to the calculated transition frame, the scene image corresponding to the corresponding position in the real-world 3D model is determined. Finally, a weighted average method is used to calculate the pixel value or grayscale value, thereby fusing the video frame image corresponding to the transition frame with the scene image to obtain a transition frame with higher fit to the real-world 3D model. The specific calculation formula is as follows:

[0093]

[0094] Where p is the fused transition frame, p1 is the pixel value or grayscale value of the initial transition frame, p2 is the pixel value or grayscale value of the scene image of the real-world 3D model, w1 is the weight of the initial transition frame, and w2 is the weight of the scene image.

[0095] In this embodiment, optionally, the weights of the video frame image and the scene image are determined based on the relative positional relationship between the positions corresponding to the switching frame and the scene image and the positions corresponding to the initial transition frame.

[0096] Specifically, the distance between the position corresponding to the initial transition frame and the position corresponding to the switching frame, as well as the distance between the position corresponding to the initial transition frame and the position corresponding to the scene image after the flight video is cut out, can be determined based on the spatial information corresponding to the initial transition frame.

[0097] The two are compared, and weights are assigned to the video frame image and the scene image when they are fused based on the comparison results. The closer the transition frame is to the original video frame, the more closely its content matches the video content, and the higher the weight is assigned to the calculated transition frame. Similarly, the closer the transition frame is to the real-world 3D scene, the more closely its content matches the scene image, and the higher the weight is assigned to the scene image.

[0098] When the distance between the initial transition frame and the scene image is not significantly different from the distance between the initial transition frame and the switching frame, or when the distance calculation is difficult, the total weight can be determined based on the number of initial transition frames between the switching frame and the scene image, and then the weight of each initial transition frame can be determined based on the total weight.

[0099] For example, first, the scene images, transition frames, and initial transition frames can be ordered by spatial position, such as transition frames, initial transition frame 1, initial transition frame 2, initial transition frame 3, initial transition frame 4, initial transition frame 5, initial transition frame 6, and the 3D scene. Then, the total weight can be determined as 7 based on the total number of initial transition frames. Finally, the weight of initial transition frame 1 can be directly assigned as 6, and the corresponding scene image can be assigned as 1. Initial transition frame 2 can be directly assigned as 5, and the corresponding scene image can be assigned as 2, and so on, to determine the weights corresponding to all initial transition frames.

[0100] like Figure 5 The flowchart shown illustrates a method for fusing and displaying real-world 3D models and videos. This method includes:

[0101] Step 1: Obtain the real-world 3D model and multiple sequences of flight videos corresponding to the target area, as well as the equipment control parameters recorded by the UAV when collecting flight videos;

[0102] Step 2: Set the roaming path in the real-world 3D model;

[0103] Step 3: Select the corresponding flight video according to the roaming path, and fuse the flight video with the real-scene 3D model according to the corresponding equipment control parameters;

[0104] Step 4: Determine the corresponding transition frames, spatial information, and viewpoint information of the transition frames by using the switching frames involving scene changes and their adjacent video frames in the flight video.

[0105] Step 5: Fuse each transition frame with the real-world 3D model based on the corresponding spatial and perspective information.

[0106] Specifically, firstly, a realistic 3D model of the target area can be built using existing methods for constructing realistic 3D models, and then aerial video of the target area can be collected using drones. During the aerial video collection, the equipment control parameters can be recorded in real time.

[0107] Then, a roaming path is set in the real-world 3D model, which includes the spatial location data of the area displayed in the real-world 3D model.

[0108] Then, the corresponding flight video can be selected based on the spatial location data in the set roaming path, and the spatial and perspective information of each video frame in the selected flight video can be determined by the recorded equipment control parameters. Based on the corresponding spatial and perspective information, each video frame is fused with the real-world 3D model.

[0109] Then, by selecting switching frames in the flight video and adjacent video frames, the transition frames used for scene transitions, as well as the corresponding spatial and perspective information of the transition frames, can be calculated. Finally, based on the corresponding spatial and perspective information, the transition frames are fused with the real-world 3D model, which allows for a smooth transition from the flight video to the real-world 3D model, or vice versa, thereby reducing the abruptness and jumpiness of the transition between the video and the real-world 3D model and maintaining a smooth visual experience.

[0110] In this embodiment, optionally, step 2 may further include: performing a linear transformation on the set roaming path based on the UAV flight trajectory corresponding to the selected flight video.

[0111] Specifically, in actual fusion display, the distribution of flight videos in the real-world 3D model is discrete, and the spatial position and viewpoint information of the starting position of the next flight video may be inconsistent with the spatial position and viewpoint information of the ending position of the previous flight video.

[0112] When displaying videos continuously, it is necessary to switch perspectives and connect videos through 3D scenes. By setting a roaming path in the 3D scene, the cut-out position and perspective of the previous video are dynamically switched to the starting position and perspective of the next video. However, the trajectory of the flight video is fixed, and the perspectives at the beginning and end positions of the video are also fixed. Therefore, the roaming path in the real-world 3D scene can be linearly processed to linearly connect the roaming path and the flight trajectory, ensuring the continuity between the flight trajectory and the scene roaming path. This enables linear cut-in and cut-out between the real-world 3D scene and the flight video, thereby ensuring the continuity of perspective during scene and video switching, and achieving seamless transitions and connections between flight video and the real-world 3D scene.

[0113] In this embodiment, the linear processing of the roaming path can employ either Lagrange interpolation or spline function interpolation to fit a transition path that connects the roaming path and the flight trajectory.

[0114] For example, as shown in Figure 10, the fused display perspective linearly cuts into flight path 1 from point a, and the fused display content is flight video 1. The video is cut out from point a1 along flight path 1 to display the scene image of the real-world 3D model, and then the video is cut back to flight video 1 from point a2. After flight video 1 ends, the video is cut out from point b, and the camera motion trajectory is cut out along the tangent line at point b.

[0115] To obtain a smooth motion trajectory, one can... Figure 10 The roaming path BCD set in the middle is linearly transformed into a smooth curve, resulting in, for example, ... Figure 11 As shown in the curve bcd, the viewpoint along curve bcd displays a real-world 3D scene. During this process, the inward-facing viewpoint of flight video 1 is gradually rotated to the viewpoint of flight video 2, and at point d, a linear cut-in is made to flight video 2 to display its content. Repeating the above steps achieves the fusion display of flight video and real-world 3D scene.

[0116] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be covered within the scope of the claims and specification of the present invention.

Claims

1. A method for fusing flight video and real-world 3D scenes, characterized in that, include: Acquire a real-world 3D model and flight video of the target area, as well as the equipment control parameters recorded by the UAV when collecting the flight video. The flight video acquisition includes: The flight path and shooting angle are preset in the real-world 3D model, and a pre-rehearsal is performed according to the preset flight path and shooting angle; Adjust the flight trajectory and shooting angle based on the pre-rehearsal effect until the video shooting effect reaches the best, and determine the best flight trajectory and best shooting angle. Based on the optimal flight trajectory and optimal shooting angle, the drone is controlled to shoot the target area and acquire the flight video; The flight video is fused with the real-world 3D model according to the corresponding equipment control parameters, including: The spatial and perspective information corresponding to the switching frames involving scene switching in the flight video are determined by the corresponding equipment control parameters. Linearly match the video frame image of the switching frame with the scene image at the corresponding position in the real-world 3D model; The spatial and viewpoint information corresponding to the switching frame are corrected based on the linear matching results; The transition frames are determined by identifying the transition frames involved in scene switching in the flight video and their adjacent video frames, along with the spatial and viewpoint information corresponding to the transition frames. The determination of the transition frames includes: Based on the switching frame and its adjacent video frames, the initial transition frame, as well as the spatial and viewpoint information corresponding to the initial transition frame, are calculated using a video frame interpolation algorithm. The scene image at the corresponding location in the real-world 3D model is determined by using the spatial and perspective information corresponding to the initial transition frame; The transition frame is obtained by fusing the video frame image corresponding to the initial transition frame with the scene image using a weighted average method. The specific calculation formula is as follows: ； in, This is the transition frame after merging. The pixel value or grayscale value of the initial transition frame. These are the pixel values ​​or grayscale values ​​of the scene image in the real-world 3D model. The weights for the initial transition frames. The weights of the scene image; The transition frame is fused with the real-world 3D model based on the corresponding spatial and perspective information.

2. The method for fusing flight video and real-world 3D scene according to claim 1, characterized in that, Oblique photographic images corresponding to the target area are acquired, and the real-world 3D model is constructed using the oblique photographic images.

3. The method for fusing flight video and real-world 3D scene according to claim 1, characterized in that, Also includes: The flight video is subjected to noise reduction and / or color correction.

4. The method for fusing flight video and real-world 3D scene according to claim 1, characterized in that, Based on the switching frame and its adjacent video frames, the corresponding transition frame is calculated using a video frame interpolation algorithm.

5. The method for fusing flight video and real-world 3D scene according to claim 1, characterized in that, The weights of the video frame image and the scene image are determined based on the relative positional relationship between the positions corresponding to the switching frame and the scene image and the positions corresponding to the initial transition frame.

6. A method for fusing and displaying real-scene 3D model videos, characterized in that, include: Acquire the real-world 3D model and flight video of the target area, as well as the equipment control parameters recorded when the drone collects the flight video; Set the roaming path in the real-world 3D model; The corresponding flight video is selected according to the roaming path, and the flight video and real-scene 3D scene fusion method as described in claim 1 is used to fuse each transition frame with the real-scene 3D model.

7. The method for fusion and display of real-scene 3D model video according to claim 6, characterized in that, Also includes: The selected flight video corresponds to the drone's flight trajectory, which is then linearly transformed to form the set roaming path.

Images