Ground video stream and unmanned aerial vehicle multi-view image fusion modeling method

Abstract

The invention relates to the field of live-action three-dimensional modeling, in particular to a ground video stream and unmanned aerial vehicle multi-view image fusion modeling method, which comprises the following steps of: 1, acquiring an aerial image, arranging an image control point, and acquiring a spatial three-dimensional coordinate of the image control point; step 2, processing and modeling the aerial image; 3, extracting marked space points, and exporting space coordinate data; 4, acquiring video stream data, extracting a high-resolution image, and screening image data; 5, taking the spatial point coordinate information as a reference, carrying out aerial triangulation processing on the image model, and obtaining an aerial triangulation result under an absolute reference; step 6, according to the aerial triangulation result, combining the aerial triangulation result of the aerial image with the aerial triangulation result of the video streaming image; and 7, performing model reconstruction based on the combined aerial triangulation result, and mapping textures to obtain a final fused three-dimensional model result. The advantages of the aerial image of the unmanned aerial vehicle and the ground video stream are fused, and the multi-view image data of the target scene can be comprehensively acquired.

Description

Technical Field

[0001] The present invention relates to the field of real-scene three-dimensional modeling, and in particular to a method for fusion modeling of ground video streams and multi-view images of unmanned aerial vehicles. Background Art

[0002] Using drones for aerial modeling is a hot research direction in the current 3D modeling field and is widely used in many fields such as architecture, surveying and mapping, cultural relics protection, virtual reality, etc. The traditional method of collecting field data for drone aerial modeling is mainly to collect multi-view images with a certain degree of overlap by planning routes. The higher the image overlap, the better the 3D modeling effect. The common drone shooting method is mainly divided into two steps: the drone first shoots at a fixed distance horizontally or vertically. When completing the shooting task of a flight strip, the drone moves in the direction perpendicular to the flight strip and repeats the above two steps until the shooting task is completed.

[0003] Since the shooting angle of traditional drone aerial photography methods is high in the air, the image resolution obtained for parts close to the ground, such as the bottom of buildings, is low and easily blocked, resulting in the presence of lace and wax melting in the bottom area of ​​the constructed three-dimensional model. By collecting video streams on the ground, extracting image information based on the video streams, and fusing them with images taken by drones for modeling, the accuracy of the bottom of the model can be greatly improved. Based on this, the present invention proposes a method for fusing ground video streams with multi-view images of drones for modeling. Summary of the invention

[0004] In view of the problems existing in the prior art that, in the process of traditional drone aerial photography modeling, the bottom images obtained have low resolution and are easily blocked, resulting in low precision of the bottom of the three-dimensional model and problems such as streaking and wax melting. The purpose of the present invention is to provide a method for fusing ground video streams and multi-view images taken by drones for modeling, which extracts image information through ground video streams and fuses them with multi-view images taken by drones to improve the precision of the bottom of the model.

[0005] To achieve the above object, the technical solution adopted by the present invention is: a method for fusion modeling of ground video stream and multi-view images of unmanned aerial vehicle, comprising the following steps: Step 1: Plan the UAV route, collect multi-view aerial images of the modeling area, obtain POS data and camera parameters, arrange image control points, and collect the spatial 3D coordinates of the image control points; Step 2: Perform aerial triangulation on the aerial images and build models; Step 3: extract M spatial points with significant features from the aerial image 3D model results, mark each selected spatial point on a different aerial image, and derive spatial coordinate data of the spatial point; Step 4: Obtain continuous video stream data of the modeling area, cover the spatial point data in the video stream data information, extract high-resolution images with spatial continuity, and screen the image data extracted from the video stream according to the spatial similarity of adjacent images; Step 5: Using the coordinate information of the M spatial points as a reference, perform aerial triangulation on the image model of the video stream to obtain an aerial triangulation result under an absolute reference; Step 6: Based on the aerial triangulation results, merge the aerial triangulation results of the aerial images with the aerial triangulation results of the video stream images; Step 7: Reconstruct the model based on the merged aerial triangulation results and map the texture to obtain the final fused 3D model results.

[0006] In the above-mentioned method for fusion modeling of ground video stream and multi-view images of unmanned aerial vehicles, step 2 includes: firstly automatically matching images, executing free network aerial triangulation, then introducing image control point coordinate information to execute regional network aerial triangulation, and reconstructing three-dimensional models and texture mapping based on the aerial triangulation results that meet the accuracy requirements.

[0007] The above-mentioned method of fusion modeling of ground video stream and multi-view images of UAV performs automatic image matching according to the beam normal equation, and the expression for executing free network aerial triangulation is: , The expression is based on the free network adjustment to express the perspective projection relationship between the image point and the ground point, where are the plane coordinates of the image point, are the coordinates of the principal point of the image, is the camera focal length, is the spatial coordinate of the ground point , is the coordinate of the photography center, is the rotation matrix, defined as: , The first row of elements in the rotation matrix R Represents the projection of the rotated X-axis in the original coordinate system. The second row of elements Represents the projection of the rotated Y axis in the original coordinate system. The third row of elements Represents the projection of the rotated Z axis in the original coordinate system.

[0008] The above-mentioned method for fusion modeling of ground video stream and multi-view image of unmanned aerial vehicle, the introduction of image control point coordinate information to perform regional network aerial triangulation, constraining the measurement results of the entire regional network to the coordinate system of the image control points, after the introduction of the image control points, the mathematical model of bundle adjustment is: , where V represents the residual vector of the observation value, A is the coefficient matrix, which contains the geometric relationship between the image points and the ground points, and X is the parameter to be determined, including the external orientation elements and ground point coordinates, L is the observation value vector, which includes the measurement value of the image point and the coordinates of the image control point. X is solved by the least squares adjustment to obtain the exterior orientation elements of the image and the spatial coordinates of the ground point.

[0009] The above-mentioned method of fusion modeling of ground video stream and multi-view images of drones, the spatial points include the corner points of the house and the corner points of the ground marking lines, the spatial points are selected to be no less than 4, and the images of each spatial point are selected to be no less than 3, and the spatial coordinate data of the spatial points are obtained. .

[0010] In the above-mentioned method for fusion modeling of ground video stream and multi-view images of unmanned aerial vehicles, in step 4, the screening method is: according to the spatial similarity of two adjacent images, the threshold value of the similarity of two adjacent images is set as T, and there are a total of C images to be screened and processed, and the first The total number of spatial points in an image is N. If there are n feature points in N that match the feature points of the adjacent image, the similarity between the two images is defined as: , Starting from the first image as the reference image, the subsequent images are screened and judged in sequence. If the similarity between the second image and the first image is greater than the threshold T, the images until the If the similarity of the images is less than the threshold T, they are retained and used as new reference images. They are screened and extracted again until all the images are screened and E screened images are obtained.

[0011] In the above-mentioned method for fusion modeling of ground video stream and multi-view image of unmanned aerial vehicle, step 5 includes: in the real-scene 3D modeling software ContextCapture, in the aerial triangulation calculation project of the image extracted from the video stream, M spatial points are positioned on the image by manual marking, and each spatial point is marked on at least 4 images, and the spatial 3D coordinates of the M spatial points are used as a reference to perform aerial triangulation on the video stream image.

[0012] In the above-mentioned method for fusion modeling of ground video stream and multi-view images of unmanned aerial vehicle, step 6 includes: according to the aerial triangulation results under the absolute reference obtained in step 5, the aerial triangulation results of the unmanned aerial vehicle images and the aerial triangulation results of the video stream images have a unified spatial reference reference, wherein there are F drone aerial images and E images extracted and screened from the ground video stream, and the fusion of the two totals E+F images, and the aerial triangulation results of the two are merged in the real-scene 3D modeling software ContextCapture.

[0013] The beneficial effect of the method for fusion modeling of ground video stream and multi-view images of unmanned aerial vehicles of the present invention is: the present invention combines the advantages of unmanned aerial vehicle aerial images and ground video streams, and can comprehensively obtain multi-view image data of target scenes. The images extracted from the ground video stream are fused and modeled with the multi-view images of unmanned aerial vehicles to supplement the data missing in the near-ground area and the occluded area of ​​the aerial images, and enhance the modeling ability of complex scenes. The method is efficient and highly adaptable, and provides an effective three-dimensional reconstruction solution for various scenes. By obtaining the aerial triangulation results under the absolute benchmark, the problem of spatial benchmark consistency is solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 The present invention is a flowchart of the steps for implementing the fusion modeling of ground video stream and multi-view images of unmanned aerial vehicles. DETAILED DESCRIPTION

[0015] In order to enable those skilled in the art to better understand the technical solution of the present invention, the technical solution of the present invention is described below in conjunction with specific implementation methods and drawings.

[0016] Example 1 like Figure 1 As shown, a method for fusion modeling of ground video stream and multi-view images of unmanned aerial vehicle includes the following steps.

[0017] Step 1: Aerial image acquisition and image control point measurement. According to the conditions of the modeling area, the flight route of the drone is reasonably planned, multi-angle images of the drone in the modeling area are collected, POS data and camera parameters are obtained, image control points are arranged according to the terrain conditions of the survey area, and the spatial three-dimensional coordinates of the image control points are collected using a GNSS receiver.

[0018] Step 2: Modeling of drone aerial images. Perform aerial triangulation and modeling of drone aerial images according to the aerial photogrammetry process. First, perform automatic image matching, perform free network aerial triangulation, then introduce image control point coordinate information to perform regional network aerial triangulation, and perform 3D model reconstruction and texture mapping based on the aerial triangulation results that meet the accuracy requirements.

[0019] Step 3: Extract and export the three-dimensional coordinates of the spatial feature points of the UAV aerial image model. Extract M spatial feature points with significant characteristics from the three-dimensional model results of the UAV aerial image as spatial points, such as the corner points of the house, the corner points of the ground markings, etc. The selected spatial feature points are evenly and reasonably distributed in space. The number of selected feature points is no less than 4, that is, M ≥ 4. Each selected spatial feature point is marked on different images, including at least 3 different images, and then the spatial coordinate data of these spatial feature points are exported.

[0020] Step 4: Ground video stream acquisition and image extraction. Use the method of walking with a handheld digital camera to obtain continuous video stream data in the modeling area. The video stream information should cover the M spatial feature point data extracted in step 3. Extract high-resolution images with spatial continuity based on the obtained video stream information. Since the number of images directly extracted through the video stream is large, and the images containing the same position features on the surface of the object are too dense and there is a lot of redundancy, it is necessary to screen the images. The image data extracted from the video stream is screened according to the spatial similarity of two adjacent images to obtain E screened images.

[0021] Step 5: Aerial triangulation and benchmark conversion of ground video images. The UAV aerial images are fused with the images extracted from the screened ground video stream to build a model. The spatial feature point coordinate information extracted from the UAV aerial image model in step 3 is used as the benchmark to perform aerial triangulation on the image model of the ground video stream to obtain the aerial triangulation results under the absolute benchmark.

[0022] Step 6: Based on the aerial triangulation results under the absolute benchmark obtained in step 5, merge the aerial triangulation results of the drone with the aerial triangulation results of the ground video stream image.

[0023] Step 7: Fusion modeling, reconstruct the model based on the merged aerial triangulation results, and map the texture to obtain the final fused 3D model results.

[0024] Example 2 like Figure 1 As shown, a method for fusion modeling of ground video stream and multi-view images of unmanned aerial vehicle includes the following steps.

[0025] 1. UAV aerial image acquisition and image control point measurement.

[0026] According to the size, complexity, and expected resolution of the modeling area, the flight route of the drone is reasonably planned, and F drone images with high overlap and multiple perspectives of the modeling area are collected, and POS data and camera parameters are obtained at the same time. Image control points are arranged according to the terrain conditions of the survey area, and the spatial three-dimensional coordinates of the image control points are collected using a GNSS receiver. Assume that the acquired image set is: ①.

[0027] in, Photographed by drone images, and F is the total number of images.

[0028] 2. Modeling of drone aerial images.

[0029] According to the aerial photogrammetry process, the aerial triangulation of the drone aerial images is processed and modeled. First, the images are automatically matched and free-net aerial triangulation is performed. According to the beam method equation, it is expressed as follows: ②.

[0030] Formula (2) represents the perspective projection relationship between image points and ground points, which is the basis of free network adjustment. are the plane coordinates of the image point, are the coordinates of the principal point of the image, is the camera focal length, is the spatial coordinate of the ground point, are the coordinates of the photography center, is the rotation matrix, defined as: ③.

[0031] The first row of elements in the rotation matrix R Represents the projection of the rotated X-axis in the original coordinate system. The second row of elements Represents the projection of the rotated Y axis in the original coordinate system. The third row of elements Represents the projection of the rotated Z axis in the original coordinate system.

[0032] In formula (3), the rotation matrix R is the rotation angle of the exterior orientation element The function of Represents the rotation angle around the X axis, the rotation matrix is: .

[0033] Represents the rotation angle around the Y axis, and the rotation moment is: .

[0034] Represents the rotation angle around the Z axis, the rotation matrix is: .

[0035] After the free network aerial triangulation meets the requirements, the image control point coordinate information is introduced to perform regional network aerial triangulation. The goal at this time is to constrain the measurement results of the entire regional network to the coordinate system of the image control points to achieve the absolute orientation of the exterior orientation elements and ground point coordinates. After the image control points are introduced, the mathematical model of the bundle adjustment is: ④.

[0036] In formula (4), V represents the residual vector of the observation value (image point coordinates), A is the coefficient matrix, which contains the geometric relationship between the image point and the ground point, and X is the parameter to be determined, including the exterior orientation element and ground point coordinates, L is the observation value vector, including the measurement value of the image point and the coordinates of the image control point. By solving X through the least squares adjustment method, the exterior orientation elements of the image and the spatial coordinates of the ground point can be obtained. Finally, the three-dimensional model is reconstructed and textured based on the aerial triangulation results that meet the accuracy requirements.

[0037] 3. Extract and export the three-dimensional coordinates of the spatial feature points of the UAV aerial image model.

[0038] M spatial feature points with significant characteristics are extracted from the three-dimensional model results of drone aerial images, such as the corner points of houses and the corner points of ground markings. The selected feature points are no less than 4, that is, M≥4, and they are evenly and reasonably distributed in space. Each selected spatial feature point is marked on at least 3 different images, and then the spatial coordinate data of these spatial feature points are exported.

[0039] ⑤.

[0040] 4. Ground video stream acquisition and image extraction.

[0041] The method of walking with a handheld digital camera is used to obtain continuous video stream data of the modeling area, and the video stream information should cover the M spatial feature point data extracted in (3) above; based on the obtained video stream information, high-resolution images with spatial continuity are extracted. Since the number of images directly extracted through the video stream is large and the images containing the same position features of the object surface are too dense, there is a lot of redundancy, so the images need to be screened.

[0042] The screening method is mainly based on the spatial similarity of two adjacent images. The threshold of the similarity of two adjacent images is set as T. There are a total of C images to be screened. The total number of spatial feature points of an image is N. If n spatial feature points in N match the feature points of the adjacent image, the similarity between the two images is defined as: ⑥.

[0043] Starting from the first image as the reference image, the subsequent images are screened and judged in sequence. If the similarity between the second image and the first image is greater than the threshold T, the images until the If the similarity of the images is less than the threshold T, they will be retained and used as new reference images. They will be screened and extracted again until all the images are screened. Suppose a total of E photos are screened.

[0044] 5. Aerial triangulation and reference conversion of ground video images.

[0045] The fusion modeling of UAV images and images extracted from ground video streams needs to solve the problem of spatial reference consistency. Since UAV aerial photography can obtain adjustment results in an absolute coordinate system through image control points, and ground video streams lack spatial scale reference information and positioning information, if the images are directly adjusted after extraction, the result obtained is a relative orientation adjustment result. Therefore, the spatial feature point coordinate information extracted from the UAV aerial photography model in step (4) is used as the reference, and the image model of the video stream is processed by aerial triangulation to obtain the aerial triangulation results under the absolute reference.

[0046] The specific method is as follows: In the real-life 3D modeling software ContextCapture, in the aerial triangulation project of images extracted from video streams, M feature points are located on the images by manual marking. Each feature point is marked on at least 4 images, and the spatial 3D coordinates of the M feature points are used as a reference to perform aerial triangulation on the video stream images.

[0047] 6. The aerial triangulation results of the UAV are merged with the aerial triangulation results of the ground video stream imagery.

[0048] According to the above process, the aerial triangulation results of the UAV images and the aerial triangulation results of the video stream images have a unified spatial reference benchmark. There are F drone aerial images and E images extracted and screened from the ground video stream. The two are fused together for a total of E+F images. In the real-scene 3D modeling software ContextCapture, the aerial triangulation results of the two are merged.

[0049] 7. Fusion modeling.

[0050] Based on the merged aerial triangulation results, image dense matching and model reconstruction are performed, and finally the texture is mapped to obtain the final fused 3D model result.

[0051] By merging the aerial triangulation results of drones and ground video stream images in the ContextCapturer software, computational modeling is performed according to the photogrammetry process, that is, dense image matching, TIN triangulation network construction, and texture mapping are performed in sequence to finally form a three-dimensional model.

[0052] The above embodiments are only for illustrating the inventive concept and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made based on the essence of the content of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for fusion modeling of ground video stream and multi-view images of unmanned aerial vehicles, characterized in that: The following steps are involved: Step 1: Plan the UAV route, collect multi-view aerial images of the modeling area, obtain POS data and camera parameters, arrange image control points, and collect the spatial 3D coordinates of the image control points; Step 2: Perform aerial triangulation on the aerial images and build models; Step 3: extract M spatial points with significant features from the aerial image 3D model results, mark each selected spatial point on a different aerial image, and derive spatial coordinate data of the spatial point; Step 4: Obtain continuous video stream data of the modeling area, cover the spatial point data in the video stream data information, extract high-resolution images with spatial continuity, and screen the image data extracted from the video stream according to the spatial similarity of adjacent images; Step 5: Using the coordinate information of the M spatial points as a reference, perform aerial triangulation on the image model of the video stream to obtain an aerial triangulation result under an absolute reference; Step 6: Based on the aerial triangulation results, merge the aerial triangulation results of the aerial images with the aerial triangulation results of the video stream images; Step 7: Reconstruct the model based on the merged aerial triangulation results and map the texture to obtain the final fused 3D model results.

2. The method for fusion modeling of ground video stream and multi-view images of unmanned aerial vehicle according to claim 1 is characterized in that: The step 2 includes: firstly automatically matching the images, executing free-net aerial triangulation, then introducing the coordinate information of the image control points to execute regional-net aerial triangulation, and reconstructing the three-dimensional model and texture mapping based on the aerial triangulation results that meet the accuracy requirements.

3. The method for fusion modeling of ground video stream and multi-view images of unmanned aerial vehicle according to claim 2 is characterized in that: According to the beam method equation, the image is automatically matched and the expression for free network aerial triangulation is: ， The expression is based on the free network adjustment to express the perspective projection relationship between the image point and the ground point, where are the plane coordinates of the image point, are the coordinates of the principal point of the image, is the focal length of the camera, is the spatial coordinate of the ground point , is the coordinate of the photography center, is the rotation matrix, defined as: ， The first row of elements in the rotation matrix R Represents the projection of the rotated X-axis in the original coordinate system. The second row of elements Represents the projection of the rotated Y axis in the original coordinate system. The third row of elements Represents the projection of the rotated Z axis in the original coordinate system.

4. The method for fusion modeling of ground video stream and multi-view images of unmanned aerial vehicle according to claim 3 is characterized in that: The image control point coordinate information is introduced to perform regional network aerial triangulation, and the measurement results of the entire regional network are constrained to the coordinate system of the image control points. After the image control points are introduced, the mathematical model of the bundle adjustment is: , where V represents the residual vector of the observation value, A is the coefficient matrix, which contains the geometric relationship between the image points and the ground points, and X is the parameter to be determined, including the external orientation elements and ground point coordinates, L is the observation value vector, which includes the measurement value of the image point and the coordinates of the image control point. X is solved by the least squares adjustment to obtain the exterior orientation elements of the image and the spatial coordinates of the ground point.

5. The method for fusion modeling of ground video stream and multi-view images of unmanned aerial vehicle according to claim 1 is characterized in that: The spatial points include the corner points of the house and the corner points of the ground markings. No less than 4 spatial points are selected, and no less than 3 images of each spatial point are selected to obtain the spatial coordinate data of the spatial point. 。 6. The method for fusion modeling of ground video stream and multi-view images of unmanned aerial vehicle according to claim 1 is characterized in that: In step 4, the screening method is: according to the spatial similarity of two adjacent images, the threshold value of the similarity of two adjacent images is set as T, and there are a total of C images to be screened and processed. The total number of spatial points in an image is N. If there are n feature points in N that match the feature points of the adjacent image, the similarity between the two images is defined as: ， Starting from the first image as the reference image, the subsequent images are screened and judged in sequence. If the similarity between the second image and the first image is greater than the threshold T, the images until the If the similarity of the images is less than the threshold T, they are retained and used as new reference images. They are screened and extracted again until all the images are screened and E screened images are obtained.

7. The method for fusion modeling of ground video stream and multi-view images of unmanned aerial vehicle according to claim 1 is characterized in that: The step 5 includes: in the real-scene 3D modeling software ContextCapture, in the aerial triangulation project of the image extracted from the video stream, M spatial points are positioned on the image by manual marking, each spatial point is marked on at least 4 images, and the spatial 3D coordinates of the M spatial points are used as a reference to perform aerial triangulation on the video stream image.

8. The method for fusion modeling of ground video stream and multi-view images of unmanned aerial vehicle according to claim 1 is characterized in that: The step 6 includes: according to the aerial triangulation results under the absolute reference obtained in step 5, the aerial triangulation results of the drone image and the aerial triangulation results of the video stream image have a unified spatial reference reference, wherein F drone aerial images and E images extracted and screened from the ground video stream are fused together to form a total of E+F images, and the aerial triangulation results of the two are merged in the real-scene 3D modeling software ContextCapture.

Images