A method and system for generating three-dimensional city scenes based on video images

Abstract

The present invention relates to the field of image processing technology, and discloses a method and system for generating a three-dimensional urban scene based on a video image, the method comprising: determining a point feature histogram of each model in a scene model library corresponding to a target video image; extracting feature points in the target video image, constructing a scene dense point cloud from the target video image for clustering processing, and calculating the point feature histogram of each dense point cloud set; based on the matching relationship between the point feature histogram of each dense point cloud set and the point feature histogram of each model in the scene model library, generating a three-dimensional model of the target video image. The present invention improves parallel processing efficiency, three-dimensional scene generation efficiency, and three-dimensional scene generation quality by classifying point cloud data that originally needs to be processed as a whole, using a pre-processed scene model library for model feature matching, and using the matching results for scene generation, so that the generated three-dimensional scene has interactive characteristics.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method and system for generating a three-dimensional city scene based on video images. Background Art

[0002] At present, three-dimensional scene restoration based on video images usually adopts two methods: scene restoration technology based on oblique photography and 3D Gaussian sputtering scene restoration based on neural vector field.

[0003] Among them, scene restoration technology based on oblique photography is a method of obtaining images of an object by taking them from different angles and using these images to generate a three-dimensional model. This technology is particularly suitable for large-scale urban modeling and geographic information system (GIS) applications. Its implementation steps include: (1) Image acquisition: Use drones or other flying platforms to take high-resolution images of the target area from multiple angles (including vertical and oblique). It is recommended to cover every part of the target area and ensure that there are enough overlapping images (generally, it is recommended that the image overlap is more than 60%). (2) Image preprocessing: Preprocess the collected images, including image denoising, color correction and distortion correction, to ensure the quality and consistency of the image. (3) Feature point extraction and matching: Use computer vision algorithms (such as SIFT, SURF or ORB) to extract feature points in the image, and find the correspondence between overlapping images through feature point matching algorithms (such as BFMatcher or FLANN). (4) Structure from motion: Use structure from motion (SfM) technology to calculate the external parameters (position and posture) of the camera through matched feature points to generate a sparse point cloud. (5) Multi-view stereo (MVS) reconstruction: Use Multi-View Stereo (MVS) technology to generate dense point clouds from sparse point clouds and camera parameters. (6) 3D model generation: Generate a 3D mesh model based on the dense point cloud and perform texture mapping to apply the texture information of the image to the 3D mesh to generate a realistic 3D model. (7) Model optimization: Optimize the generated 3D model, including simplifying the mesh, removing noise, and filling holes to improve the quality and usability of the model.

[0004] Among them, the 3D Gaussian sputtering scene restoration method based on neural vector field is a method that uses deep learning and computer graphics technology to learn and generate three-dimensional scenes from two-dimensional images. This technology can efficiently capture and restore the details and complex lighting effects in the scene, and is suitable for fields such as virtual reality, film and television special effects, and game development. Its implementation steps include: (1) Data acquisition: Take multiple high-resolution images of the target scene from multiple angles. The images should cover various parts of the scene and have different perspectives and lighting conditions. (2) Data preprocessing: Preprocess the collected images, including denoising, color correction, and distortion correction to ensure image quality and consistency. (3) Neural network training: Construct and train a neural network that can learn the radiance field of the scene from the input image. The loss function used in the training process should include image reconstruction error and regularization terms to ensure the quality and stability of the generated results. (4) Radiance field representation: Use the trained neural network to generate the radiance field representation of the scene. The radiance field representation includes the color and density information of each point in the scene, which is used to describe the lighting and geometric characteristics of the scene. (5) Gaussian sputtering rendering: Use Gaussian sputtering technology to convert the radiation field representation into three-dimensional points. Each point is rendered using Gaussian distribution according to its color and density information to generate a high-quality three-dimensional image. (6) Image generation and perspective adjustment: Generate high-quality scene images from different perspectives according to user needs. Since the radiation field representation is perspective-independent, the perspective can be freely adjusted to generate images at any angle.

[0005] However, the above existing methods for restoring three-dimensional scenes based on video images have the following problems: (1) The scenes generated by the existing technology only have visual effects, no structured information, and do not support internal viewing and other complex applications. (2) The scenes generated by the existing technology are all non-repetitive models, and each model needs to be stored separately, which consumes a lot of storage space and loading space, and also consumes a lot of rendering performance. (3) The scenes generated by the existing technology can only maintain good rendering effects under specific viewing angles and cannot meet the viewing and interaction requirements of complex viewing angles.

[0006] Therefore, how to improve the processing efficiency, generation efficiency, generation quality and interactivity of three-dimensional scene restoration based on video images is a technical problem that needs to be solved urgently. Summary of the invention

[0007] The main purpose of the present invention is to provide a method and system for generating a three-dimensional urban scene based on video images to solve the above technical problems.

[0008] To achieve the above object, the present invention provides a method for generating a three-dimensional urban scene based on a video image, comprising:

[0009] Acquire a target video image, match the target video image with a scene model library, and determine a point feature histogram of each model in the scene model library;

[0010] Extract feature points in the target video image and construct a dense point cloud of the scene from the target video image;

[0011] Performing clustering processing on the dense point cloud of the scene to obtain a plurality of dense point cloud sets, and calculating a point feature histogram of each dense point cloud set;

[0012] Based on the matching relationship between the point feature histogram of each dense point cloud set and the point feature histogram of each model in the scene model library, a three-dimensional model of the target video image is generated.

[0013] Optionally, the step of obtaining a target video image and matching the target video image with a scene model library specifically includes:

[0014] Acquire a target video image, and match an access interface of a scene model library corresponding to the target video image in an access interface list based on a video type identifier attached to the target video image;

[0015] The access interface is used to access a scene model library corresponding to the target video avatar, and a plurality of scene models in the scene model library are extracted.

[0016] Optionally, the step of determining a point feature histogram of each model in the scene model library specifically includes:

[0017] Evenly scatter points on the surfaces of several scene models extracted from the scene model library, and simulate dense point clouds based on the evenly scatter point results of each scene model;

[0018] The dense point cloud set is used to calculate the point feature histogram of each scene model.

[0019] Optionally, extracting feature points from the target video image and constructing a dense point cloud of the scene from the target video image specifically includes:

[0020] Extracting feature points from the target video image, and matching the feature points between the overlapping target video images to obtain a feature point matching relationship;

[0021] Based on the feature points in the feature point matching relationship, the external parameters of the camera are calculated and a sparse point cloud is generated. The sparse point cloud and the external parameters of the camera are used to generate a dense point cloud of the scene.

[0022] Optionally, extracting feature points from the target video image and matching the feature points between the overlapping target video images to obtain a feature point matching relationship step specifically includes:

[0023] The ORB algorithm is used to extract feature points in the target video image;

[0024] The FLANN algorithm is used to match feature points, find the corresponding relationship between feature points of overlapping target video images, and obtain the matching relationship of feature points.

[0025] Optionally, based on the feature points in the feature point matching relationship, the step of calculating the external parameters of the camera and generating a sparse point cloud specifically includes: using structure from motion technology, through the feature points in the feature point matching relationship, calculating the external parameters of the camera and generating a sparse point cloud.

[0026] Optionally, the step of generating a dense point cloud of the scene using the sparse point cloud and the external parameters of the camera specifically includes: using multi-view stereo technology to generate a dense point cloud using the sparse point cloud and the external parameters of the camera.

[0027] Optionally, the dense point cloud of the scene is clustered to obtain several dense point cloud sets, and the point feature histogram of each dense point cloud set is calculated, specifically including: using the DBSCAN algorithm to cluster and segment the dense point cloud of the scene to obtain several dense point cloud sets; calculating the point feature histogram of each dense point cloud set after segmentation.

[0028] Optionally, the step of generating a three-dimensional model of the target video image based on the matching relationship between the point feature histogram of each dense point cloud and the point feature histogram of each model in the scene model library specifically includes:

[0029] Traverse the point feature histogram of each dense point cloud set in all dense point cloud sets, compare it with the point feature histogram of each model in the scene model library, and find the matching item with the smallest Euclidean distance;

[0030] The spatial position of the dense point cloud corresponding to the matching item with the smallest Euclidean distance is calculated, and a model corresponding to the matching item is generated at the spatial position, so as to finally obtain a three-dimensional model of the target video image.

[0031] In addition, in order to achieve the above purpose, a three-dimensional city scene generation system based on video images includes:

[0032] A matching module, used to obtain a target video image, match the scene model library of the target video image, and determine a point feature histogram of each model in the scene model library;

[0033] A construction module, used for extracting feature points in a target video image and constructing a dense point cloud of a scene from the target video image;

[0034] A calculation module, used for performing clustering processing on the dense point cloud of the scene to obtain a plurality of dense point cloud sets, and calculating a point feature histogram of each dense point cloud set;

[0035] A generation module is used to generate a three-dimensional model of a target video image based on a matching relationship between a point feature histogram of each dense point cloud set and a point feature histogram of each model in the scene model library.

[0036] The beneficial effects of the present invention are as follows: a method and system for generating a three-dimensional urban scene based on a video image is proposed, the method comprising obtaining a target video image, matching a scene model library of the target video image, and determining a point feature histogram of each model in the scene model library; extracting feature points in the target video image, and constructing a dense point cloud of the scene from the target video image; clustering the dense point cloud of the scene to obtain a number of dense point cloud sets, and calculating a point feature histogram of each dense point cloud set; based on the matching relationship between the point feature histogram of each dense point cloud set and the point feature histogram of each model in the scene model library, a three-dimensional model of the target video image is generated. The present invention utilizes a dense point cloud clustering and segmentation method to classify and process point cloud data that originally needs to be processed as a whole, thereby improving parallel processing efficiency; at the same time, the traditional method of generating a three-dimensional model using point cloud data or using Gaussian sputtering rendering is abandoned, and a pre-processed scene model library is used for model feature matching, and the matching result is used for scene generation, thereby improving the efficiency of three-dimensional scene generation, improving the quality of three-dimensional scene generation, and making the generated three-dimensional scene interactive. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 Schematic diagram of the process of generating a three-dimensional city scene based on a video image in an embodiment;

[0038] Figure 2 Schematic diagram of the structure of a three-dimensional city scene generation system based on video images in an embodiment.

[0039] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0041] The embodiment of the present invention provides a method for generating a three-dimensional urban scene based on a video image. Figure 1 , Figure 1 Schematic diagram of the process of generating a three-dimensional city scene based on video images in an embodiment.

[0042] In this embodiment, the method for generating a three-dimensional city scene based on a video image includes the following steps:

[0043] S1: Acquire a target video image, match the scene model library of the target video image, and determine a point feature histogram of each model in the scene model library;

[0044] S2: extract feature points from the target video image and construct a dense point cloud of the scene from the target video image;

[0045] S3: performing clustering processing on the dense point cloud of the scene to obtain a plurality of dense point cloud sets, and calculating a point feature histogram of each dense point cloud set;

[0046] S4: Generate a three-dimensional model of the target video image based on the matching relationship between the point feature histogram of each dense point cloud and the point feature histogram of each model in the scene model library.

[0047] It should be noted that the existing three-dimensional scene restoration methods based on video images have the following problems: (1) The scenes generated by the existing technology only have visual effects, no structured information, and do not support internal viewing and other complex applications. (2) The scenes generated by the existing technology are all non-repetitive models, and each model needs to be stored separately, which consumes a lot of storage space and loading space, and also consumes a lot of rendering performance. (3) The scenes generated by the existing technology can only maintain good rendering effects under specific viewing angles and cannot meet the viewing and interaction requirements of complex viewing angles. In order to solve the above problems, this embodiment uses a dense point cloud clustering and segmentation method to classify and process the point cloud data that originally needs to be processed as a whole, thereby improving the parallel processing efficiency; at the same time, it abandons the traditional method of using point cloud data to generate a three-dimensional model or using Gaussian sputtering rendering, uses a pre-processed scene model library to match model features, and uses the matching results to generate scenes, thereby improving the efficiency of three-dimensional scene generation, improving the quality of three-dimensional scene generation, and making the generated three-dimensional scene interactive.

[0048] In order to explain the present application more clearly, a specific example of three-dimensional scene restoration based on video images of the present application is provided below.

[0049] In this specific example, a video image is obtained and combined with the collected video image to prepare a scene model library in advance. A dense point cloud of the scene is constructed from the video image using feature point extraction and matching algorithms, structure from motion algorithms, and multi-view stereo reconstruction technology. The dense point cloud is clustered to obtain multiple dense point cloud sets. For the dense point cloud set, its point feature histogram is constructed. For each dense point cloud set, the obtained point feature histogram is matched with the point feature histograms of all models in the pre-built model library, and the matching result model is generated to the corresponding spatial position. The specific steps are as follows:

[0050] (1) Scene model library preparation: With reference to video images, prepare several scene models with similar shapes in advance, and evenly scatter points on the surfaces of all scene models to simulate dense point clouds. Use point feature histograms to construct the eigenvalues ​​of the models.

[0051] (2) Feature point extraction and matching: The ORB (Oriented FAST and Rotated BRIEF) algorithm is used to extract feature points in the image, and the FLANN (Fast Library for Approximate Nearest Neighbors) algorithm is used to find the correspondence between overlapping images.

[0052] (2) Structure from Motion: Using the Structure from Motion (SfM) technique, the camera’s external parameters (position and attitude) are calculated through matching feature points, and a sparse point cloud is generated. The rule call is generated programmatically.

[0053] (3) Multi-view stereo (MVS) reconstruction: Multi-view stereo (MVS) technology is used to generate a dense point cloud through sparse point cloud and camera parameters.

[0054] (4) Dense point cloud segmentation: Use the DBSCAN (Density-Based Spatial Clustering of Applications with Noise) algorithm for clustering segmentation to obtain multiple dense point cloud sets.

[0055] (5) Dense point cloud feature calculation: Calculate the point feature histogram of all dense point cloud sets after segmentation.

[0056] (6) Model library feature matching: Traverse the point feature histograms of all dense point clouds, compare them with the point feature histograms of the pre-prepared scene model library, and find the matching item with the smallest Euclidean distance.

[0057] (7) Scene generation: Calculate the spatial position of the dense point cloud and generate the matched 3D model at the corresponding spatial position. It should be noted that the spatial position of the dense point cloud can be calculated by taking the mean of the spatial dense point cloud coordinates. In the actual process, some offset points will be removed before taking the mean of the spatial coordinates.

[0058] In this embodiment, a method for generating a three-dimensional urban scene based on video images is provided, which innovatively converts the scene model library into point cloud information, extracts scene model feature data through point feature histograms, and uses the spatial point cloud restored by the image algorithm to construct a dense point cloud, and then extracts the point feature histogram for model matching. Compared with the prior art, it has at least one of the following advantages: using the dense point cloud clustering and segmentation method, the point cloud data that originally needs to be processed as a whole is classified and processed, thereby improving the parallel processing efficiency. Abandoning the traditional method of using point cloud data to generate a three-dimensional model or using Gaussian sputtering rendering, the pre-processed scene model library is used for model feature matching, and the matching results are used for scene generation, which improves the efficiency of three-dimensional scene generation, improves the quality of three-dimensional scene generation, and makes the generated three-dimensional scene interactive.

[0059] Reference Figure 2 , Figure 2 Schematic diagram of the structure of a three-dimensional city scene generation system based on video images in an embodiment. The three-dimensional city scene generation system based on video images proposed in an embodiment of the present invention includes:

[0060] A matching module 10 is used to obtain a target video image, match the scene model library of the target video image, and determine a point feature histogram of each model in the scene model library;

[0061] A construction module 20 is used to extract feature points in the target video image and construct a scene dense point cloud from the target video image;

[0062] A calculation module 30 is used to perform clustering processing on the dense point cloud of the scene to obtain a plurality of dense point cloud sets, and calculate a point feature histogram of each dense point cloud set;

[0063] The generating module 40 is used to generate a three-dimensional model of the target video image based on the matching relationship between the point feature histogram of each dense point cloud and the point feature histogram of each model in the scene model library.

[0064] Other embodiments or specific implementations of the three-dimensional city scene generation system based on video images of the present invention can refer to the above-mentioned method embodiments, which will not be described in detail here.

[0065] It is understood that, in the description of this specification, the description with reference to the terms "one embodiment", "another embodiment", "other embodiments", or "first to Nth embodiments" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0066] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0067] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for generating a three-dimensional urban scene based on video images, characterized in that: include: Acquire a target video image, match the target video image with a scene model library, and determine a point feature histogram of each model in the scene model library; Specifically, the method includes: obtaining a target video image, and matching an access interface of a scene model library corresponding to the target video image in an access interface list based on a video type identifier attached to the target video image; using the access interface to access a scene model library corresponding to a target video avatar, and extracting a plurality of scene models from the scene model library; uniformly scattering points on the surfaces of the plurality of scene models extracted from the scene model library, and simulating a dense point cloud according to the uniform point scattering results of each scene model; and using the dense point cloud to calculate a point feature histogram of each scene model; Extract feature points in the target video image and construct a dense point cloud of the scene from the target video image; Performing clustering processing on the dense point cloud of the scene to obtain a plurality of dense point cloud sets, and calculating a point feature histogram of each dense point cloud set; Based on the matching relationship between the point feature histogram of each dense point cloud set and the point feature histogram of each model in the scene model library, a three-dimensional model of the target video image is generated; specifically, the method includes: traversing the point feature histogram of each dense point cloud set in all dense point cloud sets, comparing it with the point feature histogram of each model in the scene model library, and finding the matching item with the smallest Euclidean distance; calculating the spatial position of the dense point cloud set corresponding to the matching item with the smallest Euclidean distance, generating a model corresponding to the matching item at the spatial position, and finally obtaining the three-dimensional model of the target video image.

2. The method for generating a three-dimensional urban scene based on video images according to claim 1, characterized in that: The steps of extracting feature points from the target video image and constructing a dense point cloud of the scene from the target video image include: Extracting feature points from the target video image, and matching the feature points between the overlapping target video images to obtain a feature point matching relationship; Based on the feature points in the feature point matching relationship, the external parameters of the camera are calculated and a sparse point cloud is generated. The sparse point cloud and the external parameters of the camera are used to generate a dense point cloud of the scene.

3. The method for generating a three-dimensional urban scene based on video images according to claim 2, characterized in that: The steps of extracting feature points from the target video image and matching the feature points between the overlapping target video images to obtain a feature point matching relationship specifically include: The ORB algorithm is used to extract feature points in the target video image; The FLANN algorithm is used to match feature points, find the corresponding relationship between feature points of overlapping target video images, and obtain the matching relationship of feature points.

4. The method for generating a three-dimensional urban scene based on video images according to claim 3, characterized in that: Based on the feature points in the feature point matching relationship, the step of calculating the external parameters of the camera and generating a sparse point cloud specifically includes: using the structure from motion technology, through the feature points in the feature point matching relationship, calculating the external parameters of the camera and generating a sparse point cloud.

5. The method for generating a three-dimensional urban scene based on video images according to claim 4, characterized in that: The step of generating a dense point cloud of the scene by using the sparse point cloud and the external parameters of the camera specifically includes: using multi-view stereo technology to generate a dense point cloud through the sparse point cloud and the external parameters of the camera.

6. The method for generating a three-dimensional urban scene based on video images according to claim 1, characterized in that: The dense point cloud of the scene is clustered to obtain a plurality of dense point cloud sets, and a point feature histogram of each dense point cloud set is calculated, specifically comprising: using a DBSCAN algorithm to cluster and segment the dense point cloud of the scene to obtain a plurality of dense point cloud sets; and calculating a point feature histogram of each dense point cloud set after segmentation.

7. A three-dimensional city scene generation system based on video images, characterized in that: include: A matching module, used to obtain a target video image, match the scene model library of the target video image, and determine a point feature histogram of each model in the scene model library; Specifically, the method includes: obtaining a target video image, and matching an access interface of a scene model library corresponding to the target video image in an access interface list based on a video type identifier attached to the target video image; using the access interface to access a scene model library corresponding to a target video avatar, and extracting a plurality of scene models from the scene model library; uniformly scattering points on the surfaces of the plurality of scene models extracted from the scene model library, and simulating a dense point cloud according to the uniform point scattering results of each scene model; and using the dense point cloud to calculate a point feature histogram of each scene model; A construction module, used for extracting feature points in a target video image and constructing a dense point cloud of a scene from the target video image; A calculation module, used for performing clustering processing on the dense point cloud of the scene to obtain a plurality of dense point cloud sets, and calculating a point feature histogram of each dense point cloud set; A generation module is used to generate a three-dimensional model of a target video image based on the matching relationship between the point feature histogram of each dense point cloud set and the point feature histogram of each model in the scene model library; specifically comprising: traversing the point feature histogram of each dense point cloud set in all dense point cloud sets, comparing it with the point feature histogram of each model in the scene model library, and finding the matching item with the smallest Euclidean distance; calculating the spatial position of the dense point cloud set corresponding to the matching item with the smallest Euclidean distance, generating a model corresponding to the matching item at the spatial position, and finally obtaining the three-dimensional model of the target video image.

Images