City model generation method and device, electronic equipment and storage medium

Abstract

The application discloses a city model generation method and device, electronic equipment and a storage medium, comprising: acquiring city grid model data, and determining the main street corresponding to the city grid model data; acquiring the key view sequence corresponding to the main street, wherein the key view sequence comprises a plurality of virtual camera key views, the plurality of virtual camera key views are obtained by moving and shooting the virtual camera along the main street, and the distance interval of any two virtual camera key views in the plurality of virtual camera key views is the same; based on the key view sequence, a plurality of visual images are generated; for each visual image, each pixel point of the visual image is converted into a particle point in a world coordinate system to obtain a sparse particle field; the sparse particle field is densified to obtain a dense particle field; and based on the dense particle field, a city particle model is obtained. The application can improve the position accuracy of the particle, and accurately determine the geometric information of the building facade in the model.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computers, and particularly relates to a city model generation method and device, electronic equipment and a storage medium. BACKGROUND

[0002] In the prior art, a 3D city scene model is often generated based on a two-dimensional city bird's eye view (BEV). This method relies on the analysis of these two-dimensional images to infer the three-dimensional structure. This means that the generated model is based on the information obtained from the top view, and due to the lack of detailed information about the side of the building, the system can only "guess" the appearance of the side of the building according to the vertical information provided by the height map.

[0003] This can cause the facade of the building in the generated model to appear to be "floating" outside the structure to which it should be attached, because there is not enough information to accurately determine its exact position and shape. Therefore, there is a problem in the prior art that the geometric information of the facade cannot be accurately determined. SUMMARY

[0004] The embodiments of the present application provide a city model generation method and device, electronic equipment and a storage medium, which can improve the problem that the geometric information of the facade cannot be accurately determined in the prior art.

[0005] The embodiments of the present application provide a city model generation method, which comprises:

[0006] Obtain city grid model data and determine the main street corresponding to the city grid model data; obtain the key view sequence corresponding to the main street, wherein the key view sequence comprises a plurality of virtual camera key views, the plurality of virtual camera key views are obtained by a virtual camera moving along the main street to take pictures, and the distance interval of any two virtual camera key views in the plurality of virtual camera key views is the same; generate a plurality of visual images based on the key view sequence; for each visual image, convert each pixel point of the visual image into a particle point in a world coordinate system to obtain a sparse particle field; perform densification processing on the sparse particle field to obtain a dense particle field; and obtain a city particle model based on the dense particle field.

[0007] The embodiments of the present application provide a city model generation device, which comprises:

[0008] A data acquisition unit is configured to obtain city grid model data and determine the main street corresponding to the city grid model data.

[0009] A sequence acquisition unit is used to acquire a key view sequence corresponding to the main street, wherein the key view sequence includes multiple virtual camera key views, which are obtained by moving a virtual camera along the main street and capturing images, and the distance interval between any two virtual camera key views is the same.

[0010] A visual image generation unit is used to generate multiple visual images based on the key view sequence;

[0011] The sparse particle field unit is used to convert each pixel of the visual image into a particle point in the world coordinate system for each visual image, thereby obtaining a sparse particle field.

[0012] A dense particle field unit is used to densify the sparse particle field to obtain a dense particle field.

[0013] The particle model unit is used to obtain a city particle model based on the dense particle field.

[0014] In one embodiment, the sequence acquisition unit includes:

[0015] The view capture subunit is used to control the virtual camera to move along the main street, and to capture a key view of the virtual camera every time it moves a preset fixed distance;

[0016] The view sequence subunit is used to arrange multiple key views of the virtual camera in the order they were captured to obtain the key view sequence.

[0017] In one implementation, each of the virtual camera key views includes a depth map, a semantic map, and a normal map; correspondingly, the visual image generation unit includes:

[0018] The key view acquisition subunit is used to acquire the virtual camera key view at the end of the key view sequence.

[0019] The end image subunit is used to generate a corresponding end visual image based on the depth map, semantic map, and normal map included in the virtual camera key view at the end of the sequence.

[0020] Sub-units are generated sequentially to generate visual images corresponding to the other virtual camera key views, starting from the last visual image and following the reverse order of the key view sequence, thereby obtaining the multiple visual images.

[0021] In one implementation, the key view sequence includes m virtual camera key views, where m is a positive integer greater than 1; correspondingly, sub-units are generated sequentially, specifically for:

[0022] For the i-th virtual camera key view in the key view sequence, obtain the (i + 1)-th visual image corresponding to the (i + 1)-th virtual camera key view; where i < m and i is a positive integer;

[0023] Perform warping on the (i + 1)-th visual image to obtain the (i + 1)-th warping result;

[0024] Generate the i-th visual image based on the (i + 1)-th warping result, the depth map, semantic map, and normal map included in the i-th virtual camera key view.

[0025] In one embodiment, the sparse particle field unit includes:

[0026] The particle point calculation sub-unit is used to calculate the particle point of each pixel point in the world coordinate system according to the coordinate value of the pixel point in the visual image to which it belongs, the depth value of the pixel point, the camera intrinsic matrix, and the camera pose transformation matrix;

[0027] The particle point composition sub-unit is used to form the sparse particle field with multiple particle points.

[0028] In one embodiment, the dense particle field unit includes:

[0029] The intermediate view filling sub-unit is used to fill at least one intermediate view between any two adjacent virtual camera key views in the multiple virtual camera key views. For each intermediate view, the first distance of the intermediate view is equal to the second distance. The first distance refers to the distance between the intermediate view and its previous view, and the second distance refers to the distance between the intermediate view and its subsequent view;

[0030] The intermediate view rendering sub-unit is used to render multiple intermediate views based on the sparse particle field to obtain corresponding intermediate visual images;

[0031] The pixel point conversion sub-unit is used to convert each pixel point of each intermediate visual image into a particle point in the world coordinate system to obtain intermediate image particle points;

[0032] The dense particle field sub-unit is used to fill the sparse particle field based on the intermediate image particle points to obtain the dense particle field.

[0033] In one embodiment, the particleization model unit includes:

[0034] The rendering visual sub-unit is used to render all the virtual camera key views and all the intermediate views based on the dense particle field to obtain corresponding rendered visual images;

[0035] A noise-adding subunit is used to add noise to the rendered visual image to obtain a noisy image;

[0036] A denoising subunit is used to perform joint denoising on the noisy image to obtain a denoised image;

[0037] The loss calculation subunit is used to obtain the rendering loss based on the rendered visual image and the denoised image;

[0038] The parameter update subunit is used to update the parameters of the particles in the dense particle field in reverse based on the rendering loss, so as to obtain the city particle model.

[0039] This application also provides a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute steps in any of the city model generation methods provided in this application.

[0040] This application also provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps in any of the city model generation methods provided in this application.

[0041] The city model generation method provided in this application can acquire city grid model data and determine the main streets corresponding to the city grid model data. Then, it acquires the key view sequence corresponding to the main streets and generates multiple visual images based on the key view sequence. For each of the multiple visual images, each pixel of the visual image is converted into a particle point in the world coordinate system, thereby obtaining a sparse particle field. Subsequently, the sparse particle field is densified to obtain a dense particle field. Based on the dense particle field, a particle-based city model is obtained.

[0042] This application embodiment can utilize urban grid model data to determine the corresponding main street, obtain a key view sequence consisting of multiple virtual camera key views corresponding to the main street, and then generate multiple visual images. By performing world coordinate system transformation on each pixel of the visual images, a sparse particle field is obtained, and a dense particle field is obtained through densification processing. The particles after densification processing can be directly deployed on the grid surface of the urban grid model, thereby improving the positional accuracy of the particles and more accurately determining the geometric information of building facades in the model. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1a This is a schematic diagram illustrating the application scenario of the city model generation method provided in this application;

[0045] Figure 1b This is a flowchart illustrating the city model generation method provided in an embodiment of this application;

[0046] Figure 2 This is a flowchart illustrating a city model generation method provided in another specific embodiment of this application;

[0047] Figure 3 This is a schematic diagram of a city model generation device provided in one embodiment of this application;

[0048] Figure 4 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0049] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0050] This application provides a method, apparatus, electronic device, and storage medium for generating city models.

[0051] Specifically, the city model generation device can be integrated into an electronic device, such as a terminal or server. The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal and server can be directly or indirectly connected via wired or wireless communication; this application does not impose any restrictions.

[0052] In some embodiments, the city model generation device can also be integrated into multiple electronic devices, such as multiple servers, with multiple servers implementing the city model generation method of this application.

[0053] In some embodiments, the terminal can also be used as a server to implement some or all of the functions of a server.

[0054] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0055] Please see details Figure 1a The method provided in this application embodiment may include: acquiring urban grid model data and determining the main street corresponding to the urban grid model data; acquiring a key view sequence corresponding to the main street, wherein the key view sequence includes multiple virtual camera key views, which are obtained by moving a virtual camera along the main street and capturing images, and the distance interval between any two virtual camera key views is the same; generating multiple visual images based on the key view sequence; for each visual image, converting each pixel of the visual image into a particle point in the world coordinate system to obtain a sparse particle field; performing compaction processing on the sparse particle field to obtain a dense particle field; and obtaining an urban particle model based on the dense particle field.

[0056] In the above method, the corresponding main street can be determined using urban grid model data. A key view sequence consisting of multiple virtual camera key views corresponding to this main street can then be obtained, generating multiple visual images. A sparse particle field is obtained by transforming each pixel of the visual images to the world coordinate system, and a dense particle field is obtained through densification. The densified particles can be directly deployed on the grid surface of the urban grid model, thereby improving the particle position accuracy and more accurately determining the geometric information of building facades in the model.

[0057] The method provided in this application can be applied to fields such as digital twins, 3D computer vision, and generative artificial intelligence.

[0058] It is understood that in the embodiments of this application, data such as user information are involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent should be explicitly obtained, and the collection, use and processing of related data should comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0059] The following sections provide detailed descriptions of each example. It should be noted that the sequence numbers of the following embodiments are not intended to limit the preferred order of the embodiments.

[0060] In this embodiment, a method for generating a city model is provided, such as... Figure 1b As shown, the specific process of this method may include the following steps 110 to 160:

[0061] 110. Obtain urban grid model data and determine the main streets corresponding to the urban grid model data.

[0062] City mesh model data refers to a three-dimensional geometric representation of a city. Optionally, city mesh model data may include multiple triangular faces, the vertex coordinates of the three vertices of each triangular face, the semantic label of each vertex, and the surface normal of each vertex. The city mesh model is a 3D mesh model containing the geometric structure of urban elements such as buildings and roads (typically at LoD3 level detail). The city mesh model has semantic labels (such as "buildings" and "roads") and instance identifiers, but no texture information. In this embodiment, the city mesh model data is a geometric prior input.

[0063] Triangle meshes are the most basic units that make up 3D mesh models. Each mesh is formed by connecting three vertices to form a triangular plane. By piecing together a large number of triangle meshes, arbitrarily complex 3D surfaces (such as building facades, road undulations, etc.) can be approximated.

[0064] Each triangular facet has three corner points called vertices, and each vertex has coordinates (x, y, z) in 3D space. These coordinates are the vertex coordinates of the corresponding vertex. Particle modeling requires knowing the absolute position of each surface point in the real world in order to correctly place and render particles.

[0065] Semantic labeling refers to assigning a category label to each vertex in the mesh, indicating which type of object the vertex belongs to. Common categories can include "buildings," "roads," "sidewalks," "vegetation," "streetlights," and "vehicles." Optionally, vertices belonging to the same category can also be distinguished by their instance ID. For example, two adjacent "residential buildings" both belong to the "buildings" category, but they have different instance IDs (e.g., ID=101 and ID=102).

[0066] A surface normal is a unit vector perpendicular to a vertex on a 3D surface, which forms the surface normal of that vertex. The tangent plane is defined as follows: Let's define a tangent plane as follows: Suppose there is a vertex P on the surface. The tangent plane corresponding to vertex P is the infinitely large plane that lies at vertex P and is tangent to the surface at point P.

[0067] Alternatively, in one implementation, if the city grid model data does not include surface normals, the surface normals can be calculated using the MeshLab tool. Specifically, they can be calculated based on a weighted average of the normal vectors of adjacent faces.

[0068] Main streets refer to roads with high connectivity as reflected in urban grid model data. Optionally, main streets can be streets with a width exceeding a preset width threshold, or streets connecting important areas. Specifically, main streets can be determined from urban grid model data using graph theory algorithms (such as PageRank and centrality analysis). Optionally, main streets can be straight lines, or they can include turns (such as turns at street corners like intersections, more closely resembling real urban road networks). It should be understood that the specific street shape of main streets should not be construed as a limitation of this application.

[0069] Optionally, in one implementation, after acquiring the urban grid model data, the urban grid model data can be preprocessed first. Specifically, the preprocessing operations may include: removing redundant patches (such as tiny patches with an area of ​​<0.1㎡) to avoid particle waste; unifying the coordinate system (such as setting the origin of the world coordinate system to the city center and the Z-axis to the vertical direction); and checking the integrity of the grid (such as repairing holes in building facades) to ensure that the grid is free of geometric defects.

[0070] In a specific example, the city grid model data can be: a city grid model from the CitySample project using Unreal Engine 5, containing 3 city blocks (approximately 1km × 1km), with approximately 500,000 building vertices, and semantic tags covering "residential buildings", "commercial buildings", and "asphalt roads".

[0071] 120. Obtain the key view sequence corresponding to the main street, wherein the key view sequence includes multiple virtual camera key views, which are obtained by moving a virtual camera along the main street and taking pictures, and the distance interval between any two virtual camera key views in the multiple virtual camera key views is the same.

[0072] The key view sequence is a sequence of multiple virtual camera key views. These multiple virtual camera key views can be obtained by moving a virtual camera along the main street. The virtual camera key views are sparsely representative views selected along a preset camera path, used to initialize particle modeling; their generation quality determines the geometric alignment and appearance consistency of the subsequent particle model.

[0073] Optionally, in one implementation, the virtual camera parameters can be as follows: resolution 960×544 (16:9 aspect ratio), field of view (FoV) 45°, near clipping plane 0.5 meters, far clipping plane 100 meters (covering buildings on both sides of the street), and the intrinsic parameter matrix is ​​set according to a conventional pinhole camera model, specifically: focal length f_x=f_y=500 pixels, principal point... =(480,272). Optionally, the camera orientation can be "forward horizontal" (aligned with the street direction, 0° pitch) to ensure coverage of the main street view area.

[0074] The specific distance between two adjacent key views captured by the virtual camera can be set by the developers. For example, the distance interval can be set to 20 meters, or it can be set to other values ​​such as 10 meters. For ease of description, let's assume the distance interval is 20 meters and the total length of the main street is 200 meters, which would generate 11 virtual camera key views: C1, C2, C3, ..., C 11 C1 is the starting point of the main street, C 11 It is the end point of the main street.

[0075] Optionally, in one embodiment, step 120 may specifically include the following steps 121 to 122:

[0076] 121. Control the virtual camera to move along the main street, and take a key view of the virtual camera every time it moves a preset fixed distance.

[0077] Optionally, the movement trajectory of the virtual camera can be planned along the centerline of the main street or the edge of the sidewalk. A spline curve is used to smooth the path, resulting in a natural transition between key views of adjacent virtual cameras. Sampling points can be dynamically adjusted according to the curvature of the street, ensuring an arc distance of 20 meters rather than a straight line distance.

[0078] Continuing with the example above: Starting from the starting point C1 (0 meters), a sampling point is set every 20 meters: C2 (20 meters), C3 (40 meters), ..., C 11(200 meters). Position accuracy can be controlled to the centimeter level (±0.01 meters) to ensure the accuracy of geometric reconstruction; the position of each camera can also be precisely located using a three-dimensional spatial coordinate system (x,y,z), where the z coordinate can be fixed at 1.6 meters to simulate human eye height.

[0079] At each preset location, the scene is rendered according to the specified virtual camera parameters (i.e., 960×544 resolution, 45° FoV as mentioned above), generating multi-channel data: a depth map (encoded in color within the range of 0.5-100 meters), a normal map, and a semantic map. The depth values ​​can be accurate to the millimeter level, with a near clipping plane of 0.5 meters to ensure the capture of close-up details (such as streetlights and signs); and a far clipping plane of 100 meters to cover buildings on both sides of the street.

[0080] 122. Arrange multiple key views of the virtual camera in the order they were captured to obtain the key view sequence.

[0081] The virtual cameras can be sorted according to their physical movement along the street: C1, C2, C3, ..., C 11 This yields the key view sequence.

[0082] In the above implementation, the problem of building facade "floating" in traditional BEV methods can be improved by precisely controlled equidistant sampling and strict sequential organization. By precisely arranging 11 observation points on a 200-meter street, each equipped with standardized virtual camera parameters (45° FoV, 960×544 resolution), the geometric and appearance details of the building facade can be captured, and spatial consistency is maintained through sequential order, providing reliable multi-view basic data for subsequent particle field construction.

[0083] 130. Based on the key view sequence, generate multiple visual images.

[0084] The visual image is an image with a high-fidelity appearance and strict geometric alignment. In one implementation, the visual image can be synthesized by an image generation model. Multiple visual images can serve as input sources for subsequent particle initialization.

[0085] Optionally, in one implementation, each of the virtual camera key views includes a depth map, a semantic map, and a normal map; correspondingly, step 130 may specifically include the following steps 131 to 133:

[0086] 131. Obtain the virtual camera key view at the end of the key view sequence.

[0087] Step 131 involves selecting the last view from a predefined series of key views as the starting point. Continuing from the example above, this involves selecting C. 11 .

[0088] These key views typically include important perspectives on a specific scene or object, and each view comes with additional information such as a depth map, semantic map, and normal map. The view at the end of the sequence is chosen as the starting point because it provides a more comprehensive and important overview of the entire scene.

[0089] 132. Based on the depth map, semantic map, and normal map included in the key view of the virtual camera at the end of the sequence, generate the corresponding final visual image.

[0090] Depth maps are used to provide distance information for each pixel from the camera.

[0091] Semantic graphs are used to identify the object category (e.g., buildings, roads, etc.) to which each pixel in an image belongs.

[0092] Normal maps are used to represent the direction of the normals to a surface at each pixel, which helps in understanding lighting and material properties.

[0093] The depth map, semantic map, and normal map described above can all be obtained by rendering the mesh model from the perspective of a virtual camera. The depth map encodes depth values ​​as color images; for example, foreground is blue and background is red. Foreground can refer to 0.5 meters, and background can refer to 100 meters. Different semantic labels in the semantic map can correspond to different colors; for example, "residential buildings" is off-white, and "roads" is dark gray. The normal vectors in the normal map are encoded as RGB values; specifically, the X / Y / Z components can be mapped to the [0.255] interval.

[0094] Optionally, in one implementation, the final visual image can be generated by a first-level control model. Specifically, the aforementioned depth map, semantic map, and normal map, along with text prompts, can be input into the first-level control model, which then outputs the final visual image. The control model is a conditional control module attached to the diffusion model, which can constrain the structure and geometric alignment of the generated results based on geometric signals such as depth, semantics, and normals.

[0095] Optionally, text prompts may include descriptive terms such as "sunny day, city street view, clear building facades, asphalt roads, no pedestrians or vehicles" to control image style, lighting, object details, etc. The first-level control model can specifically be a Stable Diffusion + ControlNet model.

[0096] 133. Starting from the last visual image, in the reverse order of the key view sequence, generate the other visual images corresponding to the multiple virtual camera key views respectively, to obtain the multiple visual images.

[0097] The above-mentioned multiple visual images include the end visual image and the visual images corresponding to other multiple virtual camera key views respectively.

[0098] Step 133 describes an iterative process, that is, using the visual image generated in the previous step as a basis, traversing the key view sequence in reverse, and generating the corresponding visual image for each virtual camera key view in the sequence. The advantage of this method is that it can ensure the visual consistency between adjacent views, because each newly generated image will refer to the previous image (for example, C 10 The generation of the corresponding visual image can refer to C 11 The corresponding visual image, and the generation of the visual image corresponding to C9 can refer to C 10 The corresponding visual image…). In this way, a complete set of continuous and coordinated visual images can be constructed from a series of discrete key views.

[0099] Optionally, in one implementation, the key view sequence includes m virtual camera key views, where m is a positive integer greater than 1; correspondingly, step 133 may specifically include the following steps S1 to step S3:

[0100] S1. For the i-th virtual camera key view in the key view sequence, obtain the (i + 1)-th visual image corresponding to the (i + 1)-th virtual camera key view; where i < m and i is a positive integer.

[0101] To generate the i-th visual image corresponding to the i-th virtual camera key view, it is necessary to first obtain the previously generated (i + 1)-th visual image. Among them, the (i + 1)-th visual image is the visual image corresponding to the (i + 1)-th virtual camera key view.

[0102] S2. Perform warping on the (i + 1)-th visual image to obtain the (i + 1)-th warping result.

[0103] Warping in step S2 specifically includes: using and The known relative pose between them, project the pixels of To the image plane of For example, substituting i = 10, then C 10 And C 11 Are 20 meters apart. Project the visual image corresponding to C 11 To the image plane of C 10 You can get the warped image . In one implementation, this warped image Can directly be the (i + 1)-th warping result.

[0104] Optionally, in another implementation, since the warped image Holes are created due to changes in the field of view; the warped image can be corrected first. Perform completion processing and use the completion result as the (i+1)th warping result. The completion process can be implemented by the second-level control model.

[0105] S3. Based on the (i+1)th warping result and the depth map, semantic map, and normal map included in the i-th virtual camera key view, generate the i-th visual image.

[0106] Optionally, in one implementation, step S3 can specifically utilize the second-level control model to generate the depth map, semantic map, and normal map included in the i-th virtual camera key view as input to the second-level control model, use the (i+1)-th warping result as a reference appearance constraint, and output the i-th visual image.

[0107] Continuing with the example above, let's take C... 10 The included depth map, semantic map, and normal map are used as inputs to the second-level control model. As a reference appearance constraint for the second-level control model, C can be obtained. 10 The corresponding visual image.

[0108] Repeating steps S1 to S3 above will generate C9, C8, C7, ..., C1 in sequence.

[0109] In the above implementation, a strategy of generating from the rear view in reverse and using warp guidance can achieve high-fidelity appearance generation. Furthermore, by transmitting appearance information through warping, multi-view consistency can be achieved. Using depth information, semantic information, and normal information as constraints, precise alignment of geometry and texture can also be achieved. For example, let's consider C... 10 The corresponding visual image and C 11 For example, in the corresponding visual images, the window styles (such as white frames and brown glass) of the same residential building are consistent, and the position of the road markings (double solid yellow lines) may only shift slightly due to camera movement (shift amount < 1 pixel), with no appearance drift.

[0110] This application's embodiments can design a two-level control network to generate sparse key views using precise geometric signals (depth, semantics, normals) of a city grid model as constraints. The first-level control network takes the geometric signals of the initial key view (such as the virtual camera key view at the end of the sequence) from the grid rendering as input to generate the first key view (i.e., the corresponding final visual image). The second-level control network can generate other key views in reverse along the camera path, and additionally introduces the warping results of the preceding views as constraints during the generation process to avoid appearance drift during the expansion process. The above steps can improve the problems of poor geometric alignment and weak long-range consistency in existing technologies. The key views are directly constrained by the geometric signals of the grid model, and the generated results are strictly aligned with the grid surface; the combination of reverse expansion and warping constraints can eliminate the accumulation of errors in long-range expansion and ensure the global consistency of the key views.

[0111] 140. For each of the aforementioned visual images, each pixel of the visual image is converted into a particle point in the world coordinate system to obtain a sparse particle field.

[0112] Among them, the particle field is a 3D data structure composed of a large number of particles deployed on the surface of the urban grid. It is the core carrier of the particle model. The particle field includes parameters such as the position, scale, orientation, opacity, and color (spherical harmonic coefficient) of the particles.

[0113] Optionally, in one embodiment, step 140 may specifically include the following steps 141 to 142:

[0114] 141. For each pixel, calculate the particle point of the pixel in the world coordinate system based on the coordinate value of the pixel in the visual image, the depth value of the pixel, the camera intrinsic parameter matrix, and the camera pose transformation matrix.

[0115] Alternatively, in one implementation, step 141 can be achieved by the following formula:

[0116]

[0117] in, Here are the coordinates of a pixel, indicating that a given pixel in the image is located in the i-th row and j-th column of the image. This is the depth value of the pixel, representing the distance of the pixel from the virtual camera; These are two-dimensional homogeneous coordinates, used to convert the coordinates of pixels into three-dimensional vectors for easier matrix operations; The camera intrinsic parameter matrix is ​​used to convert pixel coordinates into direction vectors in the camera coordinate system; is the camera pose transformation matrix, used to convert pixel coordinates into direction vectors in the camera coordinate system; P is the particle point of the aforementioned pixel in the world coordinate system.

[0118] The orientation of this particle point is consistent with the normal of the corresponding point in the urban grid model data. Specifically, this can be achieved through quaternion q-coding, i.e., conversion from the normal vector, ensuring that the particle "fits" the grid surface.

[0119] The scale of this particle point can be determined based on the normal, fov, and depth. Specifically, the scale is divided into two directions: the normal direction and the tangent direction.

[0120] Normal direction ( The mesh can be fixed at 0.01 meters to prevent particles from penetrating the mesh, and it is slightly "lifted" to avoid shading.

[0121] Tangent direction ( The calculation can be dynamically performed based on the depth value, FoV, and image width. The specific calculation formula is as follows:

[0122]

[0123] For example, let's set a depth value. Take 20 meters, Given a 45-degree angle and an image width of 960 pixels, substituting these values ​​into the formula above, we get:

[0124]

[0125] The color of this particle can be obtained based on the RGB value of its corresponding pixel in the visual image. The specific calculation formula is as follows:

[0126]

[0127] in, RGB Let be the RGB value of the pixel in its corresponding visual image. The above formula can convert the RGB value of pixel (i,j) into SH coefficients.

[0128] The opacity of this particle can be manually set to 0.9.

[0129] In this embodiment, particles can be initialized directly using the geometric information (vertex coordinates, surface normals, etc.) of the urban grid model: for each pixel in the key view of the virtual camera, the depth information rendered by the grid is used to map it to the 3D coordinates of the grid surface, avoiding monocular depth error; the particle orientation is aligned with the surface normal of the urban grid model, and the scale along the normal direction is set to a minimum value to avoid "penetrating" the grid, while the scale along the tangent direction is set according to the coverage of the pixel to avoid moiré patterns; the particle color (spherical harmonic coefficient) can be directly extracted from the key view pixels, and the opacity is set to an empirical value (e.g., 0.9). The above processing can improve the problems of inaccurate particle positioning and poor geometric fit in the prior art: the particles are directly deployed on the grid surface, and the positional accuracy is consistent with the grid model, avoiding geometric misalignment.

[0130] 142. The sparse particle field is composed of multiple of the aforementioned particle points.

[0131] After completing the transformation in step 141 above, all pixels are mapped to the world coordinate system, forming a set of particle points. Since these pixels only come from the visual image corresponding to the key view of the virtual camera, not all pixels are transformed; therefore, this set is called a sparse particle field. This sparsity reflects the discrete sampling points on the surface of objects in the actual scene, rather than a continuous distribution.

[0132] Optionally, in one implementation, redundant particles can be filtered first, and then a sparse particle field can be generated based on the filtered result. Specifically, the redundant particle filtering process may include: if pixels of multiple key views are mapped to the same mesh patch (area < 0.5㎡), only the particles corresponding to the view closest to the camera are retained to avoid duplicate deployment. For example, if C 10 The corresponding visual image pixels and C 11 If the pixels of the corresponding visual image are all mapped to the same building facade, then C can be preserved. 11 The corresponding particles in the visual image are because, for the same position, C 11 The pixels of the corresponding visual image are closer to the virtual camera than the former.

[0133] Optionally, in one specific example, the sparse particle field may contain approximately 1.2 million particles, covering the grid area corresponding to all key views, with the particle positions having a fitting error of <0.05 meters to the grid surface.

[0134] 150. The sparse particle field is densified to obtain a dense particle field.

[0135] Step 150 describes how to densify the initially generated sparse particle field to obtain a more complete and detailed dense particle field. Optionally, the above densification process can be implemented using the Appearance Guided Inpainting (AGInpaint) module. The AGINpaint module is used to complete the occluded areas in the intermediate view of the particle model and optimize the particle field density. The AGINpaint module can diffuse the completion process based on the appearance constraints of the surrounding known areas.

[0136] Optionally, in one embodiment, step 150 may specifically include the following steps 151 to 154:

[0137] 151. At least one intermediate view is filled between any two adjacent virtual camera key views in the plurality of virtual camera key views, wherein, for each intermediate view, the first distance and the second distance of the intermediate view are equal, the first distance refers to the distance between the intermediate view and the view preceding the intermediate view, and the second distance refers to the distance between the intermediate view and the view following the intermediate view.

[0138] The purpose of step 151 is to increase the view density, thereby improving the spatial resolution of the particle field. This is achieved by inserting one or more intermediate views between each pair of adjacent key views. For clarity, let's continue with the example above:

[0139] In any two adjacent virtual camera key views (let's call them C...), 10 With C 11 For example, in C 10 With C 11 Between them, at 5-meter intervals, set three intermediate views (such as C). 10-1 C 10-2 C 10-3 ).

[0140] Among them, for intermediate view C 10-1 In other words, it is similar to the previous view C. 10 The distance (i.e., the first distance) is 5 meters, which is different from the distance in the next view C. 10-2 The distance (i.e., the second distance) is also 5 meters.

[0141] Similarly, for intermediate view C 10-2 In other words, it is similar to the previous view C. 10-1 The distance is 5 meters, and it is relative to the next view C. 10-3 The distance is also 5 meters.

[0142] For the intermediate view C 10-3 In other words, it is similar to the previous view C.10-2 The distance is 5 meters, and it is relative to the next view C. 11 The distance is also 5 meters.

[0143] 152. Based on the sparse particle field, render multiple intermediate views to obtain corresponding intermediate visual images.

[0144] Alternatively, in one implementation, step 152 can be specifically implemented through the following steps:

[0145] The particles of the sparse particle field are projected onto a 2D image using a selected rendering technique (such as ray tracing or rasterization). For each pixel: determine which particles contribute to that pixel. Calculate the pixel's color value based on particle attributes such as color and transparency. Handle occlusion relationships (i.e., determine which particles are in front and which are behind) to ensure correct visual hierarchy. To enhance realism, shadows and lighting effects may also be added. Finally, the processed color information is stored as an RGB image file. Each color consists of three color channels: red, green, and blue, with each channel's value ranging from 0 to 255.

[0146] Optionally, in one implementation, due to the particle field coefficient, the intermediate visual image may contain holes, such as black areas at building corners due to lack of particle coverage; it may also have blurred outlines, such as jagged edges on lampposts due to insufficient particles. Therefore, the intermediate visual image can be completed.

[0147] Specifically, the completion process can be as follows:

[0148] 1. Input preparation:

[0149] Image to be completed: Intermediate visual image of the intermediate view (to be completed);

[0150] Mask: Marks the area to be filled (e.g., black for areas to be filled, white for known areas). Specifically, the mask can be automatically generated using the "render brightness < 0.01" setting.

[0151] Reference constraints: the depth map and semantic map of the intermediate view, and the visual image of the key view adjacent to the intermediate view (for appearance style alignment).

[0152] 2. LCM bootloader completion:

[0153] The completion process: After each denoising step, the noise prediction value is optimized by gradient descent to ensure that the color and texture of the completed area are consistent with the surrounding known areas (such as the continuity of brick patterns on building walls and the uniformity of asphalt texture on roads).

[0154] 3. Output: The completed intermediate visual image, where the areas to be completed (such as holes) are repaired and the appearance is consistent with the adjacent key views.

[0155] 153. For each intermediate visual image, convert each pixel of the intermediate visual image into a particle point in the world coordinate system to obtain the intermediate image particle points.

[0156] Step 153 is the same as step 141, so it will not be described again here.

[0157] 154. Fill the sparse particle field with the intermediate image particle points to obtain the dense particle field.

[0158] The particle density of a dense particle field can be increased by 2 to 3 times, and the problems of voids and fuzziness are solved.

[0159] In step 154, all particle points extracted from the intermediate visual image can be added to the original sparse particle field to form a denser and more complete particle field. This dense particle field not only includes the information provided by the original key view, but also incorporates additional details from the intermediate view, thereby significantly improving the quality and accuracy of the entire 3D model.

[0160] Alternatively, in one implementation, step 154 ​​can deploy particles in areas where the sparse particle field coverage is insufficient (such as the mesh patch corresponding to the intermediate view completion area), with parameter settings (position, scale, orientation, color) consistent with the steps described above.

[0161] In the above implementation, by inserting intermediate views between key views and using these intermediate views to generate new particle points, the problem of detail loss caused by an overly sparse particle field can be effectively improved, laying the foundation for the final construction of a high-quality urban particle model.

[0162] Steps 151 to 154 above can render intermediate views (views between adjacent key views) based on a sparse particle field constructed from sparse key views. For occluded or hollow areas in the intermediate views, the AGINpaint module can be used for completion. Through LCM consistency constraints, the appearance of the completed area is aligned with the surrounding known areas. Simultaneously, the pixel information of the completed area is projected onto the mesh surface, and new particles are added to densify the particle field. These steps can improve the problems of poor intermediate view quality and insufficient particle field density in existing technologies. AGINpaint completes the process based on mesh geometry priors, avoiding holes; the added particles strictly adhere to the mesh surface, improving the detail density of the particle-based model.

[0163] 160. Based on the dense particle field, a city particle model is obtained.

[0164] Particle modeling is a technique for 3D scene reconstruction and rendering. It generates a large number of particles in 3D space to achieve efficient radiation field representation and real-time rendering.

[0165] Optionally, in one embodiment, step 160 may specifically include the following steps 161 to 165:

[0166] 161. Based on the dense particle field, render all the key views and all the intermediate views of the virtual camera to obtain the corresponding rendered visual image.

[0167] In step 161, based on the already densified particle field (i.e., dense particle field), all virtual camera key views and all intermediate views are rendered to generate corresponding rendered visual images (i.e., RGB images). These images can cover the effect of viewing the same scene from different angles.

[0168] 162. Noise is added to the rendered visual image to obtain a noisy image.

[0169] To simulate the impact of noise in a real-world environment and to provide a foundation for the subsequent denoising process, a forward diffusion technique is employed. Specifically, this involves performing 50 steps on the previously rendered image. A noisy image is produced by performing a noise-adding operation (50 steps).

[0170] Step 162 is used to simulate various noise interferences that an image may encounter in the real world, thereby preparing for the subsequent denoising steps.

[0171] 163. Perform joint denoising on the noisy image to obtain a denoised image.

[0172] Optionally, a Latent Consistency Model (LCM) can be used to jointly denoise all noisy images that have undergone noise addition. The goal is to preserve as much detail as possible while maintaining visual consistency across multiple views. LCM is a fast diffusion model that learns a consistent mapping of the probability flow ODE trajectory to achieve single-step or few-step denoising, improving generation efficiency and cross-view consistency.

[0173] To this end, two main constraints can be set: 1. The difference in RGB values ​​of the same object (such as a window) under different viewpoints should be less than 3% to ensure visual consistency; 2. The similarity in detail (such as edge sharpness) between the denoised image and the original rendered image should exceed 0.9 to avoid excessive blurring and information loss.

[0174] 164. Based on the rendered visual image and the denoised image, the rendering loss is obtained.

[0175] Optionally, the rendering loss is a measure of the difference between the rendered visual image and the denoised image.

[0176] The rendering loss can be expressed by the formula:

[0177]

[0178] in, Indicates rendering loss. This represents the rendering loss for any rendered visual image.

[0179] Specifically, The following formula can be used for calculation:

[0180]

[0181] in, Indicates the current particle field parameters The rendered image (i.e., the rendered visual image); This represents the denoised image corresponding to the rendered visual image; The function representing the image difference measure can be L1, L2, perceptual loss, etc.

[0182] 165. Based on the rendering loss, the parameters of the particles in the dense particle field are updated in reverse to obtain the city particle model.

[0183] Based on the rendering loss calculated in the above process, backpropagation is performed to update the attribute parameters of each particle, including but not limited to position, rotation, scaling, and transparency, in order to improve the overall model quality. After optimization, the final result is a high-precision 3D model composed of a large number of particles (i.e., the aforementioned city particle model). Each particle in this model carries detailed information, such as its coordinates in three-dimensional space, rotation state (usually represented by quaternions), size adjustment factor, transparency, and SH coefficient.

[0184] The city particle model obtained in step 165 can contain a large number of particles, stored in the format of a "particle parameter list". In the particle parameter list, each particle contains: position (x, y, z), quaternion, scaling and rotation, opacity, SH coefficient, etc., which can be directly used for 3D rendering.

[0185] In the above implementation, the above steps can form a closed-loop optimization process, from initial rendering to noise reduction, then to fine-tuning, and finally producing a high-quality 3D particle model, which can improve the efficiency and quality of 3D content creation.

[0186] Optionally, steps 161 to 165 above can be implemented by the Global ConsistencyAlignment (GCAlign) module. The GCAlign module is used to eliminate visual inconsistencies such as brightness drift and exposure differences between multiple views. The GCAlign module can optimize the global consistency of the particle model through joint denoising of multiple views.

[0187] In steps 161 to 165 above, during the particle field construction and densification process, the GCAlign module is introduced to perform forward diffusion (noise addition) on the rendering results of all key views and intermediate views; multi-view joint denoising is used with LCM to balance the exposure and brightness differences between different views (such as unifying the brightness of cloudy and sunny views); based on the denoised views, the color (spherical harmonic coefficient) and opacity parameters of the particles are optimized in reverse to ensure the consistency of the particle model under any view. The above steps can improve the problems of brightness drift and visual inconsistency in multi-view technology in existing technologies. GCAlign eliminates the appearance differences of local views through global joint optimization, ensuring the global visual uniformity of the particle model.

[0188] The city model generation method provided in this application embodiment can acquire city grid model data and determine the main streets corresponding to the city grid model data. Then, it acquires the key view sequence corresponding to the main streets and generates multiple visual images based on the key view sequence. For each of the multiple visual images, each pixel of the visual image is converted into a particle point in the world coordinate system, thereby obtaining a sparse particle field. Subsequently, the sparse particle field is densified to obtain a dense particle field. Based on the dense particle field, a particleized city model is obtained. This application embodiment can use city grid model data to determine the corresponding main streets, acquire the key view sequence corresponding to the main streets consisting of multiple virtual camera key views, generate multiple visual images, and obtain a sparse particle field by performing world coordinate system transformation on each pixel of the visual images, and then obtain a dense particle field through densification. The particles after densification can be directly deployed on the grid surface of the city grid model, which can improve the positional accuracy of the particles.

[0189] The embodiments of this application can more accurately determine the geometric information of building facades in the model.

[0190] In this embodiment, the method of this application embodiment will be described in detail. The city model generation method provided in this application embodiment can be executed by an electronic device. In this embodiment, let's take a live-streaming video of a product as an example for illustration, such as... Figure 2 As shown, the specific process of a city model generation method is as follows:

[0191] 201. Obtain the urban grid model data and determine the main street corresponding to the urban grid model data.

[0192] 202. Control the virtual camera to move along the main street, and capture a key view of the virtual camera every time it moves a preset fixed distance.

[0193] 203. Arrange multiple key views of the virtual camera in the order of shooting to obtain the key view sequence.

[0194] The key view sequence includes multiple key views of the virtual camera, which are obtained by the virtual camera moving along the main street. The distance intervals between any two of the multiple key views of the virtual camera are the same.

[0195] 204. Obtain the key view of the virtual camera at the end of the key view sequence.

[0196] 205. Generate the corresponding end visual image based on the depth map, semantic map, and normal map included in the key view of the virtual camera at the end of the sequence.

[0197] 206. Starting from the end visual image, generate the visual images corresponding to the other multiple key views of the virtual camera in the reverse order of the key view sequence to obtain the multiple visual images.

[0198] Optionally, in one implementation, the key view sequence includes m key views of the virtual camera, where m is a positive integer greater than 1; correspondingly, step 206 may specifically include the following steps: for the i-th key view of the virtual camera in the key view sequence, obtain the (i + 1)-th visual image corresponding to the (i + 1)-th key view of the virtual camera; where i < m and i is a positive integer; perform warping processing on the (i + 1)-th visual image to obtain the (i + 1)-th warping result; generate the i-th visual image based on the (i + 1)-th warping result, the depth map, semantic map, and normal map included in the i-th key view of the virtual camera.

[0199] 207. For each pixel point, calculate the particle point of the pixel point in the world coordinate system according to the coordinate value of the pixel point in the所属视觉图像 (should be "corresponding visual image" in the original text), the depth value of the pixel point, the camera internal parameter matrix, and the camera pose transformation matrix.

[0200] 208. Multiple particle points form the sparse particle field.

[0201] 209. At least one intermediate view is filled between any two adjacent virtual camera key views in the plurality of virtual camera key views, wherein, for each intermediate view, the first distance and the second distance of the intermediate view are equal, the first distance refers to the distance between the intermediate view and the view preceding the intermediate view, and the second distance refers to the distance between the intermediate view and the view following the intermediate view.

[0202] 210. Based on the sparse particle field, render multiple intermediate views to obtain corresponding intermediate visual images.

[0203] 211. For each intermediate visual image, convert each pixel of the intermediate visual image into a particle point in the world coordinate system to obtain the intermediate image particle points.

[0204] 212. The sparse particle field is filled with the intermediate image particle points to obtain the dense particle field.

[0205] 213. Based on the dense particle field, a city particle model is obtained.

[0206] Optionally, in one embodiment, step 213 may specifically include the following steps: based on the dense particle field, rendering all the key views of the virtual camera and all the intermediate views to obtain corresponding rendered visual images; adding noise to the rendered visual images to obtain noisy images; performing joint denoising on the noisy images to obtain denoised images; obtaining a rendering loss based on the rendered visual images and the denoised images; and updating the parameters of the particles in the dense particle field in reverse based on the rendering loss to obtain the city particleization model.

[0207] The specific execution process of steps 201 to 213 has been explained in detail above, and will not be repeated here.

[0208] The city model generation method provided in this application embodiment can acquire city grid model data and determine the main streets corresponding to the city grid model data. Then, it acquires the key view sequence corresponding to the main streets and generates multiple visual images based on the key view sequence. For each of the multiple visual images, each pixel of the visual image is converted into a particle point in the world coordinate system, thereby obtaining a sparse particle field. Subsequently, the sparse particle field is densified to obtain a dense particle field. Based on the dense particle field, a particleized city model is obtained. This application embodiment can use city grid model data to determine the corresponding main streets, acquire the key view sequence corresponding to the main streets consisting of multiple virtual camera key views, generate multiple visual images, and obtain a sparse particle field by performing world coordinate system transformation on each pixel of the visual images, and then obtain a dense particle field through densification. The particles after densification can be directly deployed on the grid surface of the city grid model, which can improve the positional accuracy of the particles.

[0209] The embodiments of this application can more accurately determine the geometric information of building facades in the model.

[0210] To better implement the above methods, embodiments of this application also provide a city model generation apparatus. For example... Figure 3 As shown, the device includes:

[0211] The data acquisition unit 301 is used to acquire urban grid model data and determine the main streets corresponding to the urban grid model data;

[0212] The sequence acquisition unit 302 is used to acquire the key view sequence corresponding to the main street, wherein the key view sequence includes multiple virtual camera key views, which are obtained by moving a virtual camera along the main street and taking pictures, and the distance interval between any two virtual camera key views in the multiple virtual camera key views is the same.

[0213] The visual image generation unit 303 is used to generate multiple visual images based on the key view sequence;

[0214] The sparse particle field unit 304 is used to convert each pixel of the visual image into a particle point in the world coordinate system for each visual image to obtain a sparse particle field.

[0215] The dense particle field unit 305 is used to densify the sparse particle field to obtain a dense particle field.

[0216] The particle model unit 306 is used to obtain a city particle model based on the dense particle field.

[0217] In one embodiment, the sequence acquisition unit 302 includes:

[0218] A view capture subunit, configured to control the virtual camera to move along the main street, and capture a key view of the virtual camera every time it moves a preset fixed distance;

[0219] A view sequence subunit, configured to arrange multiple key views of the virtual camera in the order of capture to obtain the key view sequence.

[0220] In one embodiment, each key view of the virtual camera includes a depth map, a semantic map, and a normal map; correspondingly, the visual image generation unit 303 includes:

[0221] A key view acquisition subunit, configured to acquire the key view of the virtual camera at the end of the sequence of the key view sequence;

[0222] An end image subunit, configured to generate a corresponding end visual image based on the depth map, the semantic map, and the normal map included in the key view of the virtual camera at the end of the sequence;

[0223] A sequentially generating subunit, configured to sequentially generate visual images corresponding to the other multiple key views of the virtual camera in reverse order from the end visual image to obtain the multiple visual images.

[0224] In one embodiment, the key view sequence includes m key views of the virtual camera, where m is a positive integer greater than 1; correspondingly, the sequentially generating subunit is specifically configured to:

[0225] For the i-th key view of the virtual camera in the key view sequence, acquire the (i + 1)-th visual image corresponding to the (i + 1)-th key view of the virtual camera; where i < m and i is a positive integer;

[0226] Perform warping processing on the (i + 1)-th visual image to obtain the (i + 1)-th warping result;

[0227] Generate the i-th visual image based on the (i + 1)-th warping result, the depth map, the semantic map, and the normal map included in the i-th key view of the virtual camera.

[0228] In one embodiment, the sparse particle field unit 304 includes:

[0229] A particle point calculation subunit, configured to calculate, for each pixel point, the particle point of the pixel point in the world coordinate system according to the coordinate value of the pixel point in the visual image to which it belongs, the depth value of the pixel point, the camera internal parameter matrix, and the camera pose transformation matrix;

[0230] The particle points form sub-units, which are used to form the sparse particle field by multiple particle points.

[0231] In one embodiment, the dense particle field unit 305 includes:

[0232] The intermediate view filling subunit is used to fill at least one intermediate view between any two adjacent virtual camera key views in the plurality of virtual camera key views. For each intermediate view, the first distance and the second distance of the intermediate view are equal. The first distance refers to the distance between the intermediate view and the view preceding it, and the second distance refers to the distance between the intermediate view and the view following it.

[0233] The intermediate view rendering subunit is used to render multiple intermediate views based on the sparse particle field to obtain corresponding intermediate visual images.

[0234] The pixel conversion subunit is used to convert each pixel of the intermediate visual image into a particle point in the world coordinate system for each intermediate visual image, so as to obtain the intermediate image particle points.

[0235] The dense particle field subunit is used to fill the sparse particle field based on the intermediate image particle points to obtain the dense particle field.

[0236] In one embodiment, the particle model unit 306 includes:

[0237] The rendering visual subunit is used to render all the key views of the virtual camera and all the intermediate views based on the dense particle field to obtain the corresponding rendered visual image.

[0238] A noise-adding subunit is used to add noise to the rendered visual image to obtain a noisy image;

[0239] A denoising subunit is used to perform joint denoising on the noisy image to obtain a denoised image;

[0240] The loss calculation subunit is used to obtain the rendering loss based on the rendered visual image and the denoised image;

[0241] The parameter update subunit is used to update the parameters of the particles in the dense particle field in reverse based on the rendering loss, so as to obtain the city particle model.

[0242] In practice, each of the above units can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units, please refer to the previous method embodiments, which will not be repeated here.

[0243] The city model generation method provided in this application embodiment can acquire city grid model data and determine the main streets corresponding to the city grid model data. Then, it acquires the key view sequence corresponding to the main streets and generates multiple visual images based on the key view sequence. For each of the multiple visual images, each pixel of the visual image is converted into a particle point in the world coordinate system, thereby obtaining a sparse particle field. Subsequently, the sparse particle field is densified to obtain a dense particle field. Based on the dense particle field, a particleized city model is obtained. This application embodiment can use city grid model data to determine the corresponding main streets, acquire the key view sequence corresponding to the main streets consisting of multiple virtual camera key views, generate multiple visual images, and obtain a sparse particle field by performing world coordinate system transformation on each pixel of the visual images, and then obtain a dense particle field through densification. The particles after densification can be directly deployed on the grid surface of the city grid model, which can improve the positional accuracy of the particles.

[0244] The embodiments of this application can more accurately determine the geometric information of building facades in the model.

[0245] This application also provides an electronic device, which can be a terminal, a server, or other similar device. The terminal can be a mobile phone, tablet computer, smart Bluetooth device, laptop computer, personal computer, etc.; the server can be a single server or a server cluster composed of multiple servers, etc.

[0246] In some embodiments, the city model generation device can also be integrated into multiple electronic devices, such as multiple servers, with multiple servers implementing the city model generation method of this application.

[0247] In this embodiment, a server will be used as an example for detailed description. For example, ... Figure 4 As shown, it illustrates a structural schematic diagram of the electronic device involved in the embodiments of this application, specifically:

[0248] The electronic device may include components such as a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a power supply 403, an input module 404, and a communication module 405. Those skilled in the art will understand that... Figure 4 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0249] The processor 401 is the control center of the electronic device, connecting various parts of the device via various interfaces and lines. It performs various functions and processes data by running or executing software programs and / or modules stored in the memory 402, and by calling data stored in the memory 402. In some embodiments, the processor 401 may include one or more processing cores; in some embodiments, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into the processor 401.

[0250] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and violation detection by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 402 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0251] The electronic device also includes a power supply 403 that supplies power to the various components. In some embodiments, the power supply 403 can be logically connected to the processor 401 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 403 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0252] The electronic device may also include an input module 404, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0253] The electronic device may also include a communication module 405. In some embodiments, the communication module 405 may include a wireless module, through which the electronic device can perform short-range wireless transmission, thereby providing users with wireless broadband internet access. For example, the communication module 405 can be used to help users send and receive emails, browse web pages, and access streaming media.

[0254] Although not shown, the electronic device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 401 in the electronic device loads the executable files corresponding to the processes of one or more applications into the memory 402 according to the following instructions, and the processor 401 runs the applications stored in the memory 402, thereby realizing the various functions in the various method embodiments of this application, as follows:

[0255] Acquire urban grid model data and determine the main streets corresponding to the urban grid model data; acquire the key view sequence corresponding to the main streets, wherein the key view sequence includes multiple virtual camera key views, which are obtained by moving a virtual camera along the main streets and capturing images, and the distance interval between any two virtual camera key views is the same; generate multiple visual images based on the key view sequence; for each visual image, convert each pixel of the visual image into a particle point in the world coordinate system to obtain a sparse particle field; perform compaction processing on the sparse particle field to obtain a dense particle field; obtain an urban particle model based on the dense particle field.

[0256] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0257] Therefore, embodiments of this application provide a computer-readable storage medium storing instructions that can be loaded by a processor to execute steps in any of the city model generation methods provided in embodiments of this application. For example, the instructions can execute the following steps:

[0258] Acquire urban grid model data and determine the main streets corresponding to the urban grid model data; acquire the key view sequence corresponding to the main streets, wherein the key view sequence includes multiple virtual camera key views, which are obtained by moving a virtual camera along the main streets and capturing images, and the distance interval between any two virtual camera key views is the same; generate multiple visual images based on the key view sequence; for each visual image, convert each pixel of the visual image into a particle point in the world coordinate system to obtain a sparse particle field; perform compaction processing on the sparse particle field to obtain a dense particle field; obtain an urban particle model based on the dense particle field.

[0259] The storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0260] According to one aspect of this application, a computer program product or computer program is provided, comprising instructions stored in a computer-readable storage medium. A processor of a computer device reads the instructions from the computer-readable storage medium and executes the instructions, causing the computer device to perform the methods provided in the various optional implementations of the above embodiments.

[0261] Since the instructions stored in the storage medium can execute the steps in any of the city model generation methods provided in the embodiments of this application, the beneficial effects that any of the city model generation methods provided in the embodiments of this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0262] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0263] The above provides a detailed description of a city model generation method, apparatus, electronic device, and computer-readable storage medium provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for generating city models, characterized in that, The method includes: Acquire urban grid model data and determine the main streets corresponding to the urban grid model data; Obtain the key view sequence corresponding to the main street, wherein the key view sequence includes multiple virtual camera key views, which are obtained by moving a virtual camera along the main street and capturing images, and the distance interval between any two virtual camera key views is the same. Based on the key view sequence, multiple visual images are generated; For each visual image, each pixel of the visual image is converted into a particle point in the world coordinate system to obtain a sparse particle field. At least one intermediate view is filled between any two adjacent virtual camera key views in the plurality of virtual camera key views. For each intermediate view, the first distance and the second distance of the intermediate view are equal. The first distance refers to the distance between the intermediate view and the view preceding it, and the second distance refers to the distance between the intermediate view and the view following it. Based on the sparse particle field, multiple intermediate views are rendered to obtain corresponding intermediate visual images; For each intermediate visual image, each pixel of the intermediate visual image is converted into a particle point in the world coordinate system to obtain the intermediate image particle point; A dense particle field is obtained by filling the sparse particle field with the particle points of the intermediate image. Based on the dense particle field, all key views and all intermediate views of the virtual camera are rendered to obtain the corresponding rendered visual image. The rendered visual image is subjected to noise processing to obtain a noisy image; The noisy image is subjected to joint denoising to obtain a denoised image; Based on the rendered visual image and the denoised image, the rendering loss is obtained; Based on the rendering loss, the parameters of the particles in the dense particle field are updated in reverse to obtain the urban particle model.

2. The method as described in claim 1, characterized in that, The step of obtaining the key view sequence corresponding to the main street includes: Control the virtual camera to move along the main street, and take a key view of the virtual camera every time it moves a preset fixed distance; The key view sequence is obtained by arranging multiple virtual camera key views in the order they were captured.

3. The method as described in claim 1, characterized in that, Each of the aforementioned key views of the virtual camera includes a depth map, a semantic map, and a normal map; The generation of multiple visual images based on the key view sequence includes: Obtain the virtual camera key view at the end of the key view sequence; Based on the depth map, semantic map, and normal map included in the key view of the virtual camera at the end of the sequence, a corresponding final visual image is generated. Starting from the last visual image, in the reverse order of the key view sequence, other visual images corresponding to the multiple virtual camera key views are generated sequentially to obtain the multiple visual images.

4. The method as described in claim 3, characterized in that, The key view sequence includes m virtual camera key views, where m is a positive integer greater than 1; Starting from the last visual image, and following the reverse order of the key view sequence, the generation of other visual images corresponding to the multiple virtual camera key views includes: For the i-th virtual camera key view in the key view sequence, obtain the (i + 1)-th visual image corresponding to the (i + 1)-th virtual camera key view; where i < m and i is a positive integer; Perform warping processing on the (i + 1)-th visual image to obtain the (i + 1)-th warping result; Generate the i-th visual image based on the (i + 1)-th warping result, the depth map, semantic map, and normal map included in the i-th virtual camera key view.

5. The method as described in claim 1, characterized in that, For each of the visual images, converting each pixel point of the visual image into a particle point in the world coordinate system to obtain a sparse particle field includes: For each pixel point, calculate the particle point of the pixel point in the world coordinate system according to the coordinate value of the pixel point in the visual image to which it belongs, the depth value of the pixel point, the camera internal parameter matrix, and the camera pose transformation matrix; Multiple of the particle points form the sparse particle field.

6. A city model generation device, characterized in that, The device includes: A data acquisition unit, configured to acquire urban grid model data and determine the main street corresponding to the urban grid model data; A sequence acquisition unit, configured to acquire a key view sequence corresponding to the main street, where the key view sequence includes multiple virtual camera key views, the multiple virtual camera key views are obtained by a virtual camera moving along the main street for shooting, and the distance intervals between any two of the multiple virtual camera key views are the same; A visual image generation unit, configured to generate multiple visual images based on the key view sequence; A sparse particle field unit, configured to, for each of the visual images, convert each pixel point of the visual image into a particle point in the world coordinate system to obtain a sparse particle field; A dense particle field unit, configured to fill at least one intermediate view between any two adjacent virtual camera key views in the multiple virtual camera key views, where for each of the intermediate views, a first distance of the intermediate view is equal to a second distance, the first distance refers to the distance between the intermediate view and the previous view of the intermediate view, and the second distance refers to the distance between the intermediate view and the next view of the intermediate view; Render multiple of the intermediate views based on the sparse particle field to obtain corresponding intermediate visual images; For each of the intermediate visual images, convert each pixel point of the intermediate visual image into a particle point in the world coordinate system to obtain intermediate image particle points; Fill the sparse particle field based on the intermediate image particle points to obtain a dense particle field; A particleized model unit, configured to render all of the virtual camera key views and all of the intermediate views based on the dense particle field to obtain corresponding rendered visual images; Perform noise addition processing on the rendered visual images to obtain a noisy image; Perform joint denoising on the noisy image to obtain a denoised image; Obtain a rendering loss based on the rendered visual images and the denoised image; Based on the rendering loss, reversely update the parameters of the particles in the dense particle field to obtain an urban particleized model.

7. An electronic device, characterized in that, It includes a processor and a memory, the memory storing instructions; the processor loads instructions from the memory to perform the steps in the city model generation method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions adapted for loading by a processor to perform the steps of the city model generation method according to any one of claims 1 to 5.

9. A computer program product, characterized in that, The method includes instructions that, when executed by a processor, implement the steps of the city model generation method according to any one of claims 1 to 5.

Images