Virtual display method and system for realizing smart city through three-dimensional visualization

By using drones to collect and process remote sensing images, a 3D model of a smart city is constructed and rendered, solving the problem of incomplete information display in the virtual display of smart cities and realizing a richer and more detailed display of urban information.

CN117593465BActive Publication Date: 2024-11-26极视创智(深圳)科技有限公司
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202311718450.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-12-13
Publication Date
2024-11-26
Estimated Expiration
2043-12-13

AI Technical Summary

Technical Problem

Existing methods for displaying smart city information cannot fully showcase the city's information, resulting in a lack of intelligence and convenience in information display.

Method used

By collecting remote sensing images using drones, image preprocessing, moving target detection and repair, overlapping building segmentation and fusion, spatial 3D coordinate construction and color rendering are performed to generate a 3D visualization model and mark city attributes to achieve a virtual display of a smart city.

Benefits of technology

It increases the amount of information displayed in smart city presentations, making the 3D virtual display richer and more detailed, and clearly showcasing the city's geographical information and attributes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117593465B_ABST
    Figure CN117593465B_ABST
Patent Text Reader

Abstract

The present invention relates to the field of virtual display technology, and discloses a method and system for realizing virtual display of a smart city by three-dimensional visualization, wherein the method comprises: collecting a smart city image and performing image preprocessing to obtain a preprocessed image; detecting and removing moving targets in the image, performing image repair on the preprocessed image to obtain a repaired image; overlapping segmentation and fusion of the repaired image based on the smart city coordinates to obtain a fused image; identifying the city center point in the fused image, constructing the spatial three-dimensional coordinates of the smart city based on the city center point, connecting the coordinates, obtaining a spatial three-dimensional structure, and performing color and texture rendering to obtain a three-dimensional visualization model; marking the city attributes in the three-dimensional visualization model, loading the three-dimensional visualization model and the city attributes into a visualization platform, performing three-dimensional virtual display of the smart city, and obtaining the three-dimensional virtual display result of the smart city. The present invention improves the amount of information displayed in the smart city.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of virtual display technology, and in particular to a virtual display method and system for realizing a smart city through three-dimensional visualization. Background Art

[0002] The virtual display of a three-dimensional city can use data visualization tools such as charts and maps to present data related to the smart city in an intuitive way.

[0003] At present, the virtual display of smart cities is mainly based on smartphone applications or websites, allowing users to quickly and easily obtain various city services and information anytime and anywhere, such as bus inquiries, parking reservations, express inquiries, etc. This display method can increase the convenience and intelligence of the city. However, this method is not comprehensive enough for the display of smart city information. Summary of the invention

[0004] In order to solve the above problems, the present invention provides a three-dimensional visualization method and system for realizing virtual display of a smart city, which can increase the amount of displayed information of the smart city.

[0005] In a first aspect, the present invention provides a three-dimensional visualization method for realizing a virtual display of a smart city, comprising:

[0006] Extracting the flight trajectory of a drone used to collect remote sensing images of a smart city, selecting a target flight trajectory from the flight trajectories based on the texture quality of the remote sensing image, collecting a target remote sensing image of the smart city and the city coordinates of the target remote sensing image using the drone based on the target flight trajectory, and performing image preprocessing on the target remote sensing image to obtain a preprocessed image;

[0007] Detecting a moving target in the preprocessed image, and after removing the moving target from the preprocessed image, performing image restoration on an image module corresponding to the moving target in the preprocessed image to obtain a restored image;

[0008] Based on the city coordinates, overlapping buildings in the repaired image are segmented to obtain a segmented image, and each segmented image in the segmented image is fused to obtain a fused image;

[0009] Identify the city center point in the fused image, construct the spatial three-dimensional coordinates of the smart city based on the first mapping relationship between the city center point and the city coordinates, and use the second mapping relationship between the pixel point coordinates in the fused image and the spatial three-dimensional coordinates to connect each spatial three-dimensional coordinate in the spatial three-dimensional coordinates to obtain a spatial three-dimensional structure;

[0010] Performing color rendering on the spatial three-dimensional structure to obtain a color rendering structure, performing texture rendering on the color rendering structure to obtain a texture rendering structure, and using the texture rendering structure as a three-dimensional visualization model of the smart city;

[0011] Mark the city attributes in the three-dimensional visualization model, load the three-dimensional visualization model and the city attributes into a pre-built visualization platform, use the visualization platform to complete the three-dimensional virtual display of the smart city, and obtain the three-dimensional virtual display result of the smart city.

[0012] In a possible implementation manner of the first aspect, performing image preprocessing on the target remote sensing image to obtain a preprocessed image includes:

[0013] The target remote sensing image is filtered using the following formula to obtain a filtered image:

[0014]

[0015] Wherein, ω represents the filtered image, M represents the total number of pixels of the target remote sensing image, arg represents the window function, (i, j) represents the pixel to be filtered in the target remote sensing image, Q(i, j) represents the pixel grayscale value of the pixel to be filtered,

[0016] The filtered image is tilt-corrected to obtain a pre-processed image.

[0017] In a possible implementation manner of the first aspect, performing tilt correction on the filtered image to obtain a preprocessed image includes:

[0018] Taking the image center point of the filtered image as the coordinate origin and taking the image diagonal point of the filtered image as the plane;

[0019] Based on the coordinate origin and the plane, a spatial coordinate system of the filtered image is constructed, and the filtered image is rotated using the following formula to obtain a rotation coordinate point:

[0020]

[0021] Among them, A 2 , B 2 , C 2 Represents the rotated coordinate point after the coordinate rotation, A 1 , B 1 , C 1 represents the original spatial coordinate point of the filtered image in the spatial coordinate system, ε A , ε B , ε C represents the rotation angle in the rotation matrix,

[0022] Based on the rotation coordinate point, the tilt angle of the filtered image is reversely rotated to obtain a tilt-corrected image.

[0023] In a possible implementation manner of the first aspect, detecting a moving target in the preprocessed image includes:

[0024] The pre-processed image is input into a pre-configured convolutional neural network to identify the image features of the pre-processed image using the convolutional neural network in combination with the following formula:

[0025]

[0026] in, represents vector features, f represents activation function, A, B, C represent the length, width, and height of the convolution kernel cube in the convolution model, is the value at the position (a, b, c) on the image and the calculated value of the k-th convolution kernel of the u-th layer, is the value at the position of the image (α+a, β+b, γ+c) and the calculated value of the mth convolution kernel in the u-1th layer of the convolution kernel, C ku represents a hyperparameter of the kth convolution kernel in the uth layer, α, β, and γ represent the offset of the position (a, b, c).

[0027] According to the image features, image information of the tilt-corrected image is identified, and a corresponding moving target in the image information is identified.

[0028] In a possible implementation manner of the first aspect, performing image restoration on an image module corresponding to the moving target in the preprocessed image to obtain a restored image includes:

[0029] Querying a historical image corresponding to the preprocessed image, extracting a region in the historical image corresponding to a missing portion of the preprocessed image, and obtaining a compensation image;

[0030] Calculating a pixel difference between the compensation image and the preprocessed image, and performing pixel correction on the compensation image based on the pixel difference to obtain a corrected image;

[0031] The correction image is used to repair the missing part in the pre-processed image to obtain a repaired image.

[0032] In a possible implementation manner of the first aspect, performing overlapping segmentation on overlapping buildings in the repaired image based on the city coordinates to obtain a segmented image includes:

[0033] Converting the repaired image into a grayscale image, performing feature matching on the grayscale image, and obtaining a feature matching result;

[0034] Based on the feature matching result, the overlapping buildings in the repaired image are determined, and the overlapping buildings are overlapped and segmented to obtain a segmented image.

[0035] In a possible implementation manner of the first aspect, the fusing each segmented image in the segmented images to obtain a fused image includes:

[0036] Marking feature points of the segmented image to obtain marked points;

[0037] A marker conversion matrix of the segmented image is constructed, and the segmented image is fused using the marker conversion matrix and the marker points to obtain a fused image.

[0038] In a possible implementation manner of the first aspect, the step of performing image fusion on the segmented image by using the marker transformation matrix and the marker points to obtain a fused image includes:

[0039] The mark conversion matrix is ​​used in combination with the following formula to convert the mark points to obtain conversion marks:

[0040]

[0041] in, represents the conversion mark, α 1 represents the first marker point before the transformation, β 1 represents the second marker point before the transformation, α 2 represents the first marker point after the transformation, β 2 represents the second marker point after the transformation, μ 1 , μ 2 represents the rotation transformation parameter, μ 3 , μ 4 represents the translation transformation parameter, μ 5 , μ 6 represents the scale transformation parameter, μ 7 , μ 8 represents the perspective transformation parameters, represents the label transformation matrix,

[0042] The segmented images are fused using the conversion marks to obtain a fused image.

[0043] In a second aspect, the present invention provides a virtual display system for realizing a smart city through three-dimensional visualization, the system comprising:

[0044] An image preprocessing module is used to extract the flight trajectory of the UAV used to collect the remote sensing image of the smart city, select a target flight trajectory from the flight trajectories based on the texture quality of the remote sensing image, collect the target remote sensing image of the smart city and the city coordinates of the target remote sensing image by using the UAV based on the target flight trajectory, and perform image preprocessing on the target remote sensing image to obtain a preprocessed image;

[0045] An image restoration module is used to detect a moving target in the preprocessed image, and after removing the moving target from the preprocessed image, perform image restoration on an image module corresponding to the moving target in the preprocessed image to obtain a restored image;

[0046] An image fusion module, configured to perform overlapping segmentation on the overlapping buildings in the repaired image based on the city coordinates to obtain a segmented image, and fuse each segmented image in the segmented image to obtain a fused image;

[0047] a three-dimensional structure construction module, configured to identify a city center point in the fused image, construct the spatial three-dimensional coordinates of the smart city based on a first mapping relationship between the city center point and the city coordinates, and connect each of the spatial three-dimensional coordinates by using a second mapping relationship between the pixel point coordinates in the fused image and the spatial three-dimensional coordinates to obtain a spatial three-dimensional structure;

[0048] The model rendering module is used to perform color rendering on the three-dimensional structure of the space to obtain a color rendering structure, perform texture rendering on the color rendering structure to obtain a texture rendering structure, and use the texture rendering structure as a three-dimensional visualization model of the smart city.

[0049] The three-dimensional display module is used to mark the city attributes in the three-dimensional visualization model, load the three-dimensional visualization model and the city attributes into a pre-built visualization platform, use the visualization platform to complete the three-dimensional virtual display of the smart city, and obtain the three-dimensional virtual display result of the smart city.

[0050] Compared with the prior art, the technical principle and beneficial effects of this solution are:

[0051] The embodiment of the present invention can provide a large number of drone flight plans for urban information collection of the smart city by extracting the flight trajectory of the drone used to collect the remote sensing image of the smart city, and then a more reasonable plan can be selected. Furthermore, the embodiment of the present invention can obtain a relatively excellent aerial photography path by selecting the target flight trajectory from the flight trajectory based on the texture quality of the remote sensing image. The embodiment of the present invention can identify the target that is easy to cause interference in the image by detecting the moving target in the preprocessed image. The moving target refers to the non-fixed building such as pedestrians and vehicles in the corresponding urban geography in the image. The embodiment of the present invention can remove the overlapping parts of the aerial image when the smart city is aerially photographed by performing overlapping segmentation on the overlapping buildings in the repaired image based on the city coordinates to obtain a segmented image. Furthermore, the embodiment of the present invention can obtain the city center point by identifying the city center point in the fused image. The smart city provides basic construction points during three-dimensional display, so that the distribution of corresponding urban buildings in the three-dimensional map is more uniform. The embodiment of the present invention can construct the basic framework for the virtual display of the smart city by constructing the spatial three-dimensional coordinates of the smart city based on the first mapping relationship between the city center point and the city coordinates, wherein the first mapping relationship refers to the correspondence between the two-dimensional coordinates of the image and the three-dimensional coordinates of the real space. Further, the embodiment of the present invention performs color rendering on the spatial three-dimensional structure to obtain a color rendering structure, which can make the color of the spatial three-dimensional structure richer and fuller, and the visual effect is better, wherein the color rendering refers to changing the color and saturation of the image and applying various visual effects to enhance the sensory expression of the image, and marking the city attributes in the three-dimensional visualization model can understand the detailed information of each region when viewing the city model, such as the name of the building, the name of the street, the river, the lake, etc. Among them, the city attribute refers to the information used to describe various data in the city, such as the name of the city road, the distance, and the name of the landmark building and the area where it is located. Therefore, the virtual display method and system of a three-dimensional visualization of a smart city proposed in the embodiment of the present invention can increase the amount of information displayed in the smart city. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0054] Figure 1A schematic diagram of a flow chart of a method for realizing a virtual display of a smart city through three-dimensional visualization provided by an embodiment of the present invention;

[0055] Figure 2 A schematic diagram of a module of a virtual display system for realizing a smart city through three-dimensional visualization provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0056] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0057] The embodiment of the present invention provides a method for realizing a virtual display of a smart city by three-dimensional visualization, and the execution subject of the method for realizing a virtual display of a smart city by three-dimensional visualization includes but is not limited to at least one of the electronic devices such as a server and a terminal that can be configured to execute the method provided by the embodiment of the present invention. In other words, the method for realizing a virtual display of a smart city by three-dimensional visualization can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, etc. The server can be an independent server, or it can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0058] See also Figure 1 FIG. 1 is a flow chart of a method for realizing a virtual display of a smart city by three-dimensional visualization provided by an embodiment of the present invention. Figure 1 The three-dimensional visualization method described in the paper realizes the virtual display method of smart city, including:

[0059] S1. Extract the flight trajectory of a drone used to collect remote sensing images of a smart city, select a target flight trajectory from the flight trajectories based on the texture quality of the remote sensing image, use the drone to collect the target remote sensing image of the smart city and the city coordinates of the target remote sensing image based on the target flight trajectory, and perform image preprocessing on the target remote sensing image to obtain a preprocessed image.

[0060] The embodiment of the present invention can provide a large number of drone flight plans for the city information collection of the smart city by extracting the flight trajectory of the drone used to collect the remote sensing image of the smart city, and then select a more reasonable plan. The flight trajectory refers to the flight path of the drone used for urban aerial photography.

[0061] Optionally, the flight trajectory of the drone of the remote sensing image of the smart city is obtained by querying a database of the drone shooting plan.

[0062] Furthermore, the embodiment of the present invention can obtain a relatively good aerial photography path by selecting a target flight trajectory from the flight trajectory based on the texture quality of the remote sensing image, wherein the texture quality refers to the clarity and richness of the texture information in the image.

[0063] Optionally, the process of selecting the target flight trajectory from the flight trajectory based on the texture quality of the remote sensing image is: identifying the image resolution of the remote sensing image, determining the texture quality of the remote sensing image based on the image resolution, and selecting the remote sensing image whose texture quality meets the preset requirements, querying the shooting order of the remote sensing images that meet the preset requirements, and determining the target flight trajectory based on the shooting order. It should be noted that the resolution of the remote sensing image is calculated using image processing software such as Photoshop, Adobe, etc.

[0064] The embodiment of the present invention can geographically locate the smart city by using the drone to collect the target remote sensing image of the smart city and the city coordinates of the target remote sensing image based on the target flight trajectory, understand the geographic information of the smart city, and provide basic data support for city digitization. Among them, the city coordinates refer to the coordinates of the drone when taking the target remote sensing image, including Z-axis height, X-axis coordinates, and Y-axis coordinates.

[0065] In the embodiment of the present invention, by performing image preprocessing on the target remote sensing image, a preprocessed image can be obtained to obtain an image with higher definition, thereby better identifying the image content. The preprocessing refers to processing such as screening, filtering, tilt correction, etc. before using the image.

[0066] As an embodiment of the present invention, the performing image preprocessing on the target remote sensing image to obtain the preprocessed image includes: performing image filtering on the target remote sensing image using the following formula to obtain a filtered image:

[0067]

[0068] Wherein, ω represents the filtered image, M represents the total number of pixels of the target remote sensing image, arg represents the window function, (i, j) represents the pixel to be filtered in the target remote sensing image, Q(i, j) represents the pixel grayscale value of the pixel to be filtered,

[0069] The filtered image is tilt-corrected to obtain a pre-processed image.

[0070] Optionally, the image filtering performed on the target remote sensing image refers to removing noise influence in the image, and the tilt correction performed on the filtered image refers to correcting the distortion of a part of the image area caused by a shooting angle.

[0071] Furthermore, in another optional embodiment of the present invention, the tilt correction of the filtered image to obtain the preprocessed image includes: taking the image center point of the filtered image as the coordinate origin, and taking the image diagonal point of the filtered image as the plane, constructing the spatial coordinate system of the filtered image based on the coordinate origin and the plane, and performing coordinate rotation on the filtered image using the following formula to obtain the rotated coordinate point:

[0072]

[0073] Among them, A 2 , B 2 , C 2 Represents the rotated coordinate point after the coordinate rotation, A 1 , B 1 , C 1 represents the original spatial coordinate point of the filtered image in the spatial coordinate system, ε A , ε B , ε C represents the rotation angle in the rotation matrix,

[0074] Based on the rotation coordinate point, the tilt angle of the filtered image is reversely rotated to obtain a tilt-corrected image.

[0075] The spatial coordinate system refers to the description of the position and direction of an object in three-dimensional space. It should be noted that due to the distortion phenomenon, the image is not a plane, and three-dimensional coordinates are needed to find the part beyond the plane.

[0076] Optionally, the spatial coordinate system of the filtered image is constructed using SQL language.

[0077] S2. Detecting a moving target in the preprocessed image, and after removing the moving target from the preprocessed image, performing image restoration on an image module corresponding to the moving target in the preprocessed image to obtain a restored image.

[0078] The embodiment of the present invention can identify targets in the image that are likely to cause interference by detecting the moving targets in the preprocessed image, wherein the moving targets refer to non-fixed buildings such as pedestrians, vehicles, etc. in the corresponding urban geography in the image.

[0079] As an embodiment of the present invention, the detecting the moving target in the preprocessed image includes: inputting the preprocessed image into a preconfigured convolutional neural network, so as to use the convolutional neural network in combination with the following formula to identify the image features of the preprocessed image:

[0080]

[0081] in, represents vector features, f represents activation function, A, B, C represent the length, width, and height of the convolution kernel cube in the convolution model, is the value at the position (a, b, c) on the image and the calculated value of the k-th convolution kernel of the u-th layer, is the value at the position of the image (α+a, β+b, γ+c) and the calculated value of the mth convolution kernel in the u-1th layer of the convolution kernel, C ku represents a hyperparameter of the kth convolution kernel in the uth layer, α, β, and γ represent the offset of the position (a, b, c).

[0082] According to the image features, image information of the tilt-corrected image is identified, and a corresponding moving target in the image information is identified.

[0083] As an embodiment of the present invention, performing image restoration on the image module corresponding to the moving target in the preprocessed image to obtain a restored image includes: querying a historical image corresponding to the preprocessed image, extracting an area in the historical image corresponding to a missing portion of the preprocessed image to obtain a compensation map, calculating a pixel difference between the compensation map and the preprocessed image, performing pixel correction on the compensation map based on the pixel difference to obtain a correction map, and performing image missing restoration on the missing portion in the preprocessed image using the correction map to obtain a restored image.

[0084] The historical image refers to the image data of the historical aerial photography of the pre-processed image, and the compensation image refers to the image used to compensate for the missing information of the pre-processed image.

[0085] Optionally, the historical image corresponding to the preprocessed image is acquired by accessing an aerial photography database corresponding to the preprocessed image, the area in the historical image corresponding to the missing part of the preprocessed image is extracted to obtain a compensation image by overlapping the historical image with the preprocessed image and stroking the missing area of ​​the preprocessed image to determine the range of the compensation image, the pixel difference between the compensation image and the preprocessed image is calculated by converting the compensation image and the preprocessed image into grayscale images to obtain a compensated grayscale image and a preprocessed grayscale image, and the pixel difference between the compensated grayscale image and the preprocessed grayscale image is calculated.

[0086] S3. Based on the city coordinates, overlapping buildings in the repaired image are segmented to obtain a segmented image, and each segmented image in the segmented image is fused to obtain a fused image.

[0087] The embodiment of the present invention performs overlapping segmentation on the overlapping buildings in the repaired image based on the city coordinates to obtain a segmented image, which can remove the overlapping parts of the aerial image when the smart city is aerially photographed.

[0088] As an embodiment of the present invention, the overlapping buildings in the repaired image are overlapped and segmented based on the city coordinates to obtain a segmented image, including: converting the repaired image into a grayscale image, performing feature matching on the grayscale image to obtain a feature matching result, determining the overlapping buildings in the repaired image based on the feature matching result, and performing overlapping segmentation on the overlapping buildings to obtain a segmented image.

[0089] Among them, the grayscale image refers to an image that converts pixel information of a color image into a grayscale level. Optionally, the feature matching of the grayscale image refers to matching the features of the two images, and a nearest neighbor search algorithm (such as brute force matching or kd tree) can be used to find the best matching point of each feature point in the other image. The overlapping buildings are overlapped and segmented to obtain a segmented image. The overlapping building part is framed by a matlab tool to obtain a framed image, and then the framed image is cut using the cutting function in the matlab tool.

[0090] Furthermore, the embodiment of the present invention can combine scattered aerial images into a complete city image by fusing each segmented image in the segmented images to obtain a fused image.

[0091] As an embodiment of the present invention, the fusing of each segmented image in the segmented images to obtain a fused image includes: marking feature points of the segmented images to obtain marking points, constructing a marking transformation matrix of the segmented images, and performing image fusion on the segmented images using the marking transformation matrix and the marking points to obtain a fused image.

[0092] The feature point refers to the highest point and the lowest point of the segmented area after image segmentation, and the fusion matrix refers to a mathematical matrix used to achieve fusion between image points.

[0093] Further, as another optional embodiment of the present invention, the step of using the label conversion matrix and the label points to perform image fusion on the segmented image to obtain a fused image includes: using the label conversion matrix in combination with the following formula to perform label conversion on the label points to obtain a conversion label:

[0094]

[0095] in, represents the conversion mark, α 1 represents the first marker point before the transformation, β 1 represents the second marker point before the transformation, α 2 represents the first marker point after the transformation, β 2 represents the second marker point after the transformation, μ 1 , μ 2 represents the rotation transformation parameter, μ 3 , μ 4 represents the translation transformation parameter, μ 5 , μ 6 represents the scale transformation parameter, μ 7 , μ 8 represents the perspective transformation parameters, Represents the marker transformation matrix.

[0096] The segmented images are fused using the conversion marks to obtain a fused image.

[0097] The conversion mark refers to a fusion mark point during image fusion to avoid mismatch in fusion.

[0098] Optionally, the segmented image is marked with feature points, and the marked points are marked by markers constructed with binary codes, the transformation matrix of the segmented image is constructed by java, and the segmented images are fused using the transformation markers to obtain a fused image by making one-to-one correspondence between the fusion marker points in the segmented images to obtain an accurate fusion position, and based on the accurate fusion position, the segmented images are spliced ​​using opencv tools to obtain a fused image.

[0099] S4. Identify the city center point in the fused image, construct the spatial three-dimensional coordinates of the smart city based on a first mapping relationship between the city center point and the city coordinates, and use a second mapping relationship between the pixel point coordinates in the fused image and the spatial three-dimensional coordinates to connect the coordinates of each spatial three-dimensional coordinate to obtain a spatial three-dimensional structure.

[0100] The embodiment of the present invention can provide a basic construction point for the smart city in three-dimensional display by identifying the city center point in the fused image, so that the distribution of corresponding urban buildings in the three-dimensional image is more uniform.

[0101] As an embodiment of the present invention, the identifying the city center point in the fused image includes: detecting image edge points of the fused image, connecting the image edge points to obtain a closed figure, and calculating the city center point of the closed figure using the following formula:

[0102]

[0103]

[0104] Wherein, (x′, y′) represents the city center point, x′ represents the horizontal coordinate of the city center point, y′ represents the vertical coordinate of the city center point, x represents the independent variable on the horizontal axis, y represents the independent variable on the vertical axis, D represents the plane area where the closed figure is located, s D represents the area of ​​the closed figure, and u(x, y) represents the density.

[0105] Optionally, the detecting of the image edge points of the fused image is performed by using a Canny image edge detection technique.

[0106] Furthermore, the embodiment of the present invention can construct a basic framework for the virtual display of the smart city by constructing the spatial three-dimensional coordinates of the smart city based on the first mapping relationship between the city center point and the city coordinates, wherein the first mapping relationship refers to the correspondence between the two-dimensional coordinates of the image and the three-dimensional coordinates of the real space.

[0107] As an embodiment of the present invention, constructing the spatial three-dimensional coordinates of the smart city based on the first mapping relationship between the city center point and the city coordinates includes: calculating the coordinate center point of the city coordinates using the following formula:

[0108]

[0109] in, represents the coordinate center point, q 1 ,q 2 ,......,q n represents the city coordinates, and n represents the total number of the city coordinates;

[0110] The first mapping matrix corresponding to the first mapping relationship between the city center point and the coordinate center point is calculated using the following formula:

[0111]

[0112] Wherein, M represents the first mapping matrix, (x′, y′) represents the city center point, represents the coordinate center point;

[0113] The pixel points in the fused image are converted into three-dimensional space points by using the first mapping matrix; the three-dimensional space points are proportionally reduced to obtain the three-dimensional coordinates of the space.

[0114] Furthermore, the embodiment of the present invention uses the second mapping relationship between the pixel point coordinates in the fused image and the spatial three-dimensional coordinates to coordinately connect each of the spatial three-dimensional coordinates, and obtains a spatial three-dimensional structure that can be used to construct a basic display layout for the three-dimensional model of the smart city. The second mapping relationship refers to the coordinate correspondence between the spatial three-dimensional coordinates of the smart city and the pixel point coordinates in the fused image.

[0115] Optionally, the second mapping relationship between the pixel point coordinates in the fused image and the spatial three-dimensional coordinates is used to connect the coordinates of each spatial three-dimensional coordinate in the spatial three-dimensional coordinates to obtain a spatial three-dimensional structure. When the pixel point coordinates in the fused image belong to the coordinates of the same building, the spatial three-dimensional coordinates corresponding to the pixel point coordinates in the fused image are connected, thereby forming a connection of the three-dimensional spatial coordinates of the same building.

[0116] S5. Perform color rendering on the spatial three-dimensional structure to obtain a color rendering structure, perform texture rendering on the color rendering structure to obtain a texture rendering structure, and use the texture rendering structure as a three-dimensional visualization model of the smart city.

[0117] The embodiment of the present invention performs color rendering on the spatial three-dimensional structure to obtain a color rendering structure, which can make the color of the spatial three-dimensional structure richer and fuller, and the visual effect better, wherein the color rendering refers to enhancing the sensory expression of the image by changing the color and saturation of the image and applying various visual effects.

[0118] Optionally, color rendering is performed on the spatial three-dimensional structure to obtain a color rendering structure, and color rendering is performed on the spatial three-dimensional structure through functions such as applying filters, adjusting tone and hue, etc. in a gimp tool.

[0119] Furthermore, the embodiment of the present invention performs texture rendering on the color rendering structure to obtain a texture rendering structure, which can allow the spatial three-dimensional structure to clearly display details and structure, and the texture area and the background area can be clearly separated.

[0120] As an embodiment of the present invention, the color rendering structure is texture rendered to obtain a texture rendering structure, including: performing gradient mapping on the color rendering structure to obtain a gradient structure, performing bump mapping on the gradient structure to obtain a bump structure, performing normal mapping on the bump structure to obtain a normal structure, and performing texture projection on the normal structure to obtain a texture rendering structure.

[0121] Optionally, performing gradient mapping on the color rendering structure to obtain a gradient structure refers to a rendering method of applying a gradient texture to the surface of a model, which is implemented by the mapping tool of 3dmax; performing concave-convex mapping on the gradient structure to obtain a concave-convex structure refers to a rendering technology of simulating tiny concave-convex details by changing the normal vector of the model surface, which is implemented by a ps tool; performing normal mapping on the concave-convex structure to obtain a normal structure refers to a technology for enhancing the surface details of a model, which is implemented by a ps tool; performing texture projection on the normal structure refers to adding effects such as shadows and reflections to the surface of the model to enhance the realism of the rendering, which is implemented by the projection function of the threejs tool.

[0122] S6. Mark the city attributes in the three-dimensional visualization model, load the three-dimensional visualization model and the city attributes into a pre-built visualization platform, use the visualization platform to complete the three-dimensional virtual display of the smart city, and obtain the three-dimensional virtual display result of the smart city.

[0123] In the embodiment of the present invention, by marking the city attributes in the three-dimensional visualization model, detailed information of each region such as building names, street names, rivers, lakes, etc. can be understood when viewing the city model. The city attributes refer to information used to describe various data in the city, such as the names of city roads, distances, and the names and areas of landmark buildings.

[0124] As an embodiment of the present invention, marking the city attributes in the three-dimensional visualization model includes: querying the geographic spatial data system of the city corresponding to the three-dimensional visualization model, using the geographic spatial data system to identify the area name of the corresponding area in the three-dimensional visualization model, and based on the area name, marking the city attributes in the three-dimensional visualization model through the geographic information system.

[0125] The geospatial data system refers to a big data system used to collect statistics on geographic information.

[0126] Furthermore, the embodiment of the present invention can display the data on the pre-built visualization platform by loading the three-dimensional visualization model and the city attributes into the visualization platform, wherein the visualization platform refers to a software tool or system for displaying and presenting data, information or knowledge, which can convert complex data into intuitive forms such as visual graphics, charts and maps to help users better understand the data.

[0127] Optionally, the three-dimensional visualization model and the city attributes are loaded into a pre-built visualization platform and connected to the pre-built visualization platform through a local data system corresponding to the three-dimensional visualization model and the city attributes for local data transmission.

[0128] The embodiment of the present invention uses the visualization platform to complete the three-dimensional virtual display of the smart city, and the three-dimensional virtual display result of the smart city can obtain a complete visualized three-dimensional model of the smart city, so that various information of the city can be observed according to the visualized three-dimensional model, such as building layout, urban construction progress, urban scenery, etc.

[0129] It can be seen that the embodiment of the present invention can provide a large number of drone flight plans for the urban information collection of the smart city by extracting the flight trajectory of the drone used to collect the remote sensing image of the smart city, and then a more reasonable plan can be selected. Furthermore, the embodiment of the present invention can obtain a relatively excellent aerial photography path by selecting the target flight trajectory from the flight trajectory based on the texture quality of the remote sensing image. The embodiment of the present invention can identify the targets in the image that are prone to interference by detecting the moving targets in the preprocessed image. The moving targets refer to non-fixed buildings in the corresponding urban geography in the image, such as pedestrians and vehicles. The embodiment of the present invention performs overlapping segmentation on the overlapping buildings in the repaired image based on the city coordinates to obtain a segmented image, which can remove the overlapping parts of the aerial image when the smart city is aerially photographed. Furthermore, the embodiment of the present invention identifies the city center point in the fused image It can provide basic construction points for the smart city in three-dimensional display, so that the distribution of corresponding urban buildings in the three-dimensional map is more uniform. The embodiment of the present invention can construct the basic framework for the virtual display of the smart city by constructing the spatial three-dimensional coordinates of the smart city based on the first mapping relationship between the city center point and the city coordinates, wherein the first mapping relationship refers to the correspondence between the two-dimensional coordinates of the image and the three-dimensional coordinates of the real space. Further, the embodiment of the present invention can make the color of the three-dimensional structure of the space richer and fuller, and the visual effect is better by color rendering the three-dimensional structure of the space to obtain the color rendering structure, wherein the color rendering refers to changing the color and saturation of the image and applying various visual effects to enhance the sensory expression of the image, and marking the city attributes in the three-dimensional visualization model can understand the detailed information of each region when viewing the city model, such as the name of the building, the name of the street, the river, the lake, etc. Among them, the city attribute refers to the information used to describe various data in the city, such as the name of the city road, the distance, and the name of the landmark building and the area where it is located. Therefore, the virtual display method and system of a three-dimensional visualization of a smart city proposed in the embodiment of the present invention can increase the amount of information displayed in the smart city.

[0130] like Figure 2 As shown, it is a functional module diagram of the virtual display system for realizing smart city through three-dimensional visualization of the present invention.

[0131] The virtual display system 200 for realizing smart city by three-dimensional visualization of the present invention can be installed in an electronic device. According to the functions implemented, the virtual display system for realizing smart city by three-dimensional visualization can include an image preprocessing module 201, an image repair module 202, an image fusion module 203, a three-dimensional structure construction module 204, a model rendering module 205 and a three-dimensional display module 206. The module of the present invention can also be called a unit, which refers to a series of computer program segments that can be executed by a processor of an electronic device and can complete fixed functions, which are stored in the memory of the electronic device.

[0132] In the embodiment of the present invention, the functions of each module / unit are as follows:

[0133] The image preprocessing module 201 is used to extract the flight trajectory of the drone used to collect the remote sensing image of the smart city, select a target flight trajectory from the flight trajectories based on the texture quality of the remote sensing image, collect the target remote sensing image of the smart city and the city coordinates of the target remote sensing image by using the drone based on the target flight trajectory, and perform image preprocessing on the target remote sensing image to obtain a preprocessed image;

[0134] The image restoration module 202 is used to detect the moving target in the pre-processed image, and after removing the moving target from the pre-processed image, perform image restoration on the image module corresponding to the moving target in the pre-processed image to obtain a restored image;

[0135] The image fusion module 203 is used to perform overlapping segmentation on the overlapping buildings in the repaired image based on the city coordinates to obtain a segmented image, and fuse each segmented image in the segmented image to obtain a fused image;

[0136] The three-dimensional structure construction module 204 is used to identify the city center point in the fused image, construct the spatial three-dimensional coordinates of the smart city based on the first mapping relationship between the city center point and the city coordinates, and use the second mapping relationship between the pixel point coordinates in the fused image and the spatial three-dimensional coordinates to connect the coordinates of each spatial three-dimensional coordinate to obtain a spatial three-dimensional structure;

[0137] The model rendering module 205 is used to perform color rendering on the spatial three-dimensional structure to obtain a color rendering structure, perform texture rendering on the color rendering structure to obtain a texture rendering structure, and use the texture rendering structure as the three-dimensional visualization model of the smart city;

[0138] The three-dimensional display module 206 is used to mark the city attributes in the three-dimensional visualization model, load the three-dimensional visualization model and the city attributes into a pre-built visualization platform, use the visualization platform to complete the three-dimensional virtual display of the smart city, and obtain the three-dimensional virtual display result of the smart city.

[0139] In detail, the modules in the virtual display system 200 for realizing smart city by three-dimensional visualization in the embodiment of the present invention are used in the same manner as above. Figure 1 The three-dimensional visualization described in the text realizes the virtual display method of the smart city by the same technical means and can produce the same technical effects, so it will not be repeated here.

[0140] The present invention further provides a storage medium, wherein the readable storage medium stores a computer program, and when the computer program is executed by a processor of an electronic device, the computer program can implement:

[0141] Extracting the flight trajectory of a drone used to collect remote sensing images of a smart city, selecting a target flight trajectory from the flight trajectories based on the texture quality of the remote sensing image, collecting a target remote sensing image of the smart city and the city coordinates of the target remote sensing image using the drone based on the target flight trajectory, and performing image preprocessing on the target remote sensing image to obtain a preprocessed image;

[0142] Detecting a moving target in the preprocessed image, and after removing the moving target from the preprocessed image, performing image restoration on an image module corresponding to the moving target in the preprocessed image to obtain a restored image;

[0143] Based on the city coordinates, overlapping buildings in the repaired image are segmented to obtain a segmented image, and each segmented image in the segmented image is fused to obtain a fused image;

[0144] Identify the city center point in the fused image, construct the spatial three-dimensional coordinates of the smart city based on the first mapping relationship between the city center point and the city coordinates, and use the second mapping relationship between the pixel point coordinates in the fused image and the spatial three-dimensional coordinates to connect each spatial three-dimensional coordinate in the spatial three-dimensional coordinates to obtain a spatial three-dimensional structure;

[0145] Performing color rendering on the spatial three-dimensional structure to obtain a color rendering structure, performing texture rendering on the color rendering structure to obtain a texture rendering structure, and using the texture rendering structure as a three-dimensional visualization model of the smart city;

[0146] Mark the city attributes in the three-dimensional visualization model, load the three-dimensional visualization model and the city attributes into a pre-built visualization platform, use the visualization platform to complete the three-dimensional virtual display of the smart city, and obtain the three-dimensional virtual display result of the smart city.

[0147] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0148] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0149] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0150] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0151] Therefore, no matter from which point of view, the embodiments should be regarded as illustrative and non-restrictive, and the scope of the present invention is limited by the appended claims rather than the above description, so it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any attached figure mark in the claims should not be regarded as limiting the claims involved.

[0152] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0153] The above description is only a specific embodiment of the present invention, so that those skilled in the art can understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but should conform to the widest scope consistent with the principles and novel features invented herein.

Claims

1. A virtual display method for realizing a smart city through three-dimensional visualization, characterized in that: The method comprises: Extracting the flight trajectory of a drone used to collect remote sensing images of a smart city, selecting a target flight trajectory from the flight trajectories based on the texture quality of the remote sensing image, collecting a target remote sensing image of the smart city and the city coordinates of the target remote sensing image using the drone based on the target flight trajectory, and performing image preprocessing on the target remote sensing image to obtain a preprocessed image; Detecting a moving target in the preprocessed image, and after removing the moving target from the preprocessed image, performing image restoration on an image module corresponding to the moving target in the preprocessed image to obtain a restored image; Based on the city coordinates, overlapping buildings in the repaired image are segmented to obtain a segmented image, and each segmented image in the segmented image is fused to obtain a fused image; Identify the city center point in the fused image, construct the spatial three-dimensional coordinates of the smart city based on the first mapping relationship between the city center point and the city coordinates, and use the second mapping relationship between the pixel point coordinates in the fused image and the spatial three-dimensional coordinates to connect each spatial three-dimensional coordinate in the spatial three-dimensional coordinates to obtain a spatial three-dimensional structure, wherein the identifying the city center point in the fused image includes: Detecting image edge points of the fused image, and connecting the image edge points to obtain a closed figure; The city center point of the closed figure is calculated using the following formula: Wherein, (x', y') represents the city center point, x' represents the horizontal coordinate of the city center point, y' represents the vertical coordinate of the city center point, x represents the independent variable on the horizontal axis, y represents the independent variable on the vertical axis, D represents the plane area where the closed figure is located, s D represents the area of ​​the closed figure, and u(x, y) represents the density; Performing color rendering on the spatial three-dimensional structure to obtain a color rendering structure, performing texture rendering on the color rendering structure to obtain a texture rendering structure, and using the texture rendering structure as a three-dimensional visualization model of the smart city; Mark the city attributes in the three-dimensional visualization model, load the three-dimensional visualization model and the city attributes into a pre-built visualization platform, use the visualization platform to complete the three-dimensional virtual display of the smart city, and obtain the three-dimensional virtual display result of the smart city.

2. The method according to claim 1, characterized in that: The step of performing image preprocessing on the target remote sensing image to obtain a preprocessed image includes: The target remote sensing image is filtered using the following formula to obtain a filtered image: Wherein, ω represents the filtered image, M represents the total number of pixels of the target remote sensing image, arg represents the window function, (i, j) represents the pixel to be filtered in the target remote sensing image, Q(i, j) represents the pixel grayscale value of the pixel to be filtered, The filtered image is tilt-corrected to obtain a pre-processed image.

3. The method according to claim 2, characterized in that The step of performing tilt correction on the filtered image to obtain a preprocessed image comprises: Taking the image center point of the filtered image as the coordinate origin and taking the image diagonal point of the filtered image as the plane; Based on the coordinate origin and the plane, a spatial coordinate system of the filtered image is constructed, and the filtered image is rotated using the following formula to obtain a rotation coordinate point: Among them, A2, B2, C2 represent the rotation coordinate points after the coordinate rotation, A1, B1, C1 represent the original spatial coordinate points of the filtered image in the spatial coordinate system, ε A , ε B , ε C represents the rotation angle in the rotation matrix, Based on the rotation coordinate point, the tilt angle of the filtered image is reversely rotated to obtain a tilt-corrected image.

4. The method according to claim 3, characterized in that The detecting of the moving target in the preprocessed image comprises: The pre-processed image is input into a pre-configured convolutional neural network to identify the image features of the pre-processed image using the convolutional neural network in combination with the following formula: in, represents vector features, f represents activation function, A, B, C represent the length, width, and height of the convolution kernel cube in the convolution model, is the value at the position (a, b, c) on the image and the calculated value of the k-th convolution kernel of the u-th layer, is the value at the position of the image (α+a, β+b, γ+c) and the calculated value of the mth convolution kernel in the u-1th layer of the convolution kernel, C ku represents a hyperparameter of the kth convolution kernel in the uth layer, α, β, and γ represent the offset of the position (a, b, c). According to the image features, image information of the tilt-corrected image is identified, and a corresponding moving target in the image information is identified.

5. The method according to claim 1, characterized in that The performing image restoration on the image module corresponding to the moving target in the pre-processed image to obtain the restored image includes: Querying a historical image corresponding to the preprocessed image, extracting a region in the historical image corresponding to a missing portion of the preprocessed image, and obtaining a compensation image; Calculating a pixel difference between the compensation image and the preprocessed image, and performing pixel correction on the compensation image based on the pixel difference to obtain a corrected image; The correction image is used to repair the missing part in the pre-processed image to obtain a repaired image.

6. The method according to claim 1, characterized in that The method of performing overlapping segmentation on the overlapping buildings in the repaired image based on the city coordinates to obtain a segmented image includes: Converting the repaired image into a grayscale image, performing feature matching on the grayscale image, and obtaining a feature matching result; Based on the feature matching result, the overlapping buildings in the repaired image are determined, and the overlapping buildings are overlapped and segmented to obtain a segmented image.

7. The method according to claim 1, characterized in that The fusing each segmented image in the segmented images to obtain a fused image comprises: Marking feature points of the segmented image to obtain marked points; A marker conversion matrix of the segmented image is constructed, and the segmented image is fused using the marker conversion matrix and the marker points to obtain a fused image.

8. The method according to claim 7, characterized in that The step of fusing the segmented image using the marker transformation matrix and the marker points to obtain a fused image includes: The mark conversion matrix is ​​used in combination with the following formula to convert the mark points to obtain conversion marks: in, represents the conversion mark, α1 represents the first mark point before the conversion, β1 represents the second mark point before the conversion, α2 represents the first mark point after the conversion, β2 represents the second mark point after the conversion, μ1, μ2 represent rotation transformation parameters, μ3, μ4 represent translation transformation parameters, μ5, μ6 represent scale transformation parameters, μ7, μ8 represent perspective transformation parameters, represents the label transformation matrix, The segmented images are fused using the conversion marks to obtain a fused image.

9. A virtual display system for realizing a smart city through three-dimensional visualization, characterized in that: The system comprises: An image preprocessing module is used to extract the flight trajectory of the UAV used to collect the remote sensing image of the smart city, select a target flight trajectory from the flight trajectories based on the texture quality of the remote sensing image, collect the target remote sensing image of the smart city and the city coordinates of the target remote sensing image by using the UAV based on the target flight trajectory, and perform image preprocessing on the target remote sensing image to obtain a preprocessed image; An image restoration module is used to detect a moving target in the preprocessed image, and after removing the moving target from the preprocessed image, perform image restoration on an image module corresponding to the moving target in the preprocessed image to obtain a restored image; An image fusion module, configured to perform overlapping segmentation on the overlapping buildings in the repaired image based on the city coordinates to obtain a segmented image, and fuse each segmented image in the segmented image to obtain a fused image; The three-dimensional structure construction module is used to identify the city center point in the fused image, construct the spatial three-dimensional coordinates of the smart city based on the first mapping relationship between the city center point and the city coordinates, and connect the coordinates of each spatial three-dimensional coordinate in the spatial three-dimensional coordinates by using the second mapping relationship between the pixel point coordinates in the fused image and the spatial three-dimensional coordinates to obtain a spatial three-dimensional structure, wherein the identifying of the city center point in the fused image includes: Detecting image edge points of the fused image, and connecting the image edge points to obtain a closed figure; The city center point of the closed figure is calculated using the following formula: Wherein, (x', y') represents the city center point, x' represents the horizontal coordinate of the city center point, y' represents the vertical coordinate of the city center point, x represents the independent variable on the horizontal axis, y represents the independent variable on the vertical axis, D represents the plane area where the closed figure is located, s D represents the area of ​​the closed figure, and u(x, y) represents the density; A model rendering module, used to perform color rendering on the three-dimensional structure of the space to obtain a color rendering structure, perform texture rendering on the color rendering structure to obtain a texture rendering structure, and use the texture rendering structure as a three-dimensional visualization model of the smart city; The three-dimensional display module is used to mark the city attributes in the three-dimensional visualization model, load the three-dimensional visualization model and the city attributes into a pre-built visualization platform, use the visualization platform to complete the three-dimensional virtual display of the smart city, and obtain the three-dimensional virtual display result of the smart city.