A 3D modeling method based on digital twin city

Abstract

The present invention discloses a 3D modeling method based on a digital twin city, belonging to the technical field of three-dimensional urban modeling, specifically including: continuously collecting a sequence of stereo images of the current city in real time, performing differential calculation between adjacent stereo images of the stereo images to obtain a to-be-determined area; performing building image recognition on the to-be-determined area to obtain all building image areas and generating a building digital surface model, obtaining a historical building digital surface model of the city and performing differential calculation with the building digital surface model to obtain a number of target areas; selecting any one of the target areas and obtaining street view photos of different perspectives of the target area, performing feature matching on the street view photos of different perspectives to generate a 3D model of the target area, and correcting the historical city 3D model according to the 3D model of the target area to obtain a final city 3D model. The present invention greatly improves the efficiency and accuracy of urban building modeling.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional urban modeling, and particularly relates to a 3D modeling method based on a digital twin city. Background Art

[0002] Digital twin refers to digital models or virtual models that simulate actual physical systems or processes and are updated and optimized using real-time data inputs. Digital twin technology can help enterprises better understand their business and systems and perform real-time optimization and decision-making. In the field of urban planning, digital twin technology can help urban planners create digital models to simulate urban development, traffic, and the environment. These digital models can receive real-time data inputs, such as population changes, traffic flow, and environmental quality, etc., so as to help urban planners better understand the urban development situation, predict possible problems, and take corresponding measures to optimize urban planning and improve the urban environment, realizing the intelligent visualization of smart city construction and providing visual support for the whole process of various applications in smart cities.

[0003] With the development of digital twin technology, people's requirements for the quality of automatic three-dimensional modeling of urban planning are getting higher and higher. The existing technology usually uses the real three-dimensional model obtained from aerial oblique images to achieve automatic three-dimensional modeling of the city. However, with the rapid development of the city, there are constantly changes in new and old buildings. Therefore, after the platform is built, it still faces the task of updating, that is, the high cost brought by expensive equipment and the inconvenience in subsequent updates, resulting in time-consuming and expensive maintenance of urban models. Summary of the Invention

[0004] The purpose of the present invention is to provide a 3D modeling method based on a digital twin city to solve the following technical problems:

[0005] However, with the rapid development of the city, there are constantly changes in new and old buildings. Therefore, after the platform is built, it still faces the task of updating, that is, the high cost brought by expensive equipment and the inconvenience in subsequent updates, resulting in time-consuming and expensive maintenance of urban models.

[0006] The purpose of the present invention can be achieved through the following technical solutions:

[0007] A 3D modeling method based on a digital twin city includes the following steps:

[0008] S1, set the observation duration T, and in the observation time T, continuously collect the sequence of stereo images Y1, Y2,..., Yn of the current city in real time. Preprocess the stereo images in sequence and calculate the pixel differences between adjacent stereo images respectively to obtain the pixel change frequency in any image area. If there is any image area where the pixel change frequency is less than or equal to the preset threshold, mark this image area as a pending area;

[0009] S2. Perform building image recognition on the to-be-determined area, obtain all building image areas, generate a building digital surface model, acquire the digital surface model of historical urban buildings, perform differential calculation with the building digital surface model, obtain the change difference result, extract the edge features of the change difference result, and obtain several target areas;

[0010] S3. Select any target area, obtain street view photos of different perspectives of the target area, perform feature matching on the street view photos of different perspectives, generate a three-dimensional model of the target area, and correct the three-dimensional model of the historical city according to the three-dimensional model of the target area to obtain the final three-dimensional model of the city.

[0011] As a further solution of the present invention: In S1, the specific process of preprocessing the stereo image is as follows:

[0012] Perform grayscale processing on the stereo image to obtain a grayscale image, perform geometric correction, radiometric correction, and atmospheric correction on the grayscale image, perform noise reduction processing on the corrected grayscale image, extract feature points in the grayscale image using Harris corner detection, perform feature point matching on the grayscale image using the nearest neighbor matching algorithm, and correct the stereo image in sequence according to the matching result.

[0013] As a further solution of the present invention: In S1, the specific generation process of the to-be-determined area is as follows:

[0014] Obtain the corrected stereo grayscale image, establish a rectangular coordinate system with the central pixel point of the stereo grayscale image as the origin, generate the coordinates (x, y) of all pixel points in the stereo grayscale image, and identify the grayscale value Pi(x, y) of all pixel points in the stereo grayscale image, where i = 1,..., n, and i represents the order of the stereo grayscale image where the pixel point is located in the sequence. Calculate the grayscale difference D between the same-position pixel points in adjacent stereo grayscale images through the grayscale difference formula D = |Pi+1(x, y) - Pi(x, y)|. If D is greater than the preset threshold, it means that the corresponding pixel point is a changed pixel; if D is less than or equal to the preset threshold, it means that the corresponding pixel point is an unchanged pixel. Obtain the changed pixel value s at any image position, calculate the pixel change frequency in any image area according to the calculation formula p = s / n - 1, and fit the areas where the pixel change frequencies of all image areas are less than or equal to the preset threshold to generate the to-be-determined area.

[0015] As a further solution of the present invention: In S1, if there is any image area where the pixel change frequency is greater than the preset threshold, then label this image area as a non-building image area;

[0016] As a further solution of the present invention: In S2, the specific process of obtaining the building area is as follows:

[0017] S11. Apply an edge extraction algorithm to the area to be determined for edge extraction. Then, based on the extracted edge features, extract the line features of the image, and use the Douglas–Peucker algorithm to simplify the line features.

[0018] S12. Select any straight line segment and calculate its length and direction to obtain the lengths and directions of all straight line segments. Obtain all straight line segments whose directions are within the preset direction interval [0, 180]. Divide the preset direction interval into several sub-direction intervals according to the preset step size, and calculate the line feature value P within any sub-direction interval in turn. The feature value where m is the number of straight lines in sub-direction interval j, n is the number of all sub-direction intervals, and qi is the length of any straight line segment; if there is a line feature value P in any sub-direction interval greater than the preset threshold, then label this sub-direction interval as the target direction interval; calculate the direction difference between target direction intervals. If there is a direction difference of 90 degrees, then label this target direction interval as the vertical interval to obtain the number N of all vertical intervals.

[0019] S13. Calculate the building coefficient A. If there is a building coefficient A in any area to be determined greater than the preset threshold, then determine that this area to be determined is a building image area; building coefficient A = (1 + N) * P0; N is the number of vertical intervals, and P0 represents the direction feature value of the direction interval of 0.

[0020] As a further solution of the present invention: in S2, it further includes classifying the difference change result. The specific process is as follows:

[0021] If the change object is a building in the historical building digital surface model and also a building in the current digital surface model, then obtain the elevation data DSM of this change object. If DSM t1 < DSM t2, then determine that the category of this change object is increase, where t1 is the historical time and t2 is the current time; if the change object is a building in the historical building digital surface model and also a building in the current digital surface model, then obtain the elevation data DSM of this change object. If DSM t1 > DSM t2, then determine that the category of this change object is decrease; if the change object is a building in the historical building digital surface model and a non-building in the current digital surface model, then the category of this object is demolition; if the change object is a non-building in the historical building digital surface model and a building in the current digital surface model, then the category of this change object is new construction.

[0022] As a further solution of the present invention: it further includes that if the classification result of any change object is increase or decrease, then directly calculate the elevation data difference dDSM of this change object and perform corresponding height adjustment on the original urban historical building digital surface model according to the calculated elevation data difference dDSM.

[0023] As a further solution of the present invention: in the step S3, the process of correcting the three-dimensional model of the historical city is as follows:

[0024] Determine the geometric relationship of the target area through the street view pictures after feature matching, optimize the dense point cloud through the accelerated dense matching algorithm, reconstruct and generate the three-dimensional model of the target area, obtain the target area and construct the minimum circumscribed rectangle to establish a buffer zone, use the buildings in the non-target area within the buffer zone to construct a reference system, obtain the position information of the current target area according to the reference system, and correct the three-dimensional model of the historical city according to the position information.

[0025] Beneficial effects of the present invention:

[0026] The present invention first sets the observation time and obtains the urban stereo images in real time, filters out the non-building targets from the urban building images according to the pixel change frequency at the same position between the stereo images to obtain the pending area, then identifies the building images in the pending area, further identifies the building targets and constructs the building digital surface model, preforms differential calculation with the surface model of the urban historical buildings to obtain the urban building change area, obtains the street view photos of different perspectives of the urban change area, performs feature matching on the street view photos of different perspectives to generate the three-dimensional model of the target area, and corrects the three-dimensional model of the historical city according to the three-dimensional model of the target area to obtain the final urban three-dimensional model. The present invention greatly improves the efficiency and accuracy of urban building modeling. Description of the drawings

[0027] The present invention will be further described below with reference to the accompanying drawings.

[0028] Figure 1 It is a schematic flowchart of a 3D modeling method based on a digital twin city of the present invention. Specific embodiments

[0029] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0030] Please refer to Figure 1 As shown, the present invention is a 3D modeling method based on a digital twin city, including the following steps:

[0031] S1, set the observation time T, collect the continuous stereo image sequence Y1, Y2, ..., Yn of the current city in real time during the observation time T, pre-process the stereo images in sequence and calculate the pixel differences between adjacent stereo images respectively, obtain the pixel change frequency in any image area, and if there is any image area with a pixel change frequency less than or equal to a preset threshold, mark the image area as a pending area;

[0032] S2, performing building image recognition on the area to be determined, obtaining all building image areas and generating building digital surface models, obtaining the digital surface models of urban historical buildings and performing differential calculation with the building digital surface models to obtain change difference results, extracting edge features of the change difference results, and obtaining a number of target areas;

[0033] S3, select any target area and obtain street view photos of the target area from different perspectives, perform feature matching on the street view photos from different perspectives, generate a three-dimensional model of the target area, and modify the historical city three-dimensional model according to the three-dimensional model of the target area to obtain a final city three-dimensional model.

[0034] The present invention first sets an observation time and acquires a city stereo image in real time, filters out non-building targets from the city building image according to the pixel change frequency at the same position between the stereo images, and obtains a pending area, then performs building image recognition on the pending area, further identifies the building target and constructs a building digital surface model, performs differential calculation with the city historical building surface model in advance, obtains the city building change area, acquires street view photos of the city change area from different perspectives, performs feature matching on the street view photos of the different perspectives, generates a three-dimensional model of the target area, and corrects the historical city three-dimensional model according to the three-dimensional model of the target area to obtain a final city three-dimensional model. The present invention greatly improves the efficiency and accuracy of urban building modeling.

[0035] It is noteworthy that the present invention can quickly identify the pending area containing the building image area by acquiring a continuous stereo image sequence in real time and calculating the pixel difference between adjacent stereo images, and greatly reduce the amount of data processing calculation by further performing building image recognition on the pending area, while realizing accurate recognition of the building image, thereby generating a building digital surface model based on the building image, and performing differential calculation with the digital surface model of the historical building, so as to accurately obtain the building changes and obtain the target area, and generate a three-dimensional model of the target area by acquiring street view photos of the target area from different perspectives and performing feature matching, and the final three-dimensional model of the city is more accurate and detailed.

[0036] In a preferred embodiment of the present invention, in S1, the specific process of preprocessing the stereoscopic image is:

[0037] Perform grayscale processing on the stereoscopic image to obtain a grayscale image. Perform geometric correction, radiometric correction, and atmospheric correction on the grayscale image. Perform noise reduction processing on the corrected grayscale image. Use Harris corner detection to extract feature points in the grayscale image. Use the nearest neighbor matching algorithm to match the feature points of the grayscale image. Correct the stereoscopic image successively according to the matching results.

[0038] By performing grayscale processing, geometric correction, radiometric correction, and atmospheric correction on the stereoscopic image, the noise and non-target features in the image can be effectively reduced, and the quality of the image can be improved. Use Harris corner detection to extract feature points in the grayscale image. Use the nearest neighbor matching algorithm to match the feature points of the grayscale image. Correct the stereoscopic image successively according to the matching results to ensure that the layouts of adjacent images are consistent.

[0039] In another preferred embodiment of the present invention, in S1, the specific generation process of the to-be-determined region is as follows:

[0040] Obtain the corrected stereoscopic grayscale image. Establish a rectangular coordinate system with the central pixel point of the stereoscopic grayscale image as the origin. Generate the coordinates (x, y) of all pixel points in the stereoscopic grayscale image, and identify the grayscale value Pi(x, y) of all pixel points in the stereoscopic grayscale image, where i = 1,..., n, and i represents the order of the stereoscopic grayscale image where the pixel point is located in the sequence. Calculate the grayscale difference value D between the pixel points at the same position in adjacent stereoscopic grayscale images through the grayscale difference formula D = |Pi+1(x, y) - Pi(x, y)|. If D is greater than the preset threshold, it means that the corresponding pixel point is a changing pixel. If D is less than or equal to the preset threshold, it means that the corresponding pixel point is an unchanged pixel. Obtain the changing pixel value s at any image position. Calculate the pixel change frequency in any image region according to the calculation formula p = s / n - 1. Fit the regions where the pixel change frequencies of all image regions are less than or equal to the preset threshold to generate the to-be-determined region.

[0041] In another preferred embodiment of the present invention, in S1, if there is any image region where the pixel change frequency is greater than the preset threshold, mark this image region as a non-building image region;

[0042] In another preferred embodiment of the present invention, in S2, the specific process of obtaining the building region is as follows:

[0043] S11, perform edge extraction on the to-be-determined region using an edge extraction algorithm. Then, based on the edge feature extraction, extract the line features of the image and simplify the line features using the Douglas–Peucker algorithm;

[0044] S12. Select any straight line segment and calculate its length and direction to obtain the lengths and directions of all straight line segments. Obtain all straight line segments whose directions are within the preset direction interval [0, 180]. Divide the preset direction interval into several sub-direction intervals according to the preset step size. Calculate the straight line eigenvalue P within any sub-direction interval in turn. The eigenvalue where m is the number of straight lines in sub-direction interval j, n is the number of all sub-direction intervals, and qi is the length of any straight line segment; if there is a straight line eigenvalue P in any sub-direction interval that is greater than the preset threshold, then label this sub-direction interval as the target direction interval; calculate the direction difference between the target direction intervals. If there is a direction difference of 90 degrees, then label this target direction interval as the vertical interval to obtain the number N of all vertical intervals;

[0045] S13. Calculate the building coefficient A. If there is a building coefficient A in any to-be-determined area that is greater than the preset threshold, then determine that this to-be-determined area is a building image area; the building coefficient A = (1 + N) * P0; N is the number of vertical intervals, and P0 represents the direction eigenvalue of the direction interval of 0.

[0046] First, detect the edges in the image. On the basis of edge extraction, further extract the straight line features in the image. To reduce the calculation amount and improve the accuracy, use the Douglas - Peucker algorithm to simplify the detected line segments. This algorithm can eliminate the points that are not the main part of the straight line. For each straight line segment, calculate its length and direction. Divide all directions into the preset interval [0, 180] and further divide it into several sub-direction intervals according to the preset step size. For each sub-direction interval, calculate an eigenvalue P. This eigenvalue is based on the lengths and quantities of the straight line segments within this interval. If the eigenvalue P of a certain sub-direction interval is greater than the preset threshold, it indicates that the straight line feature of this sub-interval is the main body. Check the direction difference between the target direction intervals. If there is a 90 - degree difference, then label it as the vertical interval. For the area where buildings exist, there is usually long linearity. Therefore, use the P0 value to measure the building coefficient. By calculating the building coefficient, it can be accurately determined whether this to-be-determined area is a building image area.

[0047] In another preferred embodiment of the present invention, in S2, it further includes classifying the difference change result. The specific process is as follows:

[0048] If the object of change is a building in the historical building digital surface model and also a building in the current digital surface model, then obtain the elevation data DSM of the object of change. If DSM t1 < DSM t2, it is determined that the category of the object of change is height increase, where t1 is the historical time and t2 is the current time; if the object of change is a building in the historical building digital surface model and also a building in the current digital surface model, then obtain the elevation data DSM of the object of change. If DSM t1 > DSM t2, it is determined that the category of the object of change is height decrease; if the object of change is a building in the historical building digital surface model and a non-building in the current digital surface model, then the category of this object is demolition; if the object of change is a non-building in the historical building digital surface model and a building in the current digital surface model, then the category of this object of change is new construction.

[0049] In another preferred embodiment of the present invention, it further includes that if the classification result of any object of change is height increase or height decrease, directly calculate the elevation data difference dDSM of the object of change and perform corresponding height adjustment on the original digital surface model of the historical urban building according to the calculated elevation data difference dDSM.

[0050] In another preferred embodiment of the present invention, in S3, the process of correcting the historical urban three-dimensional model is as follows:

[0051] Determine the geometric relationship of the target area through the street view pictures after feature matching, optimize the dense point cloud through the accelerated dense matching algorithm, reconstruct and generate the three-dimensional model of the target area, obtain the target area and construct the minimum circumscribed rectangle to establish a buffer zone, use the buildings in the non-target area within the buffer zone to construct a reference system, obtain the position information of the current target area according to the reference system, and correct the historical urban three-dimensional model according to the position information.

[0052] The above has described a specific embodiment of the present invention in detail, but the content described is only a preferred embodiment of the present invention and cannot be considered as limiting the scope of implementation of the present invention. All equivalent changes and improvements made according to the scope of the present invention application should still fall within the patent coverage scope of the present invention.

Claims

1. A 3D modeling method based on digital twin city, characterized in that: The following steps are involved: S1, set an observation time T, collect a continuous stereo image sequence Y1, Y2, ..., Yn of the current city in real time during the observation time T, where n is the total number of continuous stereo images, pre-process the stereo images in sequence and calculate the pixel differences between adjacent stereo images to obtain the pixel change frequency in any image area. If there is any image area with a pixel change frequency less than or equal to a preset threshold, mark the image area as a pending area; If there is any image area where the pixel change frequency is greater than a preset threshold, the image area is marked as a non-building image area; S2, performing building image recognition on the area to be determined, obtaining all building image areas and generating building digital surface models, obtaining the digital surface models of urban historical buildings and performing differential calculation with the building digital surface models to obtain change difference results, extracting edge features of the change difference results, and obtaining a number of target areas; S3, selecting any target area and obtaining street view photos of the target area from different perspectives, performing feature matching on the street view photos from different perspectives, generating a three-dimensional model of the target area, and correcting the historical city three-dimensional model according to the three-dimensional model of the target area to obtain a final city three-dimensional model; The specific process of preprocessing the stereo image is as follows: grayscale processing is performed on the stereoscopic image to obtain a grayscale image, geometric correction, radiation correction and atmospheric correction are performed on the grayscale image, noise reduction is performed on the corrected grayscale image, feature points in the grayscale image are extracted using Harris corner point detection, feature points of the grayscale image are matched using a nearest neighbor matching algorithm, and the stereoscopic image is corrected in sequence according to the matching results; The specific generation process of the pending area is as follows: Obtain the corrected stereo grayscale image, establish a rectangular coordinate system with the central pixel of the stereo grayscale image as the origin, generate the coordinates (x, y) of all pixels in the stereo grayscale image, and identify the grayscale values ​​P of all pixels in the stereo grayscale image. i (x, y), i = 1, ..., n, i represents the order of the three-dimensional grayscale image where the pixel is located in the sequence, and the grayscale difference formula D = |P i+1 (x, y)-P i (x, y)|Calculate the grayscale difference D of the pixels at the same position between adjacent stereo grayscale images. If D is greater than the preset threshold, it means that the corresponding pixel is a changed pixel. If D is less than or equal to the preset threshold, it means that the corresponding pixel is an unchanged pixel. The changed pixel value s of any image position is obtained. The pixel change frequency in any image area is calculated according to the calculation formula p=s / n-1. All areas in the image area where the pixel change frequency is less than or equal to the preset threshold are fitted to generate the pending area.

2. A 3D modeling method based on a digital twin city according to claim 1, characterized in that: The step S2 also includes classifying the change difference results, and the specific process is as follows: If the changed object is a building in the historical building digital surface model and is also a building in the current digital surface model, the elevation data DSM of the changed object is obtained. t1 <DSM t2 , then the change object category is determined to be increased, where t1 is the historical time and t2 is the current time; if DSM t1 >DSM t2 , then the category of the changed object is determined to be reduction; if the changed object is a building in the digital surface model of the historical building and a non-building in the current digital surface model, then the category of the changed object is demolition; if the changed object is a non-building in the digital surface model of the historical building and a building in the current digital surface model, then the category of the changed object is new construction.

3. The 3D modeling method based on digital twin city according to claim 2 is characterized in that: It also includes directly calculating the elevation data difference dDSM of any changed object if the classification result is an increase or decrease, and adjusting the original digital surface model of urban historical buildings accordingly based on the calculated elevation data difference dDSM.

4. The 3D modeling method based on digital twin city according to claim 1, characterized in that: In S3, the process of modifying the historical city three-dimensional model is as follows: The geometric relationship of the target area is determined through the street view image after feature matching, the dense point cloud is optimized by an accelerated dense matching algorithm, and a three-dimensional model of the target area is reconstructed. The target area is obtained and a minimum circumscribed rectangle is constructed to establish a buffer zone. A reference system is constructed with the buildings in the non-target area within the buffer zone. According to the reference system, the location information of the current target area is obtained and the three-dimensional model of the historical city is corrected.

Images