Settlement digital twin modeling method and system based on multimodal data fusion

Abstract

This invention provides a method and system for digital twin modeling of settlements based on multimodal data fusion, relating to the field of digital twins. The system includes: a multimodal data acquisition module for acquiring current multimodal data for settlement digitization; a digital twin establishment module for establishing a physical 3D model of the settlement based on the current multimodal data for settlement digitization, thus constructing a current digital twin model of the settlement; a development analysis module for determining multiple historical development nodes of the settlement based on the current multimodal data for settlement digitization; and the digital twin establishment module is further used to establish a 3D model of the historical development nodes of the settlement based on the predicted multiple historical development nodes and the physical 3D model of the settlement, thus constructing a digital twin model of the settlement at historical development nodes. This system has the advantage of digitizing the landscape and environmental changes of settlements.

Description

Technical Field

[0001] This invention relates to the field of digital twins, and in particular to a method and system for digital twin modeling of settlements based on multimodal data fusion. Background Technology

[0002] Settlement environments are human-created environments created through the conscious development, utilization, and transformation of nature. Applying digital twins to settlement landscapes endows settlement planning and governance with information-based and intelligent capabilities. A digital twin constructs a parallel digital world to the physical world, mapping the virtual and real worlds to accurately reflect changes in reality. In the field of landscape architecture, digital twin technology is used to simulate and model the real world. Leveraging cutting-edge information technologies such as the Internet of Things and virtual reality, and utilizing computational graphics and multi-sensor technologies, big data collected from micro-environmental observations in the physical world is used to simulate and map the state of settlements. Settlement cultural landscapes are an important component of cultural heritage, carrying rich historical, cultural, and ecological information. By generating digital twin models of settlements, this information can be permanently preserved, preventing damage and loss due to natural or human factors, thus achieving long-term protection and inheritance of cultural heritage. Settlement digital twin models provide scholars and researchers with convenient research tools. They can gain a deeper understanding of the structure, evolution, and relationship with the environment of settlement cultural landscapes through digital archives, thereby promoting the development of related disciplines. Meanwhile, digital twin models of settlements can also serve as educational resources, helping students understand and learn about cultural heritage. Beyond preservation and research, digital twin models can also support the revitalization and utilization of cultural heritage. Through digital means, settlement cultural landscapes can be presented to the public in a more vivid and intuitive way, enhancing public awareness and interest in cultural heritage and promoting its development in tourism, education, and creative industries.

[0003] Existing records of settlement cultural landscapes often rely on textual descriptions, two-dimensional images, or simple survey data, which have limitations in expressing the three-dimensional structure, dynamic changes, and relationship with the environment of the landscape.

[0004] Therefore, there is a need to provide a settlement digital twin modeling method and system based on multimodal data fusion to realize the digitization of settlement landscape and environmental changes. Summary of the Invention

[0005] This invention provides a settlement digital twin modeling system based on multimodal data fusion, comprising: a multimodal data acquisition module for acquiring current multimodal data for settlement digitization, wherein the current multimodal data for settlement digitization includes at least spatial vector data, images, laser scan data, meteorological data, vegetation data, and text data; a digital twin establishment module for establishing a physical 3D model of the settlement based on the current multimodal data for settlement digitization, thereby constructing a current digital twin model of the settlement; and a development analysis module for determining multiple historical development nodes of the settlement based on the current multimodal data for settlement digitization. The digital twin establishment module is further used to establish a historical development node 3D model of the settlement based on the predicted multiple historical development nodes and the physical 3D model of the settlement, thereby constructing a historical development node settlement digital twin model.

[0006] Furthermore, the digital twin establishment module, based on the current multimodal data used for settlement digitization, establishes a physical 3D model of the settlement and constructs a digital twin model of the settlement's current status. This includes: fusing the settlement's imagery and laser scanning data using a 3D reconstruction algorithm to generate a physical 3D model of the settlement; generating information annotations for the physical 3D model of the settlement based on text data; establishing multiple key environmental and ecological simulation scenarios for the settlement based on the settlement's meteorological and vegetation data; and generating a digital twin model of the settlement's current status based on the settlement's spatial vector data, physical 3D model, information annotations for the physical 3D model, and multiple key environmental and ecological simulation scenarios.

[0007] Furthermore, the digital twin establishment module establishes multiple key environmental and ecological simulation scenarios for the settlement based on the settlement's meteorological and vegetation data, including: acquiring meteorological and vegetation data and multiple key environmental and ecological simulation scenarios for multiple sample settlements; determining the meteorological and vegetation characteristics of the sample settlements based on the meteorological and vegetation data; determining the meteorological and vegetation characteristics of the settlements based on the meteorological and vegetation data; identifying similar sample settlements based on the meteorological and vegetation characteristics of the settlements and the meteorological and vegetation characteristics of the sample settlements; and establishing multiple key environmental and ecological simulation scenarios for the settlements based on the multiple key environmental and ecological simulation scenarios of the similar sample settlements.

[0008] Furthermore, the development analysis module determines multiple historical development nodes of the settlement based on the current multimodal data used for settlement digitization, including: tracing back the construction time of each building in the settlement based on its architectural features and textual data; determining the architectural evolution information of the settlement based on the construction time of each building; determining the road network evolution information of the settlement based on the spatial vector data of the settlement; and determining the land use type spatial evolution information of the settlement based on the spatial vector data of the settlement. The multiple historical development nodes of the settlement include at least the architectural evolution information, road network evolution information, and land use type spatial evolution information of the settlement.

[0009] Furthermore, the digital twin establishment module establishes a three-dimensional model of the historical development nodes of the settlement based on the predicted multiple historical development nodes of the settlement and the physical three-dimensional model of the settlement, and constructs a digital twin model of the historical development node settlement, including: determining multiple historical development nodes of the settlement based on the settlement's building evolution information, road network evolution information, and land use type spatial evolution information; for each historical development node, establishing a three-dimensional model of the historical development node of the settlement based on the settlement's building evolution information, road network evolution information, and land use type spatial evolution information and the physical three-dimensional model of the settlement, and constructing a digital twin model of the historical development node settlement based on multiple key environmental and ecological simulation scenarios of the settlement.

[0010] Furthermore, the digital twin creation module establishes a three-dimensional model of the historical development nodes of the settlement based on the settlement's architectural evolution information, road network evolution information, land use type spatial evolution information, and the settlement's physical three-dimensional model. This includes: extracting the architectural type, height, and distribution range of the historical development nodes; combining the architectural geometric data from the physical three-dimensional model to generate a three-dimensional model of the historical buildings; adjusting the materials and colors; generating a three-dimensional model of the road network of the historical development nodes based on historical road vector data; converting historical land use type spatial data into the texture or elevation changes of the three-dimensional terrain surface to generate an agricultural land model; and integrating the historical architectural three-dimensional model, the road network three-dimensional model, and the agricultural land model to establish a three-dimensional model of the historical development nodes of the settlement.

[0011] Furthermore, the multimodal data acquisition module acquires laser scanning data of the settlement, including: determining the key buildings of the settlement based on the images and spatial vector data of the settlement; for each key building of the settlement, determining the optimal laser scanning parameters and optimal scanning path of the key building based on the images of the key building; scanning the key building based on the optimal laser scanning parameters and optimal scanning path of the key building to acquire the laser scanning data of the key building.

[0012] Furthermore, the multimodal data acquisition module determines the key buildings of the settlement based on the settlement's images and spatial vector data, including: determining the architectural features of each building in the settlement based on the settlement's images, wherein the architectural features include at least color features and outline features; and determining the key buildings of the settlement based on the architectural features of each building in the settlement and the settlement's spatial vector data.

[0013] Furthermore, the multimodal data acquisition module determines key buildings in the settlement based on the architectural features of each building and the spatial vector data of the settlement, including: for each building in the settlement, determining the feature difference value of the building based on its architectural features; determining the target building based on the feature difference value of each building in the settlement; for each target building in the settlement, determining similar sample buildings based on the architectural features of the target building and sample buildings, determining the historical and cultural value of the target building based on the historical and cultural value of the similar sample buildings, calculating the location center value of the target building based on the spatial vector data of the settlement; and determining the key buildings in the settlement based on the historical and cultural value and location center value of each target building in the settlement.

[0014] This invention provides a settlement digital twin modeling method based on multimodal data fusion, applied to the aforementioned settlement digital twin modeling system based on multimodal data fusion. The method includes: acquiring current multimodal data for settlement digitization, wherein the current multimodal data for settlement digitization includes at least imagery, laser scanning data, environmental data, and text data; establishing a physical 3D model of the settlement based on the current multimodal data for settlement digitization, thus constructing a settlement current-state digital twin model; determining multiple historical development nodes of the settlement based on the current multimodal data for settlement digitization; and establishing a historical development node 3D model of the settlement based on the predicted multiple historical development nodes and the physical 3D model of the settlement, thus constructing a historical development node settlement digital twin model.

[0015] Compared with existing technologies, the settlement digital twin modeling method and system based on multimodal data fusion provided by this invention has at least the following beneficial effects:

[0016] 1. By extracting historical development information (such as architectural evolution, changes in road networks, and spatial land use types), the current 3D model is linked with historical data to form a dynamic archive with a time series. Users can intuitively observe the evolution of settlements from the past to the present through digital archives and understand the formation mechanism of cultural landscapes.

[0017] 2. By analyzing the color and outline features of images, combined with location information from spatial vector data, key buildings with historical and cultural value or spatial representativeness can be accurately identified. This avoids indiscriminate scanning of all buildings, reduces data redundancy, and focuses on buildings that contribute most to the digital archive of cultural landscapes. Based on the similarity of architectural features between the target building and sample buildings, and considering the historical and cultural value of the sample buildings, the historical and cultural value of the target building is assessed. This provides a scientific basis for the selection of key buildings, ensuring that the digital archive of cultural landscapes includes the most representative cultural heritage.

[0018] 3. By leveraging the architectural features and optimal laser scanning parameters of historically scanned buildings, similar historically scanned buildings are matched to key buildings, and their optimal scanning parameters are referenced. This avoids blindly setting scanning parameters, improves scanning efficiency and data quality, and ensures the accuracy of the 3D model of key buildings. Optimal laser scanning parameters include resolution, sampling rate, and scanning spacing, which can be finely adjusted according to the architectural features of key buildings. This meets the scanning needs of different building types (such as ancient buildings and modern buildings), ensuring the detailed representation of the 3D model. A 3D model is generated based on the image of the key building, and key areas are identified, providing an intuitive basis for scanning path planning. The scanning path ensures coverage of key areas, avoiding the omission of important details. Multiple scanning paths are generated, and the optimal scanning path is generated by combining path evaluation indicators (such as path length, scanning time, and coverage area) and key areas using a genetic algorithm. This reduces scanning time and labor costs while ensuring scanning quality, thus improving scanning efficiency.

[0019] 4. A physical 3D model of the settlement is established using imagery and laser scanning data, combined with textual data to generate information annotations, ensuring the model's accuracy in spatial form and cultural connotation. Key environmental and ecological simulation scenarios (such as seasonal changes and vegetation cover changes) are established based on meteorological and vegetation data and integrated into a 3D digital twin base. This enhances the realism and immersion of the sand table, helping users intuitively understand the interaction between the settlement and its natural environment. Through multi-dimensional data such as architectural features, textual data, and spatial vector data, the construction time of buildings is predicted, and information on architectural evolution, road network evolution, and land use type spatial evolution is determined, systematically revealing the historical development patterns of the settlement and providing a scientific basis for cultural heritage protection and historical research. For each historical development node, a 3D model of the historical development node is constructed based on evolutionary information and the physical 3D model, and combined with environmental and ecological simulation scenarios, a digital twin model of the settlement is generated. Users can dynamically switch between sand table scenes of different historical periods, intuitively experiencing the historical evolution of the settlement and providing decision support for cultural heritage protection and urban planning. Attached Figure Description

[0020] This specification will be further described by way of exemplary embodiments, which will be described in detail with reference to the accompanying drawings. These embodiments are not limiting; in these embodiments, the same reference numerals denote the same structures, wherein:

[0021] Figure 1 This is a flowchart illustrating a settlement digital twin modeling method based on multimodal data fusion, as shown in some embodiments of this specification.

[0022] Figure 2 This is a flowchart illustrating the process of generating multiple scan paths for key buildings according to some embodiments of this specification;

[0023] Figure 3 This is a schematic diagram of a solid three-dimensional model of a settlement according to some embodiments of this specification;

[0024] Figure 4 This is a schematic diagram of a settlement digital twin modeling system based on multimodal data fusion, as shown in some embodiments of this specification. Detailed Implementation

[0025] To more clearly illustrate the technical solutions of the embodiments in this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are merely some examples or embodiments of this specification. For those skilled in the art, these drawings can be applied to other similar scenarios without creative effort. Unless obvious from the context or otherwise specified, the same reference numerals in the drawings represent the same structures or operations.

[0026] Figure 1 This is a flowchart illustrating a settlement digital twin modeling method based on multimodal data fusion, as shown in some embodiments of this specification. Figure 1 As shown, the settlement digital twin modeling method based on multimodal data fusion may include the following steps.

[0027] Step 110: Obtain current multimodal data for settlement digitization.

[0028] The current multimodal data used for settlement digitization includes at least spatial vector data, images, laser scan data, meteorological data, vegetation data, and text data.

[0029] Specifically, spatial vector data, represented by geometric elements such as points, lines, and areas, reflects the spatial structure and functional zoning of settlements, and can include:

[0030] Building data: Building plan boundaries.

[0031] Road network: road centerline, width, grade (main road, secondary road), material (asphalt, stone slab).

[0032] Water systems and green spaces: the boundaries and attributes of rivers, lakes, parks, and protective forests.

[0033] Functional zoning: the scope and purpose of residential areas, commercial areas, and agricultural areas.

[0034] Images of settlements are used to reflect visual information about the settlement's appearance, architectural style, and spatial layout, and may specifically include:

[0035] Aerial imagery: A bird's-eye view covering the entire settlement, used to analyze spatial layout.

[0036] Ground-level photography: building facades and street details, used to identify architectural styles and materials.

[0037] Historical photographs: archival photographs reflecting the appearance of settlements at different historical development stages.

[0038] Laser scanning data can include 3D point cloud data acquired through LiDAR (Light Detection and Ranging) to reflect the spatial morphology and details of settlements. Specifically, laser scanning data can include high-precision building surface data to extract roof type and facade structure.

[0039] Meteorological data is used to reflect the microclimate conditions of the area where the settlement is located, and can include daily / monthly records of temperature, precipitation, humidity, and wind speed.

[0040] Vegetation data is used to reflect the spatial distribution of vegetation cover and land use types around settlements. Specifically, it can include the distribution and area of ​​farmland, woodland, grassland, and water bodies, the types and proportions of plants, and vegetation indices.

[0041] Text data is textual information used to reflect the history, culture, and social structure of settlements, and can specifically include:

[0042] Historical documents: local chronicles, genealogies, inscriptions, contracts, and books collected in ancestral halls and family temples.

[0043] Oral history: Resident interview records and folk legends.

[0044] Modern archives include news reports, statistical yearbooks, historical records from village history museums, and information on intangible cultural heritage.

[0045] In some embodiments, acquiring laser scanning data of a settlement includes:

[0046] Based on the imagery and spatial vector data of the settlement, the key buildings of the settlement were identified;

[0047] For each key building in the settlement, the optimal laser scanning parameters and optimal scanning path for the key building are determined based on the image of the key building. The key building is then scanned based on the optimal laser scanning parameters and optimal scanning path to obtain the laser scanning data of the key building.

[0048] Specifically, key buildings in a settlement can be those that are of historical, cultural, structural, or functional importance within the settlement, such as ancient buildings, iconic public buildings, and traditional dwellings.

[0049] In some embodiments, key buildings of a settlement are identified based on imagery and spatial vector data of the settlement, including:

[0050] Based on images of the settlement, determine the architectural features of each building in the settlement, including at least color and outline features.

[0051] For each building in the settlement, the characteristic difference value of the building is determined based on the architectural characteristics of each building in the settlement;

[0052] The target building is determined based on the characteristic differences of each building in the settlement;

[0053] For each target building in the settlement, similar sample buildings are identified based on the architectural features of the target building and the sample buildings. The historical and cultural value of the target building is determined based on the historical and cultural value of the similar sample buildings. The location center value of the target building is calculated based on the spatial vector data of the settlement.

[0054] The key buildings of the settlement are identified based on the historical and cultural value and central location of each target building in the settlement.

[0055] Specifically, architectural features are used to describe the attributes of a building's appearance and structure, including color, outline, etc.

[0056] Color features can include the dominant color tone (e.g., red, gray) and color distribution of building exteriors and roofs. For each building in a settlement, image segmentation algorithms can be used to separate the building area image, statistically analyze the frequency distribution of different colors in the building area image, and obtain a feature vector reflecting the building's color distribution by calculating the histogram of each channel in the HSV color space. The frequency of different hue values ​​(expressed as angles, ranging from 0° to 360°) in the H (hue) channel histogram is then statistically analyzed, dividing the hue range into several intervals (e.g., 10° or 20° per interval), and counting the number of pixels in each interval to form a hue histogram vector. For example, if divided into 18 intervals (each 20° interval), the hue histogram vector contains 18 data points. Similarly, the frequency of different saturation values ​​(expressed as percentages, ranging from 0% to 100%) in the S (saturation) channel histogram is statistically analyzed, dividing the saturation range into several intervals (e.g., 10% per interval), and counting the number of pixels in each interval to form a saturation histogram vector. For example, if the image is divided into 10 intervals (each interval representing 10%), the saturation histogram vector will contain 10 data points. The frequency of different brightness values ​​(usually expressed as a percentage, ranging from 0% to 100%) in the V (brightness) channel histogram is counted, dividing the brightness range into several intervals (e.g., 10% per interval). The number of pixels within each interval is counted, forming a brightness histogram vector. For example, if the image is divided into 10 intervals (each interval representing 10%), the brightness histogram vector will contain 10 data points. To eliminate the influence of the dimensions of different channel histogram vectors, the histogram vector of each channel is normalized so that its sum is 1. Thus, each data point in the feature vector represents the relative frequency of that color interval in the image. The normalized hue, saturation, and brightness histogram vectors are combined sequentially to form the color feature vector of the building.

[0057] Contour features can include the building's planar shape (e.g., rectangular, L-shaped), facade layers (e.g., single-story, multi-story), and roof type (e.g., flat roof, pitched roof). For each building in a settlement, image segmentation algorithms can be used to separate the building area image. Based on the building area image, adaptive thresholding and dual-threshold detection algorithms are used to generate the building's edges. Energy minimization is performed using gradient vector flow fields to optimize the initially extracted building contour, simplifying the contour to a polygon. The planar shape is determined by the number of vertices (e.g., 4 vertices → rectangle, 5 vertices → L-shape, etc.). The shape classification result is encoded as a discrete value (e.g., rectangle = 1, L-shape = 2) and used as part of the contour feature vector. The number of layers (e.g., each layer is approximately 3 meters high) is detected by statistically analyzing pixel density changes through vertical projection. The layer detection result is encoded as a discrete value (e.g., single layer = 1, two layers = 2, three layers = 3, etc.) and used as part of the contour feature vector. The roof area is segmented using a semantic segmentation model (such as DeepLabV3+), or the roof is located through geometric analysis (such as the difference between the contour convex hull and the building contour). The standard deviation of the height of the roof area is calculated (such as flat roof standard deviation <0.5 meters, pitched roof >1 meter). The roof form detection results are encoded as discrete values ​​(such as flat roof=1, pitched roof=2, dome=3, etc.) as part of the contour feature vector.

[0058] For each building in a settlement, the cosine similarity between the building's color feature vector and the color feature vector of any other building in the settlement can be calculated. Similarly, the cosine similarity between the building's outline feature vector and the outline feature vector of any other building in the settlement can be calculated. A weighted sum of these cosine similarities yields the first building similarity score between the building and the other buildings in the settlement. The variance of this first building similarity score is then calculated to obtain the building's feature difference value. Buildings with feature difference values ​​greater than a threshold value can be designated as target buildings. This threshold value can be determined based on human experience or large-scale statistical analysis. In essence, a target building is a building in the settlement that, based on its color and outline features and similarity analysis with other buildings, is identified as having significant feature differences and may possess high value in representing the characteristics of the settlement.

[0059] Sample buildings are architectural examples with known historical and cultural value (such as traditional dwellings and historical sites). Historical and cultural value is a comprehensive assessment of a building's historical, cultural, artistic, and scientific aspects, and can be scored by experts based on dimensions such as protection level, building age, and stylistic uniqueness. For each target building in a settlement, the second architectural similarity between the target building and the sample buildings is calculated using the above method. Sample buildings with a second architectural similarity greater than the first architectural similarity threshold are considered similar sample buildings. The historical and cultural value of the target building can be obtained by averaging the historical and cultural values ​​of the similar sample buildings. The first architectural similarity threshold can be determined through experiments or manual (e.g., expert) experience.

[0060] The location center value of a target building can be used to analyze whether it is the core of settlement development (e.g., expanding outwards from the building) or to assess its importance in the spatial layout. As an example, the average coordinates of each building in the settlement can be used as the coordinates of the settlement center. For each target building in the settlement, the distance between the target building and the settlement center is calculated based on the target building's coordinates and the settlement center's coordinates. The location center value of the target building is then calculated based on this distance; the shorter the distance between the target building and the settlement center, the larger the location center value.

[0061] For each target building in the settlement, the historical and cultural value and the location center value of the target building are weighted and summed to obtain the comprehensive value of the target building. Target buildings with a comprehensive value greater than the comprehensive value threshold are regarded as key buildings of the settlement. The comprehensive value threshold can be obtained based on human experience or big data statistical analysis.

[0062] In some embodiments, determining the optimal laser scanning parameters for a key building based on its image includes:

[0063] Acquire architectural features and optimal laser scanning parameters of historically scanned buildings;

[0064] Based on the architectural features of key buildings and historical scanned buildings, identify similar historical scanned buildings;

[0065] Based on the optimal laser scanning parameters of similar historically scanned buildings, the optimal laser scanning parameters for key buildings are determined. The optimal laser scanning parameters include at least resolution, sampling rate, and scanning spacing.

[0066] Specifically, historically scanned buildings can be those whose optimal laser scanning parameters have been determined, and these buildings do not necessarily belong to the current settlement. The optimal parameters for historically scanned buildings can be determined through experimentation or human experience (e.g., by experts).

[0067] Optimal laser scanning parameters refer to a set of parameters set for historical buildings during the LiDAR scanning process, which achieves the best balance between accuracy, efficiency, and cost in the scanning results.

[0068] Resolution: The minimum distance between adjacent points in a point cloud (e.g., 0.01 meters).

[0069] Sampling rate: Point cloud density per unit area (e.g., 1000 points / m²).

[0070] Scanning spacing refers to the physical interval between two consecutive scan positions during the movement of a laser scanner. This parameter directly affects the integrity, overlap rate, and overall efficiency of the scanned data. The scanning spacing determines the size of the overlap area between adjacent scan positions. If the spacing is too large, data gaps may appear between scanned areas, failing to completely cover the target object. If the spacing is too small, it increases scanning time and data redundancy but improves data integrity. An appropriate overlap rate (generally recommended to be 30%-50%) helps with the stitching and registration of point cloud data, reducing registration errors. Too small an overlap rate may lead to registration failure, while too large an overlap rate increases computational complexity.

[0071] For each key building in a settlement, the cosine similarity between the key building's color feature vector and the color feature vector of any historical scanned building can be calculated. The cosine similarity between the key building's contour feature vector and the contour feature vector of any historical scanned building can also be calculated. A weighted sum of these two cosine similarities yields a third building similarity between the key building and the historical scanned buildings. Historical scanned buildings with a third building similarity greater than a second building similarity threshold can be considered similar historical scanned buildings. This second building similarity threshold can be determined experimentally or through human experience (e.g., by experts).

[0072] As an example, the optimal laser scanning parameters of the most similar historically scanned building with the highest third similarity can be used as the optimal laser scanning parameters of the key building. As another example, the average of the optimal laser scanning parameters of similar historically scanned buildings can be used as the optimal laser scanning parameters of the key building.

[0073] In some embodiments, determining the optimal scanning path for a key building based on its imagery includes:

[0074] Generate a 3D model of the key building based on its imagery.

[0075] Based on the images of key buildings, identify the key areas of the 3D model of the key buildings;

[0076] Based on the 3D model of the key buildings, generate multiple scanning paths for the key buildings;

[0077] Identify key areas of the 3D model of multiple path evaluation indicators and key buildings, and establish a fitness function;

[0078] The optimal scanning path for key buildings is generated using a genetic algorithm based on a fitness function and multiple scanning paths for key buildings.

[0079] Specifically, 3D reconstruction algorithms can be used to generate 3D models of key buildings based on images of those buildings.

[0080] The key regions of a critical building's 3D model can include geometrically complex areas, functionally important areas, and accessible restricted areas. Geometrically complex areas may include areas requiring high-precision scanning, such as sculpted facades and unique roof structures. Functionally important areas may include entrances, windows, and structural support points—areas crucial to the building's structure or function. Accessible restricted areas may include occluded or difficult-to-scan areas (such as the building's rear or higher elevations). Geometrically complex areas (e.g., sculpted facades, spires) can be extracted using geometric features such as curvature and normal variations. Deep learning models (e.g., PointNet) are used for semantic segmentation of the 3D model to label functionally important areas (e.g., windows, doors). Accessible restricted areas are determined using a visibility graph or ray casting.

[0081] The scanning path refers to the sequence of trajectories that a laser scanning device moves through in three-dimensional space.

[0082] Figure 2 This is a flowchart illustrating the generation of scan paths for multiple key buildings according to some embodiments of this specification, such as... Figure 2 As shown, generating multiple scan paths for key buildings based on their 3D models can include the following process:

[0083] S11. Based on the scanning interval included in the optimal laser scanning parameters, multiple viewpoints are uniformly set on the three-dimensional model of the key building, wherein the distance between any two adjacent viewpoints is the scanning interval included in the optimal laser scanning parameters.

[0084] S12. Set the path generation constraint set, which may include the maximum total length of the scan path, the maximum total scan time, the next viewpoint selection constraint, etc. The next viewpoint selection constraint means that the next viewpoint must be the adjacent viewpoint of the current viewpoint.

[0085] S13. Use viewpoints within key areas of the 3D model of key buildings as multiple starting points;

[0086] S14. For each key region, randomly select a viewpoint within the key region as the current viewpoint;

[0087] S15. Randomly select a viewpoint from the adjacent viewpoints of the current viewpoint as the next viewpoint;

[0088] S16. Take the next viewpoint as the current viewpoint of the currently generated scan path, and determine whether the currently generated scan path meets the preset conditions. The preset conditions are that the total length of the scan path is equal to the maximum total length of the scan path, the total scanning time is equal to the maximum total scanning time, or the unscanned area corresponding to the currently generated scan path is less than the unscanned area threshold. If not, execute S15. If yes, complete the generation of a scan path and execute S17.

[0089] S17. Determine whether the number of scan paths starting from the viewpoint within the critical region is greater than the number threshold. If yes, complete the generation of scan paths for the critical region and execute S18. If no, execute S14.

[0090] S18. Determine whether the scanning path of each key area is greater than the quantity threshold. If yes, complete the generation of all scanning paths. If not, select the next key area for which no scanning path has been generated and execute S14.

[0091] Multiple path evaluation metrics may include:

[0092] Path length metric: The total length of the scanning path. The shorter the path length, the higher the scanning efficiency and the lower the energy consumption.

[0093] Scanning time metric: The total time required to complete the scan path (including movement time and viewpoint dwell time).

[0094] Significance: The shorter the scanning time, the shorter the project cycle and the lower the cost.

[0095] Data coverage metric: The percentage of key buildings covered by the scan path. Higher coverage indicates better data integrity, which is beneficial for subsequent analysis or modeling.

[0096] Key area coverage metric: The percentage of key areas covered by the scan path. Ensure data integrity in high-precision or functionally critical areas.

[0097] Path smoothness index: the smoothness of the scanning path (e.g., number of sharp turns, frequency of height changes). A smooth path reduces mechanical wear and scanning errors.

[0098] The fitness function is a mathematical model used in genetic algorithms to evaluate the quality of a path. It can be a weighted combination of the scores of multiple path evaluation metrics.

[0099] The fitness value of each individual (scanning path) in the population can be calculated based on the fitness function. A higher fitness value indicates a better path quality. Methods such as roulette wheel selection and tournament selection are used to select individuals for the next generation based on their fitness values. High-quality paths are retained, while low-quality paths are eliminated. Crossover operations (such as single-point crossover or sequential crossover) are performed on the selected individuals to generate new individuals. New path combinations are generated through genetic recombination to explore the solution space. Mutation operations (such as randomly replacing viewpoints or adjusting viewpoint order) are performed on the crossover individuals to introduce randomness. This avoids getting trapped in local optima and increases population diversity. The genetic algorithm terminates when the maximum number of iterations is reached, the fitness value converges, or a satisfactory solution is found. The individual with the highest fitness value is returned as the optimal scanning path.

[0100] Step 120: Based on the current multimodal data used for settlement digitization, establish a physical 3D model of the settlement and construct a digital twin model of the current status of the settlement.

[0101] Specifically, it includes:

[0102] Based on the images and laser scanning data of the settlement, a system was established as follows: Figure 3 The solid three-dimensional model of the settlement shown;

[0103] Based on the text data, generate information annotations for the three-dimensional physical model of the settlement;

[0104] Based on meteorological and vegetation data of the settlement, several key environmental and ecological simulation scenarios of the settlement were established.

[0105] Based on the spatial vector data of the settlement, the physical 3D model, the information annotation of the physical 3D model, and multiple key environmental and ecological simulation scenarios, a digital twin model of the current status of the settlement is generated.

[0106] Specifically, three-dimensional reconstruction algorithms (such as Structure from Motion, SfM) or specialized software (such as ContextCapture, Agisoft Metashape) are used to fuse the images and laser scanning data of the settlement to generate a solid three-dimensional model of the settlement. The solid three-dimensional model of the settlement can include at least the three-dimensional models of the settlement's buildings, and can also include three-dimensional models of infrastructure (e.g., power facilities (such as utility poles and transformers), communication facilities (such as base stations), water supply and drainage facilities (such as manhole covers and pipes)) and artificial features (e.g., sculptures, monuments, fountains, streetlights).

[0107] Use annotation tools (such as ArcGIS or QGIS) to embed text information into the 3D model of the settlement as labels, pop-ups, or layers. Annotation content can include building names, uses, historical background, and protection levels.

[0108] In some embodiments, based on meteorological and vegetation data of the settlement, multiple key environmental and ecological simulation scenarios of the settlement are established, including:

[0109] Acquire meteorological data, vegetation data, and multiple key environmental and ecological simulation scenarios from multiple sample settlements;

[0110] Based on the meteorological and vegetation data of the sample settlements, the meteorological and vegetation characteristics of the sample settlements were determined.

[0111] Based on the meteorological and vegetation data of the settlement, the meteorological environment characteristics and vegetation characteristics of the settlement are determined.

[0112] Based on the meteorological and vegetation characteristics of the settlements and the meteorological and vegetation characteristics of the sample settlements, similar sample settlements are identified.

[0113] Based on multiple key environmental and ecological simulation scenarios of similar sample settlements, multiple key environmental and ecological simulation scenarios of settlements are established.

[0114] Specifically, the sample settlement can be a settlement with multiple key environmental and ecological simulation scenarios already identified. The sample settlement can be a virtual settlement or a real settlement. The multiple key environmental and ecological simulation scenarios for the sample settlement can be multiple key environmental and ecological simulation scenarios with varying natural landscapes. These scenarios can simulate the impacts of weather events such as heavy rain, drought, typhoons, high temperatures, sunny days, and rainy days on the settlement environment and ecosystem. The key environmental and ecological simulation scenarios emphasize vegetation and meteorology as important components of the natural environment; by simulating their distribution and changes, a more realistic and vivid 3D scene can be constructed. This simulation not only helps improve the visual effect of the scene but also provides strong support for ecological research and environmental planning.

[0115] Statistical analysis is performed on meteorological and vegetation data of settlements to extract meteorological environmental characteristics (e.g., average annual temperature, seasonal precipitation distribution) and vegetation characteristics (e.g., vegetation cover, distribution ratio of different plant types, etc.). Similarly, for the meteorological and vegetation data of sample settlements, the meteorological environmental and vegetation characteristics of the sample settlements are determined. Similarity measurement methods (e.g., Euclidean distance, cosine similarity) are used to calculate the feature similarity between the meteorological and vegetation characteristics of the settlements and those of the sample settlements. Sample settlements with feature similarity scores greater than a feature similarity threshold are considered similar sample settlements. The feature similarity threshold can be determined experimentally or manually (e.g., by experts).

[0116] Multiple key environmental and ecological simulation scenarios of similar sample settlements with the highest feature similarity can be used as multiple key environmental and ecological simulation scenarios of the settlement.

[0117] Assume the current settlement is a plain settlement located in a subtropical region, with main vegetation consisting of farmland and sparse woodland. A sample settlement is a mountain settlement located in a similar climate zone. Several key environmental and ecological simulation scenarios have been identified, including stormwater flood simulation, vegetation growth simulation, and soil erosion simulation. A similar sample settlement is selected. The stormwater flood simulation scenario, included in the key environmental and ecological simulation scenarios of the similar sample settlement, is transferred to the current settlement. Due to the lower rainfall in the current settlement, the rainfall intensity is reduced by 20%. The growth rate parameters in the vegetation growth model are adjusted to suit the farmland and sparse woodland of the current settlement. Hydrological simulation is performed using the SWAT (Synergistic SWAT Model for Hydrological Simulation) model, adjusting the model parameters according to the current settlement's topography and soil characteristics. A stormwater flood simulation scenario for the current settlement is generated.

[0118] This project integrates spatial vector data of settlements (such as roads, waterways, and building outlines) with physical 3D models, information annotations, and environmental and ecological simulation scenarios. Data fusion is performed using Geographic Information Systems (GIS) or 3D visualization platforms (such as Unity and Unreal Engine). A digital twin model of the settlement is then constructed within the 3D visualization platform, showcasing its spatial layout, building forms, environmental characteristics, and ecological simulations. User interactive operations are supported, such as zooming, rotating, querying information annotations, and switching simulation scenes.

[0119] Step 130: Based on the current multimodal data used for settlement digitization, determine multiple historical development nodes of the settlement.

[0120] Specifically, it includes:

[0121] Based on the architectural features and textual data of each building in the settlement, the construction time of each building in the settlement is traced back, and the architectural evolution information of the settlement is determined based on the construction time of each building in the settlement.

[0122] Based on the spatial vector data of the settlement, determine the road network evolution information of the settlement;

[0123] Based on the spatial vector data of the settlement, the spatial evolution information of the land use type of the settlement is determined. Among them, the multiple historical development nodes of the settlement include at least the architectural evolution information, road network evolution information, and land use type spatial evolution information of the settlement.

[0124] Specifically, building feature data is transformed into numerical variables (e.g., style encoding is converted into numerical labels), and information is extracted from building-related text data using natural language processing techniques. A deep learning model is then used to predict construction time. During deep learning model training, building features and text data are used as input, and known construction times are used as output. The accuracy of the deep learning model is verified through cross-validation or hold-out methods, and the model parameters are adjusted to improve prediction accuracy. For each building, its most likely construction time range is determined based on the deep learning model's prediction results. Based on the construction time and spatial location of each building in the settlement, the architectural evolution information of the settlement is determined. This architectural evolution information can include the main characteristics and distribution of buildings at different historical development stages.

[0125] By collecting paper or electronic maps of settlements from different historical periods, digitizing the paper maps (e.g., scanning, vectorization), and extracting road information, the extracted road data can be arranged chronologically to construct a time-series dataset. Dynamic maps can then be created using GIS tools (such as ArcGIS Pro and QGIS) or WebGIS platforms (such as Leaflet and Mapbox) to display the evolution of the settlement's road network—that is, how the road network changes over time. Example: Roads in the 1950s are shown in red, newly added roads in the 1980s are shown in blue, and roads rerouted in the 2000s are shown in green.

[0126] Land use classification data (such as cultivated land, forest land, and construction land) for historical periods (such as the 1980s, 2000s, and 2020s) are obtained from sources including historical maps and remote sensing image interpretation. The total area of ​​agricultural land and the area of ​​each type (such as paddy fields and dry land) at different historical development nodes are calculated, and the distribution ratio of agricultural land in different slope and altitude ranges is statistically analyzed to determine the spatial evolution information of land use types of settlements.

[0127] Step 140: Based on the predicted multiple historical development nodes of the settlement and the physical 3D model of the settlement, establish a 3D model of the historical development nodes of the settlement and construct a digital twin model of the historical development node settlement.

[0128] Specifically, it includes:

[0129] Based on the settlement's architectural evolution information, road network evolution information, and land use type spatial evolution information, multiple historical development nodes of the settlement are identified.

[0130] For each historical development node, a three-dimensional model of the historical development node of the settlement is established based on the settlement's architectural evolution information, road network evolution information, land use type spatial evolution information, and the settlement's physical three-dimensional model. Based on multiple key environmental and ecological simulation scenarios of the settlement, a digital twin model of the historical development node settlement is constructed.

[0131] Specifically, representative time points can be selected as historical development nodes based on information on architectural evolution (such as changes in architectural style and function), road network evolution (such as road widening and new construction), and spatial evolution of land use types (such as farmland expansion and conversion of farmland to forest). For example, Node 1: 1950 (buildings were mainly adobe houses, roads were dirt roads, and agricultural land accounted for a high proportion). Node 2: 1980 (some buildings were converted to brick and tile houses, roads began to be paved, and some agricultural land was converted to industrial land). Node 3: 2020 (modernized settlements, an increase in high-rise buildings, improved road networks, and centralized agricultural land).

[0132] Extract building types, heights, and distribution ranges for each historical development node, and combine them with building geometry data from the solid 3D model to generate a 3D model of historical buildings. Based on historical road vector data, generate 3D models of road networks (e.g., dirt roads, gravel roads, asphalt roads) for different historical development nodes. Convert historical land use type spatial data (e.g., cultivated land, forest land, water areas) into 3D terrain surface textures or elevation changes. Use 3D modeling software (e.g., SketchUp, Blender) or GIS tools (e.g., ArcGIS CityEngine) to generate 3D models of historical buildings, adjusting materials and colors to reflect historical characteristics. Adjust the geometry and texture of the road 3D model according to road type (e.g., width, material), and simulate the distribution of historical agricultural land by modifying terrain elevation or overlaying vegetation textures. Merge the building, road, and agricultural land models with the solid 3D models of settlements (e.g., terrain, water systems) to form a complete 3D model of historical development nodes. Using virtual reality (VR) development platforms (such as Unity and Unreal Engine) or 3D GIS platforms (such as Cesium and ArcGIS Earth), 3D models of historical development nodes are integrated with key environmental and ecological simulation scenarios to generate digital twin models of settlements at historical development nodes.

[0133] Figure 4 This is a flowchart illustrating a settlement digital twin modeling system based on multimodal data fusion, as shown in some embodiments of this specification. Figure 4 As shown, a settlement digital twin modeling system based on multimodal data fusion can include a multimodal data acquisition module, a digital twin establishment module, and a development analysis module.

[0134] A multimodal data acquisition module is used to acquire current multimodal data for settlement digitization, wherein the current multimodal data for settlement digitization includes at least spatial vector data, images, laser scan data, meteorological data, vegetation data and text data;

[0135] The digital twin creation module is used to create a physical 3D model of the settlement based on the current multimodal data used for settlement digitization, and to construct a digital twin model of the current status of the settlement.

[0136] The development analysis module is used to identify multiple historical development nodes of a settlement based on the current multimodal data used for settlement digitization.

[0137] The digital twin creation module is also used to create a three-dimensional model of the historical development nodes of the settlement based on the predicted multiple historical development nodes of the settlement and the physical three-dimensional model of the settlement, and to construct a digital twin model of the historical development node settlement.

[0138] The settlement digital twin modeling system based on multimodal data fusion can be used to execute the settlement digital twin modeling method based on multimodal data fusion described above, which will not be elaborated here.

[0139] Finally, it should be understood that the embodiments described in this specification are merely illustrative of the principles of the embodiments described herein. Other variations may also fall within the scope of this specification. Therefore, alternative configurations of the embodiments described herein are intended to be illustrative rather than limiting, and should be considered consistent with the teachings of this specification. Accordingly, the embodiments described herein are not limited to those explicitly introduced and described herein.

Claims

1. A settlement digital twin modeling system based on multimodal data fusion, characterized in that, include: A multimodal data acquisition module is used to acquire current multimodal data for settlement digitization, wherein the current multimodal data for settlement digitization includes at least spatial vector data, images, laser scan data, meteorological data, vegetation data and text data; The digital twin creation module is used to create a physical 3D model of the settlement based on the current multimodal data used for settlement digitization, and to construct a digital twin model of the current status of the settlement. The development analysis module is used to identify multiple historical development nodes of a settlement based on the current multimodal data used for settlement digitization. The digital twin building module is also used to build a three-dimensional model of the historical development nodes of the settlement based on the predicted multiple historical development nodes of the settlement and the physical three-dimensional model of the settlement, and to construct a digital twin model of the historical development node settlement. The multimodal data acquisition module acquires laser scanning data of the settlement, including: Based on the imagery and spatial vector data of the settlement, the key buildings of the settlement were identified; For each key building in the settlement, the optimal laser scanning parameters and optimal scanning path of the key building are determined based on the image of the key building. Based on the optimal laser scanning parameters and optimal scanning path of the key building, the key building is scanned to obtain the laser scanning data of the key building. Based on the imagery of key buildings, determine the optimal scanning path for those buildings, including: Generate a 3D model of the key building based on its imagery. Based on the images of key buildings, identify the key areas of the 3D model of the key buildings; Based on the 3D model of the key buildings, generate multiple scanning paths for the key buildings; Identify key areas of the 3D model of multiple path evaluation indicators and key buildings, and establish a fitness function; The optimal scanning path for key buildings is generated using a genetic algorithm based on a fitness function and multiple scanning paths for key buildings. Based on the 3D model of the key buildings, multiple scan paths for the key buildings are generated, including the following process: S11. Based on the scanning spacing included in the optimal laser scanning parameters, multiple viewpoints are uniformly set on the three-dimensional model of the key building, wherein the distance between any two adjacent viewpoints is the scanning spacing included in the optimal laser scanning parameters. S12. Set the path generation constraint set, which includes the maximum total length of the scan path, the maximum total scan time, the next viewpoint selection constraint, etc. The next viewpoint selection constraint means that the next viewpoint must be the adjacent viewpoint of the current viewpoint. S13. Use viewpoints within key areas of the 3D model of key buildings as multiple starting points; S14. For each key region, randomly select a viewpoint within the key region as the current viewpoint; S15. Randomly select a viewpoint from the adjacent viewpoints of the current viewpoint as the next viewpoint; S16. Take the next viewpoint as the current viewpoint of the currently generated scan path, and determine whether the currently generated scan path meets the preset conditions. The preset conditions are that the total length of the scan path is equal to the maximum total length of the scan path, the total scanning time is equal to the maximum total scanning time, or the unscanned area corresponding to the currently generated scan path is less than the unscanned area threshold. If not, execute S15. If yes, complete the generation of a scan path and execute S17. S17. Determine whether the number of scan paths starting from the viewpoint in the critical area is greater than the number threshold. If yes, complete the generation of scan paths in the critical area and execute S18. If no, execute S14. S18. Determine whether the scanning path of each key area is greater than the quantity threshold. If yes, complete the generation of all scanning paths. If not, select the next key area for which no scanning path has been generated and execute S14.

2. The settlement digital twin modeling system based on multimodal data fusion according to claim 1, characterized in that, The digital twin creation module establishes a physical 3D model of the settlement based on the current multimodal data used for settlement digitization, constructing a digital twin model of the settlement's current status, including: The three-dimensional reconstruction algorithm is used to fuse the images and laser scanning data of the settlement to generate a physical three-dimensional model of the settlement and establish the physical three-dimensional model of the settlement. Based on the text data, generate information annotations for the three-dimensional physical model of the settlement; Based on meteorological and vegetation data of the settlement, several key environmental and ecological simulation scenarios of the settlement were established. Based on the spatial vector data of the settlement, the physical 3D model, the information annotation of the physical 3D model, and multiple key environmental and ecological simulation scenarios, a digital twin model of the current status of the settlement is generated.

3. The settlement digital twin modeling system based on multimodal data fusion according to claim 2, characterized in that, The digital twin creation module establishes multiple key environmental and ecological simulation scenarios for the settlement based on its meteorological and vegetation data, including: Acquire meteorological data, vegetation data, and multiple key environmental and ecological simulation scenarios from multiple sample settlements; Based on the meteorological and vegetation data of the sample settlements, the meteorological and vegetation characteristics of the sample settlements were determined. Based on the meteorological and vegetation data of the settlement, the meteorological environment characteristics and vegetation characteristics of the settlement are determined. Based on the meteorological and vegetation characteristics of the settlements and the meteorological and vegetation characteristics of the sample settlements, similar sample settlements are identified. Based on multiple key environmental and ecological simulation scenarios of similar sample settlements, multiple key environmental and ecological simulation scenarios of settlements are established.

4. The settlement digital twin modeling system based on multimodal data fusion according to claim 3, characterized in that, The development analysis module determines multiple historical development nodes of the settlement based on the current multimodal data used for settlement digitization, including: Based on the architectural features and textual data of each building in the settlement, the construction time of each building in the settlement is traced back, and the architectural evolution information of the settlement is determined based on the construction time of each building in the settlement. Based on the spatial vector data of the settlement, determine the road network evolution information of the settlement; Based on the spatial vector data of the settlement, the spatial evolution information of the land use type of the settlement is determined. The multiple historical development nodes of the settlement include at least the architectural evolution information, road network evolution information, and land use type spatial evolution information of the settlement.

5. The settlement digital twin modeling system based on multimodal data fusion according to claim 4, characterized in that, The digital twin creation module establishes a 3D model of the historical development nodes of the settlement based on the predicted multiple historical development nodes of the settlement and the physical 3D model of the settlement, and constructs a digital twin model of the settlement based on the historical development nodes, including: Based on the settlement's architectural evolution information, road network evolution information, and land use type spatial evolution information, multiple historical development nodes of the settlement are identified. For each historical development node, a three-dimensional model of the historical development node of the settlement is established based on the settlement's architectural evolution information, road network evolution information, land use type spatial evolution information, and the settlement's physical three-dimensional model. Based on multiple key environmental and ecological simulation scenarios of the settlement, a digital twin model of the historical development node settlement is constructed.

6. The settlement digital twin modeling system based on multimodal data fusion according to claim 5, characterized in that, The digital twin creation module establishes a three-dimensional model of the historical development nodes of the settlement based on the settlement's architectural evolution information, road network evolution information, land use type spatial evolution information, and the settlement's physical three-dimensional model, including: Extract the building types, heights, and distribution ranges of historical development nodes, combine them with the building geometry data in the solid 3D model, generate a 3D model of the historical building, and adjust the materials and colors. Generate a 3D model of the road network based on historical road vector data and historical development nodes; Historical land use type spatial data is converted into texture or elevation changes of three-dimensional terrain surfaces to generate agricultural land use models; By integrating 3D models of historical buildings, road networks, and agricultural land, a 3D model of the historical development nodes of the settlement is established.

7. The settlement digital twin modeling system based on multimodal data fusion according to claim 1, characterized in that, The multimodal data acquisition module determines the key buildings of the settlement based on the settlement's imagery and spatial vector data, including: Based on images of the settlement, determine the architectural features of each building in the settlement, including at least color and outline features. Based on the architectural features of each building in the settlement and the spatial vector data of the settlement, the key buildings of the settlement are identified.

8. The settlement digital twin modeling system based on multimodal data fusion according to claim 7, characterized in that, The multimodal data acquisition module identifies key buildings in the settlement based on the architectural features of each building and the spatial vector data of the settlement, including: For each building in the settlement, the characteristic difference value of the building is determined based on the architectural characteristics of each building in the settlement; The target building is determined based on the characteristic differences of each building in the settlement; For each target building in the settlement, similar sample buildings are identified based on the architectural features of the target building and the sample buildings. The historical and cultural value of the target building is determined based on the historical and cultural value of the similar sample buildings. The location center value of the target building is calculated based on the spatial vector data of the settlement. The key buildings of the settlement are identified based on the historical and cultural value and central location of each target building in the settlement.

9. A settlement digital twin modeling method based on multimodal data fusion, characterized in that, The settlement digital twin modeling system based on multimodal data fusion, as described in any one of claims 1-8, comprises: Acquire current multimodal data for settlement digitization, wherein the current multimodal data for settlement digitization includes at least imagery, laser scanning data, environmental data, and text data; Based on the current multimodal data used for settlement digitization, a physical three-dimensional model of the settlement is established, and a digital twin model of the current status of the settlement is constructed. Based on the current multimodal data used for settlement digitization, multiple historical development nodes of the settlement are identified; Based on the predicted historical development nodes of the settlement and the physical 3D model of the settlement, a 3D model of the historical development nodes of the settlement is established, and a digital twin model of the settlement based on the historical development nodes is constructed.

Images