Multi-modal data fusion-based settlement digital twin modeling method and system

Abstract

The invention provides a settlement digital twinborn modeling method and system based on multi-modal data fusion, and relates to the field of digital twinborn, and the system comprises a multi-modal data obtaining module which is used for obtaining current situation multi-modal data used for carrying out settlement digitization; the digital twinning establishment module is used for establishing a settlement entity three-dimensional model according to the current situation multi-modal data for settlement digitalization and constructing a settlement current situation digital twinning model; the development analysis module is used for determining a plurality of historical development nodes of settlement according to the current situation multi-modal data for performing settlement digitalization; and the digital twinning establishment module is also used for establishing a historical development node three-dimensional model of the settlement according to the predicted historical development nodes of the settlement and the entity three-dimensional model of the settlement, and constructing a historical development node settlement digital twinning model, and the method has the advantage of realizing settlement landscape style and environment change digitization.

Description

Technical Field

[0001] The present invention relates to the field of digital twins, and in particular to a settlement digital twin modeling method and system based on multimodal data fusion. Background Art

[0002] Settlement environments are human settlements created through the conscious development, utilization, and transformation of nature. Applying digital twins to settlement landscapes empowers information-based and intelligent planning and management. Digital twins build a digital world parallel to the physical world, mapping the virtual and real worlds to accurately reflect changes in the real world. In landscape architecture, digital twin technology is used to simulate and simulate the real world. Leveraging cutting-edge information technologies such as the Internet of Things and virtual reality, and leveraging computational graphics and multi-sensor technologies, digital twins simulate and map the existing state of settlements based on big data collected from microenvironmental observations in the physical world. Settlement cultural landscapes are a vital component of cultural heritage, carrying a wealth of historical, cultural, and ecological information. Generating digital twin models of settlements can permanently preserve this information, preventing damage and loss due to natural or human factors, thereby ensuring the long-term protection and preservation of cultural heritage. Settlement digital twin models provide scholars and researchers with a convenient research tool. Through digital archives of settlement cultural landscapes, they can gain a deeper understanding of their structure, evolution, and interrelationships with the environment, thereby promoting the development of related disciplines. At the same time, settlement digital twins can also serve as educational resources, helping students understand and learn about cultural heritage. Settlement digital twins are not limited to preservation and research; they can also support the revitalization and utilization of cultural heritage. Through digital means, settlement cultural landscapes can be presented to the public in a more vivid and intuitive way, enhancing public awareness and interest in cultural heritage and promoting the development of cultural heritage in tourism, education, and creative industries.

[0003] Existing records of settlement cultural landscapes often rely on text descriptions, two-dimensional images or simple surveying and mapping data. These information have limitations in expressing the landscape's three-dimensional structure, dynamic changes, and relationship with the environment.

[0004] Therefore, it is necessary to provide a settlement digital twin modeling method and system based on multimodal data fusion to realize the digitization of settlement landscape features and environmental changes. Summary of the Invention

[0005] The present invention provides a settlement digital twin modeling system based on multimodal data fusion, including: a multimodal data acquisition module, used to acquire current multimodal data for settlement digitization, wherein the current multimodal data for settlement digitization includes at least spatial vector data, images, laser scanning data, meteorological data, vegetation data and text data; a digital twin establishment module, used to establish a physical three-dimensional model of the settlement based on the current multimodal data for settlement digitization, and construct a current digital twin model of the settlement; a development analysis module, used to determine multiple historical development nodes of the settlement based on the current multimodal data for settlement digitization; the digital twin establishment module is also used to establish a three-dimensional model of the historical development nodes of the settlement based on the predicted multiple historical development nodes of the settlement and the physical three-dimensional model of the settlement, and construct a digital twin model of the settlement of historical development nodes.

[0006] Furthermore, the digital twin establishment module establishes a physical three-dimensional model of the settlement based on the current multimodal data used to digitize the settlement, and constructs a digital twin model of the current status of the settlement, including: using a three-dimensional reconstruction algorithm to fuse the image and laser scanning data of the settlement to generate a physical three-dimensional model of the settlement, and establish a physical three-dimensional model of the settlement; generating information annotations for the physical three-dimensional model of the settlement based on text data; establishing multiple key environmental and ecological simulation scenarios of the settlement based on the meteorological data and vegetation data of the settlement; generating a digital twin model of the current status of the settlement based on the spatial vector data, physical three-dimensional model, information annotations of the physical three-dimensional model and multiple key environmental and ecological simulation scenarios of the settlement.

[0007] Furthermore, the digital twin establishment module establishes multiple key environmental and ecological simulation scenarios of the settlement based on the meteorological data and vegetation data of the settlement, including: obtaining meteorological data, vegetation data and multiple key environmental and ecological simulation scenarios of multiple sample settlements; determining the meteorological environmental characteristics and vegetation characteristics of the sample settlement based on the meteorological data and vegetation data of the sample settlement; determining the meteorological environmental characteristics and vegetation characteristics of the settlement based on the meteorological data and vegetation data of the settlement; determining similar sample settlements based on the meteorological environmental characteristics and vegetation characteristics of the settlement and the meteorological environmental characteristics and vegetation characteristics of the sample settlement; and establishing multiple key environmental and ecological simulation scenarios of the settlement based on multiple key environmental and ecological simulation scenarios of similar sample settlements.

[0008] Furthermore, the development analysis module determines multiple historical development nodes of the settlement based on the current multimodal data used to digitize the settlement, including: tracing back the construction time of each building in the settlement based on the architectural features and text data of each building in the settlement, and determining the architectural evolution information of the settlement based on the construction time of each building in the settlement; determining the road network evolution information of the settlement based on the spatial vector data of the settlement; and determining the spatial evolution information of the land use type of the settlement based on the spatial vector data of the settlement, wherein the multiple historical development nodes of the settlement include at least the architectural evolution information, road network evolution information and land use type spatial evolution information of the settlement.

[0009] Furthermore, the digital twin establishment module establishes a three-dimensional model of the historical development nodes of the settlement based on the predicted multiple historical development nodes of the settlement and the physical three-dimensional model of the settlement, and constructs a digital twin model of the historical development node settlement, including: determining multiple historical development nodes of the settlement based on the architectural evolution information, road network evolution information and land use type spatial evolution information of the settlement; for each historical development node, establishing a three-dimensional model of the historical development node of the settlement based on the architectural evolution information, road network evolution information and land use type spatial evolution information and the physical three-dimensional model of the settlement, and constructing a digital twin model of the historical development node settlement based on multiple key environmental and ecological simulation scenarios of the settlement.

[0010] Furthermore, the digital twin establishment module establishes a three-dimensional model of the historical development nodes of the settlement based on the architectural evolution information, road network evolution information, land use type spatial evolution information and the physical three-dimensional model of the settlement, including: extracting the building type, height, and distribution range of the historical development nodes, combining the building geometry data in the physical three-dimensional model to generate a three-dimensional model of the historical building, and adjusting the material and color; generating a three-dimensional road network model of the historical development node based on the historical road vector data; converting the historical land use type spatial data into the texture or elevation change of the three-dimensional terrain surface to generate an agricultural land model; integrating the three-dimensional historical building model, the three-dimensional road network model and the agricultural land model to establish a three-dimensional model of the historical development nodes of the settlement.

[0011] Furthermore, the multimodal data acquisition module acquires laser scanning data of the settlement, including: determining the key buildings of the settlement based on the image and spatial vector data of the settlement; for each key building in the settlement, determining the optimal laser scanning parameters and optimal scanning path of the key building based on the image of the key building; scanning the key building based on the optimal laser scanning parameters and optimal scanning path of the key building to obtain laser scanning data of the key building.

[0012] Furthermore, the multimodal data acquisition module determines the key buildings of the settlement based on the image and spatial vector data of the settlement, including: determining the architectural features of each building in the settlement based on the image of the settlement, wherein the architectural features include at least color features and contour features; and determining the key buildings of the settlement based on the architectural features of each building in the settlement and the spatial vector data of the settlement.

[0013] Furthermore, the multimodal data acquisition module determines the key buildings of the settlement based on the architectural features of each building in the settlement and the spatial vector data of the settlement, including: for each building in the settlement, determining the characteristic difference value of the building according to the architectural features of each building in the settlement; determining the target building according to the characteristic difference value of each building in the settlement; for each target building in the settlement, determining similar sample buildings based on the architectural features of the target building and the architectural features of the sample buildings, determining the historical and cultural value of the target building according to the historical and cultural values ​​of the similar sample buildings, and calculating the position center value of the target building according to the spatial vector data of the settlement; and determining the key buildings of the settlement according to the historical and cultural value and the position center value of each target building in the settlement.

[0014] The present invention provides a settlement digital twin modeling method based on multimodal data fusion, which is applied to the above-mentioned settlement digital twin modeling system based on multimodal data fusion, including: obtaining current multimodal data for settlement digitization, wherein the current multimodal data for settlement digitization includes at least images, laser scanning data, environmental data and text data; establishing a physical three-dimensional model of the settlement based on the current multimodal data for settlement digitization, and constructing a current digital twin model of the settlement; determining multiple historical development nodes of the settlement based on the current multimodal data for settlement digitization; establishing a three-dimensional model of the historical development nodes of the settlement based on the predicted multiple historical development nodes of the settlement and the physical three-dimensional model of the settlement, and constructing a digital twin model of the settlement of the historical development nodes.

[0015] Compared with the existing technology, the settlement digital twin modeling method and system based on multimodal data fusion provided by the present invention have at least the following beneficial effects: 1. By extracting historical development information (such as architectural evolution, road network changes, and land use type and space), the current 3D model is linked to historical data to form a dynamic time series archive. Through this digital archive, users can visually observe the evolution of the settlement from past to present and understand the formation mechanisms of the cultural landscape.

[0016] 2. By combining the color and outline features of the image with the location information of the spatial vector data, key buildings with historical and cultural value or spatial representativeness can be accurately identified. This avoids indiscriminate scanning of all buildings, reduces data redundancy, and focuses on those that contribute most to the cultural landscape digital archive. The historical and cultural value of the target building is assessed based on the similarity of architectural features between the target building and the sample building, combined with the historical and cultural value of the sample building. This provides a scientific basis for the selection of key buildings, ensuring that the cultural landscape digital archive contains the most representative cultural heritage.

[0017] 3. By using the architectural features of historically scanned buildings and optimal laser scanning parameters, key buildings are matched with similar historically scanned buildings and their optimal scanning parameters are leveraged. This avoids blindly setting scanning parameters, improves scanning efficiency and data quality, and ensures the accuracy of 3D models of key buildings. Optimal laser scanning parameters, including resolution, sampling rate, and scanning spacing, can be fine-tuned based on the architectural features of key buildings. This meets the scanning requirements of different building types (e.g., historic buildings and modern structures) and ensures detailed representation of 3D models. A 3D model is generated based on images of key buildings, and key areas are identified, providing an intuitive basis for scanning path planning. This ensures that the scanning path covers key areas to avoid missing important details. Multiple scanning paths are generated and, using a genetic algorithm, they combine path evaluation metrics (such as path length, scanning time, and coverage) with key areas to determine the optimal scanning path. This ensures scan quality while reducing scanning time and labor costs and improving scanning efficiency.

[0018] 4. A physical 3D model is constructed using imagery and laser scanning data of the settlement. Information annotations are generated by combining text data to ensure the accuracy of the model's spatial form and cultural connotations. Key environmental and ecological simulation scenarios (such as seasonal changes and vegetation cover variations) are created based on meteorological and vegetation data and integrated into the 3D digital twin base. This enhances the realism and immersiveness of the sand table, helping users intuitively understand the interactive relationship between the settlement and its natural environment. Using multi-dimensional data such as architectural features, text data, and spatial vector data, the system predicts building construction dates and determines the spatial evolution of buildings, road networks, and land use types. This systematically reveals the historical development patterns of the settlement, providing a scientific basis for cultural heritage preservation and historical research. For each historical development node, a 3D model is constructed based on this evolutionary information and the physical 3D model. This model is then combined with environmental and ecological simulation scenarios to generate a digital twin of the settlement. Users can dynamically switch between sand table scenarios in different historical periods, intuitively experiencing the historical evolution of the settlement and providing decision-making support for cultural heritage preservation, urban planning, and other areas. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] This specification will be further described in the form of exemplary embodiments, which will be described in detail with reference to the accompanying drawings. These embodiments are not limiting, and in these embodiments, like numbers represent like structures, wherein: Figure 1 This is a flowchart of a settlement digital twin modeling method based on multimodal data fusion according to some embodiments of this specification; Figure 2 is a schematic diagram of a process for generating scanning paths of multiple key buildings according to some embodiments of this specification; Figure 3 is a schematic diagram of a solid three-dimensional model of a settlement according to some embodiments of this specification; Figure 4 It is a module diagram of a settlement digital twin modeling system based on multimodal data fusion according to some embodiments of this specification. DETAILED DESCRIPTION

[0020] To more clearly illustrate the technical solutions of the embodiments of this specification, the following briefly describes the drawings required for describing the embodiments. Obviously, the drawings described below are merely examples or embodiments of this specification. Those skilled in the art can apply this specification to other similar scenarios based on these drawings without inventive effort. Unless otherwise apparent from the context or otherwise noted, the same reference numerals in the figures represent the same structure or operation.

[0021] Figure 1 This is a flow chart of a settlement digital twin modeling method based on multimodal data fusion according to some embodiments of this specification, such as Figure 1 As shown, the settlement digital twin modeling method based on multimodal data fusion can include the following steps.

[0022] Step 110: Acquire current multimodal data for settlement digitization.

[0023] Among them, the current multimodal data used for settlement digitization includes at least spatial vector data, images, laser scanning data, meteorological data, vegetation data and text data.

[0024] Specifically, spatial vector data is geographic information represented by geometric elements such as points, lines, and surfaces, reflecting the spatial structure and functional zoning of settlements. It can include: Building data: building plan boundaries.

[0025] Road network: road centerline, width, grade (main road, branch road), material (asphalt, slate).

[0026] Water systems and green spaces: boundaries and attributes of rivers, lakes, parks, and shelterbelts.

[0027] Functional zoning: the scope and use of residential areas, commercial areas, and agricultural areas.

[0028] Settlement images are used to reflect the visual information of the settlement's appearance, architectural style, and spatial layout. Specifically, they may include: Aerial imagery: A bird's-eye view covering the entire settlement, used to analyze spatial patterns.

[0029] Ground-level photography: building facades and street details, used to identify architectural styles and materials.

[0030] Historical photos: archival photos reflecting the appearance of the settlement at different historical development nodes.

[0031] Laser scanning data can include 3D point cloud data acquired through LiDAR (Light Detection and Ranging), reflecting the spatial form and details of settlements. Specifically, laser scanning data can include high-precision building surface data, which can be used to extract roof types and facade structures.

[0032] Meteorological data is used to reflect the microclimate conditions of the settlement area and may include daily / monthly records of temperature, precipitation, humidity, and wind speed.

[0033] Vegetation data is used to reflect the vegetation coverage and land use type space around the settlement, which can specifically include the distribution and area of ​​farmland, woodland, grassland, water area, plant types and proportions, vegetation index, etc.

[0034] Text data is used to reflect the history, culture, and social structure of the settlement, and can specifically include: Historical documents: local chronicles, family genealogies, inscriptions on steles, contract documents, and books collected in ancestral halls and family temples.

[0035] Oral history: resident interview records and folk tales.

[0036] Modern archives: news reports, statistical yearbooks, historical records of village history museums, and intangible cultural heritage information.

[0037] In some embodiments, obtaining laser scanning data of a settlement includes: Determine the key buildings of the settlement based on the settlement image and spatial vector data; For each key building in the settlement, the optimal laser scanning parameters and optimal scanning path of the key building are determined based on the image of the key building. Based on the optimal laser scanning parameters and optimal scanning path of the key building, the key building is scanned to obtain laser scanning data of the key building.

[0038] Specifically, the key buildings of a settlement can be buildings that have historical, cultural, structural or functional significance in the settlement, such as ancient buildings, iconic public buildings, traditional houses, etc.

[0039] In some embodiments, determining key buildings in a settlement based on settlement images and spatial vector data includes: Determine the architectural features of each building in the settlement based on the settlement image, wherein the architectural features at least include color features and outline features; For each building in the settlement, determine the characteristic difference value of the building based on the architectural characteristics of each building in the settlement; Determine the target building based on the characteristic difference value of each building in the settlement; For each target building in the settlement, similar sample buildings are identified based on the architectural characteristics of the target building and the sample buildings. The historical and cultural value of the target building is determined based on the historical and cultural value of the similar sample buildings. The location center value of the target building is calculated based on the spatial vector data of the settlement. Determine the key buildings of the settlement based on the historical and cultural value and location center value of each target building in the settlement.

[0040] Specifically, architectural features are used to describe the attributes of a building's appearance and structure, including color, outline, etc.

[0041] Color features can include the dominant color (e.g., red, gray) and color distribution of building exterior walls and roofs. For each building in the settlement, an image segmentation algorithm can be used to separate the building area image. The frequency distribution of different colors in the building area image is then calculated. By calculating the histogram of each channel in the HSV color space, a feature vector reflecting the building color distribution is obtained. The frequency of occurrence of different hue values ​​(expressed as angles, ranging from 0° to 360°) in the H (hue) channel histogram is counted. The hue range is divided into several bins (e.g., 10° or 20°), and the number of pixels within each bin is counted to form a hue histogram vector. For example, if the hue range is divided into 18 bins (one bin every 20°), the hue histogram vector contains 18 data points. The frequency of occurrence of different saturation values ​​(expressed as percentages, ranging from 0% to 100%) in the S (saturation) channel histogram is counted. The saturation range is divided into several bins (e.g., 10% bins), and the number of pixels within each bin is counted to form a saturation histogram vector. For example, if the image is divided into 10 intervals (one interval for every 10%), the saturation histogram vector contains 10 data points. The frequency of different brightness values ​​(usually expressed as percentages, ranging from 0% to 100%) in the V (luminance) channel histogram is counted, the brightness range is divided into several intervals (such as 10% as one interval), and the number of pixels in each interval is counted to form a brightness histogram vector. For example, if the image is divided into 10 intervals (one interval for every 10%), the brightness histogram vector contains 10 data points. To eliminate the influence of the dimensions of the histogram vectors of different channels, the histogram vectors of each channel are normalized so that their sum is 1. In this way, each data point in the feature vector represents the relative frequency of that color interval in the image. The normalized hue, saturation, and brightness histogram vectors are combined in sequence to form the color feature vector of the building.

[0042] Outline features can include the building's plan shape (e.g., rectangular, L-shaped), facade hierarchy (e.g., single-story, multi-story), and roof form (e.g., flat, sloping). For each building in the settlement, an image segmentation algorithm is used to separate the building area image. Adaptive thresholding and dual-threshold detection algorithms are then used to generate building edges based on the building area image. Energy minimization is performed using a gradient vector flow field to optimize the initially extracted building outline, simplifying it into a polygon. The planar shape is determined by the number of vertices (e.g., 4 vertices = rectangular, 5 vertices = L-shaped, etc.). The shape classification result is encoded as a discrete value (e.g., rectangle = 1, L-shaped = 2) as part of the outline feature vector. Vertical projection is used to count pixel density changes and detect the number of floors (e.g., each floor is approximately 3 meters high). The floor number detection result is encoded as a discrete value (e.g., single-story = 1, two-story = 2, three-story = 3, etc.) as part of the outline feature vector. Use a semantic segmentation model (such as DeepLabV3+) to segment the roof area, or locate the roof through geometric analysis (such as the difference between the convex hull of the outline and the building outline), calculate the height standard deviation of the roof area (such as the standard deviation of flat roofs <0.5 meters, sloped roofs >1 meter), and encode the roof form detection results into discrete values ​​(such as flat roof = 1, sloped roof = 2, dome = 3, etc.) as part of the outline feature vector.

[0043] For each building in the settlement, the cosine similarity of the building's color feature vector with the color feature vector of any other building in the settlement can be calculated. The cosine similarity of the building's outline feature vector with the outline feature vector of any other building in the settlement can also be calculated. The weighted sum of the cosine similarities of the building's color feature vector and the outline feature vector with the other buildings in the settlement is used to obtain the first building similarity between the building and the other buildings in the settlement. The variance of the first building similarity between the building and the other buildings in the settlement is then calculated to obtain the characteristic difference value of the building. Buildings with characteristic difference values ​​greater than a characteristic difference value threshold can be designated as target buildings, where the characteristic difference value threshold can be determined based on human experience or statistical analysis of big data. It is understood that target buildings are buildings in the settlement that are identified as having significant characteristic differences based on similarity analysis of their color features and outline features with other buildings. Target buildings are likely to have high value in representing the characteristics of the settlement.

[0044] Sample buildings are examples of buildings with known historical and cultural value (such as traditional dwellings and historical relics). Historical and cultural value is a comprehensive assessment of a building's historical, cultural, artistic, and scientific aspects, and can be scored by experts based on dimensions such as preservation level, construction age, and unique style. For each target building in the settlement, the second architectural similarity between the target building and the sample building is calculated using the same method as above. Sample buildings with a second architectural similarity greater than the first architectural similarity threshold are considered similar sample buildings. The historical and cultural value of these similar sample buildings is then averaged to determine the historical and cultural value of the target building. The first architectural similarity threshold can be determined experimentally or manually (e.g., by experts).

[0045] The target building's location center value can be used to analyze whether the target building is the core of the settlement's development (e.g., expansion from the building) or to assess its importance in the spatial layout. For example, the coordinates of each building in the settlement can be averaged to represent the coordinates of the settlement's center. For each target building in the settlement, the distance between the target building and the settlement's center is calculated based on the target building's coordinates and the settlement's center's coordinates. Based on this distance, the target building's location center value is calculated. The shorter the distance between the target building and the settlement's center, the greater the target building's location center value.

[0046] For each target building in the settlement, the historical and cultural value and the location center value of the target building are weighted and summed to obtain the comprehensive value of the target building. The target building with a comprehensive value greater than the comprehensive value threshold is regarded as the key building of the settlement, where the comprehensive value threshold can be obtained based on human experience or big data statistical analysis.

[0047] In some embodiments, determining optimal laser scanning parameters for the key building based on the image of the key building includes: Obtain architectural features and optimal laser scanning parameters for historically scanned buildings; Identify similar historical scanned buildings based on the architectural features of the key building and the architectural features of the historical scanned buildings; The optimal laser scanning parameters of the key building are determined based on the optimal laser scanning parameters of similar historical scanned buildings, wherein the optimal laser scanning parameters at least include resolution, sampling rate, and scanning spacing.

[0048] Specifically, the historically scanned building may be a building for which optimal laser scanning parameters have been determined, and the historically scanned building may not belong to the current settlement. The optimal parameters of the historically scanned building can be determined through experiments or manual (e.g., expert) experience.

[0049] The optimal laser scanning parameters refer to a set of parameter combinations set for historical scanned buildings during the laser scanning (LiDAR) process, so that the scanning results achieve the best balance between accuracy, efficiency and cost.

[0050] Resolution: The minimum distance between adjacent points in a point cloud (e.g., 0.01 meters).

[0051] Sampling rate: point cloud density per unit area (e.g. 1000 points / ㎡).

[0052] Scanning distance: refers to the physical distance between two consecutive scanning positions during the movement of the laser scanner. This parameter directly affects the integrity, overlap rate and overall efficiency of the scanned data. The scanning distance determines the size of the overlapping area between adjacent scanning positions. If the distance is too large, data gaps may appear between the scanning areas, and the target object may not be fully covered. If the distance is too small, the scanning time and data redundancy will increase, but the data integrity can be improved. An appropriate overlap rate (usually 30%-50% is recommended) facilitates the stitching and registration of point cloud data and reduces registration errors. An overlap rate that is too small may cause registration failure, while an overlap rate that is too large may increase computational complexity.

[0053] For each key building in the settlement, the cosine similarity between the key building's color feature vector and the color feature vector of any historically scanned building can be calculated. The cosine similarity between the key building's outline feature vector and the outline feature vector of the historically scanned building can also be calculated. The weighted sum of the cosine similarities between the key building's color feature vector and the historically scanned building's color feature vector and outline feature vector cosine similarities can be performed to obtain a third building similarity between the key building and the historically scanned building. Historically scanned buildings whose third building similarity exceeds a second building similarity threshold can be considered similar historically scanned buildings. The second building similarity threshold can be determined experimentally or manually (e.g., by experts) through experience.

[0054] As an example, the optimal laser scanning parameters of the similar historically scanned building with the greatest third similarity can be used as the optimal laser scanning parameters of the key building. As another example, the optimal laser scanning parameters of the similar historically scanned buildings can be averaged to use as the optimal laser scanning parameters of the key building.

[0055] In some embodiments, determining an optimal scanning path for a key building based on an image of the key building includes: Generate a three-dimensional model of the key building based on the image of the key building; Determine the key areas of the 3D model of the key buildings based on the images of the key buildings; Generate multiple scanning paths of key buildings based on the 3D models of key buildings; Determine multiple path evaluation indicators and key areas of the three-dimensional model of key buildings, and establish a fitness function; The optimal scanning path of key buildings is generated through genetic algorithm based on the fitness function and the scanning paths of multiple key buildings.

[0056] Specifically, a three-dimensional reconstruction algorithm may be used to generate a three-dimensional model of the key building based on the image of the key building.

[0057] Key areas of a 3D model of a critical building can include geometrically complex areas, functionally important areas, and accessibility-restricted areas within the building. Geometrically complex areas can include areas requiring high-precision scanning, such as carvings on building facades and special roof structures. Functionally important areas can include building entrances, windows, structural support points, and other areas critical to the building's structure or function. Accessibility-restricted areas can include areas that are obscured or difficult to scan directly (such as the back of a building and elevated areas). Geometrically complex areas (such as carvings and spires) can be extracted using geometric features such as curvature and normal changes. Deep learning models (such as PointNet) are used to perform semantic segmentation on the 3D model and mark functionally important areas (such as windows and doors). Accessibility-restricted areas are determined using visibility graphs or ray casting.

[0058] The scanning path refers to the trajectory sequence of the laser scanning device moving in three-dimensional space.

[0059] Figure 2 is a flow chart of generating scanning paths of multiple key buildings according to some embodiments of this specification, such as Figure 2 As shown, based on the 3D model of the key building, multiple scanning paths of the key building are generated, which may include the following process: S11. Evenly set multiple viewpoints on the three-dimensional model of the key building according to the scanning spacing included in the optimal laser scanning parameters, wherein the distance between any two adjacent viewpoints is the scanning spacing included in the optimal laser scanning parameters; S12. Setting a path generation constraint set, wherein the path generation constraint set may include a maximum total length of the scanning path, a maximum total scanning time, a next viewpoint selection constraint, etc. The next viewpoint selection constraint means that the next viewpoint must be an adjacent viewpoint of the current viewpoint; S13, using viewpoints within a key area of ​​the three-dimensional model of the key building as multiple starting points; S14. For each key region, randomly select a viewpoint within the key region as the current viewpoint; S15, randomly selecting a viewpoint from the adjacent viewpoints of the current viewpoint as the next viewpoint; S16, taking the next viewpoint as the current viewpoint of the currently generated scanning path, and determining whether the currently generated scanning path satisfies a preset condition, wherein the preset condition is that the total length of the scanning path is equal to the maximum total length of the scanning path, the total scanning time is the maximum total scanning time, or the unscanned area corresponding to the currently generated scanning path is less than the unscanned area threshold; if not, executing S15; if so, completing the generation of a scanning path, executing S17; S17: Determine whether the number of scanning paths starting from the viewpoint in the key area is greater than a threshold value. If so, complete the generation of the scanning path of the key area and execute S18. If not, execute S14.

[0060] S18, determining whether the number of scan paths in each key area is greater than a threshold; if so, completing the generation of all scan paths; if not, selecting the next key area for which no scan path has been generated, and executing S14.

[0061] Multiple path evaluation indicators can include: Path length indicator: the total length of the scanning path. The shorter the path length, the higher the scanning efficiency and the lower the energy consumption.

[0062] Scan time metric: the total time required to complete the scan path (including movement time and viewpoint dwell time).

[0063] Significance: The shorter the scanning time, the shorter the project cycle and the lower the cost.

[0064] Data coverage metric: The proportion of key buildings covered by the scan path. The higher the coverage, the better the data integrity and the more suitable it is for subsequent analysis or modeling.

[0065] Key Area Coverage: This metric measures the percentage of key areas covered by the scan path. This ensures data integrity in high-precision or functionally critical areas.

[0066] Path smoothness index: The smoothness of the scanning path (such as the number of sharp turns and the frequency of height changes). Smooth paths reduce mechanical wear and scanning errors.

[0067] The fitness function is a mathematical model used in genetic algorithms to evaluate path quality, and can be a function of a weighted combination of scores of multiple path evaluation indicators.

[0068] The fitness value of each individual (scanning path) in the population can be calculated based on the fitness function. The higher the fitness value, the better the path quality. Roulette wheel selection, tournament selection, and other methods are used to select individuals for the next generation based on their fitness values. High-quality paths are retained and low-quality paths are eliminated. Crossover operations (such as single-point crossover and sequential crossover) are performed on the selected individuals to generate new individuals. New path combinations are generated through genetic recombination to explore the solution space. Mutation operations (such as randomly replacing viewpoints and adjusting the order of viewpoints) are performed on the individuals after crossover to introduce randomness. This avoids falling into local optimality and increases population diversity. The genetic algorithm is terminated when the maximum number of iterations is reached, the fitness value converges, or a satisfactory solution is found. The individual with the highest fitness value is returned as the optimal scan path.

[0069] Step 120: Based on the current multimodal data used for digitizing the settlement, a physical three-dimensional model of the settlement is established, and a digital twin model of the current status of the settlement is constructed.

[0070] Specifically include: Based on the image and laser scanning data of the settlement, the following Figure 3 A solid three-dimensional model of the settlement shown; Generate information annotation of the settlement's physical 3D model based on text data; Based on the settlement's meteorological and vegetation data, multiple key environmental and ecological simulation scenarios for the settlement were established; Based on the settlement's spatial vector data, physical three-dimensional model, information annotation of the physical three-dimensional model, and multiple key environmental and ecological simulation scenarios, a digital twin model of the settlement's current status is generated.

[0071] Specifically, 3D reconstruction algorithms (such as Structure from Motion, SfM) or professional software (such as ContextCapture, Agisoft Metashape) are used to fuse settlement images and laser scanning data to generate a physical 3D model of the settlement. The physical 3D model of the settlement can at least include 3D models of the settlement's buildings, and may also include 3D models of infrastructure (for example, power facilities (such as utility poles and transformers), communication facilities (such as base stations), water supply and drainage facilities (such as manhole covers and pipes), etc.), and man-made features (such as sculptures, monuments, fountains, and street lights).

[0072] Use annotation tools (such as ArcGIS and QGIS) to embed text information into the physical 3D model of the settlement as labels, pop-ups, or layers. Annotations can include building names, uses, historical background, and conservation levels.

[0073] In some embodiments, multiple key environmental and ecological simulation scenarios for the settlement are established based on the settlement's meteorological and vegetation data, including: Obtain meteorological data, vegetation data, and multiple key environmental and ecological simulation scenarios for multiple sample settlements; Determine the meteorological environment characteristics and vegetation characteristics of the sample settlement based on the meteorological data and vegetation data of the sample settlement; Determine the meteorological environment characteristics and vegetation characteristics of the settlement based on the settlement's meteorological data and vegetation data; Based on the meteorological environment characteristics and vegetation characteristics of the settlement and the meteorological environment characteristics and vegetation characteristics of the sample settlement, similar sample settlements are identified; Based on multiple key environmental and ecological simulation scenarios of similar sample settlements, multiple key environmental and ecological simulation scenarios of settlements are established.

[0074] Specifically, the sample settlement can be a settlement for which multiple key environmental and ecological simulation scenarios have been identified. The sample settlement can be a virtual settlement or a real settlement. The multiple key environmental and ecological simulation scenarios of the sample settlement can be multiple key environmental and ecological simulation scenarios with different natural landscapes. Multiple key environmental and ecological simulation scenarios can simulate the impact of weather conditions such as heavy rain, drought, typhoons, high temperatures, sunny days, and rainy days on the settlement environment and ecosystem. Key environmental and ecological simulation scenarios emphasize vegetation and meteorology as important components of the natural environment. By simulating their distribution and changes, a more realistic and vivid three-dimensional scene can be constructed. This simulation not only helps to enhance the visual effect of the scene, but also provides strong support for ecological research, environmental planning, etc.

[0075] Statistical analysis is performed on the settlement's meteorological and vegetation data to extract the settlement's meteorological and environmental characteristics (e.g., average annual temperature, seasonal distribution of precipitation) and vegetation characteristics (e.g., vegetation coverage, distribution ratio of different plant types, etc.). Similarly, the meteorological and vegetation data of the sample settlement are used to determine the meteorological and vegetation characteristics of the sample settlement. Similarity metrics (e.g., Euclidean distance, cosine similarity) are used to calculate the similarity between the settlement's meteorological and environmental characteristics and those of the sample settlement. Sample settlements with similarities greater than a threshold are considered similar sample settlements. The threshold can be determined experimentally or empirically (e.g., by experts).

[0076] The multiple key environments and ecological simulation scenarios of similar sample settlements with the greatest feature similarity can be used as the multiple key environments and ecological simulation scenarios of the settlement.

[0077] Assume that the current settlement is a plain settlement located in a subtropical region, with farmland and sparse woodland as the primary vegetation. A sample settlement is a mountain settlement located in a similar climate zone. Several key environmental and ecological simulation scenarios have been identified, including stormwater flooding simulation, vegetation growth simulation, and soil erosion simulation. Select a similar sample settlement. Transfer the stormwater flooding simulation, including the multiple key environmental and ecological simulation scenarios from the similar sample settlement, to the current settlement. Because the current settlement receives less precipitation, reduce the precipitation intensity by 20%. Adjust the growth rate parameters in the vegetation growth model to accommodate the farmland and sparse woodland in the current settlement. Perform hydrological simulation using the SWAT (Hydrological Simulation Using SWAT) model, adjusting model parameters based on the current settlement's topography and soil characteristics. Generate a stormwater flooding simulation scenario for the current settlement.

[0078] Integrate the settlement's spatial vector data (such as roads, water systems, and building outlines) with the physical 3D model, information annotations, and environmental and ecological simulation scenarios. Use a geographic information system (GIS) or a 3D visualization platform (such as Unity or Unreal Engine) for data fusion. Build a digital twin model of the settlement within the 3D visualization platform, showcasing the settlement's spatial layout, architectural form, environmental characteristics, and ecological simulation. This platform supports user interaction, such as zooming, rotating, querying information annotations, and switching simulation scenarios.

[0079] Step 130 : Determine multiple historical development nodes of the settlement based on the current multimodal data used for settlement digitization.

[0080] Specifically include: Based on the architectural features and textual data of each building in the settlement, the construction time of each building in the settlement is traced back, and based on the construction time of each building in the settlement, the architectural evolution information of the settlement is determined; Determine the road network evolution information of the settlement based on the spatial vector data of the settlement; Based on the spatial vector data of the settlement, the spatial evolution information of the land use type of the settlement is determined, wherein the multiple historical development nodes of the settlement include at least the architectural evolution information, road network evolution information and land use type spatial evolution information of the settlement.

[0081] Specifically, architectural feature data is converted into numerical variables (e.g., style codes are converted into numerical labels), and information is extracted from architectural text data using natural language processing techniques. A deep learning model is then used to predict construction dates. When training the deep learning model, architectural features and text data are used as input, and known construction dates are output. The accuracy of the deep learning model is verified through cross-validation or holdout methods, and the model parameters are adjusted to improve prediction accuracy. For each building, the most likely construction date range is determined based on the deep learning model's prediction results. The architectural evolution information of the settlement is determined based on the construction date and spatial location of each building in the settlement. This information can include the main characteristics and distribution of buildings at different historical development nodes.

[0082] You can collect paper or electronic maps of the settlement from different historical periods, digitize them (e.g., scan and vectorize), extract road information, and arrange the extracted road data in chronological order to construct a time series dataset. Then, use GIS tools (e.g., ArcGIS Pro, QGIS) or WebGIS platforms (e.g., Leaflet, Mapbox) to create dynamic maps that display the evolution of the settlement's road network—that is, how the road network has changed over time. For example, roads from the 1950s are red, roads newly added in the 1980s are blue, and roads rerouted in the 2000s are green.

[0083] Obtain land use classification data (such as cultivated land, forest land, and construction land) for historical periods (such as the 1980s, 2000s, and 2020s) from sources including historical maps and remote sensing image interpretation. Calculate the total area of ​​agricultural land and the area of ​​each type (such as paddy fields and dry land) at different historical development nodes, and count the distribution ratios of agricultural land in different slope and altitude ranges to determine the spatial evolution information of land use types in settlements.

[0084] Step 140: Based on the predicted multiple historical development nodes of the settlement and the physical three-dimensional model of the settlement, a three-dimensional model of the historical development nodes of the settlement is established, and a digital twin model of the historical development node settlement is constructed.

[0085] Specifically include: Based on the settlement's architectural evolution information, road network evolution information, and land use type spatial evolution information, multiple historical development nodes of the settlement are determined; For each historical development node, a three-dimensional model of the settlement's historical development node is established based on the settlement's architectural evolution information, road network evolution information, land use type spatial evolution information, and the settlement's physical three-dimensional model. Based on multiple key environmental and ecological simulation scenarios of the settlement, a digital twin model of the settlement's historical development node is constructed.

[0086] Specifically, representative time points can be selected as historical development nodes based on information on architectural evolution (e.g., changes in architectural style and function), road network evolution (e.g., road widening and new construction), and spatial evolution of land use types (e.g., expansion of cultivated land and conversion of farmland to forest). For example, node 1: 1950 (dominated by adobe buildings, dirt roads, and a high proportion of agricultural land). Node 2: 1980 (some buildings converted to brick and tile, paved roads began, and some agricultural land was converted to industrial use). Node 3: 2020 (modern settlements, an increase in high-rise buildings, a well-developed road network, and concentrated agricultural land).

[0087] Extract the building type, height, and distribution range of each historical development node and combine it with the building geometry data from the physical 3D model to generate a 3D model of the historical buildings. Generate 3D road network models (e.g., dirt roads, gravel roads, asphalt roads) for different historical development nodes based on historical road vector data. Convert spatial data of historical land use types (e.g., cultivated land, woodland, and water bodies) into textures or elevation changes on the 3D terrain surface. Use 3D modeling software (e.g., SketchUp, Blender) or GIS tools (e.g., ArcGIS CityEngine) to generate 3D models of historical buildings, adjusting materials and colors to reflect historical characteristics. Adjust the geometry and texture of the 3D road model based on road type (e.g., width, material). Simulate the distribution of historical agricultural land by modifying terrain elevation or overlaying vegetation textures. Integrate the building, road, and agricultural land models with the physical 3D settlement model (e.g., terrain and water system) to form a complete 3D model of the historical development node. Using virtual reality (VR) development platforms (such as Unity and Unreal Engine) or 3D GIS platforms (such as Cesium and ArcGIS Earth), the 3D models of historical development nodes are integrated with key environmental and ecological simulation scenarios to generate digital twin models of historical development node settlements.

[0088] Figure 4 This is a flow chart of a settlement digital twin modeling system based on multimodal data fusion according to some embodiments of this specification, such as Figure 4 As shown, the settlement digital twin modeling system based on multimodal data fusion can include a multimodal data acquisition module, a digital twin establishment module and a development analysis module.

[0089] A multimodal data acquisition module, configured to acquire current multimodal data for settlement digitization, wherein the current multimodal data for settlement digitization includes at least spatial vector data, images, laser scanning data, meteorological data, vegetation data, and text data; The digital twin building module is used to build a three-dimensional physical model of the settlement based on the current multimodal data used for settlement digitization, and construct a digital twin model of the current settlement status; The development analysis module is used to determine multiple historical development nodes of the settlement based on the current multimodal data used for settlement digitization; The digital twin establishment module is also used to establish a three-dimensional model of the historical development nodes of the settlement based on the predicted multiple historical development nodes of the settlement and the physical three-dimensional model of the settlement, and to construct a digital twin model of the historical development node settlement.

[0090] The settlement digital twin modeling system based on multimodal data fusion can be used to execute the above-mentioned settlement digital twin modeling method based on multimodal data fusion, which will not be repeated here.

[0091] Finally, it should be understood that the embodiments described in this specification are intended only to illustrate the principles of the embodiments of this specification. Other variations may also fall within the scope of this specification. Therefore, by way of example and not limitation, alternative configurations of the embodiments of this specification may be considered consistent with the teachings of this specification. Accordingly, the embodiments of this specification are not limited to the embodiments explicitly described and illustrated in this specification.

Claims

1. The settlement digital twin modeling system based on multimodal data fusion is characterized by: include: A multimodal data acquisition module, configured to acquire current multimodal data for settlement digitization, wherein the current multimodal data for settlement digitization includes at least spatial vector data, images, laser scanning data, meteorological data, vegetation data, and text data; The digital twin building module is used to build a three-dimensional physical model of the settlement based on the current multimodal data used for settlement digitization, and construct a digital twin model of the settlement's current status; The development analysis module is used to determine multiple historical development nodes of the settlement based on the current multimodal data used for settlement digitization; The digital twin establishment module is also used to establish a three-dimensional model of the historical development nodes of the settlement based on the predicted multiple historical development nodes of the settlement and the physical three-dimensional model of the settlement, and to construct a digital twin model of the historical development node settlement.

2. The settlement digital twin modeling system based on multimodal data fusion according to claim 1 is characterized in that: The digital twin building module builds a three-dimensional physical model of the settlement based on the current multimodal data used for settlement digitization, and constructs a digital twin model of the current settlement status, including: Use 3D reconstruction algorithms to fuse settlement images and laser scanning data to generate a solid 3D model of the settlement; Generate information annotation of the settlement's physical 3D model based on text data; Based on the settlement's meteorological and vegetation data, multiple key environmental and ecological simulation scenarios for the settlement were established; Based on the settlement's spatial vector data, physical three-dimensional model, information annotation of the physical three-dimensional model, and multiple key environmental and ecological simulation scenarios, a digital twin model of the settlement's current status is generated.

3. The settlement digital twin modeling system based on multimodal data fusion according to claim 2 is characterized in that: The digital twin building module builds multiple key environmental and ecological simulation scenarios for the settlement based on the settlement’s meteorological and vegetation data, including: Obtain meteorological data, vegetation data, and multiple key environmental and ecological simulation scenarios for multiple sample settlements; Determine the meteorological environment characteristics and vegetation characteristics of the sample settlement based on the meteorological data and vegetation data of the sample settlement; Determine the meteorological environment characteristics and vegetation characteristics of the settlement based on the settlement's meteorological data and vegetation data; Based on the meteorological environment characteristics and vegetation characteristics of the settlement and the meteorological environment characteristics and vegetation characteristics of the sample settlement, similar sample settlements are identified; Based on multiple key environmental and ecological simulation scenarios of similar sample settlements, multiple key environmental and ecological simulation scenarios of settlements are established.

4. The settlement digital twin modeling system based on multimodal data fusion according to claim 3 is characterized in that: The development analysis module determines multiple historical development nodes of a settlement based on the current multimodal data used for settlement digitization, including: Based on the architectural features and textual data of each building in the settlement, the construction time of each building in the settlement is traced back, and the architectural evolution information of the settlement is determined based on the construction time of each building in the settlement; Determine the road network evolution information of the settlement based on the spatial vector data of the settlement; According to the spatial vector data of the settlement, the spatial evolution information of the land use type of the settlement is determined, wherein the multiple historical development nodes of the settlement include at least the architectural evolution information, road network evolution information and land use type spatial evolution information of the settlement.

5. The settlement digital twin modeling system based on multimodal data fusion according to claim 4 is characterized in that: The digital twin establishment module establishes a three-dimensional model of the historical development nodes of the settlement based on the predicted multiple historical development nodes of the settlement and the physical three-dimensional model of the settlement, and constructs a digital twin model of the historical development node settlement, including: Based on the settlement's architectural evolution information, road network evolution information, and land use type spatial evolution information, multiple historical development nodes of the settlement are determined; For each historical development node, a three-dimensional model of the settlement's historical development node is established based on the settlement's architectural evolution information, road network evolution information, land use type spatial evolution information, and the settlement's physical three-dimensional model. Based on multiple key environmental and ecological simulation scenarios of the settlement, a digital twin model of the settlement's historical development node is constructed.

6. The settlement digital twin modeling system based on multimodal data fusion according to claim 5 is characterized in that: The digital twin building module builds a three-dimensional model of the settlement's historical development nodes based on the settlement's architectural evolution information, road network evolution information, land use type spatial evolution information, and the settlement's physical three-dimensional model, including: Extract the building type, height, and distribution range of historical development nodes, combine the building geometry data in the solid 3D model, generate a 3D model of the historical building, and adjust the material and color; Generate a 3D road network model of historical development nodes based on historical road vector data; Convert historical land use type spatial data into texture or elevation changes of three-dimensional terrain surface to generate agricultural land use models; By integrating the 3D model of historical buildings, the 3D model of the road network and the agricultural land model, a 3D model of the historical development nodes of the settlement is established.

7. The settlement digital twin modeling system based on multimodal data fusion according to any one of claims 1 to 6, characterized in that: The multimodal data acquisition module acquires laser scanning data of the settlement, including: Determine the key buildings of the settlement based on the settlement image and spatial vector data; For each key building in the settlement, the optimal laser scanning parameters and optimal scanning path of the key building are determined based on the image of the key building. Based on the optimal laser scanning parameters and optimal scanning path of the key building, the key building is scanned to obtain laser scanning data of the key building.

8. The settlement digital twin modeling system based on multimodal data fusion according to claim 7 is characterized in that: The multimodal data acquisition module determines the key buildings of the settlement based on the settlement image and spatial vector data, including: Determine the architectural features of each building in the settlement based on the settlement image, wherein the architectural features at least include color features and outline features; Based on the architectural features of each building in the settlement and the spatial vector data of the settlement, the key buildings of the settlement are determined.

9. The settlement digital twin modeling system based on multimodal data fusion according to claim 8 is characterized in that: The multimodal data acquisition module determines the key buildings of the settlement based on the architectural features of each building in the settlement and the spatial vector data of the settlement, including: For each building in the settlement, determine the characteristic difference value of the building based on the architectural characteristics of each building in the settlement; Determine the target building based on the characteristic difference value of each building in the settlement; For each target building in the settlement, similar sample buildings are identified based on the architectural characteristics of the target building and the sample buildings. The historical and cultural value of the target building is determined based on the historical and cultural value of the similar sample buildings. The location center value of the target building is calculated based on the spatial vector data of the settlement. Determine the key buildings of the settlement based on the historical and cultural value and location center value of each target building in the settlement.

10. The settlement digital twin modeling method based on multimodal data fusion is characterized by: The settlement digital twin modeling system based on multimodal data fusion as described in any one of claims 1 to 9 comprises: Acquiring current multimodal data for settlement digitization, wherein the current multimodal data for settlement digitization includes at least images, laser scanning data, environmental data, and text data; Based on the current multimodal data used for settlement digitization, a physical three-dimensional model of the settlement is established, and a digital twin model of the current settlement is constructed; Based on the current multimodal data used for settlement digitization, multiple historical development nodes of the settlement are determined; Based on the predicted multiple historical development nodes of the settlement and the physical three-dimensional model of the settlement, a three-dimensional model of the historical development nodes of the settlement is established, and a digital twin model of the historical development node settlement is constructed.

Images