Method and device for generating a dynamic simulation environment based on a multimodal large model for a city three-dimensional modeling

Abstract

The application discloses a city three-dimensional modeling and dynamic simulation environment generation method and device based on a multi-modal large model, and comprises the following steps: acquiring city three-dimensional asset layout data; generating training corpus for training the multi-modal large model; calculating and determining a loss function of the multi-modal large model; based on the loss function of the multi-modal large model, adjusting and iteratively training parameters of the multi-modal large model to obtain a target multi-modal large model; inputting a target city aerial two-dimensional image into the target multi-modal large model, and outputting layout and feature data of three-dimensional assets of the target city by the target multi-modal large model; based on the layout and feature data of the three-dimensional assets of the target city, performing three-dimensional modeling and rendering with different levels of detail according to requirements; using historical dynamic data of the target city, generating dynamic simulation data through time series training data, and thus obtaining a city dynamic three-dimensional modeling simulation environment.

Description

Technical Field

[0001] This invention relates to the field of 3D modeling and urban planning technology, and in particular to a method and apparatus for generating urban 3D modeling and dynamic simulation environments based on multimodal large models. Background Technology

[0002] In urban planning, smart city construction, and related 3D modeling and simulation fields, traditional methods mainly rely on manual modeling, Geographic Information Systems (GIS), and 3D modeling software. These methods suffer from low modeling efficiency, poor real-time performance, difficulty in updating data, and inability to cope with dynamic environmental changes. Especially in emerging application scenarios such as low-altitude economy, autonomous driving testing, virtual reality (VR), and augmented reality (AR), even higher demands are placed on the high precision, dynamism, and scalability of the urban 3D environment.

[0003] With the rapid development of artificial intelligence technology, especially breakthroughs in deep learning and large-scale pre-trained models, intelligent systems based on large models have demonstrated significant performance in various fields such as natural language processing, computer vision, and multimodal tasks. Multimodal large models, by fusing heterogeneous data of different types such as text, images, audio, and video, possess stronger semantic understanding and information generation capabilities, and have been widely applied in tasks such as cross-modal understanding, image and text retrieval, and image caption generation.

[0004] The introduction of multimodal large models provides a new technical path for the automated and intelligent generation of urban 3D modeling and dynamic simulation environments. By combining aerial images (such as satellite images or images taken by drones) with the layout and attribute data of urban 3D assets, multimodal large models can achieve intelligent mapping from 2D images to 3D structures, thereby automatically completing 3D asset identification, modeling, and dynamic behavior prediction, providing data support and intelligent driving force for urban scene reconstruction and virtual simulation.

[0005] Logical reasoning ability, as a crucial indicator of large-scale model performance, directly impacts its performance in complex scene modeling and simulation. For example, models need to infer the spatial relationships between buildings and roads, identify the dynamic correlations of pedestrian and vehicular traffic between urban areas, and determine the changing trends of urban behavior patterns across different time dimensions. Therefore, improving the logical reasoning ability of multimodal large-scale models, especially in understanding urban three-dimensional space and dynamic behavior, is a key technical challenge for achieving high-quality urban modeling and simulation.

[0006] Currently, there is a lack of a complete method and system for generating urban 3D modeling and dynamic simulation environments based on multimodal large models. This system should be able to uniformly handle the fusion and construction of static spatial structure modeling and dynamic behavioral data, meeting the diverse needs of smart city construction. Therefore, a new technical solution is urgently needed to leverage the perception and reasoning capabilities of multimodal large models to achieve efficient, accurate, and dynamic generation of urban 3D environments, thereby promoting the development of smart cities and related applications. Summary of the Invention

[0007] The purpose of this invention is to provide a method and apparatus for generating urban 3D modeling and dynamic simulation environment based on multimodal large model, so as to overcome the shortcomings of the prior art.

[0008] To achieve the above objectives, the present invention provides the following technical solution:

[0009] This application discloses a method for generating urban 3D modeling and dynamic simulation environment based on a multimodal large model, including the following steps:

[0010] S1. Obtain urban 3D asset layout data; the urban 3D asset layout data includes an aerial 2D image of the city and the layout and feature data of the 3D assets belonging to the city contained in the 2D image;

[0011] S2. Based on the city's 3D asset layout data, generate training corpus for training a multimodal large model;

[0012] S3. Input the training corpus into the multimodal large model. The multimodal large model outputs predicted layout data. Calculate and determine the loss function of the multimodal large model based on the predicted layout data.

[0013] S4. Based on the loss function of the multimodal large model, the parameters of the multimodal large model are adjusted and iteratively trained until the training termination condition is met, and the target multimodal large model is obtained.

[0014] S5. Input the target city's top-down 2D image into the target multimodal large model, and the target multimodal large model outputs the layout and feature data of the target city's 3D assets;

[0015] S6. Based on the layout and feature data of the target city's 3D assets, perform 3D modeling and rendering with different levels of detail according to requirements; use the target city's historical dynamic data and generate dynamic simulation data through time series training data to obtain a dynamic 3D modeling and simulation environment for the city.

[0016] Preferably, step S1 specifically includes the following sub-steps:

[0017] S11. Acquire a two-dimensional aerial image of the city;

[0018] S12. Label the different areas of the city in the two-dimensional image of step S11; label the specifications and types of the three-dimensional assets of the city in the two-dimensional image of step S11.

[0019] S13. Using image segmentation and data processing techniques, the regions marked in step S12 are precisely segmented from the 3D assets to form independent region and asset datasets.

[0020] S14. Integrate the area from step S13 with the asset dataset to generate urban 3D asset layout data.

[0021] As a preferred option, step S2 is specifically performed as follows: using the aerial two-dimensional image of the city as the cue words for the multimodal large model, and combining the layout and feature data of the three-dimensional assets corresponding to the two-dimensional image to generate training corpus for training the multimodal large model.

[0022] As a preferred option, step S3 is performed as follows:

[0023] S31. Input the two-dimensional aerial images of the city as prompt words into the multi-model large model. The multi-model large model outputs the predicted layout and predicted feature data of the three-dimensional assets according to the prompt words.

[0024] S32. Calculate the cross-entropy between the predicted layout and feature data of the 3D asset and the layout and feature data of the 3D asset corresponding to the 2D image, and determine the loss function of the large model.

[0025] Preferably, the training termination condition in step S4 includes the number of training iterations or data accuracy.

[0026] Preferably, step S6 involves performing 3D modeling and rendering at different levels of detail as required, specifically including:

[0027] Collision volume modeling: suitable for path planning and basic physics interactions, generating only low-precision building volume collision models;

[0028] Low-precision textured modeling: suitable for resource-constrained scenarios requiring only simple visual interaction or simple urban layout planning, providing basic textured models of buildings, streets, and green belts;

[0029] High-precision textured modeling: Suitable for visual simulation and detailed scene display, generating models of buildings, streets and green belts with high-precision textures.

[0030] Preferably, the historical dynamic data of the city in step S6 includes the actual urban situation or the relationship between pedestrian density and time changes on streets of various transportation networks in the city, the relationship between vehicle density and time changes, the distribution of pedestrian density in the urban area, and the distribution of vehicle flow in the urban area.

[0031] The present invention also includes a device for generating urban 3D modeling and dynamic simulation environment based on a multimodal large model, comprising a training device for the multimodal large model and a modeling device based on the large model.

[0032] The training device for the multimodal large model includes an acquisition module, a generation module, a training module, and an adjustment module. The acquisition module is used to acquire urban 3D asset layout data, the generation module is used to generate training corpus for training the large model, the training module is used to train the multimodal large model and determine the loss function, and the adjustment module is used to adjust the parameters of the multimodal large model and generate the target multimodal large model.

[0033] The modeling device based on a large model includes a calling module, a dynamic module, and a modeling module. The calling module is used to call the target multimodal large model, input an aerial 2D image of the city, and output the feature data of the city's 3D assets in the 2D image. The dynamic module is used to generate dynamic simulation data using historical dynamic data of the city and time-series training data. The modeling module is used to perform 3D modeling and rendering of different levels of detail based on the feature data of the city's 3D assets generated by the large model, so as to generate a city 3D environment that meets specific application requirements.

[0034] The beneficial effects of this invention are:

[0035] 1. This invention can be applied to smart city construction, infrastructure optimization and layout planning in the low-altitude economy, simulation testing and operation scheduling of unmanned aerial vehicles and low-altitude aircraft, urban planning and optimization, virtual reality (VR) and augmented reality (AR) scene construction, and related technical fields. By utilizing the dynamic three-dimensional simulation environment generated by this invention, it can effectively support the construction of a low-altitude economic ecosystem, providing high-precision environmental data and simulation support for future low-altitude transportation, logistics distribution, and urban aerial infrastructure deployment, thereby promoting the rapid development of the low-altitude economy.

[0036] The features and advantages of the present invention will be described in detail through embodiments and in conjunction with the accompanying drawings. Attached Figure Description

[0037] Figure 1 A flowchart illustrating a training method for generating a multimodal large model for urban 3D modeling and dynamic simulation environment construction, provided in an embodiment of this disclosure;

[0038] Figure 2 This is a schematic diagram of the process for obtaining urban 3D asset layout data provided in an embodiment of this disclosure;

[0039] Figure 3 This disclosure provides a modeling process for generating a multimodal large model for urban 3D modeling and dynamic simulation environment construction.

[0040] Figure 4 A schematic diagram of the structure of a training device for generating a multimodal large model for building a dynamic simulation environment of a city, provided in an embodiment of this disclosure;

[0041] Figure 5 This is a schematic diagram of a modeling device for generating a multimodal large model for urban 3D modeling and dynamic simulation environment construction, provided in an embodiment of this disclosure. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. However, it should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of the invention. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0043] Multimodal technology, in the fields of artificial intelligence and machine learning, refers to the techniques and methods that can simultaneously process and understand information from multiple different types of data sources. Multimodal technology integrates various data formats such as text, images, audio, and video, enhancing the model's understanding and expressive capabilities by comprehensively analyzing the correlations between different modalities. Multimodal technologies are widely used in image caption generation, speech recognition and synthesis, sentiment analysis, intelligent question answering, cross-media retrieval, and many other fields.

[0044] A multimodal large model refers to a complex model that further integrates and processes data from multiple modalities, building upon a large model. These models typically possess a large parameter scale and complex structure, capable of simultaneously understanding and generating multiple data types such as text, images, and audio. Multimodal large models rely on powerful computing resources for training and deployment, and can play a crucial role in applications such as automated content generation, cross-modal retrieval, virtual reality interaction, intelligent assistants, and medical diagnosis.

[0045] This disclosure applies to various urban planning and management agencies, smart city construction projects, autonomous driving simulation testing platforms, and the development of virtual reality (VR) and augmented reality (AR) applications. Specific applications include, but are not limited to: 1. Urban planning and management: In urban planning departments, it can assist planners in urban layout design and optimization, providing standardized and professional 3D modeling and dynamic simulation data to support decision-making processes such as traffic flow analysis, public facility layout optimization, and environmental impact assessment; 2. Smart city construction: In smart city projects, it can generate and update urban 3D models in real time, integrate dynamic simulation data, and achieve efficient management and monitoring of urban resources, such as optimization of intelligent transportation systems, simulation and drills of emergency response, etc., improving the level of intelligent urban operation; 3. Autonomous driving simulation testing: In autonomous driving technology R&D and testing platforms, it provides realistic 3D urban environments and dynamic traffic flow simulations, helping developers test and optimize autonomous driving algorithms, and improving the safety and reliability of autonomous driving systems in complex urban environments; 4. Virtual Reality (VR) and Augmented Reality (AR) Applications: In VR / AR content development, high-precision 3D urban models and dynamic simulation environments are generated, supporting immersive virtual tourism, city exploration, training simulations, and other applications, enhancing the realism and interactivity of the user experience; 5. Film and Game Production: In film and television special effects production and game development, detailed 3D urban models and dynamic simulation data are provided, supporting high-quality scene construction and real-time interaction, meeting the needs of film and television production and game design for complex urban environments; 6. Environmental Monitoring and Emergency Management: Through dynamic simulation environments, the impact of natural disasters or man-made events on cities is simulated, assisting in the formulation of environmental protection measures and emergency response strategies, improving the resilience and sustainable development capabilities of cities; 7. Real Estate Development and Display: In the process of real estate development, 3D modeling and dynamic simulation environments are used for project planning and design display, providing intuitive visual effects to help developers and customers better understand and evaluate project plans. Through the above diverse application scenarios, this invention can significantly improve the generation efficiency and accuracy of 3D urban modeling and dynamic simulation environments, meet the needs of different fields for high-quality 3D urban data, and promote the development and application of related technologies.

[0046] Figure 1 This is a flowchart illustrating a training method for generating a multimodal large model for urban 3D modeling and dynamic simulation environment construction, provided by an embodiment of this disclosure. Figure 1 As shown, the training method for this multimodal large model may include:

[0047] S110, acquire urban 3D asset layout data, which includes aerial 2D images of the city (satellite or drone aerial 2D images) and layout and feature data of various 3D assets belonging to the city contained in the 2D images.

[0048] It should be noted that the execution entity of the method for acquiring urban 3D asset layout data in this embodiment of the disclosure can be a hardware device with information processing capabilities and / or the necessary software to drive the hardware device. Optionally, the execution entity may include a server, a user terminal, and other intelligent devices or terminals. Optionally, the user terminal includes, but is not limited to, computers, tablets, smartphones, etc. Optionally, the server may be a cloud server, a server of a distributed system, or a server combined with edge computing, etc. Optionally, other intelligent devices may be satellite terminals, drone terminals, or devices and terminals equipped with photographic capabilities. This embodiment of the disclosure does not impose specific limitations.

[0049] In some implementations, urban 3D asset layout data can be obtained based on urban planning databases and Geographic Information System (GIS) data. Optionally, existing 3D modeling software can be used to generate or optimize the layout and feature data of urban 3D assets. This urban 3D asset layout data includes various types of 3D assets, such as buildings, roads, green belts, and traffic signs, as well as their specific locations, specifications (height, shape characteristics, etc.), and regional information within the urban layout.

[0050] S120: Determine the prompt words for the multimodal large model, and generate training corpus for training the large model based on the prompt words and the corresponding 3D asset layout and feature data.

[0051] In some implementations, pre-defined prompts can help the multimodal large model clarify its generation target, and training corpus for training the large model can be generated based on urban 3D asset layout data and prompts. Optionally, combining prompts with urban 3D asset layout data can yield richer and more diverse training corpus. In some implementations, if it is necessary to generate a 3D urban model that includes commercial and residential areas, "aerial view images of the commercial and residential areas" can be used as prompts.

[0052] S130, the training corpus is input into the large model, which generates the predicted layout and predicted feature data of the corresponding three-dimensional assets for different prompt words, and the loss function of the large model is determined based on the difference between the predicted output and the actual layout and feature data of the urban three-dimensional assets.

[0053] In some implementations, after obtaining the training corpus, the training corpus is input into a multimodal large model for training. The large model generates corresponding 3D asset prediction layout and feature data based on each prompt word in the training corpus. The prompt words and the generated prediction data can then serve as the basis for the large model to generate a 3D city model.

[0054] Furthermore, to adjust the multimodal large model to meet high-performance requirements, loss functions can be calculated for the predicted layout, feature data, and real data to determine the loss function of the large model. Optionally, cross-entropy loss can be calculated for the predicted layout, feature data, and real data to determine the difference between the probability distribution of the large model's predicted output and the probability distribution of the real data.

[0055] S140, Based on the loss function, the parameters of the large model are adjusted and iteratively trained until the training termination condition is met, and the trained large model is obtained.

[0056] Optionally, the number of training iterations of the multimodal large model can be set as the training termination condition. When the number of training iterations of the multimodal large model reaches the set number, the training termination condition is determined to be met, and the target large model is obtained. Alternatively, the training termination condition can be determined based on the accuracy of the large model's output prediction layout and feature data. When the accuracy of the prediction data is greater than a set threshold, the training termination condition is determined to be met, and the target large model is obtained.

[0057] According to the training method for a multimodal large model for urban 3D modeling and dynamic simulation environment generation provided in this disclosure, urban 3D asset layout data is acquired, and training corpus is generated based on prompts for the multimodal large model. The multimodal large model is then trained based on the training corpus, and its loss function is calculated and adjusted until the target multimodal large model is obtained. This improves the target multimodal large model's understanding of urban 3D asset layout and feature data, enabling it to accurately generate urban 3D models and adapt to various complex and changing urban environments, exhibiting better flexibility and scalability.

[0058] Figure 2 This is a schematic diagram illustrating the process of acquiring urban 3D asset layout data provided in an embodiment of this disclosure. Figure 2 As shown, methods for obtaining urban 3D asset layout data may include:

[0059] S210, Acquire an aerial 2D image of the city. This aerial 2D image can be acquired via satellite, drone, or other aerial photography equipment, providing a comprehensive view of the target city. The aerial 2D image can also be used by the user to generate or optimize a comprehensive virtual city view based on existing 3D modeling software, according to their own needs.

[0060] S220, categorizes the various areas of the city within the range of the two-dimensional image. This can be based on Geographic Information System (GIS) data and urban planning databases, or on the user's own needs based on existing 3D modeling software to generate or optimize the zoning criteria of a virtual city. This allows for the accurate categorization of different areas (such as commercial areas, residential areas, industrial areas, park areas, etc.) in the two-dimensional image, determining the boundaries and functional divisions of each area.

[0061] S230 defines the specifications and types of urban 3D assets within the range of the 2D image. This can be achieved using existing 3D modeling software and databases, or by using virtual city data generated or optimized by the user based on existing 3D modeling software. This allows for detailed definition of the specifications (height, shape characteristics, etc.) and types (residential buildings, subway entrances, traffic signs, etc.) of various 3D assets (such as buildings, roads, green belts, traffic signs, etc.) in the 2D image, ensuring the accuracy and completeness of the asset data.

[0062] It should be noted that steps S202 and S203 can be executed in parallel without any order, that is, the specifications and types of the urban area and the 3D assets are calibrated at the same time.

[0063] S240, the city areas and assets within the range of the two-dimensional image are segmented. Through image segmentation and data processing techniques, the labeled areas and assets are accurately segmented to form independent area and asset datasets.

[0064] S250 obtains the city's 3D asset data layout, integrates the segmented regions and asset data to generate complete city 3D asset layout data, including the specific location, specifications, type and region information of various 3D assets in the city.

[0065] It should be noted that the execution entity of the method for acquiring urban 3D asset layout data in this embodiment of the disclosure can be a hardware device with information processing capabilities and / or the necessary software to drive the hardware device. Optionally, the execution entity may include a server, a user terminal, and other intelligent devices or terminals. Optionally, the user terminal includes, but is not limited to, computers, tablets, smartphones, etc. Optionally, the server may be a cloud server, a server of a distributed system, or a server combined with edge computing, etc. Optionally, other intelligent devices may be satellite terminals, drone terminals, or devices and terminals equipped with photographic capabilities. This embodiment of the disclosure does not impose specific limitations.

[0066] In some implementations, urban 3D asset layout data can be obtained based on urban planning databases and Geographic Information System (GIS) data. Optionally, users can generate or optimize the layout and feature data of urban 3D assets using existing 3D modeling software according to their own needs. This urban 3D asset layout data includes various types of 3D assets, such as buildings, roads, green belts, and traffic signs, as well as their specific locations, specifications (height, shape characteristics, etc.), and regional information within the urban layout.

[0067] Figure 3 This disclosure provides a modeling process for generating multimodal large models for urban 3D modeling and dynamic simulation environment construction; such as... Figure 3 As shown, the method for generating urban 3D modeling and dynamic simulation environment based on multimodal large model may include:

[0068] S310, invoke the target multimodal large model and input an overhead 2D image of the city into the large model. It should be noted that the target multimodal large model can be... Figure 1 The method shown is used for training, and will not be elaborated further here.

[0069] In some implementations, a target multimodal large model is invoked, and an aerial 2D image of the city is input into the large model. The large model is then guided to identify and process urban structure and asset information in the image. The fusion capability of multimodal data is used to extract the 3D feature data corresponding to the 2D image.

[0070] S320, obtain the feature data of the city's three-dimensional assets corresponding to the two-dimensional image.

[0071] In some implementations, based on the input overhead 2D image, the target multimodal large model can automatically extract feature data of various 3D assets in the city, including but not limited to building height, shape characteristics, type (such as residential, commercial, industrial, etc.), road layout, green belt distribution and other urban infrastructure information.

[0072] S330 uses feature data of urban 3D assets generated from a large model to perform 3D modeling and rendering at different levels of detail according to requirements.

[0073] In some implementations, users can choose different levels of modeling detail and rendering methods according to specific needs to meet the requirements of different application scenarios. For example: low-precision modeling and rendering: suitable for path planning and basic physical interaction, generating only basic volume models of buildings and simple textures; medium-precision modeling and rendering: suitable for general visual simulation, generating models of buildings and urban infrastructure with medium detail; high-precision modeling and rendering: suitable for detailed visual simulation and scene display, generating models of buildings, roads, green belts and other urban assets with high detail, and applying high-quality textures and materials.

[0074] S340 uses historical dynamic data of the city and time series training data to generate dynamic simulation data.

[0075] In some implementations, historical dynamic data of the city, such as pedestrian flow, vehicle flow, and event frequency, can be used to generate simulation data reflecting the dynamic changes of the city through time series analysis and training. This dynamic data includes, but is not limited to: traffic flow changes over different time periods; pedestrian density distribution during peak hours; and the impact of specific events (such as holidays) on urban dynamics.

[0076] S350 provides a dynamic 3D modeling and simulation environment for the city.

[0077] In some implementations, the generated dynamic simulation data is combined with previously constructed 3D asset models to form a 3D urban simulation environment that incorporates dynamic changing factors. This simulation environment can simulate dynamic behaviors in real cities, such as traffic flow, pedestrian activity, and other dynamic events, providing a realistic simulation foundation for smart city construction, autonomous driving testing, virtual reality (VR), and augmented reality (AR) applications.

[0078] Figure 4 This is a schematic diagram of the structure of a training device for generating a multimodal large model for building a three-dimensional urban modeling and dynamic simulation environment, provided in an embodiment of this disclosure.

[0079] like Figure 4 As shown, the large model training device 410 of this embodiment includes: an acquisition module 411, a generation module 412, a training module 413, and an adjustment module 414.

[0080] The acquisition module 411 is used to acquire urban three-dimensional asset layout data, which includes aerial two-dimensional images of the city (satellite or UAV aerial two-dimensional images) and layout and feature data of various three-dimensional assets belonging to the city contained in the two-dimensional images.

[0081] The generation module 412 is used to take the aerial two-dimensional image of the city as a prompt input and generate training corpus for training a large model based on its corresponding three-dimensional asset layout and feature data.

[0082] The generation module 413 is used to input the training corpus into the large model, so that the large model generates the predicted layout and predicted feature data of the corresponding three-dimensional assets for different prompt words, and determines the loss function of the large model based on the difference between the predicted output and the actual urban three-dimensional asset layout and feature data.

[0083] The generation module 414 is used to adjust the parameters and iteratively train the large model based on the loss function until the training termination condition is met, thus obtaining the trained large model.

[0084] Figure 5 This is a schematic diagram of a modeling device for generating a multimodal large model for urban 3D modeling and dynamic simulation environment construction, provided in an embodiment of this disclosure.

[0085] like Figure 5 As shown, the urban 3D modeling and dynamic simulation environment generation device 510 based on a multimodal large model according to this embodiment of the present disclosure includes: a calling module 511, a dynamic module 512 and a modeling module 513.

[0086] The calling module 511 is used to call the target multimodal large model and input a two-dimensional aerial image of the city into the large model, and then output the feature data of the city's three-dimensional assets in the two-dimensional image.

[0087] The dynamic module 512 uses historical dynamic data of the city and time series training data to generate dynamic simulation data. The dynamic simulation data is then added to the constructed three-dimensional city model to obtain the dynamic three-dimensional model of the city.

[0088] Modeling module 513 performs 3D modeling and rendering of varying levels of detail based on the feature data of urban 3D assets generated from a large model, in order to generate urban 3D environments that meet specific application requirements.

[0089] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions or improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for generating urban 3D modeling and dynamic simulation environment based on a multimodal large model, characterized in that, Includes the following steps: S1. Obtain urban 3D asset layout data; the urban 3D asset layout data includes an aerial 2D image of the city and the layout and feature data of the 3D assets belonging to the city contained in the 2D image; S2. Based on the city's 3D asset layout data, generate training corpus for training a multimodal large model; S3. Input the training corpus into the multimodal large model. The multimodal large model outputs predicted layout data. Calculate and determine the loss function of the multimodal large model based on the predicted layout data. S4. Based on the loss function of the multimodal large model, the parameters of the multimodal large model are adjusted and iteratively trained until the training termination condition is met, and the target multimodal large model is obtained. S5. Input the target city's top-down 2D image into the target multimodal large model, and the target multimodal large model outputs the layout and feature data of the target city's 3D assets; S6. Based on the layout and feature data of the target city's 3D assets, perform 3D modeling and rendering with different levels of detail according to requirements; use the historical dynamic data of the target city, and generate dynamic simulation data through time series training data to obtain a dynamic 3D modeling simulation environment for the city. The loss function is used to measure the difference between the predicted 3D asset layout data and the actual layout data and to optimize the parameters. Step S6 involves performing 3D modeling and rendering at different levels of detail as required, specifically including: Collision volume modeling: suitable for path planning and basic physics interactions, generating only low-precision building volume collision models; Low-precision textured modeling: suitable for resource-constrained scenarios requiring only simple visual interaction or simple urban layout planning, providing basic textured models of buildings, streets, and green belts; High-precision textured modeling: suitable for visual simulation and detailed scene display, generating models of buildings, streets and green belts with high-precision textures; The historical dynamic data of the city in step S6 includes the actual urban situation or the relationship between pedestrian density and time changes on streets of various transportation networks in the city, the relationship between vehicle density and time changes, the distribution of pedestrian density in the urban area, and the distribution of vehicle flow in the urban area.

2. The method for generating a city's 3D modeling and dynamic simulation environment based on a multimodal large model as described in claim 1, characterized in that: Step S1 specifically includes the following sub-steps: S11. Acquire a two-dimensional aerial image of the city; S12. Label the different areas of the city in the two-dimensional image of step S11; label the specifications and types of the three-dimensional assets of the city in the two-dimensional image of step S11. S13. Using image segmentation and data processing techniques, the regions marked in step S12 are precisely segmented from the 3D assets to form independent region and asset datasets. S14. Integrate the area from step S13 with the asset dataset to generate urban 3D asset layout data.

3. The method for generating a city's 3D modeling and dynamic simulation environment based on a multimodal large model as described in claim 1, characterized in that, Step S2 is performed as follows: The aerial two-dimensional image of the city is used as the cue words for the multimodal large model. Combined with the layout and feature data of the three-dimensional assets corresponding to the two-dimensional image, training corpus is generated for training the multimodal large model.

4. The method for generating a city's 3D modeling and dynamic simulation environment based on a multimodal large model as described in claim 3, characterized in that, Step S3 is performed as follows: S31. Input the two-dimensional aerial images of the city as prompt words into the multimodal large model. The multimodal large model outputs the predicted layout and predicted feature data of the three-dimensional assets according to the prompt words. S32. Calculate the cross-entropy between the predicted layout and feature data of the 3D asset and the layout and feature data of the 3D asset corresponding to the 2D image, and determine the loss function of the large model.

5. The method for generating a city's 3D modeling and dynamic simulation environment based on a multimodal large model as described in claim 1, characterized in that, The training termination criteria in step S4 include the number of training iterations or data accuracy.

6. A device for generating urban 3D modeling and dynamic simulation environment based on a multimodal large model, characterized in that: This includes training devices for multimodal large models and modeling devices based on large models; The training device for the multimodal large model includes an acquisition module, a generation module, a training module, and an adjustment module. The acquisition module is used to acquire urban 3D asset layout data, the generation module is used to generate training corpus for training the large model, the training module is used to train the multimodal large model and determine the loss function, and the adjustment module is used to adjust the parameters of the multimodal large model and generate the target multimodal large model. The modeling device based on a large model includes a calling module, a dynamic module, and a modeling module; The calling module is used to call the target multimodal large model, input a two-dimensional aerial image of the city, and output the feature data of the city's three-dimensional assets in the two-dimensional image; The dynamic module is used to generate dynamic simulation data by using historical dynamic data of the city and time series training data. The historical dynamic data of the city includes the relationship between pedestrian density and time change on streets of various traffic networks in the city, the relationship between vehicle density and time change, the distribution of pedestrian density in the urban area, and the distribution of vehicle flow in the urban area, either based on real urban conditions or user-defined data. The modeling module is used to perform 3D modeling and rendering of different levels of detail based on the feature data of urban 3D assets generated from a large model, so as to generate urban 3D environments that meet specific application requirements. Depending on the requirements, 3D modeling and rendering at different levels of detail are performed, specifically including: Collision volume modeling: suitable for path planning and basic physics interactions, generating only low-precision building volume collision models; Low-precision textured modeling: suitable for resource-constrained scenarios requiring only simple visual interaction or simple urban layout planning, providing basic textured models of buildings, streets, and green belts; High-precision textured modeling: Suitable for visual simulation and detailed scene display, generating models of buildings, streets and green belts with high-precision textures.

Images