Multi-model fusion method and system based on smart city

By using aerial photographic images to texture BIM models in smart cities, combined with oblique photogrammetry models, and employing B/S-side rendering technology and real-time perception models, the flexibility and scalability issues of multi-source data fusion are solved, improving the user browsing experience and the realism of the 3D model.

CN112560137BActive Publication Date: 2024-11-01WUHAN OPTICS VALLEY INFORMATION TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202011402153.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-12-04
Publication Date
2024-11-01
Estimated Expiration
2040-12-04

AI Technical Summary

Technical Problem

Existing technologies for multi-source data fusion in smart cities suffer from several problems, including the data source not being effectively applied to BIM model mapping, poor flexibility and reusability of multi-source data fusion operations, insufficient scalability of C/S-end software, and poor fusion effect between AI and real-world 3D scenes.

Method used

By applying textures to BIM models based on aerial photographic images, BIM models of indoor and outdoor scenes are generated. Combined with oblique photogrammetry models, B/S interface and GUP rendering technology are used for rendering and fusion to build a real-time perception model. The Cesium map engine and WebSocket real-time transmission channel are used to display moving target information.

Benefits of technology

It achieves efficient fusion of multiple models, improves the user browsing experience, enhances the flexibility and scalability of data operations, and strengthens the realism and visualization effect of the real-scene 3D model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112560137B_ABST
    Figure CN112560137B_ABST
Patent Text Reader

Abstract

The present invention provides a multi-model fusion method, system, electronic device and storage medium based on smart city, the method includes: generating BIM model and oblique photography model according to aerial photography images; rendering BIM model and oblique photography model by B / S end mode and GPU rendering mode; fusing the rendered BIM model and oblique photography model to obtain a fused real-life three-dimensional model; constructing a real-time perception model of a mobile target according to the coordinate points, movement trajectory and attribute information of indoor and outdoor mobile targets collected in real time, and fusing the constructed real-time perception model of the mobile target into the real-life three-dimensional model and displaying it. The present invention can fuse multiple models, browse and access oblique photography data, BIM data model and real-time perception model of mobile targets on the browser side through B / S end and based on GPU and server rendering technology, so as to improve the browsing experience of users.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of model fusion, and more specifically, to a multi-model fusion method and system based on a smart city. Background Art

[0002] In recent years, with the rapid development of high-tech technologies such as 5G, cloud computing, the Internet of Things and AI (artificial intelligence), GIS (Geographic Information System) has gradually been given new vitality. Currently, real-scene three-dimensional technology plays an important role in the construction of smart cities.

[0003] Among them, drone aerial photography technology is currently the main means of obtaining real-life 3D data due to its convenient and fast data acquisition method and highly automated modeling method. At the same time, as people's demand for indoor and outdoor scenes increases, the combination of oblique photography and BIM models in large scenes can effectively make up for the problem of insufficient indoor scene display. In addition, in terms of IOT and AI recognition results, the combination of IoT sensing devices and AI real-time sensing technology has enhanced the authenticity of 3D scenes to a certain extent. Summary of the invention

[0004] The embodiments of the present invention provide a multi-model fusion method and system based on a smart city that overcomes the above-mentioned problems or at least partially solves the above-mentioned problems.

[0005] According to a first aspect of the present invention, a multi-model fusion method based on a smart city is provided, comprising: mapping a BIM model according to an aerial photographic image to generate a BIM model of an indoor or outdoor scene; rendering the BIM model and the oblique photography model of the indoor or outdoor scene through a B / S terminal mode and using a GUP rendering mode; fusing the rendered BIM model and the oblique photography model to obtain a fused real-scene three-dimensional model; constructing a real-time perception model of a mobile target according to the coordinate points, movement trajectories and attribute information of the indoor or outdoor mobile target collected in real time, fusing the constructed real-time perception model of the mobile target into the real-scene three-dimensional model, and displaying it on the front end.

[0006] Based on the above technical solution, the embodiment of the present invention can also make the following improvements.

[0007] Furthermore, the method of mapping the BIM model according to the aerial photographic images to generate the BIM model of the indoor and outdoor scenes includes: parsing the IFC format used by the BIM model and splitting the nodes of the BIM model; establishing a corresponding relationship between each node of the BIM model and the aerial photographic images; importing the aerial photographic images corresponding to each node of the BIM model into the BIM model, batch mapping the BIM model, and generating the BIM model of the indoor and outdoor scenes.

[0008] Furthermore, the rendering of the BIM model and oblique photography model of indoor and outdoor scenes through the B / S end and using the GUP rendering method includes: converting the BIM model data structure into a 3D Tiles data format on the B / S end, and using the Cesium map engine to load and render the BIM model data.

[0009] Furthermore, it also includes: reducing the screen error rate and maintaining the precision of the BIM model by setting the maximum screen space error parameter in the Cesium map engine; and increasing the loading speed of the BIM model data by configuring the parameters for prioritizing the loading of the central block of the screen.

[0010] Furthermore, the rendering of the BIM model and the oblique photography model is fused to obtain a fused real-scene 3D model, including:

[0011] For the overlapping parts of the BIM model and the oblique photography model, the oblique photography model is cropped based on the Cesium 3D earth framework;

[0012] The BIM model is integrated into the cropped oblique photography model to generate a real-life 3D model.

[0013] Furthermore, for the overlapping part of the BIM model and the oblique photography model, clipping the oblique photography model based on the Cesium three-dimensional earth framework includes:

[0014] For the oblique photography model, the BIM model boundary is used as the initialization polygon clipping area, and the longitude and latitude coordinate values ​​of each edge of the polygon clipping area are recorded;

[0015] Taking the coordinate origin of the oblique photography model as a reference, matrix operation of coordinate values ​​is performed to correct the coordinates of the polygonal clipping area;

[0016] Taking the surface where each edge of the corrected polygonal clipping area is located as the clipping surface, a plurality of clipping surfaces are obtained;

[0017] Reconstructing the clipping surface according to the normal vector of any clipping surface and the shortest distance from the coordinate origin of the oblique photography model to any clipping surface, and finally obtaining a plurality of reconstructed clipping surfaces;

[0018] The oblique photography model is cropped based on the reconstructed multiple cropping planes.

[0019] Furthermore, the construction of a real-time perception model of a mobile target based on the coordinate points, movement trajectory and attribute information of indoor and outdoor mobile targets collected in real time includes: constructing a real-time perception model of a mobile target based on the coordinate points, movement trajectory and attribute information of indoor and outdoor mobile targets sent by the monitoring device through the established WebSocket real-time transmission channel; after integrating the constructed real-time perception model of the mobile target into the real-scene three-dimensional model and displaying it on the front end, it also includes: updating the position and orientation of the real-time perception model of the mobile target based on the coordinate points and movement trajectory of the mobile target updated in real time by the monitoring device, and displaying the updated real-time perception model of the mobile target on the front end.

[0020] Furthermore, the real-time perception model of the mobile target includes a human and a vehicle model, and also includes: when receiving the coordinate points, movement trajectories and attribute information of indoor and outdoor mobile targets sent by the monitoring device through the established WebSocket real-time transmission channel, establishing a correspondence between the real-time perception model ID of each mobile target and the time, position and orientation; updating the position and orientation of the real-time perception model of the mobile target according to the coordinate points and movement trajectories of the mobile target updated in real time by the monitoring device includes: inputting the real-time perception model ID, time, position and orientation of the mobile target into the update orientation function headingPitchRollQuaternion to realize the update of the position and orientation of the real-time perception model of the corresponding mobile target at the corresponding time.

[0021] According to a second aspect of an embodiment of the present invention, a multi-model fusion system is provided, including:

[0022] A mapping module is used to map the BIM model according to the aerial photography images to generate the BIM model of the indoor and outdoor scenes; a rendering module is used to render the BIM model and the oblique photography model of the indoor and outdoor scenes through the B / S terminal mode and the GPU rendering mode; a fusion module is used to fuse the rendered BIM model and the oblique photography model to obtain a fused real-scene three-dimensional model; a display module is used to construct a real-time perception model of the mobile target according to the coordinate points, movement trajectory and attribute information of the indoor and outdoor mobile targets collected in real time, fuse the constructed real-time perception model of the mobile target into the real-scene three-dimensional model, and display it on the front end.

[0023] According to a third aspect of an embodiment of the present invention, there is provided an electronic device, comprising a memory and a processor, wherein the processor is used to implement the steps of a multi-model fusion method when executing a computer management program stored in the memory.

[0024] According to a fourth aspect of an embodiment of the present invention, a computer-readable storage medium is provided, on which a computer management program is stored, and when the computer management program is executed by a processor, the steps of the multi-model fusion method are implemented.

[0025] The embodiment of the present invention provides a multi-model fusion method and system based on smart city, which generates BIM models and oblique photography models of indoor and outdoor scenes according to aerial photography images; renders the BIM model and oblique photography model through the B / S end mode and adopts GPU rendering mode; fuses the rendered BIM model and oblique photography model to obtain a fused real-life three-dimensional model; constructs a real-time perception model of the mobile target according to the coordinate points, movement trajectory and attribute information of the indoor and outdoor mobile targets collected in real time, fuses the constructed real-time perception model of the mobile target into the real-life three-dimensional model, and displays it on the front end. The embodiment of the present invention can fuse multiple models such as oblique photography models and BIM models, browse and access oblique photography data and BIM data models on the browser side through the B / S end and based on GPU and server rendering technology, so as to improve the user's browsing experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 A flow chart of a multi-model fusion method provided by an embodiment of the present invention;

[0027] Figure 2 A schematic diagram for establishing the correspondence between each node of the BIM model and the aerial image;

[0028] Figure 3 An overall flow chart of a multi-model fusion method provided by an embodiment of the present invention;

[0029] Figure 4 A structural diagram of a multi-model fusion system provided by an embodiment of the present invention;

[0030] Figure 5 A flow chart of a method for cropping an oblique photography model provided by an embodiment of the invention;

[0031] Figure 6 A schematic diagram of a possible hardware structure of an electronic device provided by an embodiment of the present invention;

[0032] Figure 7 A schematic diagram of the hardware structure of a possible computer-readable storage medium provided for an embodiment of the present invention. DETAILED DESCRIPTION

[0033] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0034] Figure 1 is a flow chart of a multi-model fusion method provided by an embodiment of the present invention, such as Figure 1 As shown, the method includes: 101. According to the aerial photography image, the BIM model is mapped to generate the BIM model of the indoor and outdoor scenes; 102. The BIM model and the oblique photography model of the indoor and outdoor scenes are rendered by the B / S end mode and the GUP rendering mode; 103. The rendered BIM model and the oblique photography model are integrated to obtain the integrated real-scene three-dimensional model; 104. According to the coordinate points, movement trajectories and attribute information of the indoor and outdoor mobile targets collected in real time, a real-time perception model of the mobile target is constructed, and the constructed real-time perception model of the mobile target is integrated into the real-scene three-dimensional model, and displayed on the front end.

[0035] It is understandable that based on the existing research on the integration methods of real-life 3D technology and artificial intelligence technology, there are several research methods: 1. Realize multi-source real-life data fusion by processing data sources; 2. Load large-scene real-life 3D in a C / S manner; 3. Integrate GIS with AI real-time perception technology.

[0036] According to the above introduction, the traditional fusion methods have the following shortcomings and areas that need to be improved: 1. Data source level: The pictures collected by drones are not effectively applied to the BIM model texture material. 2. Multi-source data fusion usually uses the method of processing source data, which reduces the flexibility and reusability of source data operations; 3. The C / S end software has poor scalability and is difficult to meet the current user customization needs; 4. The fusion method of AI and real-life 3D is single, and the visualization effect is not good.

[0037] In view of the singleness or insufficiency of traditional fusion methods, the embodiments of the present invention comprehensively consider the fusion of multi-source data and AI real-time perception technology, mainly including: making full use of aerial photography images to map the BIM model, and generating BIM models and oblique photography models of indoor and outdoor scenes. Among them, the BIM model includes various micro models, for example, a specific building and various components inside the building, and the oblique photography model includes the location coordinates of each small model. Generally speaking, the degree of refinement of the oblique photography model is higher than that of the BIM model.

[0038] After generating the BIM model and oblique photography model of the indoor and outdoor scenes, the BIM model and oblique photography model of the indoor and outdoor scenes are rendered through the B / S end and the GPU rendering method, and the rendered BIM model and oblique photography model are fused to generate a fused real-life 3D model. Based on the coordinate points, movement trajectory and attribute information of the indoor and outdoor moving targets collected in real time, a real-time perception model of the moving targets is constructed, and the constructed real-time perception model of the moving targets is fused into the real-life 3D model and displayed on the front end.

[0039] The embodiment of the present invention can integrate multiple models such as oblique photography models and BIM models, and browse and access the oblique photography data and BIM data models on the browser side through the B / S side and based on GPU and server rendering technology, thereby improving the user's browsing experience.

[0040] In a possible implementation manner, mapping is performed on a BIM model according to an aerial photographic image to generate a BIM model of indoor and outdoor scenes, including: parsing the IFC format used by the BIM model and splitting the nodes of the BIM model; establishing a correspondence between each node of the BIM model and the aerial photographic image; importing the aerial photographic image corresponding to each node of the BIM model into the BIM model, batch mapping the BIM model, and generating a BIM model of the indoor and outdoor scenes.

[0041] It is understandable that, see Figure 2 , for drone aerial images, the aerial images are identified through artificial recognition algorithms (AI). Usually when building a BIM model, it is necessary to specify the material for the BIM model, but generally speaking, the material of the modeling software cannot reflect the real texture characteristics of the BIM model. By studying the IFC format used by the BIM model and parsing it, splitting the nodes, obtaining the graphic data and spatial structure information of the BIM model, and establishing the correspondence between the base point reference in IfcMaterialResource and the AI ​​recognition image result, it can be understood as establishing the correspondence between each node of the BIM model and the aerial image.

[0042] After establishing the correspondence between each node in the BIM model and the aerial image, the aerial image corresponding to all nodes of the BIM model are imported into the BIM model. If the aerial image is an indoor or outdoor scene on the ground, the generated BIM model is an indoor or outdoor scene BIM model, so that the BIM model will automatically be mapped when it is sliced. In this way, the aerial image resources are effectively utilized, and the nodes of the BIM model are used as calculation references to achieve the batch fast mapping effect of the BIM model.

[0043] In a possible implementation manner, rendering the BIM model and oblique photography model of indoor and outdoor scenes through the B / S end and using the GUP rendering method includes: converting the BIM model data structure or the oblique photography model data structure into a 3D Tiles data format on the B / S end, and using the Cesium map engine to load and render the BIM model data or the oblique photography model data.

[0044] It is understandable that at the client rendering level, in order to meet the 3D rendering requirements of the browser, the B / S end GPU rendering method is used for scene fusion. In addition, model slicing optimization and parameter tuning can be performed to achieve the purpose of improving the loading rate.

[0045] Specifically, the loading of real-life 3D data on the B / S side usually requires optimization of its data structure. The 3DTiles data format can be used to effectively stream and render large amounts of 3D geographic data. The Cesium map engine supports the 3DTiles data format and loads it according to the LOD level, which can effectively solve the problem of rendering massive data on the browser side.

[0046] Specifically, on the B / S side, the data structure of the BIM model and the data structure of the oblique photography model are converted into the 3DTiles data format, and the Cesium map engine is used to load and render the BIM model data and the oblique photography model data.

[0047] Among them, in order to further optimize the loading efficiency of 3dtiles, the embodiment of the present invention provides a set of optimization configuration parameters. For example, when loading a Cesium3DTileset object, the performance can be controlled by adding the maximum screen space error parameter in the Cesium map engine. The higher the value, the better the performance, but there will be low visual quality. After repeated testing, controlling the parameter value between 10-16 can effectively reduce the screen error rate and maintain the model refinement, ensuring the stable operation of the three-dimensional scene; in addition, the parameter of preferentially loading the central tile of the screen can be configured at the same time to improve the loading rate.

[0048] In a possible implementation manner, the fusion of the rendered BIM model and the oblique photography model to obtain a fused real-scene three-dimensional model includes: for the overlapping part of the BIM model and the oblique photography model, cropping the oblique photography model based on the Cesium three-dimensional earth framework; and fusing the BIM model into the cropped oblique photography model to generate a real-scene three-dimensional model.

[0049] Understandably, see Figure 3In order to solve the problem that the indoor scene cannot be browsed due to the overlap between the BIM model and the oblique photography model, it is necessary to perform model clipping on the oblique photography model, and the specific method is to perform model clipping based on the Cesium 3D Earth framework. Specifically, for the oblique photography model, the BIM model boundary is used as the initial polygon clipping area, and the longitude and latitude coordinate values ​​of each edge of the polygon clipping area are recorded; the coordinates of the polygon clipping area are corrected by matrix calculation of the coordinate values ​​based on the coordinate origin of the oblique photography model; the surface where each edge of the corrected polygon clipping area is located is used as the clipping surface to obtain multiple clipping surfaces; the clipping surface is reconstructed according to the normal vector of any clipping surface and the shortest distance from the coordinate origin of the oblique photography model to any clipping surface, and finally multiple reconstructed clipping surfaces are obtained; based on the reconstructed multiple clipping surfaces, the oblique photography model is clipped.

[0050] The specific process of clipping the oblique photography model is as follows: in order to solve the problem that the indoor scene cannot be browsed due to the overlap between the BIM model and the oblique photography model, the oblique photography model needs to be clipped, and the model is clipped by building a ClippingPlane collection based on the Cesium earth framework.

[0051] Take the BIM model boundary as the initial polygon clipping area and record the latitude and longitude coordinate values ​​P of each vertex of the initial polygon clipping area. i , i = 0,1,2,…, where They are The longitude, latitude and altitude coordinate values ​​of the coordinates:

[0052] .

[0053] The inverse matrix of the oblique photography model origin matrix is ​​used as the reference, and the recorded polygon vertices are calculated to generate the correction point coordinates based on the inverse matrix of the oblique photography origin. The rotated matrix ,in is the local coordinate value, which can be calculated by Cesium:

[0054] ;

[0055] The converted coordinates ,in , , is the coordinate point The longitude, latitude, and altitude of , and so on, generate the corrected vertex coordinates:

[0056] ;

[0057] ;

[0058] .

[0059] Define an arbitrary vertical ground vector The vector formed by the corrected coordinate points Perform cross multiplication to generate the final required clipping surface normal vector , where i=0,1,2…

[0060]

[0061] The ClippingPlane collection can be directly constructed through the parameter normal vector and the clipping plane coordinate point to achieve the clipping effect.

[0062] In a possible implementation manner, constructing a real-time perception model of a mobile target based on the coordinate points, movement trajectory and attribute information of indoor and outdoor mobile targets collected in real time includes: constructing a real-time perception model of the mobile target based on the coordinate points, movement trajectory and attribute information of the indoor and outdoor mobile targets sent by the monitoring device through the established WebSocket real-time transmission channel; integrating the constructed real-time perception model of the mobile target into the real-scene three-dimensional model, and displaying it on the front end also includes: updating the position and orientation of the real-time perception model of the mobile target based on the coordinate points and movement trajectory of the mobile target updated in real time by the monitoring device, and displaying the updated real-time perception model of the mobile target on the front end.

[0063] It is understandable that after the BIM model and the oblique photography model are integrated to generate a real-life 3D model of the indoor and outdoor scenes, it is necessary to construct a detailed model of each moving target therein. At this time, the actions of each moving target in the indoor and outdoor scenes are collected according to the monitoring devices set at various locations, for example, the coordinate points, movement trajectories and their own attribute information of each moving target are collected, wherein each moving target mainly refers to people and cars on the road, and the own attribute information of people and cars mainly includes the color, size, license plate, etc. of people and cars. According to the coordinate points, movement trajectories and their own attribute information of the moving targets, a real-time perception model of the moving targets is constructed. The real-time perception models of each moving target constructed are integrated into the real-life 3D model and displayed on the front end to present the people and car models and the real-life 3D model of the entire indoor and outdoor scenes.

[0064] Among them, the mobile target moves in real time, so the real-time perception model of the mobile target needs to be updated in real time. Specifically, the coordinate points and movement trajectory of the mobile target are updated in real time according to the monitoring devices set at various locations indoors and outdoors, and the position and orientation of the real-time perception model of the mobile target are updated, and the updated real-time perception model of the mobile target is displayed on the front end.

[0065] In a possible implementation manner, the real-time perception model of the mobile target includes a human and a vehicle model, and also includes: when the coordinate points, movement trajectories and attribute information of indoor and outdoor mobile targets are sent by the monitoring device through the established WebSocket real-time transmission channel, a correspondence between the real-time perception model ID of each mobile target and the time, position and orientation is established; the position and orientation of the real-time perception model of the mobile target is updated according to the coordinate points and movement trajectories of the mobile target updated in real time by the monitoring device, including: inputting the real-time perception model ID, time, position and orientation of the mobile target into the update orientation function headingPitchRollQuaternion to realize the update of the position and orientation of the real-time perception model of the corresponding mobile target at the corresponding time.

[0066] It can be understood that the process of updating the real-time perception model of each moving target in the real-scene three-dimensional model is as follows: monitoring devices can be set up at various locations indoors and outdoors, and each monitoring device and the front end realize data stream transmission by establishing a WebSocket real-time network transmission channel.

[0067] The monitoring device sends the collected coordinate points, movement trajectories and attribute information of indoor and outdoor mobile targets to the front end through the established WebSocket real-time transmission channel. When the front end receives the coordinate points, movement trajectories and attribute information of indoor and outdoor mobile targets sent by the monitoring device, a correspondence between the real-time perception model ID of each mobile target and the time, position and orientation is established.

[0068] As time goes by, when each mobile target moves, the position and orientation of the real-time perception model of the mobile target are updated according to the coordinate points and movement trajectory of the mobile target updated in real time by the monitoring device. Specifically, the real-time perception model ID, time, position, and orientation of the mobile target are input into the updated orientation function headingPitchRollQuaternion to realize the update of the position and orientation of the real-time perception model of the corresponding mobile target at the corresponding time, which can accurately present the movement of people and vehicles in real time.

[0069] See also Figure 4 , the multi-model fusion method provided by the embodiment of the present invention is described. First, according to the aerial image, the BIM model is sliced ​​and mapped, and the oblique photography model is sliced ​​and mapped, and the BIM model and the oblique photography sensitive effect are rendered and fused to generate a rendered real-scene 3D model. According to the moving trajectory of the moving target collected in real time, the human and vehicle models in the real-scene 3D model are updated in real time, and the updated real-scene 3D model is displayed in real time.

[0070] Among them, the scenes used in the embodiment of the present invention are oblique photography near the company and the company's internal BIM scenes. The area of ​​the three-dimensional real-life model after fusion of BIM is 1 square kilometer, and there are about 80 test models of people and vehicles after fusion and recognition; the test machine is an i5-8250U processor with integrated graphics card, and the loading of the fused scene model is not stuck. Since the real-time data transmission is only related to the network bandwidth and is not affected by the machine performance, the real-time transmission of AI recognition data is not affected by the machine. The loading of the real-life three-dimensional model has certain requirements for GPU rendering and is related to the amount of real-life three-dimensional data loaded. It is recommended to use a high-performance independent graphics card machine. Therefore, the machine performance needs to be improved when the real-life model of a large scene and the real-life model of the people and vehicles are fused in real time.

[0071] The embodiment of the present invention generates BIM models and oblique photography models of indoor and outdoor scenes based on aerial photography images; renders the BIM model and oblique photography model through the B / S end mode and adopts the GPU rendering mode; fuses the rendered BIM model and oblique photography model to obtain a fused real-scene three-dimensional model; constructs a real-time perception model of the mobile target based on the coordinate points, movement trajectory and attribute information of the indoor and outdoor mobile targets collected in real time, fuses the constructed real-time perception model of the mobile target into the real-scene three-dimensional model, and displays it on the front end. The embodiment of the present invention can fuse multiple models such as oblique photography models and BIM models, establish the corresponding relationship between each node of the model and the aerial photography image to realize batch mapping of the model, browse and access the oblique photography data and BIM data model on the browser side through the B / S end and based on GPU and server rendering technology, so as to improve the user's browsing experience; establish WebSocket network communication, establish a one-to-one correspondence between time and position, and smoothly, fluently and accurately realize the dynamic update of the coordinate trajectory points of the human and vehicle models.

[0072] Figure 5 A multi-model fusion system structure diagram provided by an embodiment of the present invention, such as Figure 5 As shown, a multi-model fusion system includes a mapping module 501, a rendering module 502, a fusion module 503 and a display module 504, wherein:

[0073] A mapping module 501 is used to perform mapping processing on the BIM model according to the aerial photography image to generate a BIM model and an oblique photography model of the indoor and outdoor scenes;

[0074] A rendering module 502 is used to render the BIM model and the oblique photography model of the indoor and outdoor scenes by using a B / S terminal mode and a GPU rendering mode;

[0075] A fusion module 503 is used to fuse the rendered BIM model and the oblique photography model to obtain a fused real-scene 3D model;

[0076] The display module 504 is used to construct a real-time perception model of the mobile target based on the coordinate points, movement trajectory and attribute information of the indoor and outdoor mobile targets collected in real time, integrate the constructed real-time perception model of the mobile target into the real-scene three-dimensional model, and display it on the front end.

[0077] The multi-model fusion system provided in an embodiment of the present invention corresponds to the multi-model fusion method provided in the aforementioned embodiments. The relevant technical features of the multi-model fusion system can refer to the relevant technical features of the multi-model fusion method, which will not be repeated here.

[0078] See also Figure 6 , Figure 6 Schematic diagram of an electronic device provided in an embodiment of the present application. Figure 6 As shown, an embodiment of the present application provides an electronic device, including a memory 610, a processor 620, and a computer program 611 stored in the memory 610 and executable on the processor 620. When the processor 620 executes the computer program 611, the following steps are implemented: according to the aerial photography image, the BIM model is mapped to generate the BIM model and the oblique photography model of the indoor and outdoor scenes; the BIM model and the oblique photography model of the indoor and outdoor scenes are rendered through the B / S end mode and the GUP rendering mode; the rendered BIM model and the oblique photography model are fused to obtain a fused real-life three-dimensional model; according to the coordinate points, movement trajectories and attribute information of the indoor and outdoor moving targets collected in real time, a real-time perception model of the moving target is constructed, the constructed real-time perception model of the moving target is fused into the real-life three-dimensional model, and it is displayed on the front end.

[0079] See also Figure 7 , Figure 7 A schematic diagram of an embodiment of a computer-readable storage medium provided in an embodiment of the present application. Figure 7 As shown, the present embodiment provides a computer-readable storage medium 700 on which a computer program 711 is stored. When the computer program 711 is executed by a processor, the following steps are implemented: according to the aerial photography image, the BIM model is mapped to generate the BIM model and the oblique photography model of the indoor and outdoor scenes; the BIM model and the oblique photography model of the indoor and outdoor scenes are rendered through the B / S end mode and the GUP rendering mode; the rendered BIM model and the oblique photography model are merged to obtain a merged real-life three-dimensional model; according to the coordinate points, movement trajectories and attribute information of the indoor and outdoor moving targets collected in real time, a real-time perception model of the moving targets is constructed, the constructed real-time perception model of the moving targets is merged into the real-life three-dimensional model, and it is displayed on the front end.

[0080] It should be noted that in the above embodiments, the description of each embodiment has its own emphasis, and for parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0081] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0082] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded computer, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0083] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0084] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0085] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application.

[0086] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A multi-model fusion method based on smart city, characterized in that: include: According to the aerial photography images, the BIM model is mapped to generate the BIM model of indoor and outdoor scenes; Render the BIM model and oblique photography model of indoor and outdoor scenes through B / S end and GPU rendering; The rendered BIM model and the oblique photography model are fused to obtain a fused real-scene 3D model; According to the coordinate points, movement trajectory and attribute information of indoor and outdoor moving targets collected in real time, a real-time perception model of the moving target is constructed, and the constructed real-time perception model of the moving target is integrated into the real-scene 3D model and displayed on the front end; The fusion of the rendered BIM model and the oblique photography model to obtain a fused real-scene 3D model includes: For the overlapping parts of the BIM model and the oblique photography model, the oblique photography model is cropped based on the Cesium 3D earth framework; The BIM model is integrated into the cropped oblique photography model to generate a real-life 3D model; For the overlapping part of the BIM model and the oblique photography model, the oblique photography model is clipped based on the Cesium three-dimensional earth framework, including: For the oblique photography model, the BIM model boundary is used as the initial polygonal clipping area, and the longitude and latitude coordinate values ​​of each edge of the polygonal clipping area are recorded; Taking the coordinate origin of the oblique photography model as a reference, matrix operation of coordinate values ​​is performed to correct the coordinates of the polygonal clipping area; Taking the surface where each edge of the corrected polygonal clipping area is located as the clipping surface, a plurality of clipping surfaces are obtained; Reconstructing the clipping surface according to the normal vector of any clipping surface and the shortest distance from the coordinate origin of the oblique photography model to any clipping surface, and finally obtaining a plurality of reconstructed clipping surfaces; The oblique photography model is cropped based on the reconstructed multiple cropping planes.

2. The multi-model fusion method according to claim 1, characterized in that: The method of mapping the BIM model according to the aerial photography image to generate the BIM model of the indoor and outdoor scenes includes: Parsing the IFC format used by the BIM model and splitting the nodes of the BIM model; Establish the corresponding relationship between each node of the BIM model and the aerial photography image; The aerial photography images corresponding to each node of the BIM model are imported into the BIM model, and the BIM model is batch mapped to generate the BIM model of indoor and outdoor scenes.

3. The multi-model fusion method according to claim 1 or 2, characterized in that: The rendering of the BIM model and the oblique photography model of the indoor and outdoor scenes by using the B / S end mode and the GPU rendering mode includes: On the B / S side, the BIM model data structure is converted into 3D Tiles data format, and the Cesium map engine is used to load and render the BIM model data.

4. The multi-model fusion method according to claim 3, characterized in that: Also includes: By setting the maximum screen space error parameter in the Cesium map engine, the screen error rate can be reduced and the level of detail of the BIM model can be maintained; Improve the loading speed of BIM model data by configuring the parameters of loading the central block of the screen first.

5. The multi-model fusion method according to claim 1, characterized in that: The step of constructing a real-time perception model of a moving target based on the coordinate points, moving trajectories and attribute information of the indoor and outdoor moving targets collected in real time includes: According to the coordinate points, movement trajectories and attribute information of indoor and outdoor mobile targets sent by the monitoring device through the established WebSocket real-time transmission channel, a real-time perception model of the mobile target is constructed; The method further includes integrating the constructed real-time perception model of the moving target into the real-scene three-dimensional model and displaying it on the front end: According to the coordinate points and movement trajectory of the moving target updated in real time by the monitoring device, the position and orientation of the real-time perception model of the moving target are updated, and the updated real-time perception model of the moving target is displayed on the front end.

6. The multi-model fusion method according to claim 5, characterized in that: The real-time perception model of the mobile target includes a human and a vehicle model, and also includes: When receiving the coordinate points, movement trajectories and attribute information of indoor and outdoor mobile targets sent by the monitoring device through the established WebSocket real-time transmission channel, a corresponding relationship between the real-time perception model ID of each mobile target and the time, position and orientation is established; The updating of the position and orientation of the real-time perception model of the moving target according to the coordinate points and movement trajectory of the moving target updated in real time by the monitoring device comprises: The real-time perception model ID, time, position, and orientation of the mobile target are input into the heading update function headingPitchRollQuaternion to update the position and orientation of the real-time perception model of the corresponding mobile target at the corresponding time.

7. A multi-model fusion system, characterized in that: include: The mapping module is used to map the BIM model based on the aerial photography images to generate the BIM model of the indoor and outdoor scenes; A rendering module is used to render the BIM model and oblique photography model of indoor and outdoor scenes through the B / S end and the GPU rendering method; A fusion module is used to fuse the rendered BIM model and the oblique photography model to obtain a fused real-scene 3D model; The display module is used to construct a real-time perception model of the moving target based on the coordinate points, movement trajectory and attribute information of the indoor and outdoor moving targets collected in real time, integrate the constructed real-time perception model of the moving target into the real-scene three-dimensional model, and display it on the front end; The fusion of the rendered BIM model and the oblique photography model to obtain a fused real-scene 3D model includes: For the overlapping parts of the BIM model and the oblique photography model, the oblique photography model is cropped based on the Cesium 3D earth framework; The BIM model is integrated into the cropped oblique photography model to generate a real-life 3D model; For the overlapping part of the BIM model and the oblique photography model, the oblique photography model is clipped based on the Cesium three-dimensional earth framework, including: For the oblique photography model, the BIM model boundary is used as the initial polygonal clipping area, and the longitude and latitude coordinate values ​​of each edge of the polygonal clipping area are recorded; Taking the coordinate origin of the oblique photography model as a reference, matrix operation of coordinate values ​​is performed to correct the coordinates of the polygonal clipping area; Taking the surface where each edge of the corrected polygonal clipping area is located as the clipping surface, a plurality of clipping surfaces are obtained; Reconstructing the clipping surface according to the normal vector of any clipping surface and the shortest distance from the coordinate origin of the oblique photography model to any clipping surface, and finally obtaining a plurality of reconstructed clipping surfaces; The oblique photography model is cropped based on the reconstructed multiple cropping planes.

8. An electronic device, characterized in that: It includes a memory and a processor, and the processor is used to implement the steps of the multi-model fusion method as described in any one of claims 1 to 6 when executing a computer management program stored in the memory.