VR panoramic fusion expression method and system driven by model and data collaboration

Abstract

The present invention discloses a VR panoramic fusion expression method and system driven by model and data collaboration, which belongs to the field of virtual reality technology and solves the problem that the existing panoramic image display method lacks deep coupling with the simulation model of the geographic scene object, and it is difficult to accurately describe the complex geographic scene objects and their spatiotemporal evolution process. The present invention includes fusing and modeling each geographic scene object with the corresponding geographic scene object in the real-world panoramic image collected in real time based on the spatial semantic constraints of the geographic scene to obtain a VR panoramic fusion scene; based on the attribute semantic information rules and the user's visual area of ​​interest, the VR panoramic fusion scene is enhanced and expressed by digital-analog association to obtain a VR panoramic scene, and the VR panoramic scene is updated in real time using dynamic data. The present invention is used for VR panoramic fusion expression.

Description

Technical Field

[0001] A VR panoramic fusion expression method and system driven by model and data collaboration, used for VR panoramic fusion expression, belonging to the field of virtual reality technology. Background Art

[0002] Virtual geographic environment is the digital mapping of the real geographic world in the computer. Through advanced digital means, the geographic information and spatial relationships of the real world are accurately simulated and intuitively displayed in the virtual space, providing users with a new and immersive geographic environment experience. In recent years, with the gradual popularization of virtual reality (VR) devices, users can immersively experience the virtual geographic environment in head-mounted displays. And they can interact with the virtual environment through handheld controllers, gesture recognition devices and other sensors to enhance user experience and cognitive efficiency. Panoramic image technology, as a new form of visual expression, creates a wide-angle image that can cover a 360-degree viewing angle by seamlessly splicing multiple images. Using panoramic images to build a virtual geographic environment and display it in VR can provide users with a full range of perspectives, making them feel as if they are in a real geographic space. Compared with using three-dimensional modeling or other methods to build a virtual geographic environment, panoramic images are convenient, fast, realistic, and easy to update and maintain. Through panoramic images, users can not only obtain an overall view of geographic objects, but also observe local details in detail, providing a new way of thinking for the comprehensive cognition and analysis of geographic information.

[0003] Visualization is a method of presenting data, information or knowledge in a visual form, so as to more intuitively understand and analyze complex scientific concepts, processes or results. In the geographic environment, visualization aims to improve users' understanding and perception of geographic information, provide richer details and clearer spatial relationships. In the past, the visualization of geographic environment was often used in three-dimensional engines such as Cesium, osgEarth and Unity 3D, using visual variables such as color, symbols, text and animation to transmit and interpret information. For panoramic images, visualization is often used to reflect geographic scene information through hotspot annotation. This method lacks deep coupling with geographic simulation models, cannot accurately describe the spatiotemporal evolution of geographic phenomena, does not support geographic analysis operations such as geographic problem diagnosis and trend prediction, and has the problem of "only viewing, not using". In addition, since panoramic images have a 360° all-round perspective, constructing information annotations for the entire scene not only makes the scene content complex and numerous, consumes system performance, but also easily leads to information overload, making users face cognitive burden when interpreting spatial information.

[0004] In summary, there are few studies on the fusion expression of geographical scenes based on VR panorama, and there are the following technical problems:

[0005] 1. The traditional panoramic image display method lacks deep coupling with the simulation model of geographic scene objects, and it is difficult to accurately describe complex geographic scene objects and their spatiotemporal evolution process;

[0006] 2. In the existing panoramic image expression, the scene information is complex, but there are few means of visualization expression, the user's cognitive efficiency is low, and it is difficult to efficiently obtain key information;

[0007] 3. Traditional VR virtual scenes are inefficient in data updating and are difficult to respond to dynamic changes in the real world in real time. Summary of the invention

[0008] The purpose of the present invention is to provide a VR panoramic fusion expression method and system driven by model and data collaboration, so as to solve the problem that the existing panoramic image display method lacks deep coupling with the simulation model of geographic scene objects and is difficult to accurately describe complex geographic scene objects and their spatiotemporal evolution process.

[0009] In order to achieve the above object, the technical solution adopted by the present invention is:

[0010] A VR panoramic fusion expression method driven by model and data collaboration includes the following steps:

[0011] Step 1: Based on the spatial semantic constraints of the geographic scene, each geographic scene object is fused and modeled with the corresponding geographic scene object in the real-world panoramic image collected in real time to obtain a VR panoramic fusion scene;

[0012] Step 2: Based on the attribute semantic information rules and the user's visual area of ​​interest, the VR panoramic fusion scene is enhanced by digital-analog association to obtain a VR panoramic scene, and the VR panoramic scene is updated in real time using dynamic data.

[0013] Further, the specific steps of step 1 are:

[0014] Step 1.1, constructing spatial semantic constraint rules for the three-dimensional VR panoramic virtual scene, including spatial orientation semantic constraint rules and spatial topological semantic constraint rules. The spatial orientation semantic constraint rules refer to establishing a two-dimensional and three-dimensional spatial mapping relationship between the geographic scene object and the VR panoramic virtual scene based on spherical projection by defining the spatial position and posture, and providing a complete spatial description framework. The spatial topological semantic constraint rules refer to emitting rays to the three-dimensional space object based on the imaging characteristics of the two-dimensional panoramic image from the viewpoint of the camera, and determining the two-dimensional boundary range of the geographic scene object in the VR panoramic virtual scene, thereby obtaining the topological relationship between the various spatial objects. The geographic scene object represents the object corresponding to the real world;

[0015] Step 1.2: Based on the constructed spatial semantic constraint rules, each geographic scene object is seamlessly integrated with the overall panoramic image of the real world collected in real time. The specific steps are as follows:

[0016] Firstly, the transformation relationship of each geographic scene object from the local coordinate system to the geographic coordinate system is constructed based on the spatial orientation semantic constraint rules, and the new coordinates of each geographic scene object in the unified geographic coordinate system are obtained based on the comprehensive transformation matrix obtained based on the transformation relationship. The formula of the comprehensive transformation matrix and the new coordinates is:

[0017] T=T transiate ×R×S

[0018] P′=TP

[0019] Among them, T is the comprehensive transformation matrix, T transiate represents the translation matrix of the coordinates of any point in each geographic scene object, R represents the rotation matrix of the coordinates of any point in each geographic scene object, S represents the scaling matrix of the coordinates of any point in each geographic scene object, P represents the homogeneous coordinates (x, y, z, 1) of a point in each geographic scene object, and P′ represents the new coordinates obtained by the comprehensive transformation matrix of the homogeneous coordinates (x, y, z, 1) of any point in each geographic scene object;

[0020] Secondly, according to the characteristics of each geographic scene object projected onto the real-world panoramic image, a VR virtual scene coordinate system based on spherical projection with the camera as the origin is constructed. Combined with the spatial position and posture of the real-world panoramic image at the moment of photography, and based on the new coordinates of each geographic scene object in the unified geographic coordinate system, the simulation model corresponding to each geographic scene object is transformed into the spherical coordinates in the VR virtual scene coordinate system. Finally, the spherical coordinates are projected and transformed into two-dimensional panoramic image coordinates to obtain a two-dimensional panoramic image of the simulation model of each geographic scene object. The spherical coordinates include longitude and latitude. The simulation model of the geographic scene object represents the three-dimensional model of the geographic scene object in the virtual scene, including the building BIM model and the flood simulation model. The formulas for the longitude and latitude of the spherical coordinates are:

[0021] λ=arctan2(Y,X)

[0022]

[0023] The formula for 2D image coordinates is:

[0024]

[0025] Wherein, λ represents the longitude in the spherical coordinate system, φ represents the latitude in the spherical coordinate system, (X, Y, Z) represents the coordinates of a point of each geographic scene object simulation model in three-dimensional space, (u, v) represents the two-dimensional image coordinates, and W and H represent the width and height of the panoramic image, respectively;

[0026] Finally, based on the spatial topological semantic constraint rules, the simulation models of each geographic scene object projected into a two-dimensional panoramic image are superimposed on the collected real-world panoramic image to achieve the fusion of the simulation model of each geographic scene object with the corresponding geographic scene object in the real-world panoramic image at the corresponding moment. The fused panoramic image is then mapped to the surface of a sphere in three-dimensional space through the spherical projection method, thereby obtaining a VR panoramic fusion scene.

[0027] Further, the specific steps of step 2 are:

[0028] Step 2.1: Based on the attribute semantic information rules and the digital-analog association expression of the user's visual area of ​​interest, the specific steps are as follows:

[0029] Step 2.11, construct attribute semantic information rules describing each geographic scene object, including attribute semantic information rules describing geographic processes and attribute semantic information rules describing geographic entities. Attribute semantic information rules describing geographic entities include spatial attribute semantic information rules and non-spatial attribute semantic information rules of geographic scene objects. Spatial attribute semantic information rules include coordinate positions and geometric shapes of geographic scene objects, and non-spatial attribute semantic information rules include ID identification, subordinate categories, uses, and geographic process impacts of geographic scene objects.

[0030] Step 2.12, semantically associating the spatial position of the geographic scene object in the spatial semantic constraint rule with the ID identifier in the attribute semantic information rule;

[0031] Step 2.13: Based on the semantic association obtained in step 2.12, visual variables are constructed in the VR panoramic fusion scene to realize the associated fusion expression of data and models, that is, to realize the semantic enhanced associated fusion of spatial attribute information data and geographic scene object simulation models, wherein the visual variables include text descriptions, self-explanatory symbols, hot spot marks, color changes and dynamic flashing. The specific steps are as follows:

[0032] First, when creating a visual variable for any geographic scene object, we should first determine that the spatial position of the visual variable should be located at the center of the geometric shape of the geographic scene object based on its position in the VR panoramic fusion scene through the spatial orientation semantic constraint rules, which means that the visual variable corresponds to the geographic scene object simulation model one by one;

[0033] Secondly, after the visual variables are matched one by one with the simulation models of geographic scene objects, according to the spatial topological relationship of the described geographic scene objects, if the geographic scene object is completely blocked by other geographic scene objects at the moment of taking the panoramic image in the real world, the visual variables of this geographic scene object will be removed from the VR panoramic scene according to the spatial topological semantic constraint rules, otherwise they will not be deleted;

[0034] Finally, after deleting the visual variables corresponding to the obstructed geographic scene objects, the information displayed by the retained visual variables based on semantic association matches the described geographic scene objects, that is, the associated fusion expression of data and models in the VR panoramic fusion field is realized;

[0035] Step 2.14: Based on the associative fusion expression of data and model, the digital-analog associative expression of the user's visual area of ​​interest is performed. The specific steps are as follows:

[0036] According to the position and posture of the user's VR headset, the user's gaze point during interactive behavior is dynamically obtained. When the user's gaze point is within the two-dimensional boundary range of a certain geographic scene object, the two-dimensional boundary range of this geographic scene object is defined as the user's area of ​​interest;

[0037] The region of interest is determined by obtaining the user's gaze point and then establishing the region of interest constraints;

[0038] Based on the constraints of the region of interest and the associated fusion expression of the data and the model in the VR panoramic fusion field, the detailed attributes of the corresponding geographic scene objects are displayed to obtain the VR panoramic scene, wherein the detailed attributes are the attributes defined by the attribute semantic information rules;

[0039] Step 2.2: VR panoramic dynamic update based on data model collaborative driving, that is, taking the panoramic image frame collected in real time as the next frame in the video stream, updating the VR panoramic fusion scene, visual variables and area of ​​interest constraints to obtain the updated VR panoramic scene.

[0040] A VR panoramic fusion expression system driven by model and data collaboration, including:

[0041] VR panoramic fusion scene construction module: Based on the spatial semantic constraints of the geographic scene, each geographic scene object and the corresponding geographic scene object in the real-world panoramic image collected in real time are fused and modeled to obtain a VR panoramic fusion scene;

[0042] VR panoramic scene construction module: Based on the attribute semantic information rules and the user's visual area of ​​interest, the VR panoramic fusion scene is enhanced by digital-analog association to obtain the VR panoramic scene, and the VR panoramic scene is updated in real time using dynamic data.

[0043] Furthermore, the specific implementation steps of the VR panoramic fusion scene construction module are:

[0044] Step 1.1, constructing spatial semantic constraint rules for the three-dimensional VR panoramic virtual scene, including spatial orientation semantic constraint rules and spatial topological semantic constraint rules. The spatial orientation semantic constraint rules refer to establishing a two-dimensional and three-dimensional spatial mapping relationship between the geographic scene object and the VR panoramic virtual scene based on spherical projection by defining the spatial position and posture, and providing a complete spatial description framework. The spatial topological semantic constraint rules refer to emitting rays to the three-dimensional space object based on the imaging characteristics of the two-dimensional panoramic image from the viewpoint of the camera, and determining the two-dimensional boundary range of the geographic scene object in the VR panoramic virtual scene, thereby obtaining the topological relationship between the various spatial objects. The geographic scene object represents the object corresponding to the real world;

[0045] Step 1.2: Based on the constructed spatial semantic constraint rules, each geographic scene object is seamlessly integrated with the overall panoramic image of the real world collected in real time. The specific steps are as follows:

[0046] Firstly, the transformation relationship of each geographic scene object from the local coordinate system to the geographic coordinate system is constructed based on the spatial orientation semantic constraint rules, and the new coordinates of each geographic scene object in the unified geographic coordinate system are obtained based on the comprehensive transformation matrix obtained based on the transformation relationship. The formula of the comprehensive transformation matrix and the new coordinates is:

[0047] T=T transiate ×R×S

[0048] P′=TP

[0049] Among them, T is the comprehensive transformation matrix, T transiate represents the translation matrix of the coordinates of any point in each geographic scene object, R represents the rotation matrix of the coordinates of any point in each geographic scene object, S represents the scaling matrix of the coordinates of any point in each geographic scene object, P represents the homogeneous coordinates (x, y, z, 1) of a point in each geographic scene object, and P′ represents the new coordinates obtained by the comprehensive transformation matrix of the homogeneous coordinates (x, y, z, 1) of any point in each geographic scene object;

[0050] Secondly, according to the characteristics of each geographic scene object projected onto the real-world panoramic image, a VR virtual scene coordinate system based on spherical projection with the camera as the origin is constructed. Combined with the spatial position and posture of the real-world panoramic image at the moment of photography, and based on the new coordinates of each geographic scene object in the unified geographic coordinate system, the simulation model corresponding to each geographic scene object is transformed into the spherical coordinates in the VR virtual scene coordinate system. Finally, the spherical coordinates are projected and transformed into two-dimensional panoramic image coordinates to obtain a two-dimensional panoramic image of the simulation model of each geographic scene object. The spherical coordinates include longitude and latitude. The simulation model of the geographic scene object represents the three-dimensional model of the geographic scene object in the virtual scene, including the building BIM model and the flood simulation model. The formulas for the longitude and latitude of the spherical coordinates are:

[0051] λ=arctan2(Y,X)

[0052]

[0053] The formula for 2D image coordinates is:

[0054]

[0055] Wherein, λ represents the longitude in the spherical coordinate system, φ represents the latitude in the spherical coordinate system, (X, Y, Z) represents the coordinates of a point of each geographic scene object simulation model in three-dimensional space, (u, v) represents the two-dimensional image coordinates, and W and H represent the width and height of the panoramic image, respectively;

[0056] Finally, based on the spatial topological semantic constraint rules, the simulation models of each geographic scene object projected into a two-dimensional panoramic image are superimposed on the collected real-world panoramic image to achieve the fusion of the simulation model of each geographic scene object with the corresponding geographic scene object in the real-world panoramic image at the corresponding moment. The fused panoramic image is then mapped to the surface of a sphere in three-dimensional space through the spherical projection method, thereby obtaining a VR panoramic fusion scene.

[0057] Furthermore, the specific implementation steps of the VR panoramic scene construction module are:

[0058] Step 2.1: Based on the attribute semantic information rules and the digital-analog association expression of the user's visual area of ​​interest, the specific steps are as follows:

[0059] Step 2.11, construct attribute semantic information rules describing each geographic scene object, including attribute semantic information rules describing geographic processes and attribute semantic information rules describing geographic entities. Attribute semantic information rules describing geographic entities include spatial attribute semantic information rules and non-spatial attribute semantic information rules of geographic scene objects. Spatial attribute semantic information rules include coordinate positions and geometric shapes of geographic scene objects, and non-spatial attribute semantic information rules include ID identification, subordinate categories, uses, and geographic process impacts of geographic scene objects.

[0060] Step 2.12, semantically associating the spatial position of the geographic scene object in the spatial semantic constraint rule with the ID identifier in the attribute semantic information rule;

[0061] Step 2.13: Based on the semantic association obtained in step 2.12, visual variables are constructed in the VR panoramic fusion scene to realize the associated fusion expression of data and models, that is, to realize the semantic enhanced associated fusion of spatial attribute information data and geographic scene object simulation models, wherein the visual variables include text descriptions, self-explanatory symbols, hot spot marks, color changes and dynamic flashing. The specific steps are as follows:

[0062] First, when creating a visual variable for any geographic scene object, we should first determine that the spatial position of the visual variable should be located at the center of the geometric shape of the geographic scene object based on its position in the VR panoramic fusion scene through the spatial orientation semantic constraint rules, that is, we can obtain a one-to-one correspondence between the visual variable and the geographic scene object simulation model;

[0063] Secondly, after the visual variables are matched one by one with the simulation models of geographic scene objects, according to the spatial topological relationship of the described geographic scene objects, if the geographic scene object is completely blocked by other geographic scene objects at the moment of taking the panoramic image in the real world, the visual variables of this geographic scene object will be removed from the VR panoramic scene according to the spatial topological semantic constraint rules, otherwise they will not be deleted;

[0064] Finally, after deleting the visual variables corresponding to the obstructed geographic scene objects, the information displayed by the retained visual variables based on semantic association matches the described geographic scene objects, that is, the associated fusion expression of data and models in the VR panoramic fusion field is realized;

[0065] Step 2.14: Based on the associative fusion expression of data and model, the digital-analog associative expression of the user's visual area of ​​interest is performed. The specific steps are as follows:

[0066] According to the position and posture of the user's VR headset, the user's gaze point during interactive behavior is dynamically obtained. When the user's gaze point is within the two-dimensional boundary range of a certain geographic scene object, the two-dimensional boundary range of this geographic scene object is defined as the user's area of ​​interest;

[0067] The region of interest is determined by obtaining the user's gaze point and then establishing the region of interest constraints;

[0068] Based on the constraints of the region of interest and the associated fusion expression of the data and the model in the VR panoramic fusion field, the detailed attributes of the corresponding geographic scene objects are displayed to obtain the VR panoramic scene, wherein the detailed attributes are the attributes defined by the attribute semantic information rules;

[0069] Step 2.2: VR panoramic dynamic update based on data model collaborative driving, that is, taking the panoramic image frame collected in real time as the next frame in the video stream, updating the VR panoramic fusion scene, visual variables and area of ​​interest constraints to obtain the updated VR panoramic scene.

[0070] Compared with the prior art, the advantages of the present invention are:

[0071] The present invention uses a panoramic camera mounted on a drone to dynamically acquire panoramic images, and combines spatial semantic constraint rules to perform fusion modeling of geographic scene objects and panoramic images in a VR virtual environment. After fusion, based on the user's visual area of ​​interest, the associated fusion expression of the combined data and model is used to perform enhanced expression and dynamic update of the VR panorama in this virtual environment. The specific effects are as follows:

[0072] First, the present invention realizes the efficient generation of VR panoramic scenes by combining three-dimensional geographic scene objects and panoramic images for fusion modeling. In traditional virtual reality environments, the spatial position, posture and spatial relationship of geographic scene objects with other objects are prone to deviations, resulting in insufficient realism of virtual scenes. However, the present invention ensures the precise positioning and posture matching of different geographic scene objects in VR panoramic scenes through spatial semantic rules, making virtual scenes more consistent with the real world and significantly improving the user's sense of immersion.

[0073] Second, the present invention introduces dynamic enhanced expression based on the user's visual area of ​​interest. By tracking the user's gaze point and interactive behavior in the virtual scene, the system can dynamically determine the user's area of ​​interest in real time and enhance the expression of the spatial objects in the area, such as by means of color changes, symbol annotations, text descriptions, etc. to enhance the user's visual experience. This not only reduces the user's cognitive burden when facing complex scenes, but also enables the user to obtain key geographic information more quickly and accurately, thereby improving the usability of the virtual scene and the user's cognitive efficiency;

[0074] 3. The update speed of traditional virtual reality scenes often lags behind the changes in the real environment, and it is impossible to achieve real-time reflection of the dynamic environment. The present invention uses a panoramic camera equipped with an unmanned aerial vehicle to obtain panoramic images in real time, and combines spatial semantic constraint rules and attribute semantic information rules to dynamically update the geographic objects and their related attributes in the virtual scene, so that the virtual scene can reflect the changes in the real environment in real time, significantly improving the real-time and authenticity of the virtual environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0076] Figure 1 It is the overall idea diagram of the present invention;

[0077] Figure 2 is a schematic diagram of the spatial position and spatial posture in the present invention;

[0078] Figure 3 Schematic diagram of the spatial topological relationship in the present invention;

[0079] Figure 4 Modeling VR panoramic fusion with spatial semantic constraints in the present invention;

[0080] Figure 5 A schematic diagram of the digital-analog correlation expression based on the user's area of ​​interest in the present invention;

[0081] Figure 6 A schematic diagram of VR panoramic dynamic update driven by data model collaboration in the present invention;

[0082] Figure 7 This is a schematic diagram of the VR panoramic fusion scene obtained by mapping the fused panoramic image to the surface of a sphere in a three-dimensional space through a spherical projection method in the present invention. DETAILED DESCRIPTION

[0083] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0084] like Figure 1As shown, the method includes fusing and modeling each geographic scene object with the corresponding geographic scene object in the real-world panoramic image collected in real time based on the spatial semantic constraints of the geographic scene to obtain a VR panoramic fusion scene, enhancing the expression of the VR panoramic fusion scene by digital-analog association based on the attribute semantic information rules and the user's visual interest area to obtain a VR panoramic scene, and using dynamic data to update the VR panoramic scene in real time. On the one hand, the spatial semantic constraint rules are first constructed to provide a rigorous description framework for the fusion modeling of the VR panoramic virtual environment. Based on these constraint rules, the simulation model corresponding to the geographic scene object is converted into a two-dimensional panoramic image, and superimposed and re-projected with the geographic scene object corresponding to the real-world panoramic image, and finally the fusion modeling of the spatial geographic scene object data and the panoramic image is realized to obtain a VR panoramic fusion scene. On the other hand, focusing on the user's cognitive efficiency and interactive experience of the VR panoramic fusion scene, the constraint based on the user's visual interest area is introduced. According to the user's gaze point and interactive behavior, the user's area of ​​interest is dynamically determined, and the area is enhanced and expressed as self-explanatory symbols, hotspot markers and text descriptions in the visual variables. At the same time, the VR panoramic dynamic update of model-data association fusion is realized, that is, the panoramic image data is obtained in real time by using the panoramic camera carried by the drone, and the position, posture and attributes of the geographic scene objects and semantic annotation information in the virtual scene are dynamically updated in combination with the spatial semantic knowledge of the geographic scene objects, making it closer to the state of the real world. Figure 1 The determination of the viewpoint falling point is to determine whether the viewpoint falls within the two-dimensional boundary range of a certain geographic scene object.

[0085] A VR panoramic fusion expression method driven by model and data collaboration includes the following steps:

[0086] Based on the spatial semantic constraints of the geographic scene, each geographic scene object and the corresponding geographic scene object in the real-world panoramic image collected in real time are fused and modeled to obtain a VR panoramic fusion scene. The specific steps are:

[0087] Construction of spatial semantic constraint rules for geographic scene objects:

[0088] Construct spatial semantic constraint rules for three-dimensional VR panoramic virtual scenes, including spatial orientation semantic constraint rules and spatial topology semantic constraint rules. Geographic scene objects represent objects corresponding to the real world.

[0089] Spatial orientation semantic constraint rules refer to the establishment of two-dimensional and three-dimensional spatial mapping relationships between geographic scene objects and VR panoramic virtual scenes based on spherical projection by defining spatial position and posture, providing a complete spatial description framework. Spatial orientation semantic constraint rules involve the description of both spatial position and spatial posture. Spatial position not only includes the use of geographic coordinate systems to describe the position of geographic scene objects in three-dimensional space, but also includes the use of local or scene coordinate systems to describe the position of objects relative to their environment or parent objects. In a certain geographic coordinate system, the three-dimensional spatial position of geographic scene objects can be represented by longitude, latitude, and altitude. Different geographic coordinate systems have different geographic projections and reference ellipsoids, which may cause the coordinate values ​​of the same geographic point in different geographic coordinate systems to be different. The local coordinate system usually takes the geometric center of the three-dimensional object or a specific reference point as the origin, and the direction and position of the coordinate axis are determined relative to the characteristics of the object itself and the application requirements. The typical local coordinate system uses a three-dimensional Cartesian coordinate system, in which the position is represented by x, y, and z three-dimensional coordinates, which correspond to the horizontal, vertical, and depth positions of the object in the local coordinate system, respectively. The spatial attitude describes the direction and rotation state of the object at its location, usually represented by the pitch angle, yaw angle and roll angle. The pitch angle describes the rotation of the object around the Y axis of the local coordinate system, the yaw angle describes the rotation around the Z axis, and the roll angle describes the rotation around the X axis. Through these three angle parameters, the attitude of the object in three-dimensional space can be accurately described.

[0090] By combining spatial position and spatial posture, spatial orientation semantic constraint rules can provide a complete spatial description framework, enabling the system to accurately understand and process complex spatial relationships. Subsequently, two- and three-dimensional spatial mapping relationships based on spherical projection can be established for geographic scene objects and VR panoramic virtual scenes, unifying them into the same coordinate system, and realizing the position alignment and spatial posture matching between different geographic scene objects and VR panoramic virtual scenes.

[0091] Spatial topological semantic constraint rules refer to the imaging characteristics of two-dimensional panoramic images. Starting from the camera's viewpoint, rays are emitted to three-dimensional spatial objects to determine the two-dimensional boundary range of geographic scene objects in the VR panoramic scene, thereby obtaining the topological relationship between various spatial objects. Topological relationship semantic constraint rules specifically refer to the geometric relationship between spatial objects. These relationships are independent of the specific shape and size of the objects and mainly focus on their relative position and connection method. Common topological relationships include adjacency, inclusion, intersection, and separation. In three-dimensional space, spatial objects exist independently of each other and there is no intersecting topological relationship. Two-dimensional panoramic images and videos express three-dimensional spatial objects by projecting them onto a two-dimensional plane, usually represented by spherical or cubic projections. When three-dimensional spatial objects are projected onto a two-dimensional plane, it is necessary to consider whether there are intersecting topological relationships between different spatial objects.

[0092] Therefore, the present invention combines the imaging characteristics of the real-world panoramic image, starts from the camera's viewpoint, emits rays to the three-dimensional space object, determines the boundary range of the geographic scene object in the real-world panoramic image, and thus obtains the topological relationship between the spatial objects. Figure 3 In the VR panorama, the projection of geographic scene objects on the real-world panoramic image has four spatial relationships: adjacency, inclusion, intersection, and separation. The orange and purple cubes belong to a spatial adjacency relationship, the orange and red cubes belong to a spatial inclusion relationship, the green and blue cubes belong to a spatial intersection relationship, and the blue and purple cubes belong to a spatial separation relationship. Through these topological semantic rules, the spatial layout and spatial topological relationship of different geographic scene objects can be correctly integrated and expressed in the VR panorama, enhancing the authenticity and consistency of the virtual scene.

[0093] Based on the constructed spatial orientation semantic constraint rules, the real-time collected geographic scene objects and VR panoramic scenes are seamlessly integrated. The specific steps are as follows:

[0094] The spatial semantic constraint rules express the relative spatial relationship between geographic space objects from the two aspects of spatial orientation and spatial topology, respectively, and provide a rigorous description framework for the fusion construction of VR panoramic virtual scenes. The use of these constraint rules can ensure that the accuracy and consistency of the spatial relationship of three-dimensional geographic scene objects are maintained when they are uniformly naturalized to the VR virtual scene coordinate system, thereby providing effective support for the precise modeling of complex spatial relationships between various geographic entities. Based on the spatial orientation semantic constraint rules, the present invention first constructs the transformation relationship of geographic scene objects from the local coordinate system to the geographic coordinate system, and obtains the new coordinates of the geographic scene objects in the unified geographic coordinate system based on the comprehensive transformation matrix obtained from the transformation relationship, so as to realize the fusion expression of various geographic scene object simulation models in the unified geographic coordinate system.

[0095] Specifically, the process of converting a three-dimensional object from one three-dimensional coordinate system to another three-dimensional coordinate system usually involves geometric transformations, mainly including translation, rotation and scaling operations. Translation operations are used to move objects from one position to another, rotation operations are used to rotate objects around a certain axis by a certain angle, and scaling operations are used to adjust the size of objects. Translation, rotation and scaling are combined to form a comprehensive transformation matrix. The comprehensive transformation matrix is ​​usually expressed in the form of homogeneous coordinates to facilitate matrix operations. Homogeneous coordinates are expanded to four-dimensional vectors, and the transformation matrix is ​​also expanded to a 4×4 matrix. Finally, the comprehensive transformation matrix and the new coordinate formula are obtained as follows:

[0096] T=T transiate ×R×S

[0097] P′=TP

[0098] Among them, T is the comprehensive transformation matrix, T transiate Represents the translation matrix of the coordinates of any point in the geographic scene object, R represents the rotation matrix of the coordinates of any point in the geographic scene object, S represents the scaling matrix of the coordinates of any point in the geographic scene object, P represents the homogeneous coordinates (x, y, z, 1) of a point in the geographic scene object, and P′ represents the new coordinates obtained by the comprehensive transformation matrix of the homogeneous coordinates (x, y, z, 1) of any point in the geographic scene object;

[0099] Secondly, according to the characteristics of each geographic scene object projected onto the real-world panoramic image, a VR virtual scene coordinate system based on spherical projection with the camera as the origin is constructed. Combined with the spatial position and posture of the real-world panoramic image at the moment of photography, and based on the new coordinates of each geographic scene object in the unified geographic coordinate system, the simulation model corresponding to each geographic scene object is transformed into the spherical coordinates in the VR virtual scene coordinate system. Finally, the spherical coordinates are projected and transformed into two-dimensional panoramic image coordinates to obtain a two-dimensional panoramic image of the simulation model of each geographic scene object, where the spherical coordinates include longitude and latitude, and the geographic scene object simulation model represents the three-dimensional model of the geographic scene object in the virtual scene, including the building BIM model and the flood simulation model.

[0100] Specifically, projecting a three-dimensional object into a panoramic image usually requires a geometric projection transformation to convert the object in the three-dimensional coordinate system into pixel coordinates in the two-dimensional image. Two-dimensional panoramic images are generally represented by spherical projections (such as equidistant cylindrical projections), so the projection process involves the steps of converting from three-dimensional coordinates to spherical coordinates, and then to two-dimensional image coordinates. Assuming that the coordinates of a point of a three-dimensional object in three-dimensional space are (X, Y, Z), it first needs to be converted into longitude and latitude coordinates in a spherical coordinate system.

[0101] Therefore, the formulas for longitude and latitude in spherical coordinates are:

[0102] λ=arctan2(Y,X)

[0103]

[0104] The formula for 2D image coordinates is:

[0105]

[0106] Among them, λ represents the longitude in the spherical coordinate system, φ represents the latitude in the spherical coordinate system, (X, Y, Z) represents the coordinates of a point in the geographical scene object simulation model in three-dimensional space, (u, v) represents the two-dimensional image coordinates, and W and H represent the width and height of the panoramic image, respectively.

[0107] Finally, based on the spatial topological semantic constraint rules, the simulation models of each geographic scene object projected as a two-dimensional panoramic image are superimposed with the collected real-world panoramic image, so as to achieve the fusion of each geographic scene object simulation model with the corresponding geographic scene object in the real-world panoramic image at the corresponding moment, and then the fused panoramic image is mapped to the surface of a sphere in three-dimensional space through the spherical projection method, thereby obtaining a VR panoramic fusion scene. That is, when multiple geographic scene objects are superimposed with the corresponding geographic scene objects in the real-world panoramic image, the topological relationship between the corresponding geographic scene objects and other geographic scene objects should also be considered. For example, at the moment of shooting the real-world panoramic image, in the camera's perspective, geographic scene object A is located behind geographic scene object B, and part of geographic scene object A is blocked by geographic scene object B. At this time, the topological relationship between the two geographic scene objects is an intersection relationship from the camera's perspective. When the layers are superimposed, the panoramic image of geographic scene object A should be placed in the next layer of geographic scene object B, so that the topological relationship between the two geographic scene objects can be correctly expressed. The VR panoramic fusion scene can fuse and enhance the BIM of geographic scene objects and the geographic scene objects actually obtained in the real world that are under construction or need attention.

[0108] Based on the attribute semantic information rules and the user's visual interest area, the VR panoramic fusion scene is enhanced by digital-analog association to obtain the VR panoramic scene, and the VR panoramic scene is updated in real time using dynamic data. The specific steps are as follows:

[0109] Based on the attribute semantic information rules and the digital-analog association expression of the user's visual area of ​​interest, the specific steps are:

[0110] First, construct the attribute semantic information rules describing each geographic scene object. That is, various geographic scene objects are integrated with panoramic images to construct virtual scenes, which allows users to intuitively feel the details and characteristics of the real environment without having to visit the scene in person. It not only contains many geographic entity objects, such as bridges, buildings, mountains, etc., but also many geographic process objects such as debris flow, floods, glacier melting, etc. However, facing the virtual scene, although users can "see clearly" and "see realistically", they may not be able to "understand". In order to improve users' understanding and cognition of virtual scenes, it is necessary to introduce more auxiliary information and intelligent interactive functions to achieve enhanced expression of VR panoramic virtual scenes. In addition to rich geospatial data, non-spatial data (such as name, type, value and characteristics, etc.) used to describe geographic scene objects are also an important part of building VR panoramic virtual environments. These non-spatial data constitute the attribute semantic information rules describing geographic scene objects, which play a key role in enhancing the cognition of the geographic environment, making the virtual scene not only visible, but also easy to understand and perceive.

[0111] Specifically, attribute semantic information rules can be further divided into two categories: one describes geographic processes, and the other describes geographic entities. With the help of attribute semantic information rules that describe geographic processes, we can better understand and simulate geographic processes. Taking flood disasters as an example, its attribute semantic information rules should include the following aspects:

[0112] Time attribute: record the specific time when the disaster occurs and ends.

[0113] Category attributes: describe the nature and characteristics of the disaster, such as disaster type, impact range, etc.

[0114] Numerical attributes: provide quantitative data on the evolution of disasters, such as water level, flooded area, water flow velocity, etc.

[0115] Causal attributes: describe the driving factors that cause geographic processes to occur and how these factors affect the course of the disaster.

[0116] Table 1 Attribute semantic information rules

[0117]

[0118]

[0119] As for the attribute semantic information rules describing geographic entities, they can be expressed from two aspects: spatial attribute semantic information rules and non-spatial attribute semantic information rules. Spatial attribute semantic information mainly describes the coordinate position and geometric shape of geographic scene objects, while non-spatial attribute semantic information describes the ID identification, subordinate category, use description and geographic process impact of geographic scene objects (for example, a building, a geographic entity, will have attribute information such as "not submerged, partially submerged, completely submerged" under the influence of the geographic process of flood disasters). The specific attribute semantic information rules are shown in Table 1.

[0120] Secondly, the spatial position of the geographic scene object in the spatial semantic constraint rule is semantically associated with the ID identifier in the attribute semantic information rule, that is, all attribute information is uniformly managed using the database, and the semantic association of attribute data is realized according to the spatial position of the geographic scene object and the ID identifier in the attribute semantic information rule, thereby improving the retrieval efficiency and the fusion visualization effect of non-spatial data in the VR panoramic virtual environment.

[0121] Based on semantic association, visual variables are constructed in the VR panoramic fusion scene to realize the associated fusion expression of data and models, that is, to realize the semantic enhanced associated fusion of spatial attribute information data and geographic scene object simulation model. Among them, visual variables include text descriptions, self-explanatory symbols, hot spot marks, color changes and dynamic flashing. The specific steps are as follows:

[0122] When creating visual variables for any geographic scene object, first of all, according to its position in the VR panoramic fusion scene, the spatial position of the visual variable should be determined by the spatial orientation semantic constraint rules to be located at the center of the geometric shape of the geographic scene object, which means that the visual variable corresponds to the simulation model of the geographic scene object one by one. That is, with the VR panoramic virtual scene as the carrier, the basic visual variable design is used to create new semantic visual variables. Through the guidance of these visual effects, users can quickly identify and focus on key areas, and semantically enhance and deeply focus on scene information. These elements can not only provide rich attribute information, but also help users understand complex spatial relationships and geographic processes through intuitive visual cues. When constructing visual variables in VR panoramic virtual scenes, spatial semantic constraint rules should be combined to ensure that their spatial position relationships and attribute information are accurately expressed, and correspond one to one with the spatial objects they describe. For example, when creating a hotspot mark for a building, its position in the virtual scene should be considered first. With the help of spatial orientation semantic constraints, it is determined that the spatial position of the hotspot mark should be located at the center of the geometric shape of the building.

[0123] After the visual variables are matched one by one with the simulation models of geographic scene objects, according to the spatial topological relationship of the described geographic scene objects, if the geographic scene object is completely blocked by other geographic scene objects at the moment of taking the panoramic image in the real world, the visual variables of this geographic scene object are removed from the VR panoramic scene according to the spatial topological semantic constraint rules, otherwise they are not deleted. That is, considering the topological relationship of the geographic scene objects described by the visual variables, if this building is completely blocked by other buildings at the moment of taking the panoramic image, according to the spatial topological semantic constraint rules, the hotspot mark or other mark here should be removed from the virtual scene.

[0124] After deleting the visual variables corresponding to the occluded geographic scene objects, the information displayed by the retained visual variables based on semantic association matches the described geographic scene objects, that is, the associated fusion expression of data and models in the VR panoramic fusion field is realized.

[0125] Finally, based on the associative fusion expression of data and model, the digital-analog associative expression of the user's visual area of ​​interest is performed. The specific steps are:

[0126] According to the position and posture of the user's VR headset, the gaze point of the user's interactive behavior is dynamically obtained. When the user's gaze point is within the two-dimensional boundary range of a certain geographic scene object, the two-dimensional boundary range of this geographic scene object is defined as the user's region of interest. There are many elements in the VR panoramic virtual scene. It is not only complicated to enhance the expression of the whole scene but also easy to cause information overload, affecting the user experience. Therefore, it is particularly important to reasonably select and highlight the main information. The region of interest (ROI) refers to the area in a space or image that is of special concern due to specific purposes or needs. Visual attention theory shows that when the human visual system faces a complex scene, attention will quickly be attracted by several prominent visual objects and these objects will be processed first. By optimizing resource allocation and rendering strategies, more computing resources can be concentrated on the region of interest to avoid the interference of complex backgrounds in non-interested areas, which can significantly improve the user's immersion and interactive experience. At the same time, by dynamically annotating the region of interest and superimposing information, it can provide users with rich contextual information and real-time feedback, enhancing the practicality and user experience of the system. Based on this, the present invention dynamically obtains the user's gaze point according to the position and posture of the user's VR headset. When the user's gaze point is within the boundary range of a certain geographic scene object, the boundary range of this geographic scene object is defined as the user's area of ​​interest.

[0127] The user's visual area of ​​interest is determined by obtaining the user's gaze point, and then the area of ​​interest constraint is established. That is, by obtaining the user's gaze point information, the user's area of ​​interest is determined, and the area of ​​interest constraint is established, so that the object that the user is looking at is enhanced.

[0128] Based on the constraints of the region of interest and the associated fusion expression of data and models in the VR panoramic fusion field, the detailed attributes of the corresponding geographic scene objects are displayed to obtain a VR panoramic scene, where the detailed attributes are the attributes defined by the attribute semantic information rules. That is, based on the user's gaze point and interactive behavior, the detailed attributes or dynamic changes of the object are displayed. Allow users to actively explore and query detailed information of specific areas or objects, so as to achieve a deeper understanding and analysis. In this way, the virtual scene can not only provide a highly realistic visual experience, but also become an important tool for users to conduct in-depth exploration and scientific analysis. Allow users not only to "see clearly" and "see realistically", but also to "understand", so as to achieve a comprehensive understanding and efficient application of virtual scenes.

[0129] VR panoramic dynamic update driven by data model collaboration, that is, taking the panoramic image frame collected in real time as the next frame in the video stream, updating the VR panoramic fusion scene, visual variables and area of ​​interest constraints to obtain the updated VR panoramic scene.

[0130] In the process of constructing a virtual geographic environment, it is necessary not only to accurately map the static features of the real world, but also to capture and reflect the dynamic changes of the real world in real time. This case study uses the data acquisition equipment of a panoramic camera mounted on an unmanned aerial vehicle to obtain panoramic image data of the real world in real time. On this basis, combined with the previously defined spatial semantic knowledge (spatial semantic constraint rules), the real-time panoramic image data, the panoramic image data of the simulation model of the geographic scene object, and the spatial attribute data are deeply associated and fused, and the real-time spatial position and posture of the drone are used to dynamically update the position and posture of the geographic scene objects and semantic annotation information, so as to realize the dynamic update of the virtual reality panoramic virtual scene driven by the collaborative data model. Specifically, Figure 6 As shown:

[0131] First, in terms of the dynamic acquisition of panoramic image data, this case study used a data acquisition device equipped with a panoramic camera on a drone for shooting. During the shooting process, the drone uses its own sensors to record the time, position and posture information of each frame at the time of shooting, and transmits this information and the panoramic image to a remote cloud server in real time through a 5G wireless routing device. On the cloud server, the present invention uses the Real-Time Messaging Protocol (RTMP) to build a real-time live broadcast service for video and data, and transmits the panoramic image data and time, position and posture information to the client in real time in the form of live broadcast on the Internet, thereby realizing real-time acquisition of panoramic image data.

[0132] Secondly, in VR panoramic virtual scenes, the modeling of panoramic videos mainly relies on setting up dynamic rendering of spherical textures, so as to realize the dynamic presentation of panoramic videos in virtual scenes, that is, replacing the two-dimensional panoramic images of the spherical-projected geographical scene object simulation models and the real-world panoramic images with new ones. Specifically, this process involves mapping the acquired panoramic image frames onto a spherical model to simulate the user's perspective of looking around in the real scene, so that the user can obtain an immersive viewing experience. The present invention utilizes the uniform resource locator of network streaming media as the data source for spherical texture maps, and transmits the panoramic video acquired in real time through the network. Each frame is tiled on the spherical surface at a ratio of 2:1 on the user's VR display device, presenting a 360-degree panoramic perspective to the user. And a high-performance graphics processing unit is used for real-time rendering. Whenever a new video frame is received, the system updates the spherical texture so that the user can see the latest panoramic video content in real time, such as Figure 7 shown.

[0133] In addition, by acquiring the spatial position and attitude information of the drone in real time and using the spatial semantic knowledge of the geographic scene objects (including spatial semantic constraint rules and attribute semantic information rules), the spatiotemporal relationship between the simulation model of the geographic scene objects and the spatial attribute information data is constructed, and the dynamic update of the geographic spatial objects and semantic annotation information is realized. Specifically, during the flight, the drone continuously records the current position and attitude data through its sensor system. These data are combined with the spatial semantic knowledge of the geographic scene, so that the system can adjust the position and direction of each geographic scene object in the virtual scene in real time. In this process, the spatial semantic knowledge provides clear rules and logical frameworks to ensure that the update of the geographic spatial objects is not limited to the change of position, but also includes the relationship with the surrounding environment and other objects. Secondly, using the attribute category semantic knowledge, the spatiotemporal relationship between the simulation model of the geographic scene objects and the spatial attribute data in the evolution process is constructed. While the spatial position is updated, the semantic annotation information will also be updated according to the latest spatial data to ensure its consistency with the actual situation. Through this dynamic update mechanism, the virtual reality scene can more accurately reflect the changes in the real world and provide users with a more realistic and accurate display of geographic information.

[0134] Finally, the user's head movement and interactive behavior are combined to achieve a more personalized and precise response. By tracking real-time data of the user's head movement, the system can dynamically adjust the viewing angle and display content to give the user a more immersive experience. In addition, combined with the user's interactive behavior, such as changes in gaze point, gesture operations, etc., the system can intelligently identify the user's points of interest and needs, and then provide targeted feedback and information display. For example, when a user looks at a specific geographic scene object in a virtual environment, the system can automatically select the object and display its detailed attribute information, such as location, name, historical data, etc. If the user performs gesture operations, such as clicking or sliding, the system can adjust the scene content or perform specific operations based on these inputs, such as displaying detailed information or pausing the panoramic video.

Claims

1. A VR panoramic fusion expression method driven by model and data collaboration, characterized in that: The steps include: Step 1: Based on the spatial semantic constraints of the geographic scene, each geographic scene object is fused and modeled with the corresponding geographic scene object in the real-world panoramic image collected in real time to obtain a VR panoramic fusion scene; Step 2: Based on the attribute semantic information rules and the user's visual interest area, the VR panoramic fusion scene is enhanced by digital-analog association to obtain the VR panoramic scene, and the VR panoramic scene is updated in real time using dynamic data; The specific steps of step 2 are: Step 2.1: Based on the attribute semantic information rules and the digital-analog association expression of the user's visual area of ​​interest, the specific steps are as follows: Step 2.11, construct attribute semantic information rules describing each geographic scene object, including attribute semantic information rules describing geographic processes and attribute semantic information rules describing geographic entities. Attribute semantic information rules describing geographic entities include spatial attribute semantic information rules and non-spatial attribute semantic information rules of geographic scene objects. Spatial attribute semantic information rules include coordinate positions and geometric shapes of geographic scene objects, and non-spatial attribute semantic information rules include ID identification, subordinate categories, uses, and geographic process impacts of geographic scene objects. Step 2.12, semantically associating the spatial position of the geographic scene object in the spatial semantic constraint rule with the ID identifier in the attribute semantic information rule; Step 2.13: Based on the semantic association obtained in step 2.12, visual variables are constructed in the VR panoramic fusion scene to realize the associated fusion expression of data and models, that is, to realize the semantic enhanced associated fusion of spatial attribute information data and geographic scene object simulation models, wherein the visual variables include text descriptions, self-explanatory symbols, hot spot marks, color changes and dynamic flashing. The specific steps are as follows: First, when creating a visual variable for any geographic scene object, we should first determine that the spatial position of the visual variable should be located at the center of the geometric shape of the geographic scene object based on its position in the VR panoramic fusion scene through the spatial orientation semantic constraint rules, which means that the visual variable corresponds to the geographic scene object simulation model one by one; Secondly, after the visual variables are matched one by one with the simulation models of geographic scene objects, according to the spatial topological relationship of the described geographic scene objects, if the geographic scene object is completely blocked by other geographic scene objects at the moment of taking the panoramic image in the real world, the visual variables of this geographic scene object will be removed from the VR panoramic scene according to the spatial topological semantic constraint rules, otherwise they will not be deleted; Finally, after deleting the visual variables corresponding to the obstructed geographic scene objects, the information displayed by the retained visual variables based on semantic association matches the described geographic scene objects, that is, the associated fusion expression of data and models in the VR panoramic fusion field is realized; Step 2.14: Based on the associative fusion expression of data and model, the digital-analog associative expression of the user's visual area of ​​interest is performed. The specific steps are as follows: According to the position and posture of the user's VR headset, the user's gaze point during interactive behavior is dynamically obtained. When the user's gaze point is within the two-dimensional boundary range of a certain geographic scene object, the two-dimensional boundary range of this geographic scene object is defined as the user's area of ​​interest; The region of interest is determined by obtaining the user's gaze point and then establishing the region of interest constraints; Based on the constraints of the region of interest and the associated fusion expression of the data and the model in the VR panoramic fusion field, the detailed attributes of the corresponding geographic scene objects are displayed to obtain the VR panoramic scene, wherein the detailed attributes are the attributes defined by the attribute semantic information rules; Step 2.2: VR panoramic dynamic update based on data model collaborative driving, that is, taking the panoramic image frame collected in real time as the next frame in the video stream, updating the VR panoramic fusion scene, visual variables and area of ​​interest constraints to obtain the updated VR panoramic scene.

2. The VR panoramic fusion expression method driven by model and data collaboration according to claim 1 is characterized in that: The specific steps of step 1 are: Step 1.1, constructing spatial semantic constraint rules for the three-dimensional VR panoramic virtual scene, including spatial orientation semantic constraint rules and spatial topological semantic constraint rules. The spatial orientation semantic constraint rules refer to establishing a two-dimensional and three-dimensional spatial mapping relationship between the geographic scene object and the VR panoramic virtual scene based on spherical projection by defining the spatial position and posture, and providing a complete spatial description framework. The spatial topological semantic constraint rules refer to emitting rays to the three-dimensional space object based on the imaging characteristics of the two-dimensional panoramic image from the viewpoint of the camera, and determining the two-dimensional boundary range of the geographic scene object in the VR panoramic virtual scene, thereby obtaining the topological relationship between the various spatial objects. The geographic scene object represents the object corresponding to the real world; Step 1.2: Based on the constructed spatial semantic constraint rules, each geographic scene object is seamlessly integrated with the overall panoramic image of the real world collected in real time. The specific steps are as follows: Firstly, the transformation relationship of each geographic scene object from the local coordinate system to the geographic coordinate system is constructed based on the spatial orientation semantic constraint rules, and the new coordinates of each geographic scene object in the unified geographic coordinate system are obtained based on the comprehensive transformation matrix obtained based on the transformation relationship. The formula of the comprehensive transformation matrix and the new coordinates is: T=T transiate ×R×S P′=TP Among them, T is the comprehensive transformation matrix, T transiate represents the translation matrix of the coordinates of any point in each geographic scene object, R represents the rotation matrix of the coordinates of any point in each geographic scene object, S represents the scaling matrix of the coordinates of any point in each geographic scene object, P represents the homogeneous coordinates (x, y, z, 1) of a point in each geographic scene object, and P′ represents the new coordinates obtained by the comprehensive transformation matrix of the homogeneous coordinates (x, y, z, 1) of any point in each geographic scene object; Secondly, according to the characteristics of each geographic scene object projected onto the real-world panoramic image, a VR virtual scene coordinate system based on spherical projection with the camera as the origin is constructed. Combined with the spatial position and posture of the real-world panoramic image at the moment of photography, and based on the new coordinates of each geographic scene object in the unified geographic coordinate system, the simulation model corresponding to each geographic scene object is transformed into the spherical coordinates in the VR virtual scene coordinate system. Finally, the spherical coordinates are projected and transformed into two-dimensional panoramic image coordinates to obtain a two-dimensional panoramic image of the simulation model of each geographic scene object. The spherical coordinates include longitude and latitude. The simulation model of the geographic scene object represents the three-dimensional model of the geographic scene object in the virtual scene, including the building BIM model and the flood simulation model. The formulas for the longitude and latitude of the spherical coordinates are: λ=arctan2(Y,X) The formula for 2D image coordinates is: Wherein, λ represents the longitude in the spherical coordinate system, φ represents the latitude in the spherical coordinate system, (X, Y, Z) represents the coordinates of a point of each geographic scene object simulation model in three-dimensional space, (u, v) represents the two-dimensional image coordinates, and W and H represent the width and height of the panoramic image, respectively; Finally, based on the spatial topological semantic constraint rules, the simulation models of each geographic scene object projected into a two-dimensional panoramic image are superimposed on the collected real-world panoramic image to achieve the fusion of the simulation model of each geographic scene object with the corresponding geographic scene object in the real-world panoramic image at the corresponding moment. The fused panoramic image is then mapped to the surface of a sphere in three-dimensional space through the spherical projection method, thereby obtaining a VR panoramic fusion scene.

3. A VR panoramic fusion expression system driven by model and data collaboration, characterized in that: include: VR panoramic fusion scene construction module: Based on the spatial semantic constraints of the geographic scene, each geographic scene object and the corresponding geographic scene object in the real-world panoramic image collected in real time are fused and modeled to obtain a VR panoramic fusion scene; VR panoramic scene construction module: Based on the attribute semantic information rules and the user's visual area of ​​interest, the VR panoramic fusion scene is enhanced by digital-analog association to obtain the VR panoramic scene, and the VR panoramic scene is updated in real time using dynamic data; The specific implementation steps of the VR panoramic scene construction module are: Step 2.1: Based on the attribute semantic information rules and the digital-analog association expression of the user's visual area of ​​interest, the specific steps are as follows: Step 2.11, construct attribute semantic information rules describing each geographic scene object, including attribute semantic information rules describing geographic processes and attribute semantic information rules describing geographic entities. Attribute semantic information rules describing geographic entities include spatial attribute semantic information rules and non-spatial attribute semantic information rules of geographic scene objects. Spatial attribute semantic information rules include coordinate positions and geometric shapes of geographic scene objects, and non-spatial attribute semantic information rules include ID identification, subordinate categories, uses, and geographic process impacts of geographic scene objects. Step 2.12, semantically associating the spatial position of the geographic scene object in the spatial semantic constraint rule with the ID identifier in the attribute semantic information rule; Step 2.13: Based on the semantic association obtained in step 2.12, visual variables are constructed in the VR panoramic fusion scene to realize the associated fusion expression of data and models, that is, to realize the semantic enhanced associated fusion of spatial attribute information data and geographic scene object simulation models, wherein the visual variables include text descriptions, self-explanatory symbols, hot spot marks, color changes and dynamic flashing. The specific steps are as follows: First, when creating a visual variable for any geographic scene object, we should first determine that the spatial position of the visual variable should be located at the center of the geometric shape of the geographic scene object based on its position in the VR panoramic fusion scene through the spatial orientation semantic constraint rules, that is, we can obtain a one-to-one correspondence between the visual variable and the geographic scene object simulation model; Secondly, after the visual variables are matched one by one with the simulation models of geographic scene objects, according to the spatial topological relationship of the described geographic scene objects, if the geographic scene object is completely blocked by other geographic scene objects at the moment of taking the panoramic image in the real world, the visual variables of this geographic scene object will be removed from the VR panoramic scene according to the spatial topological semantic constraint rules, otherwise they will not be deleted; Finally, after deleting the visual variables corresponding to the obstructed geographic scene objects, the information displayed by the retained visual variables based on semantic association matches the described geographic scene objects, that is, the associated fusion expression of data and models in the VR panoramic fusion field is realized; Step 2.14: Based on the associative fusion expression of data and model, the digital-analog associative expression of the user's visual area of ​​interest is performed. The specific steps are as follows: According to the position and posture of the user's VR headset, the user's gaze point during interactive behavior is dynamically obtained. When the user's gaze point is within the two-dimensional boundary range of a certain geographic scene object, the two-dimensional boundary range of this geographic scene object is defined as the user's area of ​​interest; The region of interest is determined by obtaining the user's gaze point and then establishing the region of interest constraints; Based on the constraints of the region of interest and the associated fusion expression of the data and the model in the VR panoramic fusion field, the detailed attributes of the corresponding geographic scene objects are displayed to obtain the VR panoramic scene, wherein the detailed attributes are the attributes defined by the attribute semantic information rules; Step 2.2: VR panoramic dynamic update based on data model collaborative driving, that is, taking the panoramic image frame collected in real time as the next frame in the video stream, updating the VR panoramic fusion scene, visual variables and area of ​​interest constraints to obtain the updated VR panoramic scene.

4. A VR panoramic fusion expression system driven by model and data collaboration according to claim 3, characterized in that: The specific implementation steps of the VR panoramic fusion scene construction module are: Step 1.1, constructing spatial semantic constraint rules for the three-dimensional VR panoramic virtual scene, including spatial orientation semantic constraint rules and spatial topological semantic constraint rules. The spatial orientation semantic constraint rules refer to establishing a two-dimensional and three-dimensional spatial mapping relationship between the geographic scene object and the VR panoramic virtual scene based on spherical projection by defining the spatial position and posture, and providing a complete spatial description framework. The spatial topological semantic constraint rules refer to emitting rays to the three-dimensional space object based on the imaging characteristics of the two-dimensional panoramic image from the viewpoint of the camera, and determining the two-dimensional boundary range of the geographic scene object in the VR panoramic virtual scene, thereby obtaining the topological relationship between the various spatial objects. The geographic scene object represents the object corresponding to the real world; Step 1.2: Based on the constructed spatial semantic constraint rules, each geographic scene object is seamlessly integrated with the overall panoramic image of the real world collected in real time. The specific steps are as follows: Firstly, the transformation relationship of each geographic scene object from the local coordinate system to the geographic coordinate system is constructed based on the spatial orientation semantic constraint rules, and the new coordinates of each geographic scene object in the unified geographic coordinate system are obtained based on the comprehensive transformation matrix obtained based on the transformation relationship. The formula of the comprehensive transformation matrix and the new coordinates is: T=T transiate ×R×S P′=TP Among them, T is the comprehensive transformation matrix, T transiate represents the translation matrix of the coordinates of any point in each geographic scene object, R represents the rotation matrix of the coordinates of any point in each geographic scene object, S represents the scaling matrix of the coordinates of any point in each geographic scene object, P represents the homogeneous coordinates (x, y, z, 1) of a point in each geographic scene object, and P′ represents the new coordinates obtained by the comprehensive transformation matrix of the homogeneous coordinates (x, y, z, 1) of any point in each geographic scene object; Secondly, according to the characteristics of each geographic scene object projected onto the real-world panoramic image, a VR virtual scene coordinate system based on spherical projection with the camera as the origin is constructed. Combined with the spatial position and posture of the real-world panoramic image at the moment of photography, and based on the new coordinates of each geographic scene object in the unified geographic coordinate system, the simulation model corresponding to each geographic scene object is transformed into the spherical coordinates in the VR virtual scene coordinate system. Finally, the spherical coordinates are projected and transformed into two-dimensional panoramic image coordinates to obtain a two-dimensional panoramic image of the simulation model of each geographic scene object. The spherical coordinates include longitude and latitude. The simulation model of the geographic scene object represents the three-dimensional model of the geographic scene object in the virtual scene, including the building BIM model and the flood simulation model. The formulas for the longitude and latitude of the spherical coordinates are: λ=arctan2(Y,X) The formula for 2D image coordinates is: Wherein, λ represents the longitude in the spherical coordinate system, φ represents the latitude in the spherical coordinate system, (X, Y, Z) represents the coordinates of a point of each geographic scene object simulation model in three-dimensional space, (u, v) represents the two-dimensional image coordinates, and W and H represent the width and height of the panoramic image, respectively; Finally, based on the spatial topological semantic constraint rules, the simulation models of each geographic scene object projected into a two-dimensional panoramic image are superimposed on the collected real-world panoramic image to achieve the fusion of the simulation model of each geographic scene object with the corresponding geographic scene object in the real-world panoramic image at the corresponding moment. The fused panoramic image is then mapped to the surface of a sphere in three-dimensional space through the spherical projection method, thereby obtaining a VR panoramic fusion scene.

Images