Real-geospatial-based digital twin video holographic display method and system

Abstract

The application discloses a real geographical space-based digital twin video holographic display method and system, reads a real-time image frame of a current moment from real-time video data, divides image texture of the real-time image frame into a texture mapping overlapping area and a non-overlapping area according to visibility and occlusion between the real-time image frame and a three-dimensional real scene model, for the texture mapping overlapping area, combines multiple historical image frames as a texture source, adopts a multi-source texture fusion algorithm based on real geographical space consistency, reconstructs a texture map of the texture mapping overlapping area, dynamically maps the texture map to a surface of the three-dimensional real scene model to obtain video-model data after texture mapping, and transmits the video-model data into a holographic display system to display holographic images according to position differences of both eyes of an observer in space. The application accurately fuses the three-dimensional real scene model with real-time video data, and realizes real, real-time and spatially synchronous three-dimensional holographic presentation of a physical scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of holographic display, and particularly relates to a digital twin video holographic display method and system based on real geographic space. BACKGROUND

[0002] With the development of digital twin technology, digital twins based on three-dimensional modeling have been widely applied in urban management, industrial monitoring, simulation training and other scenarios. Traditional digital twin systems usually use static texture mapping to construct virtual scenes, which cannot reflect the dynamic changes of the physical world in real time, have single display effects, and lack real-time and interactivity. Existing video display means mainly use two-dimensional display screens to play pictures, which cannot provide users with stereoscopic immersive experience. At the same time, the textures in traditional three-dimensional modeling are usually from photographed pictures, which lack association with real-time changing geographic information, and cannot realize accurate visualization of real-time physical space. SUMMARY

[0003] The purpose of the present application is to provide a digital twin video holographic display method and system based on real geographic space, which can accurately spatially fuse three-dimensional real scene models with real-time video data, and realize real, real-time, and spatially synchronized three-dimensional holographic presentation of physical scenes.

[0004] To achieve the above purpose, the present application proposes the following technical solutions:

[0005] On the one hand, the present application provides a digital twin video holographic display method based on real geographic space, which specifically includes the following steps:

[0006] S1, obtaining a three-dimensional real scene model of a target area, and obtaining real-time video data of the target area from a video acquisition device;

[0007] S2, reading a real-time image frame at the current time from the real-time video data, and dividing the image texture of the real-time image frame into a texture mapping overlap region and a non-overlap region according to the visibility and occlusion relationship between the real-time image frame and the three-dimensional real scene model;

[0008] For the texture mapping overlap region, a multi-source texture fusion algorithm based on real geographic space consistency is used to reconstruct the texture map of the texture mapping overlap region by combining multiple historical image frames as texture sources, and the texture map is dynamically mapped to the surface of the three-dimensional real scene model;

[0009] S3, after mapping the texture mapping overlap region and the non-overlap region to the three-dimensional real scene model, obtaining video-model data after texture mapping;

[0010] S4, inputting the video-model data into a holographic display system, and displaying holographic images according to the position difference of the observer's two eyes in space.

[0011] In some specific embodiments, the specific process of judging the visibility and occlusion relationship between the real-time image frame and the three-dimensional real scene model is as follows:

[0012] S01, project each pixel point in the real-time image frame to the frustum space under the camera coordinate system through the camera intrinsic matrix of the video acquisition device, and then combine the depth information to project to the three-dimensional space to obtain a three-dimensional point;

[0013] S02, convert the three-dimensional point to a spatial point under the three-dimensional real scene model coordinate system according to the six-degree-of-freedom spatial pose information of the video acquisition device;

[0014] S03, find the corresponding triangular patch of the spatial point coordinate on the surface of the three-dimensional real scene model and store it in the triangular patch set, and judge the visibility and occlusion relationship between the pixel point of the real-time image frame and the triangular patch by combining the re-projection method of the real geographical aperture projection geometry.

[0015] In some specific embodiments, the specific process of the re-projection method is as follows:

[0016] For each pixel point coordinate, construct a unit view vector, and determine whether the unit view vector intersects with any triangular patch in the triangular patch set within the set re-projection depth interval.

[0017] If yes, it means that there is no occlusion, and the pixel point is visible. The texture region corresponding to the pixel point is a non-overlapping region.

[0018] If not, it means that there is an intermediate occlusion, and the texture region corresponding to the pixel point is a texture overlapping region.

[0019] In some specific embodiments, the specific process of the multi-source texture fusion algorithm is as follows:

[0020] S21, obtain the texture information in the N-1 historical image frames before the current time and the current texture state of the three-dimensional real scene model, take the N-1 texture information and the current texture state as texture sources, and obtain N texture sources;

[0021] S22, calculate the texture difference between the image texture of the real-time image frame and the N texture sources respectively, obtain the fusion weight in each texture source in the texture mapping overlapping region, and calculate the fusion texture value of the texture mapping overlapping region to obtain the real-time updated texture map;

[0022] S23, dynamically map the real-time updated texture map to the surface of the three-dimensional real scene model.

[0023] In some specific embodiments, the fusion weight in step S22 includes multiple fusion factors, and the calculation method of the fusion weight is as follows:

[0024] Obtaining the angle θ between the surface normal vector of the pixel point (x, y) in the texture mapping overlapping area and the i-th texture source view i , calculating the direction angle;

[0025] Obtaining the brightness value of the pixel point (x, y) under the i-th texture source L i (x, y); and based on the brightness values of the N texture sources, calculating the illumination average value of the pixel point under all texture sources and the illumination difference;

[0026] Calculating the straight line distance from the video capture device to the three-dimensional point d i (x, y); when the unit view vector of the pixel point intersects with multiple triangular facets, respectively calculating the intersection point distance of the unit view vector and each triangular facet, and selecting the shortest intersection point distance as , calculating the occlusion depth difference based on the straight line distance and the shortest intersection point distance;

[0027] According to the proportion of the unit view vector passing through the unoccluded section of the path, calculating the visibility ratio S i (x, y);

[0028] Fusing the direction angle, the illumination difference, the occlusion depth difference, and the visibility ratio after assigning different weight parameters to obtain the fusion weight W i (x, y):

[0029]

[0030] wherein, represents the weight parameter.

[0031] In some specific embodiments, the fused texture value is the weighted average of the fusion weight W i (x, y) of each texture source and the original texture value T i (x, y) of each texture source, and the calculation method is:

[0032] .

[0033] In some specific embodiments, step S22 further comprises an iterative optimization step. After obtaining the fused texture value, residual analysis is performed on the fused texture value and the original texture value of each texture source:

[0034] ;

[0035] It is judged whether the residual size R(x, y) meets the set residual threshold value, and if not, the weight parameters of each factor of the feedback adjustment fusion weight are adjusted, the fused texture value is recalculated, and the iteration optimization is performed until the residual size meets the set residual threshold value.

[0036] In some specific embodiments, for the non-overlapping region, the pixel points in the real-time image frame are mapped into the patch texture of the corresponding texture region, and the single-frame texture mapping and replacement operation is completed.

[0037] In some specific embodiments, the specific process of constructing the holographic image in step S4 is as follows:

[0038] The spatial position of the observer and the observation point of the observer to the three-dimensional real scene model are acquired in real time, and the gaze direction vector of the observer is calculated;

[0039] The gaze direction vector and the video source orientation of the video acquisition device are mapped by coordinate difference to obtain the video deflection angle difference value that needs to be adjusted;

[0040] The angle deviation of the video acquisition device is adjusted according to the video deflection angle difference value to make the observation angle and the three-dimensional real scene model in the same direction;

[0041] The position difference of the observer's two eyes in space is obtained, the projection directions corresponding to the left and right eye views of the observer are obtained respectively, the left and right views conforming to the human eye stereoscopic vision difference are generated based on the video-model data, and the holographic image is displayed for the observer through the holographic display system.

[0042] In a second aspect, the application provides a digital twin video holographic display system based on real geographic space, comprising:

[0043] A data acquisition module is configured to acquire a three-dimensional real scene model of a target region and acquire real-time video data of the target region from a video acquisition device;

[0044] An image mapping module is configured to read a real-time image frame at the current time from the real-time video data, divide the image texture of the real-time image frame into texture mapping overlapping regions and non-overlapping regions according to the visibility and occlusion relationship between the real-time image frame and the three-dimensional real scene model, and for the texture mapping overlapping regions, reconstruct the texture map of the texture mapping overlapping regions by using a multi-source texture fusion algorithm based on real geographic space consistency, and dynamically map the texture map to the surface of the three-dimensional real scene model.

[0045] An image fusion module is configured to obtain video-model data after mapping the texture mapping overlapping regions and the non-overlapping regions into the three-dimensional real scene model.

[0046] A holographic display system is configured to display a holographic image according to the video-model data and the position difference of the observer's two eyes in space.

[0047] The present application has the beneficial effects of:

[0048] The present application provides a real geographic space digital twin video holographic display technology, which realizes accurate spatial fusion of real scene models and real-time video data, combines the textures of historical multi-frame images, uses the textures of historical multi-frame images and three-dimensional real scene models as texture sources, realizes a multi-source texture fusion algorithm, makes texture mapping more accurate, has better three-dimensional effect, realizes real-time, spatially synchronous three-dimensional holographic presentation of physical scenes, and provides immersive and interactive experience for observers. BRIEF DESCRIPTION OF DRAWINGS

[0049] Figure 1 The present application provides a real geographic space digital twin video holographic display method flowchart for the embodiments of the present application;

[0050] Figure 2 The present application provides a multi-source texture fusion algorithm flowchart for the embodiments of the present application; DETAILED DESCRIPTION

[0051] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. The description of the at least one exemplary embodiment is actually only illustrative, but not as any limitation on the present application and its application or use. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of the present application.

[0052] Unless otherwise specified, the relative arrangement, numerical expressions and values of the components and steps set forth in these embodiments do not limit the scope of the present application.

[0053] At the same time, it should be understood that, for the convenience of description, the sizes of the various parts shown in the drawings are not drawn in accordance with the actual proportional relationship.

[0054] In addition, for the sake of clarity and brevity, the description of well-known structures, functions and configurations can be omitted. Those of ordinary skill in the art will recognize that various changes and modifications can be made to the examples described herein without departing from the spirit and scope of the present disclosure.

[0055] Techniques, methods and equipment known to those of ordinary skill in the relevant art can not be discussed in detail, but should be considered as part of the authorized specification when appropriate.

[0056] In all examples shown and discussed herein, any specific values should be interpreted as merely exemplary, and not as a limitation. Thus, other examples of the exemplary embodiments can have different values.

[0057] Embodiment 1

[0058] As shown in the figure, the embodiment provides a real geo-spatial based digital twin video holographic display method, specifically comprising the following steps: Figure 1

[0059] S1, acquiring a three-dimensional real scene model of a target area, and acquiring real-time video data of the target area from a video acquisition device;

[0060] Step S1 is mainly the collection of display information, including a three-dimensional real scene model and real-time video data. Among them, the three-dimensional real scene model can be obtained by a tilt photogrammetry system or a mobile measurement device. The three-dimensional real scene model established in the present application adopts a global spatial rectangular coordinate system (such as WGS-84 projection coordinate system), and a mapping relationship between the model coordinate system and the geographic coordinate system is established to realize the correspondence between the geographic spatial information and the three-dimensional real scene model;

[0061] And the real-time video data is obtained by deploying a video acquisition device in the target area, using high-precision positioning sensor technology to obtain the six-degree-of-freedom spatial pose information of the video acquisition device in real time, including spatial position and attitude angle , wherein are the rotation matrices around the x, y, z axes, and the position of the camera in the three-dimensional model coordinate system can be represented as a rigid transformation matrix .

[0062] S2, reading a real-time image frame at the current time from the real-time video data, and dividing the image texture of the real-time image frame into a texture mapping overlapping area and a non-overlapping area according to the visibility and occlusion relationship between the real-time image frame and the three-dimensional real scene model;

[0063] Among them, the specific process of judging the visibility and occlusion relationship between the real-time image frame and the three-dimensional real scene model is as follows:

[0064] S01, projecting each pixel point in the real-time image frame to the view cone space under the camera coordinate system through the camera intrinsic matrix of the video acquisition device, and then combining the depth information to project it back to the three-dimensional space to obtain a three-dimensional point;

[0065] Specifically, the real-time image frame obtained by the video acquisition device is ​Each pixel (x, y) is projected into the view cone space under the camera coordinate system by the camera intrinsic matrix K, and then combined with the depth information d(x, y) to be back-projected into the three-dimensional space. The projection process is , wherein K is a camera intrinsic matrix:

[0066]

[0067] S02, according to the six-degree-of-freedom spatial pose information of the video acquisition device, the three-dimensional point is converted into a spatial point under the three-dimensional real scene model coordinate system;

[0068] The six-degree-of-freedom spatial pose information as an external parameter includes spatial position and attitude angle ; the obtained three-dimensional point P cam The point is converted into a point under the three-dimensional model coordinate system through the external parameter, and the calculation is as follows:

[0069] ,

[0070] The spatial point P world The coordinates are used to find the corresponding triangular patches of the three-dimensional model surface. In order to ensure that the image texture of the image frame can be truly mapped to the surface of the three-dimensional model, a re-projection method combined with real geographic aperture projection geometry is used to judge the visibility and occlusion relationship;

[0071] S03, according to the spatial point coordinates, find the corresponding triangular patches of the three-dimensional real scene model surface and store them in the triangular patch set, and use the re-projection method combined with real geographic aperture projection geometry to judge the visibility and occlusion relationship between the pixel points of the real-time image frame and the triangular patches.

[0072] The specific process of the re-projection method is as follows:

[0073] For each pixel point coordinate, construct a unit view vector, and in the set depth interval, whether the unit view vector intersects with any one of the triangular patches in the triangular patch set;

[0074] If it exists, it means that there is no occlusion, and the pixel point is visible. The texture area corresponding to the pixel point is a non-overlapping area.

[0075] If it does not exist, it means that there is an intermediate occlusion, and the texture area corresponding to the pixel point is a texture overlapping area.

[0076] The re-projection method is expressed by a mathematical expression: given the pixel coordinates (x, y), construct a unit view vector:

[0077]

[0078] wherein K is a camera intrinsic matrix, R is an extrinsic rotation matrix corresponding to the pose angle; let the camera center coordinate be C, and the line-of-sight direction of the pixel be n, then the back-projection point of the pixel is:

[0079] ;

[0080] wherein λ is a back-projection depth, which depends on the intersection of the pixel and the geographic model, and let the terrain model contain a set of triangular patches , if in the interval t ∈ (0, λ), it is judged whether there is a triangular patch intersecting the line-of-sight ray for any j, the judgment function is expressed as:

[0081]

[0082] When , it indicates no occlusion, and the pixel point is visible, otherwise there is an intermediate occlusion, and the line-of-sight is blocked, and the judgment result is passed to the next step for texture mapping operation;

[0083] 1. For non-overlapping regions, the pixel points in the real-time image frame are mapped to the patch texture of the corresponding texture region, and the single-frame texture mapping and replacement operation is completed.

[0084] 2. For the texture mapping overlapping region, a multi-source texture fusion algorithm based on real geographic space consistency is used, a plurality of historical image frames are combined as texture sources, the texture of the texture mapping overlapping region is reconstructed, and the texture is dynamically mapped to the surface of the three-dimensional real scene model;

[0085] The multi-source texture fusion algorithm is based on the current texture state of the three-dimensional model and the texture information in the set N-1 historical frames, and uses the observation direction angle, illumination difference, occlusion depth difference, visibility ratio and texture confidence of the spatial pixel in the above N texture information sources to calculate the fusion weight, as shown in Figure 2 , the specific process is as follows:

[0086] S21, obtain the texture information in N-1 historical image frames before the current time and the current texture state of the three-dimensional real scene model, take the N-1 historical texture information and the current texture state as texture sources, and obtain N texture sources;

[0087] S22, calculate the texture difference between the image texture of the real-time image frame and the N texture sources respectively, obtain the fusion weight in each texture source in the texture mapping overlapping region, and calculate the fusion texture value of the texture mapping overlapping region, and obtain the real-time updated texture map;

[0088] The fusion weight in step S22 includes a plurality of fusion factors, and the calculation method of the fusion weight is:

[0089] Obtain the angle θ between the surface normal vector of the pixel point (x, y) in the texture mapping overlap region and the i-th texture source view angle i , calculate the direction angle;

[0090] Obtain the brightness value of the pixel point (x, y) under the i-th texture source L i (x, y); and based on the brightness values of the N texture sources, calculate the average illumination of the pixel point under all texture sources and the illumination difference;

[0091] Calculate the straight line distance from the video capture device to the three-dimensional point d i (x, y); when the unit view vector of the pixel point intersects with multiple triangular facets, respectively calculate the intersection point distance of the unit view vector and each triangular facet, and select the shortest intersection point distance as , calculate the occlusion depth difference based on the straight line distance and the shortest intersection point distance;

[0092] Calculate the visibility ratio according to the proportion of the unit view vector passing through the unoccluded segment in the path S i (x, y);

[0093] Fuse the direction angle, illumination difference, occlusion depth difference, and visibility ratio after assigning different weight parameters to obtain the fusion weight W i (x, y):

[0094]

[0095] wherein, represents the weight parameter.

[0096] Based on the fusion weight, calculate the fused texture value, and the fused texture value is the weighted average of the fusion weight of each texture source W i (x, y) and the original texture value of each texture source T i (x, y), and the calculation method is:

[0097] .

[0098] To avoid texture artifacts caused by high light reflection areas, set a brightness threshold L max , set the average value L i (x, y) in the texture source to the average value After the texture fusion is completed, an iterative optimization step is further included, after the fused texture value is obtained, residual analysis is performed on the fused texture value and the original texture value of each texture source:

[0099]

[0100] It is judged whether the residual size R(x, y) meets the set residual threshold, if not, the weight parameters of each factor of the feedback adjustment fusion weight are adjusted , the fused texture value is recalculated until the residual size meets the set residual threshold, and iterative optimization is performed.

[0101] S23, dynamically map the real-time updated texture map to the surface of the three-dimensional real scene model. The multi-source texture fusion algorithm proposed in the application fully utilizes the real geographical space direction, distance and occlusion relationship, improves the continuity and realism of the fused texture, and finally generates a real-time updated texture map, which is dynamically mapped to the surface of the three-dimensional real scene model;

[0102] S3, after the texture mapping overlapping area and non-overlapping area are all mapped to the three-dimensional real scene model, the video-model data after texture mapping is obtained;

[0103] S4, the video-model data is transmitted into the holographic display system, and the holographic image is displayed according to the position difference of the observer's two eyes in space. The specific process is:

[0104] Real-time acquisition of the spatial position of the observer and the observation point of the observer to the three-dimensional real scene model P tar The observation point refers to the observation point position of the human eye to the model; for example, a certain area in the model is gazed at, and the gaze direction vector of the observer is calculated :

[0105]

[0106] The gaze direction vector and the video source direction of the video acquisition device are mapped by coordinate difference to obtain the video deflection angle difference value that needs to be adjusted;

[0107] Adjust the angle offset angle of the video acquisition device according to the video deflection angle difference value After that, the observer's visual angle and the three-dimensional real scene model are in the same direction, which ensures the consistency of the observer's visual angle and the real geographical information;

[0108] The position difference of the observer's two eyes in space is obtained, the projection directions corresponding to the visual angles of the observer's left and right eyes are obtained respectively, left and right views conforming to the stereoscopic vision difference of human eyes are generated based on the video-model data, and holographic images with real depth are reconstructed through a holographic display system to realize dynamic and stereoscopic three-dimensional digital twin holographic reconstruction effect.

[0109] Embodiment 2

[0110] The embodiment provides a real geographical space-based digital twin video holographic display system, which comprises:

[0111] The data acquisition module is configured to obtain a three-dimensional real scene model of a target area and real-time video data of the target area from a video acquisition device.

[0112] The image mapping module is configured to read a real-time image frame at a current time from the real-time video data, divide image textures of the real-time image frame into texture mapping overlapping regions and non-overlapping regions according to the visibility and occlusion relationship between the real-time image frame and the three-dimensional real scene model, and reconstruct texture maps of the texture mapping overlapping regions by using a multi-source texture fusion algorithm based on real geographical space consistency, and dynamically map the texture maps to the surface of the three-dimensional real scene model.

[0113] The image fusion module is configured to obtain video-model data after the texture mapping overlapping regions and the non-overlapping regions are all mapped to the three-dimensional real scene model.

[0114] The holographic display system is configured to display holographic images according to the video-model data and the position difference of the observer's two eyes in space.

[0115] The above description is only a preferred embodiment of the present application, and does not limit the present application in any form. According to the technical essence of the present application, any simple modification, equivalent replacement and improvement of the above embodiment within the spirit and principles of the present application are still within the protection scope of the technical solution of the present application.

Claims

1. A real geo-spatial based digital twin video holographic display method, characterized in that, Specifically comprising the following steps: S1, acquire a three-dimensional real scene model of a target area, and acquire real-time video data of the target area from a video acquisition device; S2, read a real-time image frame at a current time from the real-time video data, and divide image texture of the real-time image frame into a texture mapping overlap region and a non-overlap region according to visibility and occlusion relations between the real-time image frame and the three-dimensional real scene model; For the texture mapping overlap region, a multi-source texture fusion algorithm based on real geographical space consistency is adopted to reconstruct a texture map of the texture mapping overlap region by combining multiple historical image frames as texture sources, and the texture map is dynamically mapped to a surface of the three-dimensional real scene model; The specific process of the multi-source texture fusion algorithm is as follows: S21, acquire texture information in N-1 historical image frames before the current time and a current texture state of the three-dimensional real scene model, take the N-1 historical texture information and the current texture state as texture sources, and obtain N texture sources; S22, calculate texture differences between image texture of the real-time image frame and the N texture sources respectively, obtain a fusion weight in each texture source in the texture mapping overlap region, and calculate a fusion texture value of the texture mapping overlap region to obtain a real-time updated texture map; The fusion weight in step S22 includes multiple fusion factors, and the calculation method of the fusion weight is as follows: Obtaining the included angle θ between the surface normal vector of the pixel point (x, y) in the texture mapping overlapping area and the i-th texture source view angle i , calculating the direction angle; Obtaining the brightness value of the pixel point (x, y) under the i-th texture source L i (x, y); and based on the brightness values of the N texture sources, calculating the illumination average value of the pixel point under all the texture sources and the illumination difference; Computing straight-line distance from video capture device to three-dimensional point d i (x, y); when the unit sight vector of the pixel point intersects with multiple triangular facets, the intersection point distance of the unit sight vector and each triangular facet is calculated respectively, and the shortest intersection point distance is selected and recorded as d i ( x,y ), the occlusion depth difference is calculated based on the straight-line distance and the shortest intersection point distance; According to the proportion of the unit sight line vector passing through the unobstructed section in the path, the visual ratio is calculated S i (x, y); The direction angle, illumination difference, occlusion depth difference and visual access ratio are fused after different weight parameters are assigned to obtain a fusion weight W i (x, y): wherein α , β , τ , γ , η denotes a weight parameter; S23, dynamically map the real-time updated texture map to the surface of the three-dimensional real scene model; S3, after mapping the texture mapping overlap region and the non-overlap region to the three-dimensional real scene model, obtain video-model data after texture mapping; S4, input the video-model data into a holographic display system, and display a holographic image according to a position difference of two eyes of an observer in space.

2. The real geo-spatial based digital twin video holographic display method according to claim 1, characterized in that, The specific process of judging the visibility and occlusion relations between the real-time image frame and the three-dimensional real scene model is as follows: S01, project each pixel point in the real-time image frame to a view frustum space under a camera coordinate system through a camera intrinsic matrix of the video acquisition device, and then combine depth information to project reversely to a three-dimensional space to obtain a three-dimensional point; S02, convert the three-dimensional point to a space point under a three-dimensional real scene model coordinate system according to six-degree-of-freedom space pose information of the video acquisition device; S03, find a corresponding triangular patch of the three-dimensional real scene model surface according to the space point coordinates and store the triangular patch in a triangular patch set, and judge the visibility and occlusion relations between the pixel point of the real-time image frame and the triangular patch by combining a re-projection method of real geographical aperture projection geometry.

3. The real geo-spatial based digital twin video holographic display method of claim 2, wherein, The specific process of the re-projection method is as follows: For each pixel point coordinate, construct a unit view vector, and in a set reverse projection depth interval, whether the unit view vector intersects with any one of the triangular patch set; If yes, it indicates that there is no occlusion, the pixel point is visible, and a texture region corresponding to the pixel point is a non-overlap region; If no, it indicates that there is an intermediate occlusion, and a texture region corresponding to the pixel point is a texture overlap region.

4. The real geo-spatial based digital twin video holographic display method of claim 1, wherein, blended texture value blending weights for each texture source W i (x, y) and the original texture value of each texture source T i a weighted average of (x, y) computed as 。 5. The real geo-spatial based digital twin video holographic display method according to claim 4, characterized in that, Step S22 further includes an iterative optimization step, after obtaining the fusion texture value, residual error analysis is performed on the fusion texture value and original texture values of the texture sources: It is judged whether the residual size R(x, y) meets the set residual threshold value, and if not, the weight parameters of each factor of the feedback adjustment fusion weight are adjusted, the fused texture value is recalculated until the residual size meets the set residual threshold value, and iterative optimization is performed.

6. The real geo-spatial based digital twin video holographic display method of claim 3, wherein, For non-overlapping areas, the pixel points in the real-time image frame are mapped into the face texture of the corresponding texture area to complete single-frame texture mapping and replacement operations.

7. The real geo-spatial based digital twin video holographic display method according to claim 1, wherein, The specific process of constructing the holographic image in step S4 is as follows: The spatial position of the observer and the observation point of the observer to the three-dimensional real scene model are acquired in real time, and the gaze direction vector of the observer is calculated. The gaze direction vector and the video source orientation of the video acquisition device are mapped by coordinate difference to obtain the video deflection angle difference value that needs to be adjusted. The angle deviation of the video acquisition device is adjusted according to the video deflection angle difference value to make the observer's view angle and the three-dimensional real scene model in the same direction. The position difference of the observer's two eyes in space is obtained, the projection directions corresponding to the left and right eye view angles of the observer are obtained respectively, and left and right views conforming to the human eye stereoscopic vision difference are generated based on the video-model data; and the holographic image is displayed for the observer through the holographic display system.

8. A real geo-spatial based digital twin video holographic display system, applying a real geo-spatial based digital twin video holographic display method as claimed in claim 1, characterized in that, It comprises: A data acquisition module is configured to acquire a three-dimensional real scene model of a target area and real-time video data of the target area from a video acquisition device; An image mapping module is configured to read a real-time image frame at a current time from the real-time video data, divide image textures of the real-time image frame into texture mapping overlapping areas and non-overlapping areas according to the visibility and occlusion relationship between the real-time image frame and the three-dimensional real scene model, and reconstruct texture maps of the texture mapping overlapping areas by using a multi-source texture fusion algorithm based on real geographic space consistency for the texture mapping overlapping areas, and dynamically map the texture maps to the surface of the three-dimensional real scene model; An image fusion module is configured to obtain video-model data after the texture mapping overlapping areas and the non-overlapping areas are all mapped into the three-dimensional real scene model; A holographic display system is configured to display a holographic image according to the video-model data and the position difference of the observer's two eyes in space.

Images