CAVE rescue scene construction method and system based on digital twinning and holographic technology

The construction of CAVE rescue scenarios through digital twins and holographic technology solves the problem that traditional disaster rescue training cannot provide real scene experience, and improves the response capabilities and efficiency of disaster rescue personnel.

CN118918293APending Publication Date: 2024-11-08THE FIRST AFFILIATED HOSPITAL OF ARMY MEDICAL UNIV
View PDF 0 Cites 4 Cited by

Patent Information

Application Number
CN202410965527.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-18
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

Traditional disaster rescue training and drills cannot provide real disaster scene experience, resulting in a lack of actual training for rescue workers when facing disasters, which increases the difficulty of rescue work.

Method used

CAVE rescue scenarios are built using digital twins and holographic technology, combining digital twins, 3D real-time rendering technology, holographic technology and CAVE virtual reality technology, and create a highly simulated disaster rescue training environment through real-time data integration, three-dimensional visualization and multi-angle interaction.

Benefits of technology

It improves the authenticity and effect of training, enhances the on-site perception and operational flexibility of rescue personnel, and improves the response ability and efficiency of disaster rescue.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918293A_ABST
    Figure CN118918293A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of disaster rescue, and discloses a CAVE rescue scene construction method based on a digital twinning and holographic technology, which comprises the digital twinning technology, a 3D real-time rendering technology, a holographic technology, a Cave virtual reality technology and a real-time cooperation module. A digital simulation technology is established in an informatization platform with the assistance of artificial intelligence, machine learning and software analysis, and the simulation can be updated along with the change of a physical entity according to feedback so as to ensure that a virtual model is matched with a corresponding object in the real world. According to the CAVE rescue scene construction method and system based on the digital twinning and holographic technology, digital twinning, real-time rendering and holographic Cave technologies are integrated, so that the efficiency and safety of disaster rescue are improved, the immersion and interactivity are enhanced, and disaster rescue training and drilling are more scientific, efficient and real.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of disaster relief technology, and specifically to a CAVE rescue scene construction method and system using digital twin and holographic technology. Background Art

[0002] my country is a country where natural disasters occur frequently. Every time a disaster occurs, local residents will suffer casualties of varying degrees. Whenever a disaster occurs, people will engage in arduous rescue and disaster relief work. In the process of disaster relief, a large amount of manpower and material resources are required. At the same time, before disaster relief, rescue personnel are often required to undergo a large amount of rescue simulation training.

[0003] Traditional disaster rescue training and drills are often limited and cannot provide a realistic disaster scene experience, resulting in a lack of practical training for rescue personnel in the challenges they may encounter when responding to disasters. As a result, rescue work often becomes more difficult as the disaster becomes more sudden and random. To overcome this problem, the present invention aims to use digital twin and holographic technology to construct a highly realistic and interactive virtual-reality fusion scene to improve the response capability and efficiency of disaster rescue personnel. Therefore, a CAVE rescue scene construction method and system using digital twin and holographic technology are proposed to solve the above problems. Summary of the invention

[0004] 1. Technical issues to be solved

[0005] In view of the shortcomings of the existing technology, the present invention provides a CAVE rescue scene construction method and system based on digital twin and holographic technology, which has the advantages of improving the realism of training and thus improving the training effect, and solves the problem of poor effect caused by insufficient realism of training.

[0006] (II) Technical solution

[0007] In order to achieve the above-mentioned purpose of improving the realism of training and thus improving the training effect, the present invention provides the following technical solutions: a CAVE rescue scene construction method using digital twin and holographic technology, comprising: digital twin technology, 3D real-time rendering technology, holographic technology, Cave virtual reality technology, and a real-time collaboration module;

[0008] S1. Digital twin technology: This is a technology that integrates physical feedback data and uses artificial intelligence, machine learning and software analysis to build a digital simulation within an information platform. This simulation will be updated based on feedback and as the physical entity changes to ensure that the virtual model matches its counterpart in the real world;

[0009] S2. 3D real-time rendering technology: It is a technology that generates and updates three-dimensional images in real time on display devices. 3D real-time rendering technology demonstrates high-quality rendering, integration of advanced interactive technologies, and application of emerging rendering technologies.

[0010] S3. Holographic technology: Holographic technology uses the principles of interference and diffraction to record and reproduce the real three-dimensional image of an object. This technology can capture all the information of the light emitted by the object, thereby reconstructing a high-quality three-dimensional image in a virtual environment;

[0011] S4. CAVE virtual reality technology: CAVE systems usually use a four-sided projection structure, including the front, left side, right side and ground, to provide an immersive visual experience. In disaster relief scenarios, this technology can simulate different perspectives, such as walking, driving or drone patrol, to enhance the on-site perception of rescue personnel;

[0012] S5. Combining these technologies, a highly simulated disaster rescue training and simulation environment (i.e., real-time collaboration module) can be constructed, including:

[0013] 1) Real-time data integration: Through digital twin technology, on-site sensor data is input into the virtual model in real time to ensure dynamic updating and accuracy of the model;

[0014] 2) 3D visualization: Using holographic technology to generate high-resolution 3D images, rescuers can visually see the situation in the disaster area in a virtual environment;

[0015] 3) Multi-angle interaction: CAVE technology provides a multi-angle visual experience, allowing rescue workers to observe the disaster area from different positions and heights, increasing operational flexibility and efficiency.

[0016] Preferably, the segmentation and fusion generation of the CAVE space screen by the CAVE system in step S4 is a complex process involving multiple steps and calculations. The following is a simplified and structured overview, as well as the relevant calculation formulas and steps:

[0017] 1. Image Segmentation

[0018] Purpose: To divide the space in the 3D virtual scene into multiple spatial picture modules for subsequent projection mapping;

[0019] step:

[0020] Determine the space to be segmented and the required segmentation accuracy;

[0021] Use spatial segmentation algorithms to segment spatial objects into multiple modules through various physical or logical means;

[0022] For each screen module, record the coordinates and attributes it contains;

[0023] 2. Determine the projection direction and coordinate system

[0024] Purpose: To determine the projection direction and coordinate system for each screen module block in order to calculate their mapping positions in the real space;

[0025] step:

[0026] Select a direction perpendicular to the virtual scene object as the projection direction;

[0027] Establish a unified coordinate system to describe the spatial relationship between the virtual scene and the wall;

[0028] 3. Mapping calculation

[0029] Purpose: Calculate the mapping position of each screen module in the actual space according to the projection geometry relationship;

[0030] Calculation formula:

[0031] Use the perspective transformation matrix to calculate the coordinates of each screen module in the real space. The perspective transformation matrix is ​​a 3x3 matrix used to describe the projection mapping relationship from the virtual scene to the real space.

[0032] For each screen module, multiply its coordinates with the perspective transformation matrix to obtain the mapping coordinates on the wall;

[0033] 4. Projection Fusion

[0034] Purpose: To merge the images projected by multiple projectors into a coherent, seamless 2D picture.

[0035] step:

[0036] Ensure that each projector is accurately calibrated and configured to ensure the consistency and authenticity of the projected image;

[0037] Use projection blending software or hardware solutions to blend images from multiple projectors; this typically involves adjusting parameters such as projector brightness, contrast, and color balance to ensure smooth transitions and consistency between images.

[0038] Preferably, the data modification and generation of the space venue and the projection screen in step S4 is a process involving multiple steps and techniques. The following is a clear, point-by-point representation and summary:

[0039] 1. CAVE space data modification

[0040] Data collection:

[0041] Conduct on-site measurements of the space site to obtain basic data such as the location coordinates and dimensions of the space site;

[0042] Data import and processing:

[0043] Import the collected spatial site data into the software, edit and organize the data to ensure the accuracy and completeness of the data;

[0044] Data modification:

[0045] Modify the spatial site data according to project requirements, such as adding new spatial elements, modifying the position or attributes of existing elements, etc.;

[0046] Data export and storage:

[0047] Export the modified spatial site data to a format suitable for projection, such as an image file (JPEG, PNG, etc.) or vector data (SVG, DWG, etc.);

[0048] 2. Projection screen data generation

[0049] Projection settings:

[0050] Adjust the projection position and angle of the projector to ensure that the projection image can accurately cover the target site;

[0051] According to the configuration of the CAVE system and projection requirements, set the resolution of the projector and the physical size of the image, length, width, and height parameters;

[0052] Data conversion:

[0053] Convert spatial site data (image or vector data) into a format suitable for projection;

[0054] If necessary, use software to further edit and modify the projection image, such as adding text, annotations, icons, etc.;

[0055] Real-time data integration:

[0056] If the projection screen needs to display real-time data, the real-time data can be integrated into the projection screen through the data interface or API;

[0057] Use programming scripts or software tools to capture, process and display real-time data;

[0058] Interactive modification:

[0059] If the audience needs to participate in the interaction, you can set interactive elements in the projection screen (such as gesture recognition, spatial positioning, etc.);

[0060] Use interactive programming technology to achieve audience interaction and real-time feedback;

[0061] 3. Real-time rendering and presentation

[0062] A high-performance graphics processor (GPU) and rendering engine are used to render the virtual scene in real time. The rendered image is projected onto the wall through a projector. Factors such as the curvature of the wall and the focal length of the projector are considered to ensure that the shape and size of the projected image match the objects in the virtual scene. The calculated mapping position is rendered onto the wall in real time and presented.

[0063] Preferably, the real-time change of the viewing point of the CAVE screen in step S4 mainly depends on the spatial positioning technology and system design. The following are the specific implementation steps and key elements:

[0064] The CAVE immersive space needs to be equipped with a spatial positioning and motion capture system. The system uses infrared or laser radar technologies to accurately track the user's movement trajectory and map the user's movements to the virtual environment in real time, ensuring that the viewing point (i.e. the user's position) can be captured and identified by the system in real time and accurately.

[0065] When the user moves in the venue, the system calculates the user's spatial position in real time and quickly updates the projection content to ensure that the user sees a coherent, delay-free, and distortion-free CAVE image at different locations.

[0066] By spatial positioning, the viewing point of the CAVE screen can be changed in real time, which can bring users a more realistic and vivid immersive experience. Users can move and explore freely in the CAVE space and feel as if they are in a virtual world.

[0067] Preferably, the specific steps of the Cave virtual reality technology are as follows:

[0068] Step 1: Use digital twin technology to establish a 3D disaster rescue scene and obtain 3D scene data resources;

[0069] Step 2: Based on the physical dimensions of the length, width and height of each imaging surface of the site and the pixel resolution of the final image, a number of virtual physical cameras equal to the number of imaging surfaces of the actual site is established in the scene. The images are extracted and segmented by different virtual cameras and then fused and spliced ​​into a complete image.

[0070] Step 3: Based on the results of step 1 and step 2, the above results are rendered in real time through the 3D real-time rendering technology of the 3D real-time rendering engine;

[0071] Step 4: The real-time rendering results are output to multiple projectors through the graphics workstation, and projected to different imaging surfaces (walls, floors) through the projectors to form an immersive spatial visual experience of the CAVE. When the venue changes, step 4 returns to step 2, and the parameters of the background are changed to match the changes of the on-site venue, and the screen is automatically split and re-fused and spliced;

[0072] Step 5: Combine the Internet of Things, real-time network communication and other technologies to push the on-site equipment data and interactive feedback results to the graphics workstation in real time. The system changes the content in the scene in real time based on the data feedback, and relies on 3D real-time rendering technology to change the visual effects of the on-site CAVE immersive space in real time.

[0073] A CAVE rescue scene construction system based on digital twin and holographic technology.

[0074] Including system modules, control end graphics workstation, projection graphics workstation, camera equipment and other Internet equipment;

[0075] The output end of the system module is connected to the input end of the control end graphic workstation by a two-way signal. The control end graphic workstation is used to initiate the digital twin scene synchronization service, and enable the client to communicate and feedback interactive information, while the system module is used for rendering, display and interactive communication, and the client obtains information and interacts;

[0076] The projection graphics workstation is used to feedback device status information, and the system module is used to communicate data with the projection graphics workstation, receive scene synchronization data rendering, and generate Cave fusion images;

[0077] The projection graphics workstation and camera equipment are used to transmit image signals to the projector output;

[0078] The system module consists of a digital twin 3D resource module, a holographic cave screen generation module, a real-time rendering module, a simulation interaction module, a network communication module, a data service storage module, an auxiliary AI module and an Internet of Things module;

[0079] The system module is used with other IoT devices for digital twin virtual interaction and real-time feedback of IoT data.

[0080] (III) Beneficial effects

[0081] Compared with the prior art, the present invention provides a CAVE rescue scene construction method and system based on digital twin and holographic technology, which has the following beneficial effects:

[0082] 1. The CAVE rescue scene construction method and system of digital twin and holographic technology, through real-time simulation and prediction capabilities: Digital twin technology can provide real-time simulation of objects or systems in the real world, not only to observe their current status, but also to predict possible future situations. This capability is particularly important in disaster relief because it can help rescue teams better understand the on-site situation and make quick decisions.

[0083] 2. The CAVE rescue scene construction method and system based on digital twin and holographic technology, through efficient data processing and analysis: Digital twin technology uses big data and Internet of Things technology to realize the analysis and diagnosis of the current status and infer future development trends. This means that in disaster relief, key information can be quickly obtained to optimize rescue processes and resource allocation.

[0084] 3. The CAVE rescue scene construction method and system based on digital twin and holographic technology, through immersive experience: Holographic technology can provide naked-eye 3D effects, and you can enjoy a vivid real-time visual experience without wearing any equipment. This immersive experience is extremely valuable for disaster rescue personnel because it can enhance their understanding of the on-site environment and thus improve rescue efficiency.

[0085] 4. The CAVE rescue scene construction method and system of digital twin and holographic technology, cross-border integration and adaptive capabilities: Digital twin technology is not limited to a specific field. It can cross-border integrate data and technologies from different fields to achieve intelligence and adaptability. In disaster relief, this capability can help integrate information from different sources and provide more comprehensive and accurate rescue support.

[0086] 5. The digital twin and holographic technology CAVE rescue scene construction method and system are cost-effective: Although the CAVE system requires a higher initial investment, in the long run, it can reduce human errors and maintenance costs, thereby reducing overall costs. In addition, the application of holographic technology can also improve the presentation of exhibits, making the display more attractive, thereby increasing public attention and support for disaster relief work. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] Figure 1 A schematic diagram of the process of constructing a CAVE rescue scene using digital twin and holographic technology proposed in the present invention;

[0088] Figure 2 This is a schematic diagram of the system framework for building a CAVE rescue scene using digital twin and holographic technology proposed in the present invention. DETAILED DESCRIPTION

[0089] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0090] The purpose is to use digital twin and holographic technology to build a highly realistic interactive virtual-real fusion scene to improve the response ability and efficiency of disaster rescue personnel. Therefore, a CAVE rescue scene construction method based on digital twin and holographic technology is proposed, including: digital twin technology, 3D real-time rendering technology, holographic technology, Cave virtual reality technology, and real-time collaboration module;

[0091] S1. Digital twin technology: This is a technology that integrates physical feedback data and uses artificial intelligence, machine learning and software analysis to build a digital simulation within an information platform. This simulation will be updated based on feedback and as the physical entity changes to ensure that the virtual model matches its counterpart in the real world;

[0092] S2. 3D real-time rendering technology: It is a technology that generates and updates three-dimensional images in real time on display devices. 3D real-time rendering technology demonstrates high-quality rendering, integration of advanced interactive technologies, and application of emerging rendering technologies.

[0093] S3. Holographic technology: Holographic technology uses the principles of interference and diffraction to record and reproduce the real three-dimensional image of an object. This technology can capture all the information of the light emitted by the object, thereby reconstructing a high-quality three-dimensional image in a virtual environment;

[0094] S4. CAVE virtual reality technology: CAVE systems usually use a four-sided projection structure, including the front, left side, right side and ground, to provide an immersive visual experience. In disaster relief scenarios, this technology can simulate different perspectives, such as walking, driving or drone patrol, to enhance the on-site perception of rescue personnel;

[0095] S5. Combining these technologies, a highly simulated disaster rescue training and simulation environment (i.e., real-time collaboration module) can be constructed, including:

[0096] 1) Real-time data integration: Through digital twin technology, on-site sensor data is input into the virtual model in real time to ensure dynamic updating and accuracy of the model;

[0097] 2) 3D visualization: Using holographic technology to generate high-resolution 3D images, rescuers can visually see the situation in the disaster area in a virtual environment;

[0098] 3) Multi-angle interaction: CAVE technology provides a multi-angle visual experience, allowing rescue workers to observe the disaster area from different positions and heights, increasing operational flexibility and efficiency.

[0099] The segmentation and fusion generation of the CAVE space image by the CAVE system is a complex process involving multiple steps and calculations. The following is a simplified and structured overview, as well as the relevant calculation formulas and steps:

[0100] 1. Image Segmentation

[0101] Purpose: To divide the space in the 3D virtual scene into multiple spatial picture modules for subsequent projection mapping;

[0102] step:

[0103] Determine the space to be segmented and the required segmentation accuracy;

[0104] Use spatial segmentation algorithms to segment spatial objects into multiple modules through various physical or logical means;

[0105] For each screen module, record the coordinates and attributes it contains;

[0106] 2. Determine the projection direction and coordinate system

[0107] Purpose: To determine the projection direction and coordinate system for each screen module block in order to calculate their mapping positions in the real space;

[0108] step:

[0109] Select a direction perpendicular to the virtual scene object as the projection direction;

[0110] Establish a unified coordinate system to describe the spatial relationship between the virtual scene and the wall;

[0111] 3. Mapping calculation

[0112] Purpose: Calculate the mapping position of each screen module in the actual space according to the projection geometry relationship;

[0113] Calculation formula:

[0114] Use the perspective transformation matrix to calculate the coordinates of each screen module in the real space. The perspective transformation matrix is ​​a 3x3 matrix used to describe the projection mapping relationship from the virtual scene to the real space.

[0115] For each screen module, multiply its coordinates with the perspective transformation matrix to obtain the mapping coordinates on the wall;

[0116] 4. Projection Fusion

[0117] Purpose: To merge the images projected by multiple projectors into a coherent, seamless 2D picture.

[0118] step:

[0119] Ensure that each projector is accurately calibrated and configured to ensure the consistency and authenticity of the projected image;

[0120] Use projection blending software or hardware solutions to blend images from multiple projectors; this typically involves adjusting parameters such as projector brightness, contrast, and color balance to ensure smooth transitions and consistency between images.

[0121] The data modification and generation of the CAVE system space venue and projection screen in step S4 is a process involving multiple steps and technologies. The following is a clear, point-by-point representation and summary:

[0122] 1. CAVE space data modification

[0123] Data collection:

[0124] Conduct on-site measurements of the space site to obtain basic data such as the location coordinates and dimensions of the space site;

[0125] Data import and processing:

[0126] Import the collected spatial site data into the software, edit and organize the data to ensure the accuracy and completeness of the data;

[0127] Data modification:

[0128] Modify the spatial site data according to project requirements, such as adding new spatial elements, modifying the position or attributes of existing elements, etc.;

[0129] Data export and storage:

[0130] Export the modified spatial site data to a format suitable for projection, such as an image file (JPEG, PNG, etc.) or vector data (SVG, DWG, etc.);

[0131] 2. Projection screen data generation

[0132] Projection settings:

[0133] Adjust the projection position and angle of the projector to ensure that the projection image can accurately cover the target site;

[0134] According to the configuration of the CAVE system and projection requirements, set the resolution of the projector and the physical size of the image, length, width, and height parameters;

[0135] Data conversion:

[0136] Convert spatial site data (image or vector data) into a format suitable for projection;

[0137] If necessary, use software to further edit and modify the projection image, such as adding text, annotations, icons, etc.;

[0138] Real-time data integration:

[0139] If the projection screen needs to display real-time data, the real-time data can be integrated into the projection screen through the data interface or API;

[0140] Use programming scripts or software tools to capture, process and display real-time data;

[0141] Interactive modification:

[0142] If the audience needs to participate in the interaction, you can set interactive elements in the projection screen (such as gesture recognition, spatial positioning, etc.);

[0143] Use interactive programming technology to achieve audience interaction and real-time feedback;

[0144] 3. Real-time rendering and presentation

[0145] A high-performance graphics processor (GPU) and rendering engine are used to render the virtual scene in real time. The rendered image is projected onto the wall through a projector. Factors such as the curvature of the wall and the focal length of the projector are considered to ensure that the shape and size of the projected image match the objects in the virtual scene. The calculated mapping position is rendered onto the wall in real time and presented.

[0146] It should be noted that the real-time change of the viewing point of the CAVE screen mainly depends on the spatial positioning technology and system design. The following are the specific implementation steps and key elements:

[0147] The CAVE immersive space needs to be equipped with a spatial positioning and motion capture system. The system uses infrared or laser radar technologies to accurately track the user's movement trajectory and map the user's movements to the virtual environment in real time, ensuring that the viewing point (i.e. the user's position) can be captured and identified by the system in real time and accurately.

[0148] When the user moves in the venue, the system calculates the user's spatial position in real time and quickly updates the projection content to ensure that the user sees a coherent, delay-free, and distortion-free CAVE image at different locations.

[0149] By spatial positioning, the viewing point of the CAVE screen can be changed in real time, which can bring users a more realistic and vivid immersive experience. Users can move and explore freely in the CAVE space and feel as if they are in a virtual world.

[0150] In this embodiment, the specific steps of Cave virtual reality technology are as follows ( Figure 1 shown):

[0151] Step 1: Use digital twin technology to establish a 3D disaster rescue scene and obtain 3D scene data resources;

[0152] Step 2: Based on the physical dimensions of the length, width and height of each imaging surface of the site and the pixel resolution of the final image, a number of virtual physical cameras equal to the number of imaging surfaces of the actual site is established in the scene. The images are extracted and segmented by different virtual cameras and then fused and spliced ​​into a complete image.

[0153] Step 3: Based on the results of step 1 and step 2, the above results are rendered in real time through the 3D real-time rendering technology of the 3D real-time rendering engine;

[0154] Step 4: The real-time rendering results are output to multiple projectors through the graphics workstation, and projected to different imaging surfaces (walls, floors) through the projectors to form an immersive spatial visual experience of the CAVE. When the venue changes, step 4 returns to step 2, and the parameters of the background are changed to match the changes of the on-site venue, and the screen is automatically split and re-fused and spliced;

[0155] Step 5: Combine the Internet of Things, real-time network communication and other technologies to push the on-site equipment data and interactive feedback results to the graphics workstation in real time. The system changes the content in the scene in real time based on the data feedback, and relies on 3D real-time rendering technology to change the visual effects of the on-site CAVE immersive space in real time.

[0156] Implementation method: The digital twin model module uses sensor data to build a digital twin model, the holographic projection technology module realizes holographic projection of virtual scenes, and the real-time collaboration module supports collaborative interaction among different participants:

[0157] The following steps are involved:

[0158] Digital Twin Modeling:

[0159] Use digital twin technology to accurately model disaster scenarios in three dimensions, including information such as terrain, buildings, and personnel locations;

[0160] Collect field data and sensor information, combine image processing and data analysis methods, and build a complete and real digital twin model. The main methods of digital twin modeling include oblique photography, 3D scanning, panoramic modeling, and manual modeling:

[0161] Oblique photography 3D scene technology is an advanced 3D reconstruction technology. It uses special oblique angles and a large number of overlapping photos to capture and record the shape and details of buildings or geographical scenes. This technology can not only obtain detailed side information from multiple viewpoints, but also provide high resolution and a wide field of view, so as to more realistically reflect the situation around the objects. In practical applications, oblique photography technology is widely used in urban planning, geographic information system (GIS), land management, architectural design and cultural heritage protection. For example, it is used in urban refined management, urban planning and design, BIM design, urban security, land, building and people management, etc. In addition, this technology can also be used for the production of real-scene 3D models in highway construction to improve construction efficiency and quality.

[0162] 3D scanning technology is a method of capturing three-dimensional information of an object or environment through various sensors and devices. These technologies can be divided into two categories: contact and non-contact. Contact scanning usually uses a physical probe to directly contact the scanned object to obtain data on its surface or internal structure, while non-contact scanning uses optical, laser or other invisible bands to measure the shape and appearance of objects, such as laser scanning and structured light scanning. In terms of establishing a 3D scene, it is first necessary to collect three-dimensional data of the target object through a 3D scanner. This step can use different technologies, such as laser scanners, which create accurate point cloud data by emitting lasers and measuring the time and angle of reflection back. In addition, there is also scanning technology based on structured light, which reconstructs the shape of an object by projecting a specific pattern of light and capturing the resulting interference pattern;

[0163] 3D modeling of scenes through panoramic shooting mainly involves capturing real-world scenes through photography technology, and converting these images into 3D models using 3D modeling software and technology. First, a panoramic camera is needed to shoot 360-degree panoramic video, which can be achieved through a single-lens or multi-lens system. Panoramic video usually requires post-processing to splice into a seamless spherical structure, and specific software is used to convert 360-degree panoramic video into a 3D model. With the advancement of technology, 3D modeling of panoramic videos will become more sophisticated and efficient. Future development may focus on improving the realism and interactivity of the model, as well as reducing costs and improving production efficiency.

[0164] The model is created by artificial modeling and professional 3D modeling software. Commonly used software includes 3dsMax, MAYA, Blueender, SketchUp, Solidwork, and Ci nema 4D. These software allow users to estimate the outline and height information of the building based on 2D vector graphics, image data or directly from photos taken on site, thereby constructing a preliminary 3D model. After the model is built, the model needs to be textured to increase the visual realism of the model. This usually involves taking photos from the real world and pasting them onto the 3D model in an appropriate format. Finally, the scene is further simulated and optimized.

[0165] 3D real-time rendering technology:

[0166] 3D real-time rendering technology is an efficient graphics processing method that allows 3D models to be updated and displayed instantly during user interaction, thereby providing a smooth visual experience. Commonly used rendering engines include Unity3D, Unreal Engine, CryEngin, Cocos Creator, and Eggert Engine. Compared with offline rendering technology, which takes several minutes to several hours to render an image, real-time rendering technology can achieve a frame rate requirement of 60 images per second. Its advantages are mainly reflected in interactivity and low lag. This technology allows users to interact in real time in a virtual environment and can instantly see changes in the scene, which is very important for application scenarios that require fast feedback, meaning that users can instantly see the results of their operations without having to wait for a long rendering process.

[0167] Application of holographic projection technology:

[0168] Combined with holographic technology, the digital twin model is projected into the CAVE system to achieve stereoscopic display and interactive experience, allowing rescuers to experience highly realistic disaster scenes in a virtual environment. Holographic projection CAVE technology is a highly immersive virtual reality display system that combines multiple technologies such as high-resolution stereoscopic projection technology, three-dimensional computer graphics technology, audio technology, and sensor technology to create a fully immersive environment. The CAVE system uses multiple projectors and fusion technology to ensure the integrity of the picture and the shock of the visual effect. In addition, CAVE technology supports multi-channel vision synchronization technology, so that in a room-sized four-sided cube projection display space, all participants can be fully immersed in a unified virtual environment. This technology not only improves the resolution, but also enhances the layering and interactive experience of the picture, thereby providing a real immersive experience. Holographic projection technology itself uses the principles of interference and diffraction to record and reproduce the real three-dimensional image of the object. The audience can see the three-dimensional effect without wearing any glasses, providing users with an unprecedented virtual reality experience. This experience is not limited to visual enjoyment, but also includes comprehensive stimulation of hearing and touch, so that each participant can deeply feel the emotions and information brought by the content.

[0169] Real-time data update and synchronization:

[0170] Integrate real-time data sources, such as sensor data, communication information, etc., update the data in the virtual disaster scene in real time, maintain the synchronization of the virtual and real environments, and use network technology and data processing algorithms to ensure the accuracy and timeliness of virtual environment information. Digital twin technology first needs to collect data from physical objects or systems in the real world through various sensors and devices. These data include but are not limited to temperature, pressure, speed, position and other information. These data are then transmitted to the cloud or local server for processing. The collected data is used to build a virtual model corresponding to the physical objects or systems in the real world. Digital twin technology can realize real-time synchronization of virtual models with real-world entities, the so-called "virtual-reality mapping". In addition, it also supports two-way interaction, that is, users can perform operations in the physical world, and these operations will be fed back to the virtual environment in real time.

[0171] Interactive Application Development:

[0172] Develop interactive applications to provide users with real-time simulation, drills, and decision-making support functions in a virtual environment, and design user-friendly interfaces and operating methods to facilitate rescue personnel to conduct emergency response training and decision-making simulation in a virtual environment.

[0173] In-depth simulation and analysis:

[0174] Based on digital twin technology, in-depth simulation and analysis of disaster scenarios can be achieved to provide a scientific basis for rescue decisions. Data mining and machine learning algorithms can be used to analyze and predict data in virtual environments to provide rescue effect evaluation and optimization suggestions.

[0175] System Integration and Optimization:

[0176] Realize overall system integration of various technologies, ensure stable and efficient operation of virtual-reality fusion scenarios, continuously optimize system performance, enhance the realism and user experience of virtual environments, and continuously improve the functions and performance of applications.

[0177] The above is a patented technical solution for constructing a CAVE-style disaster rescue virtual-reality fusion scene based on digital twin technology and holographic technology. It aims to provide disaster rescue personnel with an efficient and realistic training environment, improve their ability and efficiency in responding to disaster challenges, improve the training and emergency response capabilities of disaster rescue personnel, enhance disaster rescue efficiency and accuracy, and bring new experiences and possibilities to disaster rescue training and drills.

[0178] See also Figure 2 : A CAVE rescue scene construction system based on digital twin and holographic technology,

[0179] Including system modules, control end graphics workstation, projection graphics workstation, camera equipment and other Internet equipment;

[0180] The output end of the system module is connected to the input end of the control end graphic workstation by a two-way signal. The control end graphic workstation is used to initiate the digital twin scene synchronization service, and enable the client to communicate and feedback interactive information, while the system module is used for rendering, display and interactive communication, and the client obtains information and interacts;

[0181] The projection graphics workstation is used to feedback device status information, and the system module is used to communicate data with the projection graphics workstation, receive scene synchronization data rendering, and generate Cave fusion images;

[0182] The projection graphics workstation and camera equipment are used to transmit image signals to the projector output;

[0183] The system module consists of a digital twin 3D resource module, a holographic cave screen generation module, a real-time rendering module, a simulation interaction module, a network communication module, a data service storage module, an auxiliary AI module and an Internet of Things module;

[0184] The system module is used with other IoT devices for digital twin virtual interaction and real-time feedback of IoT data.

[0185] The beneficial effects of the present invention are: improving rescue efficiency and safety: through digital twin technology, a virtual model that is highly similar to the disaster scene in the real world can be created, which enables commanders and rescue teams to conduct training and drills without actual risks. This method can significantly improve the ability of emergency response and scientific decision-making, while enhancing the response and coordinated rescue capabilities of the rescue team.

[0186] Optimize the picture production rendering process and cycle: Use real-time rendering technology to render digital twin scene images, so that the 3D scene model can be updated and displayed in real time during user interaction, thereby providing a smooth visual experience. Compared with the traditional offline rendering output film production method, it also greatly simplifies the complicated production process and the large increase in workload and cost caused by content modification or site adjustment changes.

[0187] Enhanced immersion and interactivity: The CAVE system organically combines high-resolution stereoscopic projection technology, three-dimensional computer graphics technology, and sound technology to produce a fully immersive virtual environment. This immersive experience is particularly important for simulating complex disaster scenarios because it allows participants to better understand and respond to possible situations.

[0188] Accurately simulate the development process of disasters: Using holographic technology, the real three-dimensional images of objects can be recorded and reproduced. This technology can achieve stereoscopic effects and realism. In disaster relief, this means that the development process of disasters can be simulated very accurately, providing more accurate information support for rescue operations.

[0189] Integration of multi-disciplinary, multi-physical, multi-scale, and multi-probability simulation: Digital twin technology is not just a simple digital replication, but an integration of multi-disciplinary, multi-physical, multi-scale, and multi-probability simulation processes, which makes disaster relief simulation more comprehensive and scientific, and can take into account more variables and conditions, thereby improving the accuracy and reliability of the simulation.

[0190] Real-time data updates and feedback: A key advantage of digital twin technology is its real-time nature, which is the ability to update virtual models in real time to reflect changes in the real world. This is particularly important for dynamically changing disaster relief scenarios as it allows rescue teams to adjust strategies and actions based on the latest situation.

[0191] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A CAVE rescue scene construction method based on digital twin and holographic technology, comprising: Digital twin technology, 3D real-time rendering technology, holographic technology, Cave virtual reality technology, real-time collaboration module; Features: S1. Digital twin technology: This is a technology that integrates physical feedback data and uses artificial intelligence, machine learning and software analysis to build a digital simulation within an information platform. This simulation will be updated based on feedback and as the physical entity changes to ensure that the virtual model matches its counterpart in the real world; S2. 3D real-time rendering technology: It is a technology that generates and updates three-dimensional images in real time on display devices. 3D real-time rendering technology demonstrates high-quality rendering, integration of advanced interactive technologies, and application of emerging rendering technologies. S3. Holographic technology: Holographic technology uses the principles of interference and diffraction to record and reproduce the real three-dimensional image of an object. This technology can capture all the information of the light emitted by the object, thereby reconstructing a high-quality three-dimensional image in a virtual environment; S4. CAVE virtual reality technology: CAVE systems usually use a four-sided projection structure, including the front, left side, right side and ground, to provide an immersive visual experience. In disaster relief scenarios, this technology can simulate different perspectives, such as walking, driving or drone patrol, to enhance the on-site perception of rescue personnel; S5. Combining these technologies, a highly simulated disaster rescue training and simulation environment (i.e., real-time collaboration module) can be constructed, including: 1) Real-time data integration: Through digital twin technology, on-site sensor data is input into the virtual model in real time to ensure dynamic updating and accuracy of the model; 2) 3D visualization: Using holographic technology to generate high-resolution 3D images, rescuers can visually see the situation in the disaster area in a virtual environment; 3) Multi-angle interaction: CAVE technology provides a multi-angle visual experience, allowing rescue workers to observe the disaster area from different positions and heights, increasing operational flexibility and efficiency.

2. According to claim 1, a CAVE rescue scene construction method using digital twin and holographic technology is characterized in that: The segmentation and fusion generation of the CAVE space image by the CAVE system in step S4 is a complex process involving multiple steps and calculations. The following is a simplified and structured overview, as well as the relevant calculation formulas and steps:

1. Image Segmentation Purpose: To divide the space in the 3D virtual scene into multiple spatial picture modules for subsequent projection mapping; step: Determine the space to be segmented and the required segmentation accuracy; Use spatial segmentation algorithms to segment spatial objects into multiple modules through various physical or logical means; For each screen module, record the coordinates and attributes it contains; 2. Determine the projection direction and coordinate system Purpose: To determine the projection direction and coordinate system for each screen module block in order to calculate their mapping positions in the real space; step: Select a direction perpendicular to the virtual scene object as the projection direction; Establish a unified coordinate system to describe the spatial relationship between the virtual scene and the wall; 3. Mapping calculation Purpose: Calculate the mapping position of each screen module in the actual space according to the projection geometry relationship; Calculation formula: Use the perspective transformation matrix to calculate the coordinates of each screen module in the real space. The perspective transformation matrix is ​​a 3x3 matrix used to describe the projection mapping relationship from the virtual scene to the real space. For each screen module, multiply its coordinates with the perspective transformation matrix to obtain the mapping coordinates on the wall; 4. Projection Fusion Purpose: To merge the images projected by multiple projectors into a coherent, seamless 2D picture. step: Ensure that each projector is accurately calibrated and configured to ensure the consistency and authenticity of the projected image; Use projection blending software or hardware solutions to blend images from multiple projectors; this typically involves adjusting parameters such as projector brightness, contrast, and color balance to ensure smooth transitions and consistency between images.

3. According to claim 2, a CAVE rescue scene construction method using digital twin and holographic technology is characterized in that: The data modification and generation of the space venue and the projection screen in step S4 is a process involving multiple steps and technologies. The following is a clear, point-by-point representation and summary:

1. CAVE space data modification Data collection: Conduct on-site measurements of the space site to obtain basic data such as the location coordinates and dimensions of the space site; Data import and processing: Import the collected spatial site data into the software, edit and organize the data to ensure the accuracy and completeness of the data; Data modification: Modify the spatial site data according to project requirements, such as adding new spatial elements, modifying the position or attributes of existing elements, etc.; Data export and storage: Export the modified spatial site data to a format suitable for projection, such as an image file (JPEG, PNG, etc.) or vector data (SVG, DWG, etc.); 2. Projection screen data generation Projection settings: Adjust the projection position and angle of the projector to ensure that the projection image can accurately cover the target site; According to the configuration of the CAVE system and projection requirements, set the resolution of the projector and the physical size of the image, length, width, and height parameters; Data conversion: Convert spatial site data (image or vector data) into a format suitable for projection; If necessary, use software to further edit and modify the projection image, such as adding text, annotations, icons, etc.; Real-time data integration: If the projection screen needs to display real-time data, the real-time data can be integrated into the projection screen through the data interface or API; Use programming scripts or software tools to capture, process and display real-time data; Interactive modification: If the audience needs to participate in the interaction, you can set interactive elements in the projection screen (such as gesture recognition, spatial positioning, etc.); Use interactive programming technology to achieve audience interaction and real-time feedback; 3. Real-time rendering and presentation A high-performance graphics processor (GPU) and rendering engine are used to render the virtual scene in real time. The rendered image is projected onto the wall through a projector. Factors such as the curvature of the wall and the focal length of the projector are considered to ensure that the shape and size of the projected image match the objects in the virtual scene. The calculated mapping position is rendered onto the wall in real time and presented.

4. The method for constructing a CAVE rescue scene using digital twin and holographic technology according to claim 3 is characterized in that: The real-time change of the viewing point of the CAVE screen in step S4 mainly depends on the spatial positioning technology and system design. The following are the specific implementation steps and key elements: The CAVE immersive space needs to be equipped with a spatial positioning and motion capture system. The system uses infrared or laser radar technologies to accurately track the user's movement trajectory and map the user's movements to the virtual environment in real time, ensuring that the viewing point (i.e. the user's position) can be captured and identified by the system in real time and accurately. When the user moves in the venue, the system calculates the user's spatial position in real time and quickly updates the projection content to ensure that the user sees a coherent, delay-free, and distortion-free CAVE image at different locations. By spatial positioning, the viewing point of the CAVE screen can be changed in real time, which can bring users a more realistic and vivid immersive experience. Users can move and explore freely in the CAVE space and feel as if they are in a virtual world.

5. According to claim 4, a CAVE rescue scene construction method using digital twin and holographic technology is characterized in that: The specific steps of the Cave virtual reality technology are as follows: Step 1: Use digital twin technology to establish a 3D disaster rescue scene and obtain 3D scene data resources; Step 2: Based on the physical dimensions of the length, width and height of each imaging surface of the site and the pixel resolution of the final image, a number of virtual physical cameras equal to the number of imaging surfaces of the actual site is established in the scene. The images are extracted and segmented by different virtual cameras and then fused and spliced ​​into a complete image. Step 3: Based on the results of step 1 and step 2, the above results are rendered in real time through the 3D real-time rendering technology of the 3D real-time rendering engine; Step 4: The real-time rendering results are output to multiple projectors through the graphics workstation, and projected to different imaging surfaces (walls, floors) through the projectors to form an immersive spatial visual experience of the CAVE. When the venue changes, step 4 returns to step 2, and the parameters of the background are changed to match the changes of the on-site venue, and the screen is automatically split and re-fused and spliced; Step 5: Combine the Internet of Things, real-time network communication and other technologies to push the on-site equipment data and interactive feedback results to the graphics workstation in real time. The system changes the content in the scene in real time based on the data feedback, and relies on 3D real-time rendering technology to change the visual effects of the on-site CAVE immersive space in real time.

6. A CAVE rescue scene construction system based on digital twin and holographic technology, characterized by: Including system modules, control end graphics workstation, projection graphics workstation, camera equipment and other Internet equipment; The output end of the system module is connected to the input end of the control end graphic workstation by a two-way signal. The control end graphic workstation is used to initiate the digital twin scene synchronization service, and enable the client to communicate and feedback interactive information, while the system module is used for rendering, display and interactive communication, and the client obtains information and interacts; The projection graphics workstation is used to feedback device status information, and the system module is used to communicate data with the projection graphics workstation, receive scene synchronization data rendering, and generate Cave fusion images; The projection graphics workstation and camera equipment are used to transmit image signals to the projector output; The system module consists of a digital twin 3D resource module, a holographic cave screen generation module, a real-time rendering module, a simulation interaction module, a network communication module, a data service storage module, an auxiliary AI module and an Internet of Things module; The system module is used for digital twin virtual interaction and real-time feedback of IoT data with other IoT devices.

Citation Information

Cited By

  • Immersive hardware and software path verification method and system

    CN119739290A

  • Elevator digital twin modeling method

    CN120217470A

  • Mine accident emergency rescue deduction system and method

    CN120494651A

  • Monitoring equipment layout method and device based on digital twinning, equipment and medium

    CN121173935A