Virtual reality and augmented reality technology construction simulation system

Abstract

The invention relates to the technical field of virtual reality and augmented reality, in particular to a virtual reality and augmented reality technology construction simulation system. The image is generated by arranging the sensing module such as the radar sensor, the penetrability of the radar sensor is utilized, the image processing module is combined, in the complex and changeable construction environment, the definition, the detail expressive force and the object recognition accuracy of the image are remarkably improved, and reliable data support and excellent user experience are provided for construction simulation. And meanwhile, the performance of the construction simulation system in a dynamic environment is improved, and the safety and high efficiency of the construction process are ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of virtual reality and augmented reality, and in particular to a construction simulation system using virtual reality and augmented reality technologies. Background Art

[0002] Virtual reality (VR) and augmented reality (AR) are two different technologies. Both use computer technology to create an environment that interacts with the real world to varying degrees. Virtual reality (VR) is a computer-generated simulated environment that allows users to immerse themselves in a completely fictional environment through a head-mounted display or similar device. In VR, users can interact with objects or scenes in the virtual environment, feeling as if they are in a completely fictional world. VR technology typically enhances the user's sense of immersion and interactive experience through stereoscopic vision and audio feedback, as well as devices such as handles and gloves. Augmented reality (AR) superimposes virtual information on the real world, displaying real-time virtual information through smart devices (such as mobile phones and AR glasses), allowing users to see and interact with real-world objects.

[0003] After applying virtual reality and augmented reality technologies to construction, due to the complex and changeable visual information of the construction site and the unfavorable conditions such as dust, mist and poor lighting in the environment, traditional construction simulations are difficult to provide clear and detailed visual data, resulting in limitations in spatial mapping, object positioning and construction process monitoring. Summary of the Invention

[0004] In view of the shortcomings of the prior art described above, the purpose of the present invention is to provide a construction simulation system using virtual reality and augmented reality technology to solve the problem that traditional construction simulation in the prior art is difficult to provide clear and detailed visual data, thereby having limitations in spatial mapping, object positioning and construction process monitoring.

[0005] To achieve the above-mentioned and other related purposes, the present invention provides a construction simulation system using virtual reality and augmented reality technologies, the construction simulation system comprising:

[0006] A sensing module for real-time scanning of buildings and building environments and capturing motion data, including a radar sensor, a depth camera, an RGB camera, and a laser scanner;

[0007] An image processing module includes a multi-source data fusion unit and an edge detection and object recognition unit. The multi-source data fusion unit is used to process and enhance the image information generated by the sensing module using a multi-source data fusion algorithm. The edge detection and object recognition unit is used to process and analyze the image information generated by the sensing module using edge detection technology and deep learning technology.

[0008] Optionally, the construction simulation system further includes a gesture recognition module, which is used to recognize the gestures of construction workers using a convolutional neural network and an optical flow algorithm.

[0009] Optionally, the construction simulation system further includes a spatial mapping module and an AR overlay module, wherein the spatial mapping module is used to create a three-dimensional map of the building environment based on the image information generated by the sensing module, and the AR overlay module is used to overlay virtual construction elements onto the user's real-world view based on the three-dimensional map created by the spatial mapping module.

[0010] Optionally, the construction simulation system further includes a VR display module and a VR imaging module, wherein the VR display module is used to provide a display screen to display virtual images and virtual construction elements superimposed on the real-world view, and the VR imaging module is used to generate images that can be displayed on the VR display module.

[0011] Optionally, the VR display module is provided with an adjustable lens, an eye tracking sensor, a head tracking sensor and a light sensor. The adjustable lens is used to adjust the user's vision to provide a clear field of view. The eye tracking sensor is used to detect the user's eye movement to adjust the image display. The head tracking sensor is used to track the movement of the user's head to synchronize the image with the user's line of sight. The light sensor adjusts the display brightness according to the brightness of the surrounding environment.

[0012] Optionally, the VR imaging module includes an image rendering engine, a graphics processing unit and a binocular rendering unit, the image rendering engine is used to render a three-dimensional scene in real time, the graphics processing unit is used to execute the computing tasks of the image rendering engine, and the binocular rendering unit is used to render slightly different images for the user's left eye and right eye respectively.

[0013] Optionally, the construction simulation system further includes an interactive control module, and the interactive control module is used to realize real-time interaction between the user and the virtual world.

[0014] Optionally, the construction simulation system further includes a collaboration module, which is used to achieve synchronous collaboration and communication among multiple users.

[0015] Optionally, the construction simulation system further includes a data synchronization and integration module, which is used to convert and adapt data of different formats and structures into a format that can be recognized and processed by the system, and to process real-time data streams from the sensing module and user interaction.

[0016] In this virtual reality and augmented reality construction simulation system, radar sensors and other sensing modules are used to generate images. Leveraging the radar sensor's penetrating properties, combined with image processing modules, this significantly improves image clarity, detail, and object recognition accuracy in complex and changing construction environments, providing reliable data support and an excellent user experience for construction simulation. This system also enhances its performance in dynamic environments, ensuring the safety and efficiency of the construction process. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a principle block diagram of the construction simulation system of virtual reality and augmented reality technology of the present invention. DETAILED DESCRIPTION

[0018] Refer to the following Figure 1 To describe a virtual reality and augmented reality construction simulation system of the present invention. In the description of this embodiment, the reference terms "one embodiment," "some embodiments," "illustrative embodiments," "example," "specific example," or "some examples" mean that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples.

[0019] like Figure 1 As shown, an embodiment of the present invention provides a construction simulation system using virtual reality and augmented reality technologies, including a sensing module and an image processing module. The sensing module is used to scan buildings and building environments in real time and capture motion data to generate image information. The sensing module includes a radar sensor, a depth camera, an RGB camera, and a laser scanner. The image processing module includes a multi-source data fusion unit and an edge detection and object recognition unit. The multi-source data fusion unit is used to process and enhance the image information generated by the sensing module using a multi-source data fusion algorithm, and the edge detection and object recognition unit is used to process and analyze the image information generated by the sensing module using edge detection technology and deep learning technology.

[0020] Radar sensors can be used to scan the construction site's terrain and building structures in real time, generating high-precision 3D models. They can also monitor and identify construction workers and equipment on the site. Depth cameras can also capture the 3D structure of the construction site environment, RGB cameras capture color images of the construction site environment, and laser scanners use laser beams to scan the construction site environment. Radar sensors have excellent penetration capabilities, allowing them to penetrate certain materials such as mist and dust, and operate even in construction environments with poor visibility. Furthermore, radar sensors can provide accurate distance and shape information even in dim light or at night, enabling precise distance and speed measurements, ensuring accurate spatial mapping and object positioning in construction simulations. Furthermore, radar sensors can effectively track moving objects, accurately capturing dynamically changing objects in simulated construction environments and enabling real-time monitoring of the construction process. Furthermore, radar sensors can operate in various weather conditions, including rain, snow, and fog, making the system suitable for construction simulations in diverse environments. By deploying radar sensors, reliable image information can be provided even under the complex conditions of a construction site.

[0021] Using a multi-source data fusion algorithm, the radar sensor's image information is combined with that of the depth camera, RGB camera, and laser scanner to generate an image with detailed and depth information, improving image clarity and information content. Specifically, the radar sensor's image information, along with that of the depth camera, RGB camera, and laser scanner, is aligned. Then, a weighted average fusion method is applied to combine the image's brightness, chromaticity, and depth information to generate a composite image, providing users with a clearer and more intuitive visual experience in virtual and augmented reality environments.

[0022] Edge detection and deep learning technologies are used to process and analyze image information from radar sensors, as well as image information obtained by depth cameras, RGB cameras, and laser scanners, to accurately identify the location and status of building materials, tools, and construction personnel, enabling monitoring and identification of construction personnel and equipment. Specifically, edge detection technology is first used to extract significant features from image information. Then, deep learning technologies, such as combining convolutional neural networks (CNN) and recurrent neural networks (RNN), are used to deeply analyze and process the captured motion data to identify the skeletal posture and motion status of construction personnel. CNN is used to extract spatial features from motion data, while RNN is used to capture and understand the changes in these features in the temporal dimension, thereby achieving accurate recognition of complex postures and continuous movements. This not only improves the management efficiency of the construction site, but also provides reliable data support for further optimization of the construction process.

[0023] In summary, the image generation through sensing modules such as radar sensors, combined with image processing modules, significantly improves image clarity, detail, and object recognition accuracy in complex and changing construction environments, providing reliable data support and an excellent user experience for construction simulation. This also enhances the performance of construction simulation systems in dynamic environments, ensuring safety and efficiency during the construction process.

[0024] Furthermore, the signal processing algorithms involved in the radar sensor operation are as follows:

[0025]

[0026] Where X(k) is the frequency domain signal, x(n) is the time domain signal, N is the number of sampling points, and j is the imaginary unit.

[0027] Furthermore, the construction simulation system also includes a gesture recognition module. This module utilizes convolutional neural networks and optical flow algorithms to analyze hand motion trajectories in real time, enabling rapid recognition and response to a variety of complex gestures. This significantly enhances the system's interactivity and user experience, and strengthens the construction simulation system's dynamic monitoring and interactive capabilities. It also provides strong support for construction site safety management and efficiency improvements, demonstrating its superior performance and application prospects in complex construction environments.

[0028] Furthermore, the construction simulation system also includes a spatial mapping module and an AR overlay module. The spatial mapping module is used to create a three-dimensional map of the building environment based on the image information generated by the sensing module, and provide spatial information for the subsequent AR overlay module. The AR overlay module is used to overlay virtual construction elements onto the user's real-world view based on the three-dimensional map created by the spatial mapping module. Specifically, the AR overlay module overlays virtual information, images or objects onto the real-world view seen by the user, and uses computer vision technology to identify and track objects or markers in the real world, and uses depth cameras, laser scanning or SLAM technology to create three-dimensional maps, and dynamically generates or selects appropriate virtual content for overlay based on the user's perspective and environmental information.

[0029] Furthermore, the construction simulation system also includes a VR display module and a VR imaging module. The VR display module is used to provide a display screen for displaying virtual images and virtual construction elements superimposed on the real-world view. The VR imaging module is used to generate images that can be displayed on the VR display module.

[0030] Optionally, the VR display module is a VR helmet, which is convenient to wear and easy to operate.

[0031] Furthermore, the VR display module is provided with an adjustable lens, an eye-tracking sensor, a head-tracking sensor, and a light sensor. The adjustable lens, also known as a focus-adjustable lens, adjusts the user's vision by adjusting the focal length to adapt to the different user vision differences and usage habits, so that the user can obtain a clearer field of view. The eye-tracking sensor is used to detect the user's eye movements and adjust the image display according to the rotation angle of the eyeball, so that the user can observe a clear picture from different perspectives. The head-tracking sensor is used to track the movement of the user's head to synchronize the image with the user's line of sight. The light sensor is used to adjust the display brightness according to the brightness of the surrounding environment to improve the comfort of the picture. Through the combination of the adjustable lens, eye-tracking sensor, head-tracking sensor, and light sensor, the VR display module can provide a more realistic and immersive experience, and at the same time, through dynamic adjustment, the user can interact with the virtual environment more naturally.

[0032] Furthermore, the VR imaging module includes an image rendering engine, a graphics processing unit, and a binocular rendering unit. The image rendering engine is used to render 3D scenes in real time, the graphics processing unit is used to perform the image rendering engine's computational tasks, and the binocular rendering unit is used to render slightly different images for the user's left and right eyes. This image processing provides high-quality VR images, offering a better user experience.

[0033] Furthermore, the image rendering engine in the VR imaging module renders 3D scenes in real time, which involves graphics rendering algorithms such as rasterization, shading, and texture mapping. The shading algorithm uses the Phong reflection model to calculate the lighting effect, as follows:

[0034]

[0035] Among them, L o is the outgoing light from the observation point, L a is the ambient light, L i is the light of the i-th light source, n is the surface normal, l i is the direction from the surface point to the light source, and k is the glossiness.

[0036] The construction simulation system also includes an interactive control module, which enables real-time user interaction with the virtual world. Specifically, the interactive control module utilizes a sensor module, combined with depth perception technology, to track and recognize user gestures, allowing users to interact with virtual content through gestures. It also monitors the user's eye movements to select, navigate, or focus on objects in the virtual environment, thereby enabling human-computer interaction and enhancing immersion and operational efficiency.

[0037] Furthermore, depth perception technology involves using the depth camera and laser scanner in the sensor module to scan the user's surroundings, capturing the three-dimensional structure of the space. Raw data from the sensor module is then cleaned and filtered to remove noise and outliers. Key features are extracted from the processed data, and a three-dimensional map or model of the environment is constructed using the extracted features and sensor module data. The sensor module also identifies objects in the environment, tracks their position and movement in three-dimensional space, and calculates the distance between the user and objects in the environment, providing spatial information.

[0038] Furthermore, the construction simulation system includes a collaboration module, which enables simultaneous collaboration and communication among multiple users. By synchronizing all user perspectives and interactions in real time within the system, users in different locations can collaborate within the same virtual or augmented reality environment to complete design, planning, and construction, thereby improving construction efficiency.

[0039] Furthermore, the construction simulation system also includes a data synchronization and integration module, which is used to convert and adapt data of different formats and structures into a format that the system can recognize and process. It also processes real-time data streams from the sensor module and user interactions to ensure instant data updates. A data fusion algorithm is used to process real-time data streams from the sensor module and user interactions. A Kalman filter is used to estimate and predict the state. The Kalman filter update method is as follows:

[0040]

[0041] in, is the estimated value of time k, K k is the Kalman gain, z k is the observation value, and H is the observation matrix.

[0042] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the present invention. Anyone skilled in the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by one of ordinary skill in the art without departing from the spirit and technical principles disclosed herein are intended to be covered by the claims of the present invention.

Claims

1. A construction simulation system using virtual reality and augmented reality technology, characterized in that: The construction simulation system includes: A sensing module for real-time scanning of buildings and building environments and capturing motion data, including a radar sensor, a depth camera, an RGB camera, and a laser scanner; An image processing module includes a multi-source data fusion unit and an edge detection and object recognition unit. The multi-source data fusion unit is used to process and enhance the image information generated by the sensing module using a multi-source data fusion algorithm. The edge detection and object recognition unit is used to process and analyze the image information generated by the sensing module using edge detection technology and deep learning technology.

2. The virtual reality and augmented reality construction simulation system according to claim 1, characterized in that: The construction simulation system also includes a gesture recognition module, which is used to recognize the gestures of construction workers using a convolutional neural network and an optical flow algorithm.

3. The virtual reality and augmented reality construction simulation system according to claim 1, characterized in that: The construction simulation system also includes a spatial mapping module and an AR overlay module. The spatial mapping module is used to create a three-dimensional map of the building environment based on the image information generated by the sensing module. The AR overlay module is used to overlay virtual construction elements onto the user's real-world view based on the three-dimensional map created by the spatial mapping module.

4. The virtual reality and augmented reality construction simulation system according to claim 3, characterized in that: The construction simulation system also includes a VR display module and a VR imaging module. The VR display module is used to provide a display screen to display virtual images and virtual construction elements superimposed on the real-world view. The VR imaging module is used to generate images that can be displayed on the VR display module.

5. The virtual reality and augmented reality construction simulation system according to claim 4, characterized in that: The VR display module is provided with an adjustable lens, an eye tracking sensor, a head tracking sensor and a light sensor. The adjustable lens is used to adjust the user's vision to provide a clear field of view. The eye tracking sensor is used to detect the user's eye movement to adjust the image display. The head tracking sensor is used to track the movement of the user's head to synchronize the image with the user's line of sight. The light sensor adjusts the display brightness according to the brightness of the surrounding environment.

6. The virtual reality and augmented reality construction simulation system according to claim 4, characterized in that: The VR imaging module includes an image rendering engine, a graphics processing unit, and a binocular rendering unit. The image rendering engine is used to render three-dimensional scenes in real time, the graphics processing unit is used to perform the computing tasks of the image rendering engine, and the binocular rendering unit is used to render slightly different images for the user's left eye and right eye respectively.

7. The virtual reality and augmented reality construction simulation system according to claim 5, characterized in that: The construction simulation system further comprises an interactive control module, which is used to realize real-time interaction between the user and the virtual world.

8. The virtual reality and augmented reality construction simulation system according to claim 7, characterized in that: The construction simulation system further includes a collaboration module, which is used to achieve synchronous collaboration and communication among multiple users.

9. The virtual reality and augmented reality construction simulation system according to claim 7, characterized in that: The construction simulation system also includes a data synchronization and integration module, which is used to convert and adapt data of different formats and structures into a format that can be recognized and processed by the system, and at the same time process real-time data streams from the sensing module and user interaction.

Images