Digital twin system and method based on pixel streaming

Abstract

The invention discloses a digital twin system and method based on pixel streaming, and relates to the technical field of image processing, and the system comprises a data collection layer which collects the sensor data of a physical entity, the touring data of a tourist, and the environment data through a sensor network; the data processing layer is used for preprocessing the sensor data and analyzing the touring data of the tourists and the environment data; the digital twin model layer firstly constructs a geometric model of a physical entity, then maps the geometric model of the physical entity into a virtual model, and updates the virtual model; the pixel streaming layer is used for transmitting the virtual model or the updated virtual model to a virtual terminal by adopting a coding compression technology; the user interaction layer is used for receiving a user behavior instruction and real-time environment data sent by a virtual terminal, controlling the content of the virtual model sent by the pixel streaming layer and selecting to send the virtual model or the updated virtual model; the method is used for solving the problems that a traditional model is poor in authenticity and a virtual space model is low in construction efficiency.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a digital twin system and method based on pixel streaming. Background Art

[0002] With the development of technology, virtual tourism has gradually attracted attention as an emerging form of tourism. However, current virtual tourism technology has many shortcomings in terms of realism and immersion. For one thing, existing virtual tourism projects struggle to deliver an experience comparable to in-person tourism. Authentic walking postures are difficult to perceive in virtual tourism scenes, and the quality of virtual images differs significantly from that of real-world scenic spots, failing to meet tourists' expectations for virtual tourism.

[0003] On the other hand, when it comes to achieving three-dimensional restoration of virtual tourism scenes, existing technical means have defects: for example, although oblique photography technology can perform panoramic scanning of scenic spots to collect model data, it is affected by the change of light over time, and the three-dimensional point cloud is very easy to lose information when expressing the concave and convex details of the texture, resulting in the inability to accurately express the microscopic level of the close-up; at the same time, the point cloud model architecture has a huge number of polygons, the topology process is difficult and labor-intensive, and a large amount of three-dimensional model information data is lost after topology, which is not conducive to the restoration of three-dimensional scenes.

[0004] In addition, achieving walking movement in infinite virtual space within a limited physical space is also an urgent problem to be solved. Existing methods, such as walking correction algorithms, require a large physical space and the device wires are easily interfered with. The method of using virtual reality wearable devices and movement restriction devices to control the displacement posture of the human body is extremely inconvenient and costly, lacks convenience and economy, and cannot be widely promoted.

[0005] A Chinese patent with publication number CN117555426B discloses a virtual reality interaction system based on digital twin technology. Based on the display range, the system directly screens and combines the pre-acquired twin simulation data of all physical scenes and the twin simulation data of interactive physical entities in different response states for display. This eliminates the need to build and render the scene every time an interactive instruction input by the user is received, thereby reducing the amount of data processing in the virtual reality display video generation process, improving the generation efficiency of the display video in the virtual reality display device, and also improving the refresh rate of the virtual reality display video in the head-mounted display device. However, the system relies too much on pre-rendered static data, resulting in storage redundancy, poor dynamic adaptability, and high retrieval latency.

[0006] Therefore, we propose a system that can enhance tourists’ real-world experience and improve the efficiency of building virtual space models. Summary of the Invention

[0007] The purpose of the present invention is to provide a digital twin system and method based on pixel streaming, which is used to solve the problems of poor authenticity of traditional models and low efficiency in constructing virtual space models.

[0008] The present invention is achieved through the following technical solutions: The digital twin system based on pixel streaming includes a data acquisition layer that collects sensor data of physical entities, tourist travel data, and environmental data through a sensor network; The data processing layer is used to pre-process sensor data and analyze tourists' travel data and environmental data; The digital twin model layer first builds a geometric model of the physical entity, then maps the physical entity's geometric model into a virtual model based on processed sensor data, and updates the virtual model based on environmental data; The pixel streaming layer uses coding and compression technology to transmit the virtual model or the updated virtual model to the virtual terminal; The user interaction layer is used to receive user behavior instructions and real-time environmental data sent by the virtual terminal, control the virtual model content sent by the pixel streaming layer based on the user behavior instructions and the visitor's tour data, and control the pixel streaming layer to select the virtual model or the updated virtual model to be sent based on the real-time environmental data.

[0009] Furthermore, the acquisition frequency of the sensor network is adjusted according to the changing characteristics of the acquisition target and business requirements.

[0010] Furthermore, the data processing layer uses edge computing nodes to pre-process or analyze sensor data, tourists' travel data and environmental data, and the edge computing nodes are equipped with multi-core processors and caches.

[0011] Furthermore, the data processing layer analyzes and processes the collected data, specifically: Preprocessing the collected sensor data; Fusion of pre-processed sensor data based on time synchronization; Spatiotemporal grid is divided using spatiotemporal grid feature extraction algorithm; The spatiotemporal density features are extracted based on tourists' tour behavior data, and the environmental fluctuation features are extracted based on environmental data.

[0012] Furthermore, the specific steps of constructing a geometric model of a physical entity, mapping the geometric model of the physical entity into a virtual model based on the processed sensor data, and updating the virtual model based on the environmental data are as follows: Use 3D modeling software to build geometric models; Convert point cloud data in sensor data into a triangular mesh model; The geometric model and the triangular mesh model are fused through feature matching algorithm to obtain a virtual model; Update virtual models based on real-time environmental data.

[0013] Furthermore, the surface of the virtual model is rendered using a physically based rendering method.

[0014] Furthermore, a collision algorithm is provided in the virtual model.

[0015] Furthermore, the encoding compression technology adopts VP8 or VP9.

[0016] Furthermore, the user interaction layer includes a data feedback module for receiving operation instructions sent by the virtual terminal, and feeding back the operation instructions to the data processing layer and the digital twin model layer to perform corresponding adjustments.

[0017] The digital twin method based on pixel streaming includes: Use sensor networks to collect sensor data of physical entities and environmental data; Preprocess sensor data and analyze environmental data; Build a geometric model of the physical entity, then map the physical entity's geometric model into a virtual model based on the processed sensor data, and update the virtual model based on the environmental data; Using coding compression technology to transmit the virtual model or the updated virtual model to the virtual terminal; Receive user behavior instructions and real-time environment data sent by the virtual terminal; The virtual model content sent by the pixel streaming layer is controlled according to user behavior instructions, and the pixel streaming layer is controlled to select the virtual model or the updated virtual model to be sent according to real-time environmental data.

[0018] The technical solution of the present invention has at least the following advantages and beneficial effects: The present invention discloses a digital twin system and method based on pixel streaming. By adopting coding compression technology, it can effectively reduce the data transmission delay between the system and the virtual terminal, reduce network bandwidth occupancy, ensure that data can be transmitted quickly and stably, and meet the needs of real-time application scenarios.

[0019] By first constructing a geometric model based on the physical entity, then generating a triangular network model from multi-source sensor data, and then fusing the geometric model and the triangular network model, a digital twin virtual model can be constructed that can accurately reflect the geometric characteristics and operating status of the physical entity, thereby improving the authenticity of the virtual model and providing a reliable basis for subsequent analysis, decision-making and optimization. In this way, the construction speed of the virtual model is also faster. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A schematic flow chart of a method of the present invention is shown; Figure 2 A schematic diagram of a data processing layer process of the present invention; Figure 3 This is a flow chart of a digital twin model layer of the present invention. DETAILED DESCRIPTION

[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0022] Example 1 like Figure 1-Figure 3 The pixel streaming-based digital twin system shown includes a data acquisition layer that collects sensor data of physical entities, tourist data, and environmental data through a sensor network; The sensor network can clearly define the physical entities or phenomena that need to be monitored for specific application scenarios. For example, topographic features include: mountain / rock features: slope, crack displacement (accuracy ≤ 5mm), vegetation cover (resolution 0.1m / pixel); water features: water level (accuracy ±1cm), water quality parameters (pH, turbidity, sensor accuracy ±2%), and water flow velocity (ultrasonic sensor, error ≤ 5%); vegetation features: tree crown morphology (3D point cloud density 100 points / ㎡), leaf wetness (capacitive sensor, accuracy ±3%), and pest and disease areas (multispectral camera identification, accuracy ≥ 90%).

[0023] Specifically, set an appropriate data collection frequency based on the changing characteristics of the collection target and business needs. For parameters that change slowly, such as topographic features and vegetation, collection can be done every 10-15 hours; for parameters that require real-time monitoring, such as water bodies and ambient temperature, collection can be done every 15-20 minutes.

[0024] In addition, tourists’ travel data includes their location trajectory and length of stay, and environmental data includes temperature, humidity, light and weather.

[0025] In addition, the data transmission in this system needs to be transmitted through the transmission network, which includes wired networks and wireless networks. The wired network adopts the deployment of 10 Gigabit Ethernet (10Gbps) + single-mode optical fiber to build a star topology. It uses core switches (such as Huawei CE6865) to directly connect edge computing nodes and cloud servers to ensure zero packet loss transmission of sensor data and environmental data to the data processing layer.

[0026] The wireless network adopts 5G+Wi-Fi 6 hybrid wireless coverage, mainly realizing the transmission of virtual models between this system and virtual terminals. The 5G private network (Sub-6GHz) is: Deploy Huawei's 5G AAU base station, supporting 300Mbps downlink bandwidth, meeting the requirements of real-time streaming transmission (single-channel bit rate ≤ 120Mbps) for 8K panoramic cameras (30FPS), and end-to-end latency ≤ 50ms.

[0027] For areas with high tourist density (such as observation decks), 5G slicing technology is enabled to allocate dedicated bandwidth (≥50Mbps) for VR / AR devices to ensure stable transmission of binocular stereo streams (2K×2K per eye, 90Hz).

[0028] Wi-Fi 6 hotspots (802.11ax) are: Deploy high-density APs (such as Cisco Catalyst 9130) in rest areas and visitor centers. A single AP supports 200+ concurrent connections, with a transmission rate of ≥1.2Gbps, and is suitable for 1080P streaming loading (bitrate ≤6Mbps) and offline cache updates for mobile apps.

[0029] The data processing layer is used to pre-process sensor data and analyze tourists' travel data and environmental data; The preprocessing process specifically includes: Noise removal: Filtering algorithms are used to address noise in the collected data. For time series data, such as temperature and pressure data collected by sensors over time, a moving average filter can be used. An appropriate window size is set, for example, using five time points as a window. The average value of the data within the window is calculated and used to replace the data at the middle time point of the window. This smoothes the data and removes random noise. For noise in image data, a median filter algorithm can be used to replace the value of each pixel in the image with the median value of its neighboring pixels, effectively removing salt and pepper noise, etc.

[0030] Outlier processing: Data thresholds are set to identify outliers. For outlier processing, if the outlier is due to sensor failure, it can be eliminated. If the data is likely to be caused by a sudden equipment abnormality, it can be marked and further analyzed and judged in conjunction with other equipment operating parameters. Statistical methods can also be used, such as Z-score-based outlier detection. A data point is identified as an outlier when its absolute Z-score exceeds a certain threshold (e.g., 3).

[0031] Sensor calibration: Regularly calibrate sensors to ensure data accuracy. For temperature sensors, compare their measurements with those of a standard thermometer under the same conditions, calculate the deviation, and then use this deviation to correct subsequent sensor measurements. For some high-precision sensors, return them to the manufacturer or a professional calibration agency for recalibration.

[0032] Data normalization: When processing multiple data sources with different dimensions, data normalization is performed. The minimum-maximum normalization method is used to linearly transform the data of each indicator to the [0,1] interval. The formula is:

[0033] in, is the original data, and These are the minimum and maximum values ​​of the indicator data. Normalization makes the data of different indicators comparable, facilitating subsequent data analysis and modeling. Finally, the normalized sensor data are fused based on time synchronization.

[0034] The process of analyzing tourists' travel data and environmental data is as follows: First, the scenic area's geographic space is divided into 5m×5m dynamic grids. Each grid is associated with 3D coordinates, timestamps (accurate to 10ms), and multi-source data labels (sensor type, visitor interaction type) to form spatiotemporal feature primitives. ,in: are the grid space coordinates, is a time series (resolution 1 second), For sensor data (such as temperature and humidity , passenger flow density ); Then extract the spatiotemporal density features: Calculate the length of time tourists stay in the grid and visit frequency , construct the density matrix:

[0035] in ,and , , and i and j represent the grid row and column indexes, which are used to locate the scenic spot and calculate the tourist density Used to identify popular attractions ( ) and unpopular areas ( ), when users arrive at popular attractions, they drive the pre-loading of digital twin virtual models.

[0036] Finally, extract the environmental fluctuation characteristics: Perform differential processing on environmental data such as temperature, humidity, and light within the grid to extract dynamic fluctuation characteristics:

[0037] in is the historical average, is the environmental sensor weight coefficient, To sum index variables, is the total number of environmental data points, and sgn(·) is a sign function used to detect sudden environmental changes (such as a sudden increase in humidity caused by rainfall). That is, when an environmental change occurs, the weather effects of the digital twin virtual model are triggered to render in real time. Represents a timestamp index, Indicates that the current value is higher than the previous moment, Indicates that the current value is lower than the previous value.

[0038] It should be noted that after preprocessing and analysis, the data will be initially stored. For data with a high degree of structure and clear correlation between data, relational databases such as MySQL and Oracle are selected; for unstructured or semi-structured data, as well as scenarios with high requirements for data read and write performance, non-relational databases are used.

[0039] In addition, develop a sound data backup strategy, including full and incremental backups. For important data, perform a full backup weekly and incremental backups daily. A full backup copies all data to the backup storage device, while an incremental backup only backs up data that has changed since the last backup. This ensures data integrity while reducing backup time and storage space.

[0040] Regular data recovery drills are conducted to simulate data loss or corruption scenarios and verify the availability of backup data and the correctness of the recovery process. During these drills, metrics such as recovery time and recovered data integrity are recorded to continuously optimize the recovery process. Recovered data is also verified to ensure accuracy and consistency, such as through data checksums and comparisons with original data.

[0041] The digital twin model layer first builds a geometric model of the physical entity, then maps the physical entity's geometric model into a virtual model based on processed sensor data, and updates the virtual model based on environmental data; Specifically including: using 3D modeling software to build geometric models; Convert point cloud data in sensor data into a triangular mesh model; The geometric model and the triangular mesh model are fused through feature matching algorithm to obtain a virtual model; Update virtual models based on real-time environmental data.

[0042] Using professional 3D modeling software (such as 3ds Max and Blender) or point cloud-based modeling tools (such as MeshLab), a basic geometric model is constructed based on preprocessed point cloud data or image data obtained by scanning the target scenic area with a laser scanner. A point cloud meshing algorithm is then used to convert the discrete point cloud data into a continuous triangular mesh model, initially outlining the object's shape. During the modeling process, the mesh accuracy is appropriately set, and the basic geometric model and the triangular mesh model are fused using a feature matching algorithm to create a virtual model. This processing method balances model accuracy with computational efficiency.

[0043] In addition, you can add details to the virtual model, such as using texture images acquired through photogrammetry and applying texture mapping technology to the model surface to make the model more realistic. You can also refine key parts of the model and smooth the model to eliminate jagged edges and blemishes on the surface, improving the visual effect.

[0044] In addition, the virtual model is dynamically updated based on the analyzed environmental data to obtain real-time rendering of weather effects.

[0045] Physically Based Rendering (PBR) parameters are used to render the surface of the virtual model: Rendering includes defining the diffuse color: based on texture image mean sampling, with an error of ≤2% of the RGB value; Definition of roughness: Calculated by image gradient, the formula is:

[0046] Where N is the pixel value and I is the grayscale value.

[0047] Define lighting baking: Use UE engine Lightmass for global illumination baking, generate lightmaps (resolution 256px / m²), and support real-time light and shadow tracing (delay ≤ 10ms).

[0048] In addition, the digital twin virtual model can have the function of interactive feedback. The specific logic is as follows: Visual feedback: Interactive operations trigger model highlighting (e.g., when clicking on an artifact, the edge flashes yellow for 2 seconds), achieved through dynamic material switching (Shader switching delay ≤ 5ms).

[0049] Auditory feedback: Plays corresponding sound effects based on the interaction type (e.g., a "clacking" sound is played when stepping on a cobblestone road, with the volume attenuating with distance).

[0050] Haptic feedback: The VR terminal's handle adjusts vibration intensity based on the weight of the object being grasped (50Hz for light objects, 200Hz for heavy objects, with an error of ≤10%).

[0051] The pixel streaming layer uses coding and compression technology to transmit the virtual model or the updated virtual model to the virtual terminal; In particular, the coding compression technology adopts VP8 or VP9, ​​which can significantly reduce the amount of data transmission while ensuring image quality, lower the requirement for network bandwidth, and ensure smooth pixel stream transmission in different network environments; It should be noted that the virtual terminal includes the PC terminal, which supports users to access and interact with the digital twin system through a personal computer; Mobile terminal, that is, allowing users of mobile devices such as smartphones and tablets to experience the virtual tour and other functions of this digital twin system; VR / AR devices provide immersive interactive experiences for virtual reality (VR) and augmented reality (AR) devices. These devices usually need to process large amounts of graphics information to create realistic virtual environments.

[0052] The user interaction layer is used to receive user behavior instructions and real-time environment data sent by the virtual terminal, control the virtual model content sent by the pixel streaming layer according to the user behavior instructions, and control the pixel streaming layer to select the virtual model or updated virtual model to be sent according to the real-time environment data; The user's current physical location is determined based on user behavior instructions. User behavior instructions are VR device gesture recognition. The VR device determines user movement, mobile terminal click operations, or PC terminal mouse movement instructions. The location is transmitted to the virtual terminal through the pixel streaming layer. This does not affect the virtual model content displayed by the virtual device, and can ensure the speed of virtual model transmission. As for real-time environmental data, if it is judged based on the real-time environmental data that the current scenic area environment reaches the above-mentioned sudden environmental changes, the real-time rendering of the weather effects of the digital twin virtual model will be triggered, that is, the updated virtual model, that is, the pixel streaming layer is controlled to stop transmitting the general virtual model to the virtual terminal, and at the same time, the pixel streaming layer is controlled to transmit the updated virtual model to the virtual terminal.

[0053] In addition, the user interaction layer includes a data feedback module, which is used to receive operation instructions sent by the virtual terminal and feed back the operation instructions to the data processing layer and the digital twin model layer to perform corresponding adjustments.

[0054] Example 2 As an embodiment, a digital twin method based on pixel streaming specifically includes: Use sensor networks to collect sensor data of physical entities and environmental data; Preprocess sensor data and analyze environmental data; Build a geometric model of the physical entity, then map the physical entity's geometric model into a virtual model based on the processed sensor data, and update the virtual model based on the environmental data; Using coding compression technology to transmit the virtual model or the updated virtual model to the virtual terminal; Receive user behavior instructions and real-time environment data sent by the virtual terminal; The virtual model content sent by the pixel streaming layer is controlled according to user behavior instructions, and the pixel streaming layer is controlled to select the virtual model or the updated virtual model to be sent according to real-time environmental data.

[0055] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A digital twin system based on pixel streaming, characterized by: It includes a data collection layer that collects sensor data of physical entities, tourists' travel data and environmental data through sensor networks; The data processing layer is used to pre-process sensor data and analyze tourists' travel data and environmental data; The digital twin model layer first builds a geometric model of the physical entity, then maps the physical entity's geometric model into a virtual model based on processed sensor data, and updates the virtual model based on environmental data; The pixel streaming layer uses coding and compression technology to transmit the virtual model or the updated virtual model to the virtual terminal; The user interaction layer is used to receive user behavior instructions and real-time environmental data sent by the virtual terminal, control the virtual model content sent by the pixel streaming layer based on the user behavior instructions and the visitor's tour data, and control the pixel streaming layer to select the virtual model or the updated virtual model to be sent based on the real-time environmental data.

2. The digital twin system based on pixel streaming according to claim 1, characterized in that: The acquisition frequency of the sensor network is adjusted according to the changing characteristics of the acquisition target and business requirements.

3. The digital twin system based on pixel streaming according to claim 1, characterized in that: The data processing layer uses edge computing nodes to pre-process or analyze sensor data, tourists' travel data and environmental data, and the edge computing nodes are equipped with multi-core processors and caches.

4. The digital twin system based on pixel streaming according to claim 1, characterized in that: The data processing layer performs the following steps to analyze and process the collected data: Preprocessing the collected sensor data; Fusion of pre-processed sensor data based on time synchronization; Spatiotemporal grid is divided using spatiotemporal grid feature extraction algorithm; The spatiotemporal density features are extracted based on tourists' tour behavior data, and the environmental fluctuation features are extracted based on environmental data.

5. The digital twin system based on pixel streaming according to claim 1, characterized in that: The specific steps of constructing a geometric model of a physical entity, mapping the geometric model of the physical entity into a virtual model based on the processed sensor data, and updating the virtual model based on the environmental data are as follows: Use 3D modeling software to build geometric models; Convert point cloud data in sensor data into a triangular mesh model; The geometric model and the triangular mesh model are fused through feature matching algorithm to obtain a virtual model; Update virtual models based on real-time environmental data.

6. The digital twin system based on pixel streaming according to claim 1, characterized in that: The surface of the virtual model is rendered using a physically based rendering method.

7. The digital twin system based on pixel streaming according to claim 1, characterized in that: A collision algorithm is provided in the virtual model.

8. The digital twin system based on pixel streaming according to claim 1, characterized in that: The coding compression technology adopts VP8 or VP9.

9. The digital twin system based on pixel streaming according to claim 1, characterized in that: The user interaction layer includes a data feedback module, which is used to receive operation instructions sent by the virtual terminal and feed the operation instructions back to the data processing layer and the digital twin model layer to perform corresponding adjustments.

10. A digital twin method based on pixel streaming, characterized in that: Specifically include: Use sensor networks to collect sensor data of physical entities and environmental data; Preprocess sensor data and analyze environmental data; Build a geometric model of the physical entity, then map the physical entity's geometric model into a virtual model based on the processed sensor data, and update the virtual model based on the environmental data; Using coding compression technology to transmit the virtual model or the updated virtual model to the virtual terminal; Receive user behavior instructions and real-time environment data sent by the virtual terminal; The virtual model content sent by the pixel streaming layer is controlled according to user behavior instructions, and the pixel streaming layer is controlled to select the virtual model or the updated virtual model to be sent according to real-time environmental data.

Images