Virtual-real double-wheel data-driven real-time visualization method, device and medium

Abstract

The application provides a virtual-real double-wheel data-driven real-time visualization method and device, and a medium, and has the characteristics that multi-source physical sensor data streams are received and analyzed in real time, CAD design data and CAE simulation data frames are received and analyzed, and dynamic association of the three is established through a virtual channel ID; a lightweight treatment is performed on an original three-dimensional design model, and a high-precision space mapping relationship between physical measuring points and vertexes of the lightweight model is established; a multi-level rendering pipeline is constructed, real-time data streams are bound to corresponding vertexes of the model, dynamic particle special effects are synchronously driven in a three-dimensional scene, and two-dimensional analysis charts are superimposed and rendered; and based on user interaction instructions, intelligent switching is performed between a plurality of preset visualization modes. The application realizes high-precision and low-delay fusion visualization of test data and simulation models, and effectively improves the monitoring efficiency and decision support capability of a test process of a complex system such as a spacecraft.

Description

Technical Field

[0001] This document relates to the field of data visualization technology, and in particular to a real-time visualization method, device and medium driven by both virtual and real data. Background Technology

[0002] In the design, testing, and manufacturing of complex products such as spacecraft and high-end equipment, real-time, fused, and high-precision visualization of physical test and digital simulation data is a crucial step for status monitoring, anomaly diagnosis, and decision support. However, existing traditional test data visualization technologies suffer from the following major pain points and technical bottlenecks: 1. The problem of data silos is serious, lacking dynamic correlation: Sensor-acquired measured data, CAE simulation analysis data, and 3D models are typically processed and displayed by different independent systems. The lack of an effective dynamic correlation mechanism between these data makes real-time comparison and analysis impossible. For example, CAE simulation results cannot be dynamically compared with rapidly changing measured data within a unified scene at the millisecond level, making it difficult for designers to quickly discern differences between virtual and real data and pinpoint the root cause of problems.

[0003] 2. Inefficient visualization and insufficient real-time performance: Current 3D model refresh rates are generally less than 20Hz, making it difficult to achieve smooth dynamic effects. When loading large-scale CAD models, the latency typically exceeds 3 seconds, severely impacting the real-time performance of monitoring. This performance bottleneck cannot meet the demands of high frame rate and low latency real-time data monitoring, posing a significant risk, especially in experimental scenarios requiring rapid response.

[0004] 3. Weak scenario generalization ability and fragmented multi-dimensional information: Current display methods often separate two-dimensional charts from three-dimensional model views, requiring users to manually switch between different views and software interfaces. This information fragmentation prevents users from intuitively and simultaneously viewing multi-dimensional information such as heat flux distribution cloud maps, strain curves, and spectrum analysis within a three-dimensional scene, significantly increasing cognitive load and reducing analysis efficiency.

[0005] For example, existing patents CN202311687519.7 proposes a method, device, product, and medium for three-dimensional visualization of physical field data based on WebGL, and CN202211555073.8 proposes a method for online display of finite element simulation results based on WebGL. Although these methods improve versatility, they only perform three-dimensional visualization of physical field model data and finite element model data. They lack real-time modeling and display of real-time experimental data, as well as the fusion and matching with simulation data and the fusion display of simulation models. Their accuracy and response timeliness, among other key indicators, are insufficient to meet the stringent requirements of high precision and high real-time performance in fields such as aerospace and aviation.

[0006] In summary, existing technologies cannot effectively support the high-precision, real-time, integrated, and autonomously controllable virtual and physical data visualization requirements of high-end industrial sectors, especially the aerospace field. Therefore, there is an urgent need for an innovative method and facility to break down data barriers, improve visualization efficiency and accuracy, and achieve autonomous, secure, and controllable operation throughout the entire process. Summary of the Invention

[0007] This invention provides a real-time visualization method, device, and medium driven by both virtual and real data, aiming to solve the above-mentioned problems.

[0008] According to an embodiment of the present invention, a real-time visualization method driven by both virtual and real data is provided, comprising: The system receives real-time data streams from multiple sensors via a high-speed data bus, associates the spatial location of physical measurement points with virtual channel IDs, matches the corresponding CAD model spatial coordinates, synchronously parses their time series labels, dynamically matches the corresponding CAE simulation data frames, encapsulates the data using a lightweight protocol, and writes it in parallel to a dual storage system. The dual storage system consists of a database that supports structured queries and a database that supports unstructured storage. The original 3D design model is lightweighted, and a high-precision mapping relationship is established between the physical measurement point positions and the vertices of the lightweight model. Error correction is applied through spatial coordinate system meshing algorithm and measurement point coordinate inverse correction algorithm to control the mapping error within the preset accuracy range. A multi-layered rendering pipeline, including a base layer, a dynamic layer, and an overlay layer, is constructed to achieve real-time data-driven dynamic particle effects and two-dimensional analysis charts overlaid in the same frame in a 3D scene.

[0009] According to an embodiment of the present invention, an electronic device is provided, comprising: Processor; and, A memory is configured to store computer-executable instructions, which, when executed, cause the processor to perform the steps of the aforementioned real-time visualization method driven by virtual and real dual-wheel data.

[0010] According to an embodiment of the present invention, a storage medium is provided for storing computer-executable instructions, which, when executed, implement the steps of the above-described real-time visualization method driven by virtual and real dual-wheel data.

[0011] This invention, through a multi-layered hybrid rendering pipeline, achieves seamless overlay of 2D charts, 3D models, and particle effects from the same viewpoint, completely resolving industry pain points such as view separation and manual switching, and realizing true integrated visualization. While designing the parallel processing pipeline, the core mathematical library of the engine is manually assembled or inlined for optimization, resulting in several times the performance improvement. Through spatial mapping algorithms, data mapping errors are controlled to the millimeter level, meeting aerospace-grade rendering accuracy requirements. From the underlying hardware and operating system to the upper-level rendering engine, the entire chain has been adapted and optimized for domestic production, with core technologies being independent and free from "bottleneck" risks, achieving complete self-reliance and control. It provides multi-modal interaction methods such as touch and voice, and supports multiple view presets such as expert mode and command mode, greatly improving user experience and decision-making efficiency, and providing a powerful interactive experience. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in one or more embodiments of this specification or in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart of a real-time visualization method driven by virtual and real dual-wheel data, according to an embodiment of the present invention. Figure 2 This is a schematic diagram of a typical scheme of the real-time visualization method driven by virtual and real dual-wheel data according to an embodiment of the present invention; Figure 3 This is a flowchart illustrating the data receiving and preprocessing process according to an embodiment of the present invention. Figure 4 This is a flowchart of the data processing engine in an embodiment of the present invention; Figure 5 This is a flowchart of the three-dimensional rendering module according to an embodiment of the present invention; Figure 6 This is an interactive control flowchart of an embodiment of the present invention. Detailed Implementation

[0014] To enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of this document.

[0015] Method Implementation Examples According to embodiments of the present invention, a real-time visualization method driven by both virtual and real data is provided. Figure 1 This is a flowchart of a real-time visualization method driven by virtual and real dual-wheel data according to an embodiment of the present invention. Figure 1 As can be seen, the real-time visualization method driven by virtual and real dual-wheel data in this embodiment of the invention specifically includes: S1. Receive real-time data streams from multiple sources via a high-speed data bus, associate the spatial location of physical measurement points based on virtual channel IDs, match the corresponding CAD model spatial coordinates, synchronously parse their time series labels, dynamically match the corresponding CAE simulation data frames, encapsulate the data using a lightweight protocol, and write it in parallel to a dual storage system. The dual storage system consists of a database that supports structured queries and a database that supports unstructured storage. Furthermore, the CAE simulation data frame corresponding to the dynamic matching specifically includes: The CAE simulation data frames are preloaded into the managed memory state of the real-time stream processing engine and organized using the virtual channel ID and time period as primary keys. During data processing, the collaborative processing function or interval connection operator of the real-time stream processing engine is used to find and associate the most matching CAE simulation data within the corresponding time window, based on the timestamp of the multi-source sensor data stream, so as to achieve millisecond-level time alignment between the two.

[0016] Furthermore, the parallel writing to the dual-storage system consisting of a relational database and a file database is specifically implemented as follows: The parsed and encapsulated data is written to both a relational database and a file database simultaneously. The relational database is used to support high-frequency structured data queries and transaction operations, while the file database is used to store unstructured or semi-structured time-series data and document data to achieve high data availability and differentiated query efficiency.

[0017] S2. Lightweighting the original 3D design model, establishing a high-precision mapping relationship between the physical measurement point positions and the vertices of the lightweight model, and applying error correction through the spatial coordinate system meshing algorithm and the measurement point coordinate inverse correction algorithm to control the mapping error within the preset accuracy range. The specific process of applying error correction through spatial coordinate system gridding algorithm and measurement point coordinate inverse correction algorithm is as follows: The vertex space of the lightweight model surface is divided into a grid to form a spatial coordinate grid; Based on the actual physical coordinates of the physical measurement points, interpolation calculations are performed in the spatial coordinate grid to determine the model vertices of the initial mapping. The reverse correction algorithm calculates and applies an error correction amount based on interpolation compensation or least squares principle, accurately mapping the data of the physical measurement point to the correction position of the initial mapping vertex, wherein the error correction amount is controlled to be no greater than 0.1 mm.

[0018] Specifically: (a) Constructing the error objective function: Let the actual physical coordinates of the physical measurement point be the first coordinate point, and the preliminary mapped vertex coordinates obtained through interpolation be the second coordinate point. Define the spatial difference between the two, and minimize the sum of squares of this difference as the optimization objective; (b) Local Surface Fitting and Normal Projection: Centered on the initially mapped vertex, several adjacent vertices around it are selected on the surface of the lightweight model to form a local point set. A surface fitting algorithm is used to perform quadratic surface fitting on this local point set to obtain the local parametric surface of the region. The physical measurement points are then projected perpendicularly onto the surface along the normal direction to obtain the projection points; (c) Calculation of Inverse Error Correction: Calculate the spatial offset between the projected point and the initial mapped vertex on the model surface; this is the error correction for the current iteration step. Apply this correction to the initial mapped vertex to obtain the new mapped position. (d) Iterative optimization and accuracy control: Repeat steps (b) and (c), using the newly obtained mapping position as the current initial mapping vertex for iterative calculation, until the correction magnitude obtained in two adjacent iterations is less than a preset threshold. The final position obtained at this point is the accurate mapping vertex. Through the above iterative optimization process, the data of the physical measurement points are accurately mapped to the final mapping vertex, and the final total mapping error is controlled to be no greater than 0.1 mm.

[0019] S3. Construct a multi-layered rendering pipeline including a base layer, a dynamic layer, and an overlay layer to achieve real-time data-driven dynamic particle effects and two-dimensional analysis charts overlaid in the same frame in a 3D scene.

[0020] The construction of the multi-level rendering pipeline specifically includes: The base layer is responsible for rendering high-fidelity 3D models; The dynamic layer binds the real-time data stream to the corresponding vertex of the model, driving the dynamic particle effects to change dynamically as the data changes, wherein the dynamic particle effects include at least one of tail flame simulation and fuel leak simulation. The overlay layer synchronously overlays and renders two-dimensional analysis charts from real-time data or processing results within the same viewport of the three-dimensional scene. The two-dimensional analysis charts include at least one of line charts, bar charts, and heat flow distribution cloud maps.

[0021] The real-time visualization method driven by virtual and real dual-wheel data in this embodiment of the invention also includes an intelligent switching step for multiple view modes: It switches between different visualization modes in response to user commands or preset conditions; In expert mode, a 3D cloud map, a spectrum analysis map, and key parameter curves are displayed simultaneously on the same interface. In command mode, the screen is displayed in full screen, focusing on rendering the macroscopic situation of the ballistic trajectory, and supplemented by key parameter dashboards at the edge of the screen or in the overlay.

[0022] In the step of constructing a multi-layered rendering pipeline, rendering efficiency and frame rate stability are ensured by implementing rendering optimization strategies, specifically including: On the central processing unit, all objects in the 3D scene are traversed, and a fast intersection test is performed using their bounding boxes and the camera's view frustum to skip rendering invisible objects; at the same time, models of different detail levels are dynamically selected for rendering based on the distance between the objects and the camera. Before submitting the rendering command, objects with the same material and texture will be batched together to reduce the number of draw calls.

[0023] The real-time visualization method driven by virtual and real dual-wheel data in this embodiment of the invention also includes a data security transmission step: before sending the encapsulated data through the network, the built-in national cryptographic algorithm acceleration engine is used to perform SM4 encryption algorithm on the data packet to achieve hardware-level real-time data encryption and ensure the security and controllability of the data transmission process.

[0024] like Figure 2 This is a schematic diagram of a typical embodiment of the present invention. Figure 2 As can be seen, the real-time visualization method driven by virtual and real dual-wheel data described in this invention includes the following steps: 1. Real-time reception and parsing of experimental data: Data streams from multiple sensors, including temperature, strain, and vibration signals, are received via a high-speed data bus (network latency ≤50μs), and their time-series labels are parsed simultaneously. Corresponding CAE simulation data frames are dynamically matched, using virtual channel IDs to associate the spatial coordinates of physical measurement points during matching. Figure 3 This is a flowchart of data reception and preprocessing according to an embodiment of the present invention. Figure 3 As can be seen, this module receives data from multiple industrial buses, ultimately packages and sends it to a high-speed network, and processes multiple types of data in parallel to meet the requirements of low latency and high reliability for aerospace test data. The core processing flow is as follows: (1) Data reception and hardware preprocessing layer a. Multi-protocol interface access: The module connects directly to the physical bus via a dedicated protocol controller chip (such as the domestically produced 1553B or RS422 protocol chip) and a PXIe interface chip.

[0025] Each protocol controller operates independently and receives data in parallel, achieving load distribution of high-speed data streams.

[0026] b. FPGA / CPLD coprocessing: The received raw data stream first enters the FPGA or CPLD (Field-Programmable Gate Array / Complex Programmable Logic Device) for low-level preprocessing. Functions include: Frame verification: Perform CRC verification on 1553B and RS422 data frames, discard erroneous frames, and ensure data validity.

[0027] Protocol parsing: Parses specific frame headers, addresses, instruction words, data words, and other information of each bus protocol, strips away the protocol shell, and extracts the payload data.

[0028] Data Packaging: Data from different bus sources is packaged into a unified internal format data packet after being tagged with internal tags (source, timestamp).

[0029] Data buffer: A FIFO (First In First Out) buffer is established within the chip to cope with the impact of instantaneous data flow.

[0030] (2) Loongson core processing and data transmission layer a. High-speed data transmission to the Loongson core: The preprocessed data packets are directly written to the memory buffer specified by the Loongson chip via PCIe or Loongson's proprietary LSPI high-speed interface using DMA (Direct Memory Access). Employing DMA eliminates the need for CPU intervention in data transfer, significantly reducing CPU utilization and ensuring real-time performance and processing efficiency.

[0031] b. Software data processing: The real-time operating system (such as Loongnix RT) and dedicated drivers running on the Loongson chip read data from the memory buffer. The application layer program performs the following functions: Data filtering / cleaning: Filter invalid data according to preset rules.

[0032] Precise timestamp alignment: Data from different buses is given a unified and precise timestamp (using the high-precision timer of the Loongson chip), providing a foundation for subsequent multi-source data fusion analysis.

[0033] Data compression: An optional step to perform lossless or lightweight compression of data, reducing network transmission pressure.

[0034] Encapsulation and Packet Assembly: Encapsulate the processed data into a format suitable for network transmission (such as Google Protobuf, custom binary format).

[0035] c. Hardware-level security encryption: By utilizing the national cryptographic algorithm acceleration engine built into the Loongson chip, encryption algorithms such as SM4 are directly executed on the data in memory to achieve real-time hardware-level encryption of the data, ensuring data security. This step can be performed before or after packet assembly.

[0036] (3) Global Management and Support Layer a. Network output: The finally processed data packet is sent to the host computer, data center or digital twin platform via TCP / IP or UDP protocol through the gigabit Ethernet controller integrated in the Loongson chip.

[0037] To ensure real-time performance, the UDP protocol is typically used with a custom application-layer retransmission mechanism, or a high-priority TCP channel is employed.

[0038] b. System Management and Monitoring: The entire process is uniformly scheduled and monitored by the operating system.

[0039] Watchdog timer: Monitors program execution status to prevent crashes.

[0040] Status reporting: The module can report its own health status, load status, error logs and other information to the host.

[0041] Dynamic configuration: The host can dynamically issue commands to configure the parameters of each bus port (such as baud rate, terminal address, etc.).

[0042] Data is encapsulated using a lightweight JSON protocol and written in parallel to a dual-storage system consisting of a relational database (MySQL) and a file database (MongoDB) to ensure high data availability and query efficiency.

[0043] 2. Dynamic mapping of 3D models: Based on a self-developed open-kernel CAD model lightweight engine, which supports Catia, SolidWorks, FreeCAD, etc., the engine converts original 3D design models into an editable lightweight format, achieving file size compression of over 80%. It establishes a high-precision mapping relationship between physical measurement point positions and the vertices of the lightweight model, achieving an error correction Δx ≤ 0.1mm. Figure 4 This is a flowchart of the data processing engine according to an embodiment of the present invention. Figure 4 As can be seen, this engine adopts a typical Lambda architecture, combining the speed layer and the batch layer to simultaneously meet the needs of real-time stream processing and big data batch processing. The core processing flow of the data processing engine is as follows: (1) Data access and buffering submodule: To decouple data production and consumption, handle traffic spikes, and ensure no data loss, this submodule first writes all raw data to an Apache Kafka cluster, persisting massive amounts of data with millisecond-level latency. The embedded acquisition terminal does not need to wait for a response from the processing end; it can continue acquisition immediately after successful transmission, greatly improving the overall system's disaster recovery capabilities and throughput. Kafka's persistence mechanism ensures that data is not lost in the event of a data processing engine restart or failure. This significantly enhances the high reliability of aerospace test data acquisition.

[0044] (2) Real-time stream processing submodule To enable real-time processing of data streams at the millisecond to second level and to implement business logic, this submodule uses Apache Flink as its core engine to achieve high throughput and low latency. The design perfectly balances both, easily achieving a throughput requirement of ≥1Gbps, ensuring that each piece of data is processed accurately once even in the event of a failure, without omission or duplication, meeting the rigid requirements of aerospace-grade applications. At the same time, it has built-in state storage, which facilitates complex operations such as window aggregation and pattern matching.

[0045] The data processing pipeline implementation process is as follows: a. Data Analysis: Consume raw byte data from Kafka.

[0046] Deserialization and decoding are performed according to predefined formats (such as JSON, Protobuf, and custom binary protocols).

[0047] Perform data cleaning, validation (such as checking field integrity and value range validity), and standardization (such as unifying the timestamp format).

[0048] b. Dynamic matching of virtual and real data: Based on the "virtual channel ID" and high-precision timestamp in the data packet, real-time physical sensor data is associated and matched with CAE simulation data frames pre-loaded into memory.

[0049] Data loading: Simulation data is preloaded into Flink's managed state key, which is the virtual channel ID and the time period.

[0050] Time alignment: Using operators such as Co-Process Function or Interval Join, the most matching simulation data within the corresponding time window is found based on the timestamp of real-time data to achieve millisecond-level alignment.

[0051] Rule matching: Use Flink's CEP (Complex Event Processing) library to detect complex event patterns in the data stream (e.g., alarm if three consecutive vibrations exceed the limit).

[0052] Related queries: Through the Async I / O function, asynchronous queries can be made to external databases (such as Redis) or dimension tables to enrich the data (such as querying the location information of a device based on its ID).

[0053] Window calculation: Real-time aggregation of data within sliding and scrolling windows (e.g., calculating average temperature and maximum pressure per second).

[0054] c. Packaging and Distribution: The processed result data is encapsulated into the target format.

[0055] Write to multiple sinks (output terminals) in parallel: Real-time storage: Writes to Redis or ClickHouse to support low-latency queries for real-time monitoring dashboards.

[0056] Batch storage: Write data to a data lake warehouse (such as Apache Iceberg) to provide a high-quality, structured data foundation for subsequent batch analysis and machine learning.

[0057] Message queue: Can be written to Kafka again for consumption by other downstream systems (such as alarm systems and business systems).

[0058] (3) Batch processing and persistence submodule This submodule performs deep and complex offline computations on massive amounts of historical data, compensating for the limitations of stream processing in terms of computing power. Data stored in the data lake warehouse is processed periodically in batches by Apache Spark. This includes batch verification and repair of data quality, construction of more complex machine learning features, and generation of historical statistical reports and aggregated datasets. The results can then be stored in a data warehouse or OLAP database for querying by the visualization module.

[0059] (4) Cluster Management and Support Submodule This submodule is mainly used for service coordination, monitoring optimization, and configuration management.

[0060] a. Coordination services The main tasks include master node election, configuration management, and high availability management of Flink / Spark clusters based on ZooKeeper or Etcd.

[0061] b. Monitoring and Optimization: Metrics collection: Use Prometheus to collect various metrics of the Flink / Spark cluster (throughput, latency, backpressure, CPU / Memory utilization).

[0062] Visualization and Alerts: Use Grafana to create a monitoring dashboard and set alert rules (such as high latency or sudden drop in throughput).

[0063] JVM optimization: JVM tuning for Flink / Spark TaskManager / Executor, including: Use the G1 or ZGC garbage collector to minimize GC pause time.

[0064] Properly configure the ratio of memory inside and outside the heap to avoid OutOfMemoryError (OOM).

[0065] Configure code caching and JIT compilation parameters to improve execution efficiency.

[0066] c. Configuration Management: All data parsing rules, matching rules, and computational logic (such as Flink jobs) should be versioned and managed through a configuration center to achieve dynamic updates without downtime.

[0067] 3. Visualization of virtual vs. real: Construct a multi-layered rendering pipeline: Base layer: Renders high-fidelity 3D models. Dynamic layer: Binds real-time data streams to corresponding vertices of the model, driving dynamic changes in particle effects (such as exhaust plumes and fuel leaks). Overlay layer: Synchronously overlays and renders 2D analysis charts (such as line charts, bar charts, and cloud maps) in the 3D scene. Supports intelligent switching between multiple view modes: Expert mode: Simultaneously displays multi-dimensional professional views such as 3D cloud maps and spectrum analysis charts. Command mode: Full-screen display of macroscopic situations such as ballistic trajectories, supplemented by key parameter dashboards.

[0068] The facilities for implementing the above method include the following modules: 1) Data receiving module: The embedded acquisition terminal is equipped with a domestic Loongarch architecture chip and supports multiple industrial bus protocols such as RS422, 1553B, and PXIe.

[0069] 2) Data processing engine: A real-time computing cluster based on JVM optimization, with a data processing throughput of ≥1Gbps, responsible for data parsing, matching, encapsulation and storage.

[0070] 3) 3D rendering module: A domestically developed rendering engine based on Unreal Engine, adapted to the Loongnix operating system, responsible for achieving 40Hz 3D scene refresh at 4K resolution.

[0071] 4) Interactive control terminal: A multi-screen workstation that supports touch and voice commands (resolution supports 7680×4320), providing an immersive interactive experience.

[0072] like Figure 5 This is a flowchart of the 3D rendering module in an embodiment of the present invention. This module, through a carefully designed parallel processing pipeline, ensures stable 4K@40Hz rendering output on a domestically produced hardware and software platform. Its core process is as follows: (1) Resource loading and preprocessing submodule This submodule efficiently loads 3D assets (models, textures, materials) into memory and converts them into a format usable by the engine. It includes: Format parsing: Develop domestically produced model and texture format parsing plugins to replace certain native format support in UE, ensuring that the data source is independent and controllable.

[0073] High-precision spatial data processing: Model data: Parse vertices, normals, UV coordinates, and index buffers, and map data (such as temperature and strain) to model vertex displacement, color change, or material parameters (such as self-illumination intensity) to dynamically generate 3D cloud maps and upload them to GPU memory.

[0074] High-precision spatial coordinate mapping and error correction: Using a spatial coordinate system gridding algorithm and a measurement point coordinate inverse correction algorithm, interpolation calculations are performed in the vertex grid of the model based on the physical coordinates of the measurement points, and an error correction amount (Δx≤0.1mm) is applied, so that the data points are accurately "attached" to the correct position on the model surface.

[0075] Texture data: Decode image files (such as PNG) into GPU-supported formats (such as BCn / ASTC) and generate a Mipmap chain.

[0076] Material data: Read the material definition file (such as JSON), dynamically create the corresponding material instance within the engine, and associate it with the corresponding shader and texture.

[0077] Resource Management: All resources are stored in a unified resource pool, and reference counting and lifecycle management are performed to avoid duplicate loading.

[0078] (2) Scene Management and Visibility Clipping (CPU - Business Threads) This submodule determines which objects need to be rendered in the current frame and optimizes them to reduce GPU load. This includes: Scene traversal: Traverse all objects in the scene and calculate their world transformation matrix.

[0079] Frustum culling: Using a fast intersection test with the bounding box and the camera's frustum, objects completely outside the frustum are skipped and not submitted for rendering. This is key to reducing draw calls.

[0080] Level of Detail (LOD) selection: Choose a model with different levels of detail based on the distance between the object and the camera. For objects at greater distances, use a model with fewer faces, significantly reducing the number of triangles.

[0081] Batch merging: Try to statically batch small objects using the same material and texture, or perform GPU instantiation rendering of dynamic objects to further merge DrawCalls.

[0082] (3) Rendering command preparation (CPU - rendering thread) This submodule translates the scene state into a sequence of commands that the GPU can understand. This includes: Constructing drawing calls: Generate a drawing command for each object element that needs to be rendered. The command contains all the necessary resource handles (vertex buffers, index buffers, shaders, textures, etc.).

[0083] Set rendering state: Configure the GPU state required for this drawing command (such as depth test, blending mode, stencil test, etc.).

[0084] Update constant buffer: Update CPU-side variables (such as model-view projection matrix, camera position, and light parameters) to the GPU's constant buffer.

[0085] Command queue submission: Submit all rendering commands sequentially to the command buffer, which is then ultimately submitted to the GPU by the driver.

[0086] (4) GPU rendering pipeline execution (GPU) This submodule performs the actual geometric and pixel calculations to generate the final image. It includes: Input assembly: The GPU reads vertex and index data and assembles them into triangles.

[0087] Vertex shader: processes each vertex and performs coordinate transformations (from model space to screen space).

[0088] Rasterization: Converting triangles into pixel fragments on the screen.

[0089] Pixel shader: processes each pixel fragment and calculates the final color (samples texture, calculates lighting, etc.).

[0090] Output merging: Blending pixel colors with existing colors in the frame buffer and performing depth and stencil tests.

[0091] (5) Output and Synchronization (Display Hardware) This submodule outputs the rendered image to the display and stabilizes the frame rate. It includes: Frame buffer swapping: A rendered frame of image is swapped to the front buffer for display.

[0092] Vertical Sync Control: The engine must be strictly synchronized with the monitor's refresh rate. The goal is a stable 40Hz, meaning each frame's latency must be strictly controlled to within 25 milliseconds. This is achieved by using eglSwapBuffers or similar functions and setting the swap interval to lock the frame rate and prevent screen tearing.

[0093] Performance monitoring: Real-time monitoring of the time taken for each frame. If a frame takes longer than 25ms, the system automatically reduces the Level of Detail (LOD) and decreases the quality of special effects to ensure that the next frame is completed on time, thus maintaining a stable frame rate.

[0094] Figure 6 The diagram below illustrates the interactive control process of this invention. As an immersive interactive portal for the digital twin platform, the core of the interactive control lies in the efficient fusion processing of multimodal inputs and the precise control of ultra-high-definition multi-screen output. The interactive process is as follows: (1) Multimodal input acquisition a. Touch input: Hardware: Employs a high-precision capacitive or electromagnetic induction screen, supporting multi-touch (at least 20 points) and handwriting input (including pressure sensitivity).

[0095] Driver: A customized driver continuously collects raw touch point coordinates, pressure, area and other data, and reports them to the system at a high frequency (≥120Hz).

[0096] b. Voice input: Hardware: Integrated domestic microphone array (4-6 microphones), supporting far-field sound pickup and noise reduction (such as DSP chip hardware noise reduction).

[0097] Driver: The audio driver acquires multi-channel raw audio streams and performs preliminary acoustic preprocessing (such as echo cancellation (AEC) and noise suppression (ANS)).

[0098] c. Other inputs: Compatible with traditional keyboards, mice, spatial controllers and other peripherals as auxiliary interaction methods.

[0099] (2) Core processing and integration a. Input preprocessing: Touch point trajectory tracking: Aggregates continuous touch points into "gesture trajectories", performs smoothing and prediction, and reduces jitter.

[0100] Voice activation detection: Real-time monitoring of the audio stream to identify "wake words" (such as "spaceship"), and subsequent audio is only sent to the recognition engine after the wake-up is activated, saving computing power.

[0101] Beamforming: Utilizes a microphone array to calculate the direction of the sound source and combines it with a camera for sound source localization, improving the recognition rate in noisy environments.

[0102] b. Parallel recognition: Gesture recognition: Matches touch trajectories with a predefined gesture library (such as zoom, rotate, drag, and line drawing) to identify user intent.

[0103] Speech recognition: A locally deployed speech recognition engine converts the speech stream into text in real time. To ensure data security, all recognition processes are completed locally.

[0104] Natural Language Understanding: Semantic parsing of the identified text to extract operation commands and parameters (such as "zoom in", "switch to engine view", "raise the temperature to 500 degrees").

[0105] c. Multimodal fusion decision-making: This is the intelligent core of the system. The fusion engine combines touch and voice commands based on timestamps and context.

[0106] Example: A user says "zoom in on this area" while drawing a circle on the screen with their finger. The fusion engine will bind the "zoom in" command to the "circle" area, generating a precise strong command to "zoom in on this area".

[0107] d. Instruction mapping and distribution: The generated standardized instructions are mapped to specific API calls for digital twin platform applications (such as camera.zoomTo(region)).

[0108] The instructions are placed in the application command queue, where they are received and executed by the main application.

[0109] (3) Rendering and Output a. Application responsiveness and rendering: Digital twin applications (such as Unreal Engine) update scene states (such as camera pose, model state, and display parameters) based on instructions.

[0110] The rendering engine begins rendering calculations for a new frame based on the new scene state. To support 8K resolution, optimization techniques such as tiered rendering and dynamic resolution rendering are required.

[0111] b. Multi-screen display composition and management: The display compositor is responsible for compositing and outputting the final image.

[0112] Work mode: Extended Mode: Combines multiple screens into an ultra-wide virtual desktop for displaying large-scale models or panoramic data dashboards.

[0113] Copy mode: Displays the same content on all screens, suitable for collaborative discussions among multiple users.

[0114] Hybrid mode: The main screen displays a 3D scene, while the secondary screen displays 2D data instruments, operation logs, or control menus.

[0115] Using domestically produced graphics card drivers and display interfaces (such as DP 1.4), the rendered images are stably output to various 8K displays at a refresh rate of 60Hz.

[0116] (4) Support from domestic platforms a. Hardware foundation: The terminal is equipped with a Loongson 3C6000 series processor, whose powerful multi-core performance and memory bandwidth are sufficient to meet the computing power requirements of 8K display and multimodal processing.

[0117] b. Operating System: Run the Loongnix operating system, and deeply customize and optimize its kernel, display services (such as Wayland), and audio services to ensure low-latency interaction.

[0118] c. Driver and algorithm autonomy: Develop our own graphics card drivers, touch drivers, and audio drivers.

[0119] It integrates proprietary speech recognition, NLU, and fusion decision algorithms to ensure that all core technologies are independently controllable.

[0120] This invention provides an innovative method and facility for high-precision, real-time, integrated, and autonomously controllable virtual and real data visualization, targeting high-end industrial fields, especially the aerospace field. It breaks down data barriers, improves visualization efficiency and accuracy, and achieves autonomous, secure, and controllable operation throughout the entire process.

[0121] By employing the embodiments of the present invention, the following beneficial effects are achieved: 1. Through a multi-layered hybrid rendering pipeline, seamless overlay of 2D charts, 3D models, and particle effects is achieved from the same viewpoint, completely solving the industry pain points of view separation and manual switching, and realizing true integrated visualization.

[0122] 2. While carefully designing the parallel processing pipeline, the core mathematical library of the engine (matrix operation, vector calculation, quaternion, skeleton skinning) was manually assembled or inlined using Loongson's LSX / LASX vector instruction set, resulting in a performance improvement of several times. This enabled high-definition high-refresh-rate rendering at 4K@40Hz in a fully domestically produced environment, supporting ultra-large-scale scenes with 60 million polygons. The performance far exceeded that of traditional solutions (below 20Hz), providing excellent rendering performance.

[0123] 3. By using a spatial mapping algorithm, the data mapping error is controlled to the millimeter level, meeting the requirements for aerospace-grade rendering accuracy.

[0124] 4. From the underlying hardware and operating system to the upper-level rendering engine, the entire chain has been adapted and optimized for domestic production. The core technologies are independent, there is no risk of being "strangled", and complete self-reliance and control have been achieved.

[0125] 5. It provides multimodal interaction methods such as touch and voice, and supports multiple view presets such as expert mode and command mode, which greatly improves user experience and decision-making efficiency, and provides a powerful interactive experience.

[0126] Device Example 1 According to an embodiment of the present invention, an electronic device is provided, comprising: Processor; and, A memory is configured to store computer-executable instructions, which, when executed, cause the processor to perform the steps of the aforementioned real-time visualization method driven by virtual and real dual-wheel data.

[0127] Device Example 2 According to an embodiment of the present invention, a storage medium is provided for storing computer-executable instructions, which, when executed, implement the steps of the above-described real-time visualization method driven by virtual and real dual-wheel data.

[0128] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A real-time visualization method driven by both virtual and real data, characterized in that... include: The system receives real-time data streams from multiple sensors via a high-speed data bus, associates the spatial location of physical measurement points with virtual channel IDs, matches the corresponding CAD model spatial coordinates, synchronously parses their time series labels, dynamically matches the corresponding CAE simulation data frames, encapsulates the data using a lightweight protocol, and writes it in parallel to a dual storage system. The dual storage system consists of a database that supports structured queries and a database that supports unstructured storage. The original 3D design model is lightweighted, and a high-precision mapping relationship is established between the physical measurement point positions and the vertices of the lightweight model. Error correction is applied through spatial coordinate system meshing algorithm and measurement point coordinate inverse correction algorithm to control the mapping error within the preset accuracy range. A multi-layered rendering pipeline, including a base layer, a dynamic layer, and an overlay layer, is constructed to achieve real-time data-driven dynamic particle effects and two-dimensional analysis charts overlaid in the same frame in a 3D scene.

2. The method according to claim 1, characterized in that, The CAE simulation data frames corresponding to the dynamic matching specifically include: The CAE simulation data frames are preloaded into the managed memory state of the real-time stream processing engine and organized using the virtual channel ID and time period as primary keys. During data processing, the collaborative processing function or interval connection operator of the real-time stream processing engine is used to find and associate the most matching CAE simulation data within the corresponding time window, based on the timestamp of the multi-source sensor data stream, so as to achieve millisecond-level time alignment between the two.

3. The method according to claim 1, characterized in that, The specific process of applying error correction through spatial coordinate system gridding algorithm and measurement point coordinate inverse correction algorithm is as follows: The vertex space of the lightweight model surface is divided into a grid to form a spatial coordinate grid; Based on the actual physical coordinates of the physical measurement points, interpolation calculations are performed in the spatial coordinate grid to determine the model vertices of the initial mapping. The reverse correction algorithm calculates and applies an error correction amount to accurately map the data of the physical measurement point to the correction position of the initial mapping vertex, wherein the error correction amount is controlled to be no greater than 0.1 mm.

4. The method according to claim 1, characterized in that, The construction of the multi-level rendering pipeline specifically includes: The base layer is responsible for rendering high-fidelity 3D models; The dynamic layer binds the real-time data stream to the corresponding vertex of the model, driving the dynamic particle effects to change dynamically as the data changes, wherein the dynamic particle effects include at least one of tail flame simulation and fuel leak simulation. The overlay layer synchronously overlays and renders two-dimensional analysis charts from real-time data or processing results within the same viewport of the three-dimensional scene. The two-dimensional analysis charts include at least one of line charts, bar charts, and heat flow distribution cloud maps.

5. The method according to claim 1, characterized in that, The method also includes an intelligent switching step for multiple view modes: It switches between different visualization modes in response to user commands or preset conditions; In expert mode, a 3D cloud map, a spectrum analysis map, and key parameter curves are displayed simultaneously on the same interface. In command mode, the screen is displayed in full screen, focusing on rendering the macroscopic situation of the ballistic trajectory, and supplemented by key parameter dashboards at the edge of the screen or in the overlay.

6. The method according to claim 1, characterized in that, The parallel writing to the dual-storage system consisting of a relational database and a file database is specifically implemented as follows: The parsed and encapsulated data is written to both a relational database and a file database simultaneously. The relational database is used to support high-frequency structured data queries and transaction operations, while the file database is used to store unstructured or semi-structured time-series data and document data to achieve high data availability and differentiated query efficiency.

7. The method according to claim 1, characterized in that, In the step of constructing a multi-layered rendering pipeline, rendering efficiency and frame rate stability are ensured by implementing rendering optimization strategies, specifically including: On the central processing unit, all objects in the 3D scene are traversed, and a fast intersection test is performed using their bounding boxes and the camera's view frustum to skip rendering invisible objects; at the same time, models of different levels of detail are dynamically selected for rendering based on the distance between the object and the camera. Before submitting the rendering command, objects with the same material and texture will be batched together to reduce the number of draw calls.

8. The method according to claim 1, characterized in that, The method also includes a data security transmission step: before sending the encapsulated data over the network, the built-in national cryptographic algorithm acceleration engine is used to perform SM4 encryption on the data packets to achieve hardware-level real-time data encryption and ensure the security and controllability of the data transmission process.

9. An electronic device, comprising: processor; as well as, A memory configured to store computer-executable instructions, which, when executed, cause the processor to perform the steps of the real-time visualization method driven by virtual and real dual-wheel data as described in any one of claims 1-8.

10. A storage medium for storing computer-executable instructions, which, when executed, implement the steps of the real-time visualization method driven by virtual and real dual-wheel data as described in any one of claims 1-8.

Images