An intelligent AR visual visualization system and method thereof

Abstract

The present invention discloses an intelligent AR visual visualization system and method thereof, which relates to the technical field of AR visualization system, including a data acquisition sensor fusion module, an image processing module and an intelligent reasoning engine module, wherein the image processing module is used to process and identify the collected image data; the image processing module includes: a real-time image processing unit, a deep learning model unit and an object recognition unit. The present invention realizes the function of real-time processing of collected image data by designing an image processing module, solves the problem of computationally intensive processing of complex images, improves image quality and clarity, makes the visualization system perceive the environment more accurately, and ensures that the images seen by users in the visualized AR environment are more realistic and clear.

Description

Technical Field

[0001] The present invention relates to the technical field of AR visualization systems, and in particular to an intelligent AR visual visualization system and a method thereof. Background Art

[0002] In the context of the rapid development of science and technology today, augmented reality (AR) technology has attracted widespread attention as an interactive experience that integrates virtual information into the real world. Intelligent AR visualization systems integrate hardware sensors, image processing, deep learning, and user interaction technologies to provide a more immersive and intelligent virtual reality experience. The continuous innovation of sensor technology, the development of image processing and deep learning, and the support of powerful computing power provide the prerequisites for building intelligent AR systems.

[0003] Existing AR systems still face challenges in the accuracy of perceiving the environment and user behavior, especially in complex scenarios where inaccurate perception can lead to misalignment between virtual information and the real environment. Existing AR systems lack adaptability to changes in lighting, scenarios, and user behavior. Hardware integration, lightweight, and real-time performance of AR devices remain a challenge.

[0004] Therefore, it is very necessary for the present application to propose an intelligent AR visual visualization system and method thereof to solve the technical weaknesses in the prior art.

[0005] 1. Patent document CN109947052B discloses a virtual factory manufacturing execution system based on digital AR technology. The above patent realizes the issuance of work instructions in the factory and remote visual monitoring to achieve the purpose of mastering, guiding and controlling the production process, but the above patent cannot realize the function of real-time processing of collected image data.

[0006] 2. Patent document CN110971678B discloses an immersive visualization campus system based on a 5G network. The above patent realizes the construction of a knowledge kingdom that is visible and immersive to the human eye in a real campus in a combination of virtual and real ways, providing teachers and students in the school with a new three-dimensional immersive working, learning and living space. However, the above patent cannot realize the multi-dimensional image data collection function of environmental information.

[0007] 3. Patent document CN116894930B discloses a digital work guidance system based on AI+AR. The above patent realizes the simulation and visualization of the virtual environment, provides more intuitive and real-time visual information, and helps users to practice and make decisions in the virtual environment. However, the above patent cannot achieve a deep understanding and reasoning function of the environment.

[0008] 4. Patent document CN115035767B discloses a spinal surgery teaching and training system based on AR and anthropomorphic models. The above patent realizes real-time autonomous interactive teaching guidance, has realistic visual and force feedback, and automatically conducts quantitative assessment and evaluation of surgical operation skill levels and conducts long-term tracking, thereby reducing the cost of spinal surgery training and shortening the spinal surgery training cycle. However, the above patent cannot realize the data fusion function of image features and environmental information.

[0009] In summary, the above patents cannot realize the functions of real-time processing of collected image data, multi-dimensional image data collection of environmental information, deep understanding and reasoning of the environment, and data fusion of image features and environmental information, resulting in the inability to accurately perceive environmental information, reliably model the environment, deeply learn and reason about environmental data, and organize and fuse large amounts of data to provide users with real and clear visual images.

[0010] To this end, the present application proposes an intelligent AR visual visualization system and method thereof that can realize real-time processing of collected image data, multi-dimensional image data collection of environmental information, deep understanding and reasoning of the environment, and data fusion of image features and environmental information. Summary of the invention

[0011] The purpose of the present invention is to provide an intelligent AR visual visualization system and method thereof, so as to solve the technical problems proposed in the above-mentioned background technology that the function of real-time processing of collected image data, the function of multi-dimensional image data collection of environmental information, the function of deep understanding and reasoning of the environment, and the function of data fusion of image features and environmental information cannot be realized, resulting in the inability to accurately perceive environmental information, the inability to reliably model the environment, the inability to deeply learn and reason about environmental data, and the inability to organize and fuse large amounts of data to provide users with real and clear visualization images.

[0012] To achieve the above object, the present invention provides the following technical solutions: an intelligent AR visual visualization system, comprising a data acquisition sensor fusion module, an image processing module and an intelligent reasoning engine module, wherein the image processing module is used to process and identify the collected image data;

[0013] The image processing module includes: a real-time image processing unit, a deep learning model unit and an object recognition unit;

[0014] The real-time image processing unit is integrated with a DSP, and the DSP accelerates the real-time image processing algorithm, and the real-time image processing algorithm includes a real-time edge detection algorithm and an image enhancement algorithm;

[0015] The deep learning model unit is integrated with an NVIDIA GPU, and the deep learning model unit uses the NVIDIA GPU to run a deep learning model ResNet, and the deep learning model ResNet is used for complex object recognition and scene understanding. The DSP transmits the processed image data to the NVIDIA GPU for deep learning model operation;

[0016] The object recognition unit is integrated with an object recognition coprocessor, and the object recognition coprocessor is integrated with TPU dedicated hardware to achieve real-time object recognition. The NVIDIA GPU is connected to the object recognition coprocessor via a high-speed bus.

[0017] Preferably, the visualization system is designed with an AR device module, and the AR device module is used to collect and identify environmental data information in real time;

[0018] The AR device module includes: a smart glasses unit, a camera unit, and a sensor unit;

[0019] The smart glasses unit uses HoloLens2 smart glasses, which are equipped with Azure Kinect sensors, a depth sensor. The smart glasses unit uses the depth sensor to perceive the environment in real time and collect depth maps and RGB maps.

[0020] The camera unit is integrated with a dual-camera module, which combines binocular stereo vision technology to improve scene depth perception and object recognition accuracy;

[0021] The sensor unit is integrated with IMU, which uses IMU to obtain the acceleration, gyroscope and magnetometer information of the AR device in real time for head tracking and posture control.

[0022] Preferably, the data acquisition sensor fusion module is used to perform feature extraction and data fusion on the data information collected by the AR device module, and the data acquisition sensor fusion module is connected to the AR device module via a dedicated interface and a high-speed data bus;

[0023] The data acquisition sensor fusion module includes: a data acquisition unit, a sensor fusion unit and an environment model unit;

[0024] The data acquisition unit is integrated with an embedded processor, which uses an ARM architecture processor to process the image data collected by the camera unit and the data information collected by the sensor unit, and extract image features and environmental information;

[0025] The sensor fusion unit is integrated with FPGA, which is connected to the embedded processor through the PCle interface. The FPGA transmits the image processing results and is connected to the IMU through a high-speed serial port for updating the environmental model. FPGA realizes hardware-level sensor data fusion to improve real-time performance and computing efficiency.

[0026] The environmental model unit is integrated with an SSD, which is connected to the FPGA through a SATA interface to store the environmental model and system configuration. The environmental model is stored in the SSD, which supports fast reading and writing and improves the system response speed.

[0027] Preferably, the data acquisition sensor fusion module is also designed with an intelligent reasoning engine module, which is used to reason and dynamically adjust and optimize the machine learning model;

[0028] The intelligent reasoning engine module includes: machine learning model unit, contextual reasoning unit and real-time learning unit;

[0029] The machine learning model unit integrates a multi-core CPU cluster, uses the TensorFlow framework to train and deploy machine learning models, and supports offline and online reasoning;

[0030] The contextual reasoning unit is integrated with an FPGA accelerator card, which is connected to a multi-core CPU cluster through a high-speed interconnect bus. The parameters of the machine learning model are passed to the FPGA, and the FPGA accelerator card is used for efficient contextual reasoning, which reduces the load on the main processor.

[0031] The real-time learning unit is integrated with a deep incremental learning module, which supports deep incremental learning and uses dynamic weights to adjust the model to adapt to environmental changes.

[0032] Preferably, the visualization system is also designed with an AR visualization interface module, and the AR visualization interface is used to render and display the fusion of virtual information and real scenes;

[0033] The AR visualization interface module includes: a 3D rendering engine unit, a virtual information display unit, and a user interaction unit;

[0034] The 3D rendering engine unit is integrated with a Vulkan graphics interface, which is connected to the FPGA through a high-speed graphics bus. The rendered 3D scene is passed to the 3D rendering engine unit, and the Vulkan API is used for high-performance 3D rendering to ensure the integration of virtual information and real scenes.

[0035] The virtual information display unit is integrated with a Micro LED display. The 3D rendering engine unit is connected to the Micro LED display via a data cable. The Micro LED display technology provides high resolution and high brightness, ensuring that the virtual information is clearly visible under different lighting conditions.

[0036] The user interaction unit is integrated with mixed reality gloves, which are connected to the visualization system via wireless communication to transmit user gesture information and integrate gesture recognition sensors to allow users to interact through gestures, providing an intuitive experience.

[0037] Preferably, the data acquisition unit is also designed with a data security and privacy module, which is used to protect the security of data information and user information;

[0038] The data security and privacy module includes: data encryption unit, user authority management unit and anonymization processing unit;

[0039] The data encryption unit is integrated with TPM, which provides hardware-level data encryption and authentication to ensure the security of sensor data and user information.

[0040] The user rights management unit integrates a biometric recognition module, including face recognition and fingerprint recognition. The biometric recognition module is connected to the TPM through a dedicated interface to transmit user authorization information, use biometric recognition to perform user authorization, and manage access rights to sensitive data;

[0041] The anonymization processing unit is integrated with a privacy filter to anonymize personal identity information during the data collection stage to protect user privacy.

[0042] Preferably, the intelligent reasoning engine module is also designed with a system optimization extension module, and the system optimization extension module is used to monitor and optimize the performance of the system;

[0043] The system optimization extension modules include: real-time performance optimization unit, model update iteration unit and open interface application ecosystem unit;

[0044] The real-time performance optimization unit is integrated with a real-time monitoring unit to monitor the performance of each component of the system in real time and dynamically adjust the working frequency of the image processing and reasoning engine according to real-time requirements;

[0045] The model update iteration unit integrates an OTA module, which is connected to the cloud through a network interface to receive new firmware and algorithm updates. It supports OTA updates through the cloud, allowing the system to quickly adapt to new algorithms and functions.

[0046] A developer platform is set up within the open interface application ecosystem unit, providing developer-friendly SDKs and APIs to encourage third parties to develop AR applications and expand the system's functions and ecosystem.

[0047] Preferably, the visualization system is also designed with a technical verification test module, which is used to perform performance testing and scenario simulation on the visualization system;

[0048] The technical verification test modules include: simulation environment test unit, field test unit and user feedback improvement unit;

[0049] The simulation environment test unit uses the Unity Engine to create a realistic virtual environment through a simulator workstation to perform system performance testing and scene simulation;

[0050] The field test unit uses mobile test vehicles to conduct system tests in the real world, simulating the user experience in different mobile scenarios;

[0051] The user feedback improvement unit is integrated with a user feedback platform. A user feedback system is designed to collect user opinions, analyze large amounts of user data through machine learning, and guide system improvements.

[0052] Preferably, the method comprises the following steps:

[0053] S1. Data collection: Obtain environmental depth, image and device posture data from the AR device's HoloLens2 smart glasses, camera unit, and sensor unit hardware modules;

[0054] S2, Perception Modeling: Use real-time image processing units and deep learning models to perform real-time image analysis, object recognition, and scene understanding, generate environmental perception data, and perceive objects, scenes, and spatial layouts around the user in real time;

[0055] S3, learning reasoning: using deep learning models to understand and reason about the environment at a higher level, and to achieve recognition of complex scenes and objects;

[0056] S4. Visual interaction: Use the 3D rendering engine to integrate the environmental modeling results with the virtual information to generate a realistic mixed reality scene. Through the Vulkan graphics interface, the processed 3D scene is passed to the MicroLED display and displayed in front of the user.

[0057] Preferably, the method further comprises the following steps:

[0058] S11, using embedded processors and FPGAs to perform real-time data preprocessing, including image denoising and fusion of depth maps and RGB images;

[0059] S21, integrate the real-time perceived environmental data, and generate accurate environmental modeling information through the collaboration of the embedded processor and FPGA module;

[0060] S31, use FPGA acceleration card for contextual reasoning, realize dynamic weight adjustment through real-time learning module, and improve the adaptability of the system;

[0061] S41, integrated mixed reality gloves, realize user gesture interaction through gesture recognition sensors, and feed back user behavior to the system to realize interaction between users and virtual environment.

[0062] Compared with the prior art, the present invention has the following beneficial effects:

[0063] 1. The present invention realizes the function of real-time processing of collected image data by designing an image processing module, solves the problem of computationally intensive processing of complex images, improves image quality and clarity, makes the visualization system perceive the environment more accurately, and ensures that the images seen by users in the visualized AR environment are more realistic and clear;

[0064] 2. The present invention realizes the multi-dimensional image data acquisition function of environmental information by designing an AR device module, solves the hardware fusion problem of AR devices, provides high-quality sensor data, and provides a reliable data basis for the perception and understanding of the entire visualization system;

[0065] 3. The present invention realizes the deep understanding and reasoning function of the environment by designing an intelligent reasoning engine module, solves the recognition and user understanding problems in complex scenes, improves the intelligent perception ability of the visualization system, and can accurately interact with users;

[0066] 4. The present invention realizes the data fusion function of image features and environmental information by designing a data acquisition sensor fusion module, constructs an accurate environmental model, solves the technical problem of real-time data integration, improves the accuracy of the visualization system's cognition and modeling of the environment, and makes the visualization system more stable and reliable. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 is a schematic diagram of an image processing module of the present invention;

[0068] Figure 2 It is a schematic diagram of the AR device module of the present invention;

[0069] Figure 3 It is a schematic diagram of the data acquisition sensor fusion module of the present invention;

[0070] Figure 4 It is a schematic diagram of the data acquisition sensor fusion module of the present invention;

[0071] Figure 5 It is a schematic diagram of the AR visualization interface module of the present invention. DETAILED DESCRIPTION

[0072] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0073] Example 1

[0074] See also Figure 1 and Figure 2 , an embodiment provided by the present invention: an intelligent AR visual visualization system, comprising a data acquisition sensor fusion module, an image processing module and an intelligent reasoning engine module, wherein the image processing module is used to process and recognize the collected image data;

[0075] The image processing module includes: a real-time image processing unit, a deep learning model unit and an object recognition unit;

[0076] The real-time image processing unit is integrated with a DSP, and the DSP accelerates the real-time image processing algorithm, and the real-time image processing algorithm includes a real-time edge detection algorithm and an image enhancement algorithm;

[0077] The deep learning model unit is integrated with an NVIDIA GPU, and the deep learning model unit uses the NVIDIA GPU to run a deep learning model ResNet, and the deep learning model ResNet is used for complex object recognition and scene understanding. The DSP transmits the processed image data to the NVIDIA GPU for deep learning model operation;

[0078] The object recognition unit is integrated with an object recognition coprocessor, and the object recognition coprocessor is integrated with TPU dedicated hardware to achieve real-time object recognition. The NVIDIA GPU is connected to the object recognition coprocessor via a high-speed bus;

[0079] Furthermore, the DSP accelerates the real-time edge detection and image enhancement image processing algorithms. The FPGA is connected to the DSP through the PCIe interface and passes the preliminary results of image processing to the DSP. The DSP passes the processed image data to the NVIDIA GPU for deep learning model operations. The deep learning model unit uses the NVIDIA GPU to run the ResNet deep learning model for complex object recognition and scene understanding. The NVIDIA GPU is connected to the object recognition coprocessor through a high-speed bus. The object recognition coprocessor integrates TPU dedicated hardware for real-time object recognition.

[0080] Example 2

[0081] See also Figure 1 and Figure 2 , an embodiment provided by the present invention: an intelligent AR visual visualization system, wherein the visualization system is designed with an AR device module, and the AR device module is used to collect and identify environmental data information in real time;

[0082] The AR device module includes: a smart glasses unit, a camera unit, and a sensor unit;

[0083] The smart glasses unit uses HoloLens2 smart glasses, which are equipped with Azure Kinect sensors, a depth sensor. The smart glasses unit uses the depth sensor to perceive the environment in real time and collect depth maps and RGB maps.

[0084] The camera unit is integrated with a dual-camera module, which combines binocular stereo vision technology to improve scene depth perception and object recognition accuracy;

[0085] The sensor unit is integrated with an IMU, which uses the IMU to obtain the acceleration, gyroscope, and magnetometer information of the AR device in real time for head tracking and posture control.

[0086] Furthermore, HoloLens 2 collects depth maps and RGB maps in real time through the built-in Azure Kinect sensor. The smart glasses are connected to the embedded processor through the USB-C interface, and the image and depth data collected by the sensor unit are passed to the embedded processor. The dual-camera module uses binocular stereo vision technology to improve scene depth perception and object recognition accuracy. The dual-camera module transmits image data to the embedded processor through a high-speed data bus to provide richer information for environmental perception. The IMU obtains the device's acceleration, gyroscope and magnetometer information in real time for head tracking and posture control. The IMU data is transmitted to the data acquisition unit through a dedicated interface to support real-time tracking of the user's head in a virtual environment.

[0087] Example 3

[0088] See also Figure 1 , Figure 2 and Figure 3 , an embodiment provided by the present invention: an intelligent AR visual visualization system, the data acquisition sensor fusion module is used to extract features and fuse data from data information collected by the AR device module, and the data acquisition sensor fusion module is connected to the AR device module through a dedicated interface and a high-speed data bus;

[0089] The data acquisition sensor fusion module includes: a data acquisition unit, a sensor fusion unit and an environment model unit;

[0090] The data acquisition unit is integrated with an embedded processor, which uses an ARM architecture processor to process the image data collected by the camera unit and the data information collected by the sensor unit, and extract image features and environmental information;

[0091] The sensor fusion unit is integrated with FPGA, which is connected to the embedded processor through the PCle interface. The FPGA transmits the image processing results and is connected to the IMU through a high-speed serial port for updating the environmental model. FPGA realizes hardware-level sensor data fusion to improve real-time performance and computing efficiency.

[0092] The environmental model unit is integrated with an SSD, which is connected to the FPGA through a SATA interface to store the environmental model and system configuration. The environmental model is stored in the SSD, which supports fast reading and writing, and improves the system response speed.

[0093] Furthermore, the embedded processor uses an ARM architecture processor to process the images collected by the camera unit and the data collected by the sensor unit, and extract image features and environmental information. The embedded processor is connected to the FPGA through the PCIe interface to transmit the image processing results. The FPGA implements hardware-level sensor data fusion to improve real-time performance and computing efficiency. The IMU data is transmitted to the FPGA through the high-speed serial port for updating the environmental model. The SSD is connected to the FPGA through the SATA interface to store the environmental model and system configuration. The environmental model is stored in the SSD, supports fast reading and writing, and improves the system response speed. The FPGA passes the preliminary results of image processing to the real-time image processing unit through the PCIe interface, and the IMU data is transmitted to the FPGA through the high-speed serial port for updating the environmental model.

[0094] Example 4

[0095] See also Figure 1 , Figure 3 and Figure 4, an embodiment provided by the present invention: an intelligent AR visual visualization system, wherein the data acquisition sensor fusion module is further designed with an intelligent reasoning engine module, and the intelligent reasoning engine module is used to reason and dynamically adjust and optimize the machine learning model;

[0096] The intelligent reasoning engine module includes: machine learning model unit, contextual reasoning unit and real-time learning unit;

[0097] The machine learning model unit integrates a multi-core CPU cluster, uses the TensorFlow framework to train and deploy machine learning models, and supports offline and online reasoning;

[0098] The contextual reasoning unit is integrated with an FPGA accelerator card, which is connected to a multi-core CPU cluster through a high-speed interconnect bus. The parameters of the machine learning model are passed to the FPGA, and the FPGA accelerator card is used for efficient contextual reasoning, which reduces the load on the main processor.

[0099] The real-time learning unit is integrated with a deep incremental learning module, which supports deep incremental learning and uses dynamic weights to adjust the model to adapt to environmental changes;

[0100] Furthermore, the multi-core CPU cluster uses the TensorFlow or PyTorch framework to train and deploy machine learning models, supporting offline and online reasoning. The multi-core CPU cluster is connected to the FPGA accelerator card through a high-speed interconnect bus, and the parameters of the machine learning model are passed to the FPGA accelerator card. The FPGA accelerator card is used for efficient contextual reasoning to reduce the load on the main processor, support deep incremental learning, and use dynamic weights to adjust the model to adapt to environmental changes. The FPGA receives data from the real-time learning module through a dedicated interface and dynamically adjusts the model parameters. The FPGA transmits the results of contextual reasoning to the system bus for the next step of AR visualization.

[0101] Example 5

[0102] See also Figure 2 , Figure 4 and Figure 5 , an embodiment provided by the present invention: an intelligent AR visual visualization system, wherein the visualization system is further designed with an AR visualization interface module, and the AR visualization interface is used to render and display the fusion of virtual information and real scenes;

[0103] The AR visualization interface module includes: a 3D rendering engine unit, a virtual information display unit, and a user interaction unit;

[0104] The 3D rendering engine unit is integrated with a Vulkan graphics interface, which is connected to the FPGA through a high-speed graphics bus. The rendered 3D scene is passed to the 3D rendering engine unit, and the Vulkan API is used for high-performance 3D rendering to ensure the integration of virtual information and real scenes.

[0105] The virtual information display unit is integrated with a Micro LED display. The 3D rendering engine unit is connected to the Micro LED display via a data cable. The Micro LED display technology provides high resolution and high brightness, ensuring that the virtual information is clearly visible under different lighting conditions.

[0106] The user interaction unit is integrated with mixed reality gloves, which are connected to the visualization system through wireless communication to transmit user gesture information and integrate gesture recognition sensors to allow users to interact through gestures, providing an intuitive experience;

[0107] Furthermore, the FPGA is connected to the Vulkan interface through a high-speed graphics bus to pass the rendered 3D scene to the 3D rendering engine. The 3D rendering engine passes the rendering result to the MicroLED display to display it in front of the user. The mixed reality gloves are connected to the system through wireless communication or a dedicated interface to transmit user gesture information. The user feedback system is connected to the system through a network interface to receive user feedback and pass the analysis results to the system optimization module.

[0108] Working principle: First, the intelligent AR visualization system starts with the AR device module. Through hardware modules such as smart glasses, camera units and sensor units, the depth map, RGB map and device posture data are collected in real time. Smart glasses obtain depth maps and RGB maps through the Azure Kinect sensor. The dual camera unit uses binocular stereo vision technology to improve depth perception. At the same time, the sensor unit obtains IMU data in real time. These data are transmitted to the embedded processor to establish the basic data source of the system.

[0109] Then, the data acquisition and sensor fusion module comes into play. The embedded processor uses an ARM architecture processor to process camera images and sensor data, extract image features and environmental information, and transmit the data to the FPGA through the PCIe interface for sensor data fusion, including the integration of IMU data and the construction of the environmental model. At the same time, the environmental model unit uses SSD to store the environmental model in a high-speed solid-state drive for fast reading and writing, thereby improving the system response speed. The data acquisition sensor fusion module connects to the user's mobile device through Bluetooth, receives the user's personalized settings and instructions, and realizes interaction with the user's mobile device.

[0110] Finally, the deep learning model unit and the intelligent reasoning engine module come into play. The DSP in the real-time image processing unit processes the real-time image, the NVIDIA GPU executes the deep learning model, and the object recognition coprocessor accelerates object recognition, achieving a deep understanding of the environment. At the same time, the FPGA accelerator card performs contextual reasoning, and dynamic learning dynamically adjusts the model's weights to adapt to environmental changes. This enables the system to understand user needs more intelligently and update environmental cognition in real time.

[0111] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. An intelligent AR visual visualization system, characterized by: It includes a data acquisition sensor fusion module, an image processing module and an intelligent reasoning engine module, wherein the image processing module is used to process and identify the collected image data; The image processing module includes: a real-time image processing unit, a deep learning model unit and an object recognition unit; The real-time image processing unit is integrated with a DSP, and the DSP accelerates the real-time image processing algorithm, and the real-time image processing algorithm includes a real-time edge detection algorithm and an image enhancement algorithm; The deep learning model unit is integrated with an NVIDIA GPU, and the deep learning model unit uses the NVIDIA GPU to run a deep learning model ResNet, and the deep learning model ResNet is used for complex object recognition and scene understanding. The DSP transmits the processed image data to the NVIDIA GPU for deep learning model operation; The object recognition unit is integrated with an object recognition coprocessor, and the object recognition coprocessor is integrated with TPU dedicated hardware to achieve real-time object recognition. The NVIDIA GPU is connected to the object recognition coprocessor via a high-speed bus; The data acquisition sensor fusion module is also designed with an intelligent reasoning engine module, which is used to reason and dynamically adjust and optimize the machine learning model; The intelligent reasoning engine module includes: machine learning model unit, contextual reasoning unit and real-time learning unit; The machine learning model unit integrates a multi-core CPU cluster, uses the TensorFlow framework to train and deploy machine learning models, and supports offline and online reasoning; The contextual reasoning unit is integrated with an FPGA accelerator card, which is connected to a multi-core CPU cluster through a high-speed interconnect bus. The parameters of the machine learning model are passed to the FPGA, and the FPGA accelerator card is used to perform efficient contextual reasoning, reducing the load on the main processor. The real-time learning unit is integrated with a deep incremental learning module, which supports deep incremental learning and uses dynamic weights to adjust the model to adapt to environmental changes; The intelligent reasoning engine module is also designed with a system optimization extension module, which is used to monitor and optimize the performance of the system; The system optimization extension modules include: real-time performance optimization unit, model update iteration unit and open interface application ecosystem unit; The real-time performance optimization unit is integrated with a real-time monitoring unit to monitor the performance of each component of the system in real time and dynamically adjust the working frequency of the image processing and reasoning engine according to real-time requirements; The model update iteration unit integrates an OTA module, which is connected to the cloud through a network interface to receive new firmware and algorithm updates. It supports OTA updates through the cloud, allowing the system to quickly adapt to new algorithms and functions. The open interface application ecosystem unit has a developer platform that provides developer-friendly SDKs and APIs to encourage third parties to develop AR applications and expand the system's functions and ecosystem. The visualization system is also designed with a technical verification test module, which is used to perform performance testing and scenario simulation on the visualization system; The technical verification test modules include: simulation environment test unit, field test unit and user feedback improvement unit; The simulation environment test unit uses the Unity Engine to create a realistic virtual environment through a simulator workstation to perform system performance testing and scene simulation; The field test unit uses mobile test vehicles to conduct system tests in the real world, simulating the user experience in different mobile scenarios; The user feedback improvement unit integrates a user feedback platform, designs a user feedback system to collect user opinions, analyzes a large amount of user data through machine learning, and guides system improvements; The visualization system is designed with an AR device module, which is used to collect and identify environmental data information in real time; The AR device module includes: a smart glasses unit, a camera unit, and a sensor unit; The smart glasses unit uses HoloLens2 smart glasses, which are equipped with a depth sensor AzureKinect sensor. The smart glasses unit uses the depth sensor to perceive the environment in real time and collect depth maps and RGB maps. The camera unit is integrated with a dual-camera module, which combines binocular stereo vision technology to improve scene depth perception and object recognition accuracy; The sensor unit is integrated with an IMU, which uses the IMU to obtain the acceleration, gyroscope, and magnetometer information of the AR device in real time for head tracking and posture control. The data acquisition sensor fusion module is used to extract features and fuse data from the data information collected by the AR device module. The data acquisition sensor fusion module is connected to the AR device module through a dedicated interface and a high-speed data bus; The data acquisition sensor fusion module includes: a data acquisition unit, a sensor fusion unit and an environment model unit; The data acquisition unit is integrated with an embedded processor, which uses an ARM architecture processor to process the image data collected by the camera unit and the data information collected by the sensor unit, and extract image features and environmental information; The sensor fusion unit is integrated with FPGA, which is connected to the embedded processor through the PCle interface. The FPGA transmits the image processing results and is connected to the IMU through a high-speed serial port for updating the environmental model. FPGA realizes hardware-level sensor data fusion to improve real-time performance and computing efficiency. The environmental model unit is integrated with an SSD, which is connected to the FPGA through a SATA interface to store the environmental model and system configuration. The environmental model is stored in the SSD, which supports fast reading and writing, and improves the system response speed. The visualization system is also designed with an AR visualization interface module, and the AR visualization interface is used to render and display the fusion of virtual information and real scenes; The AR visualization interface module includes: a 3D rendering engine unit, a virtual information display unit, and a user interaction unit; The 3D rendering engine unit is integrated with a Vulkan graphics interface, which is connected to the FPGA through a high-speed graphics bus. The rendered 3D scene is passed to the 3D rendering engine unit, and the Vulkan API is used for high-performance 3D rendering to ensure the integration of virtual information and real scenes. The virtual information display unit is integrated with a Micro LED display. The 3D rendering engine unit is connected to the Micro LED display via a data cable. The Micro LED display technology provides high resolution and high brightness, ensuring that the virtual information is clearly visible under different lighting conditions. The user interaction unit is integrated with mixed reality gloves, which are connected to the visualization system via wireless communication to transmit user gesture information and integrate gesture recognition sensors to allow users to interact through gestures, providing an intuitive experience.

2. An intelligent AR visual visualization system according to claim 1, characterized in that: The data collection unit is also designed with a data security and privacy module, which is used to protect the security of data information and user information; The data security and privacy module includes: data encryption unit, user authority management unit and anonymization processing unit; The data encryption unit is integrated with TPM, which provides hardware-level data encryption and authentication to ensure the security of sensor data and user information. The user rights management unit integrates a biometric recognition module, including face recognition and fingerprint recognition. The biometric recognition module is connected to the TPM through a dedicated interface to transmit user authorization information, use biometric recognition to perform user authorization, and manage access rights to sensitive data; The anonymization processing unit is integrated with a privacy filter to anonymize personal identity information during the data collection stage to protect user privacy.

3. An intelligent AR visual visualization method, applicable to an intelligent AR visual visualization system according to claim 1 or 2, characterized in that: The method comprises the following steps: S1. Data collection: Obtain environmental depth, image and device posture data from the AR device's HoloLens2 smart glasses, camera unit, and sensor unit hardware modules; S2, Perception Modeling: Use real-time image processing units and deep learning models to perform real-time image analysis, object recognition, and scene understanding, generate environmental perception data, and perceive objects, scenes, and spatial layouts around the user in real time; S3, learning reasoning: using deep learning models to understand and reason about the environment at a higher level, and to achieve recognition of complex scenes and objects; S4. Visual interaction: Use the 3D rendering engine to integrate the environmental modeling results with the virtual information to generate a realistic mixed reality scene. Through the Vulkan graphics interface, the processed 3D scene is passed to the MicroLED display and displayed in front of the user.

4. The intelligent AR visual visualization method according to claim 3, characterized in that: The method further comprises the following steps: S11, using embedded processors and FPGAs to perform real-time data preprocessing, including image denoising and fusion of depth maps and RGB images; S21, integrate the real-time perceived environmental data, and generate accurate environmental modeling information through the collaboration of the embedded processor and FPGA module; S31, use FPGA acceleration card for contextual reasoning, realize dynamic weight adjustment through real-time learning module, and improve the adaptability of the system; S41, integrated mixed reality gloves, realize user gesture interaction through gesture recognition sensors, and feed back user behavior to the system to realize interaction between users and virtual environment.

Images