Real-time AR data superposition and interaction system for industrial inspection robot

Abstract

The invention discloses a real-time AR data superposition and interaction system for an industrial inspection robot, and relates to the technical field of industrial man-machine interaction, and the system comprises the steps: collecting multi-dimensional inspection context data obtained by a sensor carried by the inspection robot, and building a unified space reference system; updating of the AR terminal in the industrial scene is completed; and the corresponding control command is issued to the inspection robot for execution. According to the invention, high-precision, low-delay and semantic-consistent closed-loop coordination between data sensed by the inspection robot and AR terminal display is realized, and the information presentation intuition, the man-machine coordination fluency and the field decision accuracy in the industrial inspection process are improved; the defects of interaction lag, virtual-real dislocation, operation interruption and the like existing in a traditional system are effectively overcome, and the urgent requirement for efficient and immersive man-machine interaction equipment in a high-risk complex industrial scene is met.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of industrial human-computer interaction, and in particular to a real-time AR data superimposition and interaction system for an industrial inspection robot. BACKGROUND

[0002] Industrial inspection is accelerating towards intelligent and unmanned development, and inspection robots are increasingly widely used in high-risk or complex scenarios such as power, petrochemical and rail transit. At the same time, the demand of operators for human-computer collaborative efficiency and intuitive information perception is continuously increasing, and the traditional two-dimensional monitoring interface has been difficult to meet the requirements of real-time decision-making and precise intervention. Augmented reality (AR) technology, with its ability to seamlessly superimpose virtual information on the real environment, is becoming a key means to improve human-computer interaction experience, and the demand for efficient, intuitive and immersive human-computer interaction devices is becoming increasingly urgent.

[0003] However, the existing industrial inspection systems generally have the problems of single human-computer interaction mode, lagging information presentation and lack of spatial context association. Most systems still rely on tablets, PC terminals or remote monitoring large screens for data viewing, and cannot realize real-time AR visualization interaction between on-site operators and inspection robots. Even if some systems introduce AR display, there are often defects such as high data superimposition delay, poor virtual-real registration accuracy and inconvenient interactive instruction input, which makes it difficult for operators to quickly understand the equipment status and make accurate responses during dynamic inspection, seriously restricting the inspection efficiency and safety. SUMMARY

[0004] In view of the problems existing in the prior art real-time AR data superimposition and interaction system for an industrial inspection robot, the present application is proposed.

[0005] Therefore, the problem to be solved by the present application is that the AR data superimposition in the existing industrial inspection system has high delay, poor virtual-real registration accuracy, single human-computer interaction mode and lack of real-time collaboration ability with the inspection task context.

[0006] To solve the above technical problems, the present application provides the following technical solutions:

[0007] In a first aspect, the present application provides a real-time AR data superimposition and interaction system for an industrial inspection robot, comprising: a multi-source data fusion module, configured to collect multi-dimensional inspection context data obtained by sensors carried by an inspection robot, and to perform spatio-temporal semantic collaborative alignment on the multi-dimensional inspection context data to generate a structured real-time data stream;

[0008] The space registration and pose solving module is configured to solve the pose of the AR terminal worn by the operator in the three-dimensional space of the industrial site in real time through a visual-inertial fusion algorithm, and establish a unified space reference system in combination with the target device coordinate information reported by the inspection robot.

[0009] The real-time AR rendering engine module is configured to dynamically superimpose the inspection augmented information in the structured real-time data stream in the form of text to the corresponding physical device position in the field of view of the AR terminal based on the unified space reference system, and complete the update of the AR terminal in the industrial scene by using a real-time rendering optimization mechanism.

[0010] The natural interaction instruction analysis module is configured to receive the interaction intention signal of the operator, cooperatively analyze the natural interaction instruction in combination with the current multi-dimensional inspection context data, generate a corresponding control command, and deliver the control command to the inspection robot for execution.

[0011] As a preferred scheme of the real-time AR data superimposition and interaction system for the industrial inspection robot, the multi-source data fusion module comprises a multi-dimensional data acquisition sub-module, a timestamp synchronization sub-module, a space semantic alignment sub-module and a structured stream generation sub-module.

[0012] The multi-dimensional data acquisition sub-module is configured to acquire the original data output by a plurality of sensors carried by the inspection robot, and form a heterogeneous multi-source data set.

[0013] The timestamp synchronization sub-module is configured to perform time alignment on each data item in the heterogeneous multi-source data set output by the multi-dimensional data acquisition sub-module according to the acquisition time, eliminate the time sequence deviation caused by the difference in sampling frequency of the sensors and the transmission delay by using a hybrid time sequence calibration mechanism, and generate a time-consistent multi-modal data sequence.

[0014] The space semantic alignment sub-module is configured to map each item in the time-consistent multi-modal data sequence to a unified industrial scene three-dimensional coordinate system based on the robot space context information of the inspection robot, and perform object semantic anchoring on each item of data in the time-consistent multi-modal data sequence in combination with the inspection semantic context.

[0015] The structured stream generation sub-module is configured to encapsulate the data structure specification of the multi-dimensional inspection context data processed by the space semantic alignment sub-module, and generate a structured real-time data stream.

[0016] As a preferred scheme of the real-time AR data superimposition and interaction system for the industrial inspection robot, the space registration and pose solving module comprises an AR terminal pose solving sub-module, a target device coordinate receiving sub-module and a unified reference system construction sub-module.

[0017] The AR terminal pose solver module is configured to collect visual inertial sensing data through a visual inertial sensing unit arranged on the AR terminal worn by the operator, solve the six-degree-of-freedom pose of the AR terminal in the three-dimensional space of the industrial site in real time based on a visual inertial fusion algorithm, and output a pose estimation result;

[0018] The target device coordinate receiving sub-module is configured to receive three-dimensional coordinate information of the target device reported by the inspection robot in real time through a wireless communication link, the three-dimensional coordinate information being generated by the inspection robot based on joint indexing and positioning assets matching;

[0019] The unified reference system construction sub-module is configured to perform coordinate system unification processing on the AR terminal local pose output by the AR terminal pose solver module and the target device global coordinate obtained by the target device coordinate receiving sub-module, map the AR terminal local pose and the target device global coordinate to the same industrial scene three-dimensional space reference frame through a dynamic coordinate transformation matrix, and establish a geometric alignment relationship between the field of view of the AR terminal and the perception world of the inspection robot.

[0020] As a preferred scheme of the real-time AR data superimposition and interaction system for the industrial inspection robot, the real-time AR rendering engine module comprises a space mapping sub-module, an augmented content generation sub-module, a low-latency rendering scheduling sub-module, and a frame-level display updating sub-module.

[0021] The space mapping sub-module is configured to perform spatial binding on the inspection augmentation information in the structured real-time data stream and the three-dimensional scene representation of the industrial site based on the unified space reference system.

[0022] The augmented content generation sub-module is configured to convert the inspection augmentation information into a renderable AR graphical element in the form of text according to the projection coordinates output by the space mapping sub-module, dynamically adjust the AR graphical element, and generate visual content adapted to the industrial site environment.

[0023] The low-latency rendering scheduling sub-module is configured to implement a rendering resource scheduling strategy on the AR graphical element output by the augmented content generation sub-module, adopt a pose timing cooperative compensation mechanism in combination with the terminal rendering running state of the AR terminal, and control the end-to-end delay of the rendering pipeline from data reception to picture output.

[0024] The frame-level display updating sub-module is configured to perform pixel-level fusion on the AR graphical element processed by the low-latency rendering scheduling sub-module and the real scene video stream captured by the camera of the AR terminal in each video frame period, and push the AR graphical element to the display unit through an AR display driving interface.

[0025] As a preferred embodiment of the real-time AR data overlay and interaction system for industrial inspection robots described in this invention, the natural interaction command parsing module includes a multimodal intent capture submodule, a context-aware semantic understanding submodule, an intent behavior mapping submodule, and a robot command issuing submodule.

[0026] The multimodal intent capture submodule is used to collect the interaction intent signals input by the operator through the multimodal natural interaction perception unit integrated in the AR terminal;

[0027] The context-aware semantic understanding submodule is used to combine the current multi-dimensional inspection context data to perform joint semantic parsing on the multimodal raw interaction data stream and identify the operator's interaction intent in a specific inspection scenario.

[0028] The intent-behavior mapping submodule is used to map the abstract intent into control actions based on the interactive intent output by the context-aware semantic understanding submodule and according to the predefined inspection operation behavior library, thereby generating robot control commands.

[0029] The robot command issuing submodule is used to encapsulate robot control commands through a secure communication protocol and transmit them to the inspection robot in real time via a wireless network, triggering the inspection robot to perform corresponding operations.

[0030] As a preferred embodiment of the real-time AR data overlay and interaction system for industrial inspection robots described in this invention, the context-aware semantic understanding submodule includes a multimodal signal fusion unit, a dynamic task state matching unit, and an ambiguity resolution decision unit.

[0031] The multimodal signal fusion unit is used to align time-aligned multimodal interaction signals and fuse them into a joint interaction representation; the dynamic task state matching unit is used to filter candidate intentions that conform to the inspection logic by combining the inspection task context state in the current multidimensional inspection context data; the ambiguity resolution decision unit is used to perform spatial pointing disambiguation on fuzzy instructions based on candidate intentions.

[0032] As a preferred embodiment of the real-time AR data overlay and interaction system for industrial inspection robots described in this invention, the specific formula of the visual-inertial fusion algorithm is as follows:

[0033]

[0034] in, Indicates the first The posterior state estimate after time-mapping. Indicates the first The prior state estimate at time t. Indicates the first a multi-modal observation vector at the time instant, denotes a nonlinear observation function, denotes the first a Kalman gain matrix at the time instant.

[0035] In a second aspect, an embodiment of the present application provides a real-time AR data superimposition and interaction method for an industrial inspection robot, which comprises: collecting multi-dimensional inspection context data obtained by a sensor carried by the inspection robot, performing spatio-temporal semantic collaborative alignment on the multi-dimensional inspection context data, and generating a structured real-time data stream; solving a pose of an AR terminal worn by an operator in a three-dimensional space of an industrial site in real time through a visual-inertial fusion algorithm, establishing a unified space reference system in combination with target device coordinate information reported by the inspection robot; dynamically superimposing inspection enhancement information in the structured real-time data stream in the form of text to a corresponding physical device position in a field of view of the AR terminal based on the unified space reference system, and completing updating of the AR terminal in the industrial scene by using a real-time rendering optimization mechanism; receiving an interaction intention signal of the operator, performing intention behavior collaborative analysis on a natural interaction instruction in combination with current multi-dimensional inspection context data, generating a corresponding control command, and delivering the control command to the inspection robot for execution.

[0036] In a third aspect, an embodiment of the present application provides a computer device, comprising a memory and a processor, and the memory stores a computer program, wherein: the processor implements any step of the real-time AR data superimposition and interaction system for the industrial inspection robot when executing the computer program.

[0037] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program, wherein: the computer program is executed by a processor to implement any step of the real-time AR data superimposition and interaction system for the industrial inspection robot.

[0038] The present application has the following beneficial effects: the present application realizes high-precision, low-delay, and semantic-consistent closed-loop collaboration between perception data of the inspection robot and display of the AR terminal by constructing four core modules of multi-source data fusion, space registration and pose solving, real-time AR rendering engine, and natural interaction instruction analysis. The system can accurately superimpose device state, fault warning, operation guide, and other inspection enhancement information in the form of text to a corresponding physical device position in the AR field of view of the operator, and supports gesture, voice, eye movement, and other hands-free natural interaction modes to control the robot in real time, significantly improving the intuitiveness of information presentation, the fluency of human-computer collaboration, and the accuracy of on-site decision-making in the industrial inspection process, effectively overcoming the defects of interaction lag, virtual-real misplacement, and operation interruption in the traditional system, and meeting the urgent needs of high-efficiency and immersive human-computer interaction devices in high-risk and complex industrial scenes. BRIEF DESCRIPTION OF DRAWINGS

[0039] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0040] Figure 1 The flowchart illustrates a real-time AR data overlay and interaction system for industrial inspection robots, as provided in an embodiment of the present invention.

[0041] Figure 2 This is a schematic diagram of a real-time AR data overlay and interaction system for industrial inspection robots, provided by an embodiment of the present invention. Detailed Implementation

[0042] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0043] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0044] Secondly, the term "an embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0045] This invention is described in detail with reference to the schematic diagrams. When detailing the embodiments of this invention, for ease of explanation, the cross-sectional views illustrating the device structure may be partially enlarged, not adhering to the usual scale. Furthermore, the schematic diagrams are merely examples and should not be construed as limiting the scope of protection of this invention. In actual fabrication, the three-dimensional spatial dimensions of length, width, and depth should be included.

[0046] Furthermore, in the description of this invention, it should be noted that the terms "upper," "lower," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. These terms are used solely for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. In addition, the terms "first," "second," or "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0047] Unless otherwise explicitly specified and limited, the terms "installation," "connection," and "joining" in this invention should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; similarly, they can refer to mechanical connections, electrical connections, or direct connections, or indirect connections through an intermediate medium, or internal connections between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0048] Example

[0049] Reference Figure 1 and Figure 2 This is the first embodiment of the present invention, which provides a real-time AR data overlay and interaction system for industrial inspection robots, including:

[0050] S1: Multi-source data fusion module, used to collect multi-dimensional inspection context data obtained by the sensors on the inspection robot, perform spatiotemporal semantic collaborative alignment of the multi-dimensional inspection context data, and generate structured real-time data stream.

[0051] The multi-source data fusion module includes a multi-dimensional data acquisition submodule, a timestamp synchronization submodule, a spatial semantic alignment submodule, and a structured stream generation submodule.

[0052] The multidimensional data acquisition submodule is used to acquire raw data output from various sensors on the inspection robot, forming a heterogeneous multi-source data set;

[0053] The timestamp synchronization submodule is used to time-align each data item in the heterogeneous multi-source data set output by the multidimensional data acquisition submodule according to the acquisition time. It adopts a hybrid timing calibration mechanism to eliminate timing deviations caused by differences in sensor sampling frequencies and transmission delays, and generates a time-consistent multimodal data sequence.

[0054] The spatial semantic alignment submodule is used to map each item in a time-consistent multimodal data sequence to a unified industrial scene 3D coordinate system based on the robot spatial context information of the inspection robot. Combined with the inspection semantic context, it performs object semantic anchoring on each data item in the time-consistent multimodal data sequence.

[0055] The structured stream generation submodule is used to encapsulate the data structure specifications of the multidimensional inspection context data processed by the spatial semantic alignment submodule, and generate a structured real-time data stream.

[0056] Furthermore, the multi-source data fusion module achieves efficient transformation from raw perception to structured AR-ready data through a four-level collaborative processing flow: First, the multi-dimensional data acquisition submodule synchronously accesses the raw signals output by various sensors on the inspection robot, such as infrared thermal imagers, visible light cameras, lidar, vibration sensors, and gas detectors, to construct a heterogeneous multi-source data set covering equipment status, environmental characteristics, and task information; Second, the timestamp synchronization submodule performs high-precision alignment of the above heterogeneous data according to their respective acquisition times, and uses a hybrid time-series calibration mechanism combining hardware triggering or software interpolation to effectively compensate for time-series offsets caused by inconsistent sensor sampling rates and communication transmission delays, forming a strictly time-synchronized multimodal data sequence; Subsequently, the spatial semantic alignment submodule maps the temporally consistent multimodal data item by item to a unified industrial scene 3D coordinate system based on the robot's own pose and the environment map it is in (i.e., robot spatial context information). It also combines equipment asset information and the current inspection task target (i.e., inspection semantic context) to bind each data item to a specific physical equipment object and its semantic role in the task, completing object-level semantic anchoring. Finally, the structured stream generation submodule standardizes and encapsulates the spatiotemporally and semantically aligned data according to the predefined data structure specifications of multidimensional inspection context data, outputting a structured real-time data stream with a unified format, complete fields, and low redundancy, providing highly consistent input for subsequent AR visualization and interaction.

[0057] Furthermore, the multi-source data fusion module organizes four sub-modules in an end-to-end pipeline manner to ensure that the raw perception data is accurately transformed into a structured semantic stream that the AR system can directly call: First, the multi-dimensional data acquisition sub-module gathers data from various sensors on the inspection robot in real time, including the equipment temperature field output by the infrared thermal imager, the appearance image captured by the visible light camera, the point cloud generated by the lidar, the mechanical vibration waveform recorded by the vibration sensor, and the environmental component concentration fed back by the gas detector. It also simultaneously acquires contextual information from the task scheduling system (such as the current inspection plan and the target equipment list), forming a heterogeneous multi-source data set covering equipment status, environmental perception, and task intent. Next, the timestamp synchronization sub-module performs fine alignment of the data items from different sensors based on their original acquisition timestamps. For the rate difference between high-frequency IMU and low-frequency visual or gas data, it uses hardware synchronization signals to trigger keyframe acquisition, or uses linear / spline interpolation for software compensation when there is no hardware support, thereby eliminating the timing misalignment caused by transmission jitter and asynchronous sampling, and outputting strictly according to a unified time. The system consists of a multimodal data sequence arranged along axes. Based on this, the spatial semantic alignment submodule utilizes the real-time positioning results of the inspection robot and its constructed or loaded 3D industrial scene map (i.e., robot spatial context information) to transform each time-aligned data item from the sensor's local coordinate system to the global industrial coordinate system. Furthermore, it combines the equipment ID, installation location, and type attributes from the equipment asset database, as well as the inspection items of interest in the current task phase (i.e., inspection semantic context), to assign a clear object label (e.g., #5 transformer A-phase bushing) and semantic role (e.g., temperature measurement point or leak detection area), achieving dual anchoring of cross-modal data at both the physical entity and task logic levels. Finally, the structured stream generation submodule, according to the preset data structure specifications for multidimensional inspection context data, encapsulates the spatiotemporally bound and semantically labeled data items into a standard field combination containing equipment identifier, status value, 3D coordinates, timestamp, and semantic tags. This generates a low-redundancy, highly cohesive structured real-time data stream that can be directly parsed by the AR engine, providing a reliable and consistent data foundation for subsequent precise virtual-real overlay and natural interaction.

[0058] S2: Spatial registration and pose calculation module, used to calculate the pose of the AR terminal worn by the operator in the three-dimensional space of the industrial site in real time through visual-inertial fusion algorithm, and to establish a unified spatial reference system by combining the target equipment coordinate information reported by the inspection robot.

[0059] The spatial registration and pose calculation module includes an AR terminal pose calculation submodule, a target device coordinate receiving submodule, and a unified reference system construction submodule.

[0060] The AR terminal pose calculation submodule is used to collect visual inertial perception data through the visual inertial sensing unit deployed on the AR terminal worn by the operator, and calculate the six-degree-of-freedom pose of the AR terminal in the three-dimensional space of the industrial site online based on the visual inertial fusion algorithm, and output the pose estimation result.

[0061] The target device coordinate receiving submodule is used to receive the target device's three-dimensional coordinate information in the global map coordinate system, which is reported in real time by the inspection robot through the wireless communication link. The three-dimensional coordinate information includes information generated by the inspection robot based on the joint index matching of the positioning assets.

[0062] The Unified Reference System Construction Submodule is used to perform coordinate system unification processing on the local pose of the AR terminal output by the AR terminal pose calculation submodule and the global coordinates of the target device obtained by the target device coordinate receiving submodule. Through a dynamic coordinate transformation matrix, the local pose of the AR terminal and the global coordinates of the target device are mapped to the same three-dimensional spatial reference frame of the industrial scene, establishing a geometric alignment relationship between the AR terminal's field of view and the world perceived by the inspection robot.

[0063] Furthermore, the spatial registration and pose calculation module achieves high-precision alignment between the AR terminal and the inspection robot in a unified spatial coordinate system through three closely cooperating sub-modules: First, the AR terminal pose calculation sub-module utilizes the vision-inertial sensing unit (including camera and IMU) integrated on the AR terminal worn by the operator to collect environmental images and motion sensing data in real time, and estimates the six-degree-of-freedom pose (including position and orientation) of the AR terminal in the three-dimensional space of the industrial site online based on a tightly coupled vision-inertial fusion algorithm, outputting local pose results with high frequency and low drift characteristics; Second, the target equipment coordinate receiving sub-module continuously receives the three-dimensional coordinates of the target equipment reported by the inspection robot through a wireless communication link. The information, these coordinates are not the original sensor readings, but rather the precise location with clear semantic identification and situated in the global map coordinate system, generated by the inspection robot after matching its own global positioning results with the equipment asset database; finally, the unified reference system construction submodule incorporates the local pose of the AR terminal and the global coordinates of the target device into the same spatial framework. By dynamically calculating the coordinate transformation relationship between the two (such as rotation and translation parameters), a unified spatial reference system is constructed that updates in real time as the AR terminal moves, thereby ensuring that the real device seen in the AR terminal's field of view and its corresponding virtual information are strictly aligned in geometric space, establishing a consistent spatial cognitive foundation between humans (AR perspective) and machines (robot perception).

[0064] Furthermore, the spatial registration and pose calculation module achieves dynamic and precise alignment between the AR operator and the inspection robot in a shared 3D space through a three-level progressive processing mechanism: First, the AR terminal pose calculation submodule relies on the AR terminal's built-in visual-inertial sensing unit to continuously capture image frame sequences of the operator's environment and acceleration and angular velocity data output by the IMU. Using a tightly coupled visual-inertial fusion algorithm, it calculates the AR terminal's six-degree-of-freedom pose (i.e., 3D position and 3D orientation) relative to its startup time or local map origin in real time without external positioning assistance. This pose has a high update frequency (e.g., above 100Hz) and low cumulative drift characteristics, suitable for complex lighting and dynamic occlusion scenarios in industrial environments. Second, the target equipment coordinate receiving submodule receives the target equipment coordinate information actively reported by the inspection robot in real time through a low-latency wireless communication channel (e.g., 5G or industrial Wi-Fi). This information is not the original sensor measurement value, but rather the inspection robot's own high-precision global positioning (e.g., based on SLAM or UHF). Based on WB positioning, and combined with a pre-stored equipment asset database (including equipment ID, installation location, geometric contour, etc.), semantic matching is performed to generate 3D spatial coordinates with clear equipment identification and converted to the same global map coordinate system, ensuring global consistency and recognizability. Finally, the unified reference system construction submodule dynamically constructs a unified industrial scene 3D spatial reference framework covering the current work area by calculating the rigid body transformation relationship (including rotation matrix and translation vector) between the two heterogeneous but semantically related spatial data, namely the local pose of the AR terminal (expressed in local coordinate system) and the global coordinates of the target equipment (expressed in map coordinate system) online. This framework is continuously updated as the AR terminal moves and serves as the sole spatial reference for the subsequent AR rendering engine to overlay virtual information, thereby ensuring that the text prompts, alarm signs, and other content seen by the operator in the AR field of view can be stably and accurately attached to the corresponding positions of the real physical equipment, truly realizing human-machine co-visualization and consistent virtual-real spatial collaborative perception.

[0065] S3: Real-time AR rendering engine module, which dynamically overlays the inspection enhancement information in the structured real-time data stream into the corresponding physical device location in the AR terminal's field of view in the form of text, based on a unified spatial reference system. It adopts a real-time rendering optimization mechanism to complete the update of the AR terminal in industrial scenarios.

[0066] The real-time AR rendering engine module includes a spatial mapping submodule, an enhanced content generation submodule, a low-latency rendering scheduling submodule, and a frame-level display update submodule.

[0067] The spatial mapping submodule is used to spatially bind the inspection enhancement information in the structured real-time data stream with the 3D scene representation of the industrial site based on a unified spatial reference system.

[0068] The enhanced content generation submodule is used to convert the inspection enhancement information into renderable AR graphic elements in text form based on the projection coordinates output by the spatial mapping submodule, dynamically adjust the AR graphic elements, and generate visual content adapted to the industrial site environment.

[0069] The low-latency rendering scheduling submodule is used to implement rendering resource scheduling strategies for AR graphic elements output by the enhanced content generation submodule. Combined with the terminal rendering operation status of the AR terminal, a pose and timing collaborative compensation mechanism is adopted to control the end-to-end latency of the rendering pipeline from data reception to screen output.

[0070] The frame-level display update submodule is used to perform pixel-level fusion of the AR graphic elements processed by the low-latency rendering scheduling submodule with the real scene video stream captured by the AR terminal camera within each video frame cycle, and push it to the display unit through the AR display driver interface.

[0071] Furthermore, the real-time AR rendering engine module, through four collaborative sub-modules, achieves low-latency, high-precision, and environmentally adaptive visualization overlay of inspection information on the AR terminal: First, the spatial mapping sub-module, based on a unified spatial reference system established by the spatial registration and pose calculation modules, precisely binds each piece of inspection enhancement information (such as equipment temperature, alarm status, operation instructions, etc.) in the structured real-time data stream to the spatial position of its corresponding physical equipment in the 3D scene representation of the industrial site, determining the anchor point coordinates of the information in 3D space; subsequently, the enhanced content generation sub-module, based on the anchor point coordinates and the current viewpoint of the AR terminal, converts the text-based inspection enhancement information into renderable AR graphic elements (such as labels with backgrounds, highlighted boxes, or directional arrows), and dynamically adjusts the font size, color contrast, transparency, and layout according to the light intensity, background complexity, and user gaze area of ​​the real scene, ensuring that the text is clear in various industrial environments. The AR graphics elements are clearly readable without obscuring key equipment details. Next, the low-latency rendering scheduling submodule intelligently schedules the generated AR graphics elements. Combining the AR terminal's current frame rate target, GPU load, memory usage, and other terminal rendering status, it employs a pose-temporal collaborative compensation mechanism (including IMU-predicted future pose pre-rendering and asynchronous time warp) to prioritize rendering tasks and pre-allocate resources. This effectively suppresses screen stuttering or virtual-real offset caused by system fluctuations, ensuring that the end-to-end latency from data reception to screen output is stably controlled within milliseconds imperceptibly to the human eye. Finally, the frame-level display update submodule performs pixel-level fusion of the optimized AR graphics elements with the real-time video stream captured by the AR terminal's camera within each video frame cycle, generating a composite image of virtual and real overlay. This image is then pushed to the display unit with high synchronization through the AR display driver interface, ensuring that what the operator sees is what they get, achieving a smooth, stable, and accurate industrial AR interactive experience.

[0072] Furthermore, the real-time AR rendering engine module constructs a complete pipeline from data to visual presentation with frame-level precision, ensuring stable, accurate, and seamless virtual-real fusion of inspection enhancement information in complex industrial environments: First, the spatial mapping submodule, relying on a unified spatial reference system (which has aligned the AR terminal pose with the global coordinates of the equipment reported by the inspection robot), spatially binds each piece of inspection enhancement information in the structured real-time data stream (such as #3 pump bearing temperature: 85℃ or valve V201 abnormally open) to the geometric center or key feature point of the corresponding physical equipment in the 3D scene representation of the industrial site, generating a one-to-one spatial binding with world coordinate attributes. The information anchor point is then used. Subsequently, the enhanced content generation submodule, based on the world coordinates of this anchor point and the AR terminal's current six-DOF viewpoint, calculates its two-dimensional screen position on the camera image plane through perspective projection. Based on this, it generates lightweight text-based AR graphic elements, including anti-aliased fonts, semi-transparent backgrounds, outlines, or directional indicators. Furthermore, based on real-time perceived ambient lighting (e.g., strong backlight, low illumination), background texture complexity (e.g., dense pipe areas), and the user's gaze focus provided by eye tracking, it adaptively adjusts text size, color contrast (e.g., highlighted red warning, gray normal state), transparency, and layout to prevent information from being obscured or overlapping with other elements. Background obfuscation; based on this, the low-latency rendering scheduling submodule dynamically prioritizes all AR graphics elements to be rendered, for example, placing alarm-type information above the normal state. It also combines the AR terminal's current frame rate target (e.g., 60fps), GPU utilization, memory bandwidth, and other terminal rendering statuses to initiate a pose-timing collaborative compensation mechanism: on the one hand, it uses high-frequency IMU data to predict the terminal pose at the next frame display time, completing viewpoint correction and content pre-rendering in advance; on the other hand, it triggers asynchronous time warp (ATW) when the main rendering thread is delayed, performing geometric reprojection on the already rendered image, thus ensuring a smooth transition from receiving structured data to final display. End-to-end latency is compressed to less than 20 milliseconds, eliminating the misalignment between virtual and real objects caused by motion. Finally, the frame-level display update submodule performs subpixel-level alignment and alpha mixing of the optimized AR graphic elements and the real scene video stream synchronously captured by the AR terminal camera within each video frame cycle (e.g., 16.7ms@60fps) to generate a seamlessly integrated augmented reality image. This image is then pushed to the near-eye display unit via a dedicated AR display driver interface in a hard-synchronized manner, ensuring that the virtual information remains firmly attached to the real device as the operator moves, turns their head, or scans quickly, achieving truly immersive and highly reliable industrial AR human-machine interaction.

[0073] S4: Natural Interaction Command Parsing Module, used to receive the operator's interaction intent signal, combine it with the current multi-dimensional inspection context data to perform intention behavior collaborative parsing of natural interaction commands, generate corresponding control commands and send them to the inspection robot for execution.

[0074] The natural interaction instruction parsing module includes a multimodal intent capture submodule, a context-aware semantic understanding submodule, an intent behavior mapping submodule, and a robot instruction issuance submodule.

[0075] The multimodal intent capture submodule is used to collect the interaction intent signals input by the operator through the multimodal natural interaction perception unit integrated into the AR terminal;

[0076] The context-aware semantic understanding submodule is used to combine the current multi-dimensional inspection context data to perform joint semantic parsing on the multimodal raw interaction data stream and identify the operator's interaction intent in a specific inspection scenario;

[0077] The Intent-Behavior Mapping Submodule is used to map the abstract intent into control actions based on the interactive intent output by the Context-Aware Semantic Understanding Submodule and a predefined inspection operation behavior library, thereby generating robot control commands.

[0078] The robot command issuing submodule is used to encapsulate robot control commands through a secure communication protocol and transmit them to the inspection robot in real time via a wireless network, triggering the inspection robot to perform corresponding operations.

[0079] Furthermore, the natural interaction command parsing module achieves a semantic closed loop from the operator's natural behavior to the precise control of the inspection robot through a four-level progressive processing flow: First, the multimodal intent capture submodule utilizes the multimodal natural interaction perception unit integrated into the AR terminal (including gesture recognition sensors, microphone arrays, and eye-tracking modules) to simultaneously collect the operator's gesture trajectories, voice commands, and gaze focus positions during the operation, forming raw multimodal interaction signals containing spatial orientation, semantic keywords, and attention focus; subsequently, the context-aware semantic understanding submodule deeply integrates this raw signal with the current multidimensional inspection context data (such as the equipment being inspected, the equipment's real-time status, the current task stage, and historical operation records), and through joint semantic understanding... The system resolves ambiguities in vague expressions such as "view this" or "check it out," accurately identifying the operator's true interaction intent in a specific industrial scenario (e.g., performing infrared temperature measurement on the currently viewed transformer). Next, the intent-behavior mapping submodule, based on a predefined inspection operation behavior library (containing a set of standard operations matching equipment type and task flow, such as initiating temperature measurement, requesting video feedback, and marking anomalies), transforms abstract interaction intents into structured, executable robot control commands, ensuring that the instructions comply with safety regulations and operational logic. Finally, the robot command issuing submodule encapsulates the generated control commands using industrial-grade secure communication protocols (such as TLS encryption or MQTT-SN) and transmits them in real-time to the inspection robot via a low-latency wireless network (such as a 5G private network or Wi-Fi 6), triggering the robot to immediately execute the corresponding action, thus completing end-to-end closed-loop control from natural human-machine interaction to robot behavior response.

[0080] Preferably, the context-aware semantic understanding submodule includes a multimodal signal fusion unit, a dynamic task state matching unit, and an ambiguity resolution decision unit;

[0081] The multimodal signal fusion unit is used to align time-aligned multimodal interaction signals and fuse them into a joint interaction representation; the dynamic task state matching unit is used to filter candidate intentions that conform to the inspection logic by combining the inspection task context state in the current multidimensional inspection context data; the ambiguity resolution decision unit is used to perform spatial pointing disambiguation on fuzzy instructions based on candidate intentions.

[0082] The specific formula for the visual-inertial fusion algorithm is as follows:

[0083]

[0084] in, Indicates the first The posterior state estimate after time-mapping. Indicates the first The prior state estimate at time t. Indicates the first The multimodal observation vector at time 1. Represents a nonlinear observation function. Indicates the first The Kalman gain matrix at time t.

[0085] In a preferred embodiment, a real-time AR data overlay and interaction method for industrial inspection robots includes: collecting multi-dimensional inspection context data acquired by sensors mounted on the inspection robot; performing spatiotemporal semantic co-alignment on the multi-dimensional inspection context data to generate a structured real-time data stream; using a visual-inertial fusion algorithm to calculate the pose of the AR terminal worn by the operator in the three-dimensional space of the industrial site in real time, and establishing a unified spatial reference system by combining the target equipment coordinate information reported by the inspection robot; based on the unified spatial reference system, dynamically overlaying the inspection enhancement information in the structured real-time data stream onto the corresponding physical equipment position in the AR terminal's field of view in text form, and using a real-time rendering optimization mechanism to update the AR terminal in the industrial scenario; receiving the operator's interaction intent signal, performing intention behavior co-analysis on the natural interaction instructions in combination with the current multi-dimensional inspection context data, generating corresponding control commands, and issuing them to the inspection robot for execution.

[0086] The above-mentioned unit modules can be embedded in the processor of the computer device in hardware form or independent of it, or they can be stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of the above modules.

[0087] In one embodiment, a computer device is provided, which may be a terminal. The computer device includes a processor, memory, a communication interface, a display screen, and an input device connected via a system bus. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage medium. The communication interface of the computer device is used for wired or wireless communication with external terminals. Wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen of the computer device may be an LCD screen or an e-ink display screen. The input device of the computer device may be a touch layer covering the display screen, or buttons, a trackball, or a touchpad located on the casing of the computer device, or an external keyboard, touchpad, or mouse, etc.

[0088] In summary, this invention achieves high-precision, low-latency, and semantically consistent closed-loop collaboration between the inspection robot's perceived data and the AR terminal display by constructing four core modules: multi-source data fusion, spatial registration and pose calculation, real-time AR rendering engine, and natural interaction command parsing. The system can accurately overlay enhanced inspection information such as equipment status, fault warnings, and operation instructions onto the corresponding physical equipment locations in the operator's AR field of view in text form. It also supports real-time robot control via handheld natural interaction methods such as gestures, voice, and eye tracking, significantly improving the intuitiveness of information presentation, the smoothness of human-machine collaboration, and the accuracy of on-site decision-making during industrial inspections. This effectively overcomes the shortcomings of traditional systems, such as interaction lag, virtual-real misalignment, and operation interruption, meeting the urgent need for efficient and immersive human-machine interaction equipment in high-risk and complex industrial scenarios.

[0089] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A real-time AR data overlay and interaction system for industrial inspection robots, characterized in that: include, The multi-source data fusion module is used to collect multi-dimensional inspection context data obtained by the sensors on the inspection robot, perform spatiotemporal semantic collaborative alignment on the multi-dimensional inspection context data, and generate a structured real-time data stream. The spatial registration and pose calculation module is used to calculate the pose of the AR terminal worn by the operator in the three-dimensional space of the industrial site in real time through the visual-inertial fusion algorithm. Combined with the target equipment coordinate information reported by the inspection robot, a unified spatial reference system is established. The real-time AR rendering engine module is used to dynamically overlay the inspection enhancement information in the structured real-time data stream into the corresponding physical device location in the AR terminal's field of view in the form of text, based on a unified spatial reference system. It adopts a real-time rendering optimization mechanism to complete the update of the AR terminal in industrial scenarios. The natural interaction command parsing module is used to receive the operator's interaction intent signal, combine it with the current multi-dimensional inspection context data to perform intention behavior collaborative parsing of the natural interaction command, generate the corresponding control command, and send it to the inspection robot for execution.

2. The real-time AR data overlay and interaction system for industrial inspection robots as described in claim 1, characterized in that: The multi-source data fusion module includes a multi-dimensional data acquisition submodule, a timestamp synchronization submodule, a spatial semantic alignment submodule, and a structured stream generation submodule; The multidimensional data acquisition submodule is used to acquire raw data output by various sensors mounted on the inspection robot, forming a heterogeneous multi-source data set; The timestamp synchronization submodule is used to time-align each data item in the heterogeneous multi-source data set output by the multidimensional data acquisition submodule according to the acquisition time. It adopts a hybrid timing calibration mechanism to eliminate timing deviations caused by differences in sensor sampling frequencies and transmission delays, and generates a time-consistent multimodal data sequence. The spatial semantic alignment submodule is used to map each item in the time-consistent multimodal data sequence to a unified industrial scene three-dimensional coordinate system based on the robot spatial context information of the inspection robot. Combined with the inspection semantic context, it performs object semantic anchoring on each data item in the time-consistent multimodal data sequence. The structured stream generation submodule is used to encapsulate the data structure specifications of the multidimensional inspection context data processed by the spatial semantic alignment submodule, and generate a structured real-time data stream.

3. The real-time AR data overlay and interaction system for industrial inspection robots as described in claim 2, characterized in that: The spatial registration and pose calculation module includes an AR terminal pose calculation submodule, a target device coordinate receiving submodule, and a unified reference system construction submodule. The AR terminal pose calculation submodule is used to collect visual inertial perception data through the visual inertial sensing unit deployed on the AR terminal worn by the operator, calculate the six-degree-of-freedom pose of the AR terminal in the three-dimensional space of the industrial site online based on the visual inertial fusion algorithm, and output the pose estimation result. The target device coordinate receiving submodule is used to receive the target device's three-dimensional coordinate information in the global map coordinate system, which is reported in real time by the inspection robot through the wireless communication link. The three-dimensional coordinate information includes information generated by the inspection robot based on the joint index matching of the positioning assets. The unified reference system construction submodule is used to perform coordinate system unification processing on the local pose of the AR terminal output by the AR terminal pose calculation submodule and the global coordinates of the target device obtained by the target device coordinate receiving submodule. Through a dynamic coordinate transformation matrix, the local pose of the AR terminal and the global coordinates of the target device are mapped to the same three-dimensional spatial reference frame of the industrial scene, and a geometric alignment relationship is established between the AR terminal's field of view and the world perceived by the inspection robot.

4. The real-time AR data overlay and interaction system for industrial inspection robots as described in claim 3, characterized in that: The real-time AR rendering engine module includes a spatial mapping submodule, an enhanced content generation submodule, a low-latency rendering scheduling submodule, and a frame-level display update submodule. The spatial mapping submodule is used to spatially bind the inspection enhancement information in the structured real-time data stream with the three-dimensional scene representation of the industrial site based on a unified spatial reference system. The enhanced content generation submodule is used to convert the inspection enhancement information into renderable AR graphic elements in text form according to the projection coordinates output by the spatial mapping submodule, dynamically adjust the AR graphic elements, and generate visual content adapted to the industrial site environment. The low-latency rendering scheduling submodule is used to implement rendering resource scheduling strategies for AR graphic elements output by the enhanced content generation submodule. Combined with the terminal rendering operation status of the AR terminal, a pose and timing collaborative compensation mechanism is adopted to control the end-to-end latency of the rendering pipeline from data reception to screen output. The frame-level display update submodule is used to perform pixel-level fusion of the AR graphic elements processed by the low-latency rendering scheduling submodule with the real scene video stream captured by the AR terminal camera within each video frame period, and push it to the display unit through the AR display driver interface.

5. The real-time AR data overlay and interaction system for industrial inspection robots as described in claim 4, characterized in that: The natural interaction instruction parsing module includes a multimodal intent capture submodule, a context-aware semantic understanding submodule, an intent behavior mapping submodule, and a robot instruction issuing submodule; The multimodal intent capture submodule is used to collect the interaction intent signals input by the operator through the multimodal natural interaction perception unit integrated in the AR terminal; The context-aware semantic understanding submodule is used to combine the current multi-dimensional inspection context data to perform joint semantic parsing on the multimodal raw interaction data stream and identify the operator's interaction intent in a specific inspection scenario. The intent-behavior mapping submodule is used to map the abstract intent into control actions based on the interactive intent output by the context-aware semantic understanding submodule and according to the predefined inspection operation behavior library, thereby generating robot control commands. The robot command issuing submodule is used to encapsulate robot control commands through a secure communication protocol and transmit them to the inspection robot in real time via a wireless network, triggering the inspection robot to perform corresponding operations.

6. The real-time AR data overlay and interaction system for industrial inspection robots as described in claim 5, characterized in that: The context-aware semantic understanding submodule includes a multimodal signal fusion unit, a dynamic task state matching unit, and an ambiguity resolution decision unit; The multimodal signal fusion unit is used to align time-aligned multimodal interaction signals and fuse them into a joint interaction representation; the dynamic task state matching unit is used to combine the inspection task context state in the current multidimensional inspection context data to filter candidate intentions that conform to the inspection logic. The ambiguity resolution decision unit is used to perform spatial pointing disambiguation on fuzzy instructions based on candidate intentions.

7. The real-time AR data overlay and interaction system for industrial inspection robots as described in claim 6, characterized in that: The specific formula for the visual-inertial fusion algorithm is as follows: in, Indicates the first The posterior state estimate after time-mapping. Indicates the first The prior state estimate at time t. Indicates the first The multimodal observation vector at time 1. Represents a nonlinear observation function. Indicates the first The Kalman gain matrix at time t.

8. A real-time AR data overlay and interaction method for industrial inspection robots, based on the real-time AR data overlay and interaction system for industrial inspection robots as described in any one of claims 1 to 7, characterized in that: include, Collect multi-dimensional inspection context data acquired by the sensors mounted on the inspection robot, perform spatiotemporal semantic co-alignment on the multi-dimensional inspection context data, and generate a structured real-time data stream; The pose of the AR terminal worn by the operator in the three-dimensional space of the industrial site is calculated in real time by visual-inertial fusion algorithm. Combined with the target equipment coordinate information reported by the inspection robot, a unified spatial reference system is established. Based on a unified spatial reference system, the inspection enhancement information in the structured real-time data stream is dynamically overlaid in text form onto the corresponding physical device location in the AR terminal's field of view. A real-time rendering optimization mechanism is used to complete the update of the AR terminal in industrial scenarios. It receives the operator's interaction intent signal, combines it with the current multi-dimensional inspection context data to perform intention behavior collaborative parsing of natural interaction instructions, generates corresponding control commands, and sends them to the inspection robot for execution.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the real-time AR data overlay and interaction system for industrial inspection robots as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the real-time AR data overlay and interaction system for industrial inspection robots as described in any one of claims 1 to 7.

Images