Inspection interaction control method based on three-dimensional live-action virtual-real mapping

Abstract

The invention discloses an inspection interaction control method based on three-dimensional live-action virtual-real mapping, belongs to the technical field of industrial inspection and digital operation and maintenance, and aims to solve the problems of virtual-real space interaction splitting, model updating lagging and control closed loop deficiency in a traditional inspection mode. A system integrating laser scanning and positioning is used for collecting live-action data and constructing a high-fidelity three-dimensional live-action model, then high-precision feature matching and space alignment are carried out on the high-fidelity three-dimensional live-action model and a preset virtual inspection model, a stable virtual-real mapping relation is established, an operator can input a control instruction in a natural interaction mode, and the real-time performance of the system is improved. According to the mapping relation, the instruction is analyzed into specific control parameters for the physical scanning system, the physical scanning system is driven to adjust the scanning posture or path so as to pertinently collect and update data, and the three-dimensional live-action model is fused and updated in real time. According to the method, accurate linkage of virtual and real spaces and dynamic enhancement of the model in the inspection process are realized, and the interactivity, the real-time performance and the intelligent level of inspection are remarkably improved.

Description

Technical Field

[0001] This invention discloses an inspection interactive control method based on three-dimensional real-scene virtual-real mapping, belonging to the field of industrial inspection and digital operation and maintenance technology. Background Technology

[0002] In many fields such as industrial equipment operation and maintenance and infrastructure management, regular and efficient inspections are crucial for ensuring facility safety and preventing malfunctions. Traditional inspection methods mainly rely on manual on-site inspection and recording, which is not only inefficient but also limited by personnel experience, environmental complexity, and safety hazards, making it difficult to achieve comprehensive and accurate detection of complex structures or hidden areas. With the development of digital technology, using 3D scanning to obtain real-world models and constructing virtual digital models for comparative analysis has become an important direction for improving inspection quality.

[0003] Currently, existing technologies attempt to combine 3D reality models with pre-built virtual design models to assist in inspection analysis. However, current methods typically treat data acquisition, model building, and interactive analysis as separate processes. The construction of reality models is often a one-time or static process, with subsequent analysis and interaction primarily based on the completed data. When specific areas require focused attention during inspections, or when equipment status changes, existing technologies struggle to dynamically and specifically drive data acquisition devices to perform local rescans and model updates. This results in the inability to effectively maintain the real-time correspondence between virtual and reality models, leading to information lag.

[0004] Furthermore, existing inspection interactive control methods are mostly limited to operations on virtual models. Their control commands cannot directly and accurately feed back to and control data acquisition devices in the physical space, lacking a closed-loop control link from "virtual interaction intent" to "dynamic acquisition of real-world data." This disconnect between virtual and physical space interaction limits the flexibility, real-time performance, and intelligence of inspections, making it difficult to proactively adjust data acquisition strategies based on clues discovered on-site or pre-defined, precise needs, thus affecting the depth and efficiency of inspections. Therefore, there is an urgent need for an inspection method that can achieve precise mapping between virtual and physical spaces and support dynamic control of real-world data acquisition through natural interaction. Summary of the Invention

[0005] To achieve the above objectives, this application provides the following technical solution: According to a first aspect of the present invention, the present invention claims protection for an inspection interactive control method based on three-dimensional real-scene virtual-real mapping, comprising the following steps: S1 provides a three-dimensional scanning system, which integrates a laser scanning module and a positioning module for collecting three-dimensional geometric data and spatial location data of the target inspection area; S2, Based on the three-dimensional geometric data and spatial location data, a three-dimensional real-scene model of the target inspection area is constructed using a point cloud processing algorithm. The three-dimensional real-scene model includes geometric structure and surface texture. S3, Load the virtual inspection model corresponding to the target inspection area. The virtual inspection model is a digital model created based on design drawings or historical data. S4, execute the virtual-real mapping process, align the three-dimensional real scene model and the virtual inspection model in space through feature matching and coordinate transformation, and establish a one-to-one mapping relationship between the real scene and the virtual scene; S5, on the inspection interaction platform, receive inspection control commands input through the interactive interface, the inspection control commands including view control, path navigation or object selection; S6, based on the virtual-real mapping relationship, the inspection control command is parsed into a control parameter adjustment command for the three-dimensional scanning system; S7. According to the control parameter adjustment command, drive the actuator of the three-dimensional scanning system to adjust the scanning posture or scanning path to change the data acquisition area. S8: Real-time acquisition of updated data collected by the 3D scanning system after adjustment, and integration of the updated data into the 3D real-scene model to achieve dynamic updating of the model.

[0006] Furthermore, the laser scanning module of the three-dimensional scanning system is a line laser scanner or an area array lidar, and the positioning module is a combined positioning unit of an inertial measurement unit and a global navigation satellite system. The steps for collecting three-dimensional geometric data and spatial position data of the target inspection area include: controlling the laser scanning module to emit laser beams at a preset sampling frequency and receiving echoes to generate discrete point cloud data; at the same time, the positioning module records the spatial coordinates and attitude angles corresponding to each point cloud data in real time, and binds and stores the point cloud data with the spatial coordinates and attitude angles.

[0007] Furthermore, the step of constructing a 3D reality model based on 3D geometric data and spatial location data includes: The three-dimensional geometric data collected by the laser scanning module and the spatial location data recorded by the positioning module are spatiotemporally fused to generate a point cloud dataset with global coordinate system labels, in which each point contains three-dimensional coordinates and reflection intensity information. Spatial sorting and redundancy removal are performed on the point cloud dataset. Point cloud fragments collected at different times are stitched together into a continuous and complete point cloud surface representation through a point cloud registration algorithm to ensure the consistency of point clouds in overlapping areas. A surface reconstruction algorithm based on the Poisson equation is applied to triangulate the point cloud surface to construct an initial geometric mesh model, which consists of vertices, edges, and triangular faces, while preserving the topological structure. From the color image sequence synchronously acquired by the laser scanning module, the image pixels are mapped onto the corresponding triangular facets of the geometric mesh model through camera calibration and extrinsic parameter calibration. Texture mapping is performed using ultraviolet coordinates to form a preliminary textured 3D model. The initial 3D model is geometrically refined and smoothed, and the density and shape of the triangular facets are adjusted to eliminate holes and distortions. At the same time, the color differences between texture images are integrated to output the optimized final 3D reality model, which supports real-time rendering and interactive operation.

[0008] Furthermore, the steps of performing the virtual-real mapping process include: Feature elements with significant geometric or textural features, including corner points, edge contours, and planar regions, are extracted from the 3D real-world model and the virtual inspection model, respectively. A feature descriptor describing the location and attributes of each feature element is generated. The similarity between the feature descriptors of the 3D real scene model and the feature descriptors of the virtual inspection model is calculated. The correspondence between feature elements is established by nearest neighbor search, and incorrect matching pairs are eliminated to improve the matching reliability. Based on the correspondence pair, the iterative nearest point optimization algorithm is used to calculate the spatial transformation parameters from the virtual inspection model to the 3D real scene model. The spatial transformation parameters include rotation matrix, translation vector and scaling factor to ensure that feature alignment error is minimized. The spatial transformation parameters are applied to the coordinates of all vertices of the virtual inspection model to perform an overall rigid transformation or non-rigid deformation on the virtual inspection model, so that it is precisely aligned with the key feature areas of the three-dimensional real scene model in three-dimensional space. The mapping accuracy is evaluated by calculating the distance between the sampled points of the aligned model and analyzing the visual overlap. If the accuracy does not meet the preset tolerance standard, the feature extraction sub-step is automatically returned for iterative optimization until the mapping accuracy meets the requirements, and the final virtual-real mapping relationship matrix is ​​solidified.

[0009] Furthermore, the interactive interface is a gesture recognition interface in a virtual reality headset or an eye-tracking interface in augmented reality glasses, and the inspection control command is generated through gesture, eye movement, or voice input. The step of receiving inspection control commands input by the operator through the interactive interface includes: The system captures the operator's hand gestures, eye movements, or voice commands through sensors, converts them into digital signals, and identifies them as specific view control commands, path navigation commands, or object selection commands through the command parsing module.

[0010] Furthermore, the step of parsing the inspection control command into control parameter adjustment commands for the 3D scanning system includes: Based on the type and parameters of the inspection control commands, the predefined command mapping rule library is queried to parse the view control commands into the pitch and yaw angle adjustment values ​​of the laser scanning module, the path navigation commands into the speed and direction control sequence of the mobile platform, and the object selection commands into the boundary coordinates and resolution settings of the scanning area.

[0011] Furthermore, the step of adjusting the scanning posture or scanning path by the actuator of the driving 3D scanning system includes: The servo motor of the gimbal is controlled to rotate to change the laser emission direction of the laser scanning module, or the linear motor is controlled to drive the moving platform to move along the Bézier curve trajectory in three-dimensional space. At the same time, the scanning frequency and scanning range of the laser scanning module are adjusted to ensure the continuity and integrity of data acquisition.

[0012] Furthermore, the step of integrating the updated data into the 3D reality model includes: The newly acquired point cloud data is matched with the point cloud of the existing 3D reality model using an octree-based spatial index matching. The overlapping areas are merged and new area point clouds are added using a point cloud fusion algorithm. Then, the triangular mesh vertices and texture images of the corresponding areas are updated, and the mesh normals are recalculated to maintain the lighting consistency of the model.

[0013] Furthermore, the step of generating a combined virtual and real inspection view through fusion rendering includes: In the graphics rendering pipeline of the display terminal, the 3D real scene model is used as the base layer for real-time lighting rendering, and the virtual inspection model is used as the overlay layer for highlighting and contour enhancement rendering. The two images are superimposed through transparency blending technology, and interactive controls and status information are superimposed on the view to form an interactive virtual-real fusion visualization interface.

[0014] Furthermore, the method also includes a mapping maintenance step: During the inspection process, verification point cloud data of key areas is periodically collected through a 3D scanning system and compared with the virtual inspection model for local features. When the deviation in the mapping relationship is detected to exceed the dynamic threshold, the local remapping subprocess is automatically triggered to update the virtual-real mapping relationship matrix and write the adjustment record to the log file for traceability analysis.

[0015] This invention discloses an interactive control method for inspection based on 3D real-scene virtual-real mapping, belonging to the field of industrial inspection and digital operation and maintenance technology. It aims to solve the problems of fragmented interaction between virtual and real spaces, lagging model updates, and lack of control closed loops in traditional inspection methods. By integrating laser scanning and positioning systems to collect real-scene data and construct a high-fidelity 3D real-scene model, it then performs high-precision feature matching and spatial alignment with a preset virtual inspection model, establishing a stable virtual-real mapping relationship. Operators can input control commands through natural interaction. Based on this mapping relationship, the commands are parsed into specific control parameters for the physical scanning system, driving it to adjust its scanning posture or path to collect and update data in a targeted manner, and the 3D real-scene model is updated in real time. This invention achieves precise linkage between virtual and real spaces and dynamic model enhancement during the inspection process, significantly improving the interactivity, real-time performance, and intelligence level of the inspection. Attached Figure Description

[0016] Figure 1 The flowchart illustrates the process of an inspection interactive control method based on three-dimensional real-scene virtual-real mapping, as claimed in an embodiment of the present invention. Figure 2 The second flowchart is shown for an inspection interactive control method based on three-dimensional real-scene virtual-real mapping, as claimed in the embodiments of the present invention. Figure 3 The third flowchart is a method for interactive inspection control based on three-dimensional real-scene mapping, as claimed in an embodiment of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0018] The terms "first," "second," and "third" in this application are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first," "second," or "third" may explicitly or implicitly include at least one of those features. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified. All directional indications in the embodiments of this application, such as up, down, left, right, front, back, etc., are only used to explain the relative positional relationships and movements between components in a specific orientation as shown in the accompanying drawings. If the specific orientation changes, the directional indications will change accordingly. Furthermore, the terms "including" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0019] References to embodiments herein mean that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0020] According to a first embodiment of the present invention, the present invention claims protection for an inspection interactive control method based on three-dimensional real-scene virtual-real mapping, referring to... Figure 1 This includes the following steps: S1 provides a three-dimensional scanning system, which integrates a laser scanning module and a positioning module for collecting three-dimensional geometric data and spatial location data of the target inspection area; S2, Based on the three-dimensional geometric data and spatial location data, a three-dimensional real-scene model of the target inspection area is constructed using a point cloud processing algorithm. The three-dimensional real-scene model includes geometric structure and surface texture. S3, Load the virtual inspection model corresponding to the target inspection area. The virtual inspection model is a digital model created based on design drawings or historical data. S4, execute the virtual-real mapping process, align the three-dimensional real scene model and the virtual inspection model in space through feature matching and coordinate transformation, and establish a one-to-one mapping relationship between the real scene and the virtual scene; S5, on the inspection interaction platform, receive inspection control commands input through the interactive interface, the inspection control commands including view control, path navigation or object selection; S6, based on the virtual-real mapping relationship, the inspection control command is parsed into a control parameter adjustment command for the three-dimensional scanning system; S7. According to the control parameter adjustment command, drive the actuator of the three-dimensional scanning system to adjust the scanning posture or scanning path to change the data acquisition area. S8: Real-time acquisition of updated data collected by the 3D scanning system after adjustment, and integration of the updated data into the 3D real-scene model to achieve dynamic updating of the model.

[0021] Furthermore, the laser scanning module of the three-dimensional scanning system is a line laser scanner or an area array lidar, and the positioning module is a combined positioning unit of an inertial measurement unit and a global navigation satellite system. The steps for collecting three-dimensional geometric data and spatial position data of the target inspection area include: controlling the laser scanning module to emit laser beams at a preset sampling frequency and receiving echoes to generate discrete point cloud data; at the same time, the positioning module records the spatial coordinates and attitude angles corresponding to each point cloud data in real time, and binds and stores the point cloud data with the spatial coordinates and attitude angles.

[0022] This embodiment details the application of a three-dimensional real-scene virtual-real mapping-based inspection interactive control method in the inspection of large-scale chemical equipment such as heat exchanger tube bundles.

[0023] An integrated 3D scanning system is provided, mounted on a mobile inspection robot platform. The system's laser scanning module employs a high-precision line laser scanner, while the positioning module utilizes a positioning unit combining an inertial measurement unit (IMU) and a differential global navigation satellite system (GNSS). An operator remotely controls the robot to enter the target heat exchanger area and activates the system. The laser scanner emits a laser beam at a preset high frequency and receives the echoes reflected from the equipment surface, generating dense point cloud data. Simultaneously, the positioning module continuously calculates and outputs the scanner probe's precise 3D coordinates and attitude angles (pitch, yaw, roll) in the global coordinate system. Each acquired point cloud data point is bound to its spatial coordinates and attitude angles at the time of acquisition, forming a raw dataset with spatiotemporal labels.

[0024] Furthermore, referring to Figure 2 The steps for constructing a 3D reality model based on 3D geometric data and spatial location data include: The three-dimensional geometric data collected by the laser scanning module and the spatial location data recorded by the positioning module are spatiotemporally fused to generate a point cloud dataset with global coordinate system labels, in which each point contains three-dimensional coordinates and reflection intensity information. Spatial sorting and redundancy removal are performed on the point cloud dataset. Point cloud fragments collected at different times are stitched together into a continuous and complete point cloud surface representation through a point cloud registration algorithm to ensure the consistency of point clouds in overlapping areas. A surface reconstruction algorithm based on the Poisson equation is applied to triangulate the point cloud surface to construct an initial geometric mesh model, which consists of vertices, edges, and triangular faces, while preserving the topological structure. From the color image sequence synchronously acquired by the laser scanning module, the image pixels are mapped onto the corresponding triangular facets of the geometric mesh model through camera calibration and extrinsic parameter calibration. Texture mapping is performed using ultraviolet coordinates to form a preliminary textured 3D model. The initial 3D model is geometrically refined and smoothed, and the density and shape of the triangular facets are adjusted to eliminate holes and distortions. At the same time, the color differences between texture images are integrated to output the optimized final 3D reality model, which supports real-time rendering and interactive operation.

[0025] In this embodiment, after receiving the raw dataset, the central processing unit first performs data fusion. The system strictly synchronizes the point cloud data with the positioning data according to the timestamp, ensuring that the geometric position information of each point is associated with the correct global coordinates. Next, point cloud integration is performed. The algorithm automatically identifies overlapping areas between consecutive scan segments and, through a feature-based point cloud registration method, accurately aligns and stitches together point cloud segments collected at different times and from different perspectives into a complete and seamless point cloud model of the outer surface of the heat exchanger tube bundle. Then, in the mesh generation stage, the system uses a surface reconstruction algorithm to process the point cloud, generating an initial mesh model composed of numerous triangular facets that can fully represent the geometry of the device. In the texture bonding stage, the system calls the high-resolution color images synchronously acquired by the scanner. Using the calibrated camera intrinsic and extrinsic parameters, the image pixels are accurately projected and pasted onto the corresponding mesh triangles, giving the model realistic color and texture. Finally, model optimization is performed. The system automatically detects and fills small holes caused by occlusion in the mesh, smooths jagged edges, and balances the texture color difference from different image sources, outputting a high-fidelity 3D real-world model that can be used for real-time rendering.

[0026] Furthermore, referring to Figure 3 The steps of performing the virtual-real mapping process include: Feature elements with significant geometric or textural features, including corner points, edge contours, and planar regions, are extracted from the 3D real-world model and the virtual inspection model, respectively. A feature descriptor describing the location and attributes of each feature element is generated. The similarity between the feature descriptors of the 3D real scene model and the feature descriptors of the virtual inspection model is calculated. The correspondence between feature elements is established by nearest neighbor search, and incorrect matching pairs are eliminated to improve the matching reliability. Based on the correspondence pair, the iterative nearest point optimization algorithm is used to calculate the spatial transformation parameters from the virtual inspection model to the 3D real scene model. The spatial transformation parameters include rotation matrix, translation vector and scaling factor to ensure that feature alignment error is minimized. The spatial transformation parameters are applied to the coordinates of all vertices of the virtual inspection model to perform an overall rigid transformation or non-rigid deformation on the virtual inspection model, so that it is precisely aligned with the key feature areas of the three-dimensional real scene model in three-dimensional space. The mapping accuracy is evaluated by calculating the distance between the sampled points of the aligned model and analyzing the visual overlap. If the accuracy does not meet the preset tolerance standard, the feature extraction sub-step is automatically returned for iterative optimization until the mapping accuracy meets the requirements, and the final virtual-real mapping relationship matrix is ​​solidified.

[0027] In this embodiment, the original design model of the heat exchanger is loaded from the database as a virtual inspection model. The virtual-to-real mapping process is then executed. First, the feature extraction sub-step extracts points, lines, and surfaces with significant geometric features from 3D reality models such as the edges of pipe holes on a pipe sheet, the outline of a flange, and the virtual design model, respectively, and generates feature descriptors describing the spatial distribution of these features. Then, the feature matching sub-step establishes a large number of reliable corresponding point pairs between the reality model and the virtual model by calculating the similarity between the descriptors. Next, the transformation calculation sub-step uses these corresponding point pairs and an optimization algorithm to calculate an optimal spatial transformation matrix containing rotation and translation. The model alignment sub-step applies this transformation matrix to the entire virtual design model, making it completely coincident and aligned with the newly constructed 3D reality model in 3D space. The mapping verification sub-step evaluates the accuracy by calculating the average distance between key points on the surfaces of the two models after alignment. In this embodiment, this distance is controlled at a very low level, indicating successful mapping, and the mapping relationship matrix is ​​saved.

[0028] Furthermore, the interactive interface is a gesture recognition interface in a virtual reality headset or an eye-tracking interface in augmented reality glasses, and the inspection control command is generated through gesture, eye movement, or voice input. The step of receiving inspection control commands input by the operator through the interactive interface includes: The system captures the operator's hand gestures, eye movements, or voice commands through sensors, converts them into digital signals, and identifies them as specific view control commands, path navigation commands, or object selection commands through the command parsing module.

[0029] Furthermore, the step of parsing the inspection control command into control parameter adjustment commands for the 3D scanning system includes: Based on the type and parameters of the inspection control commands, the predefined command mapping rule library is queried to parse the view control commands into the pitch and yaw angle adjustment values ​​of the laser scanning module, the path navigation commands into the speed and direction control sequence of the mobile platform, and the object selection commands into the boundary coordinates and resolution settings of the scanning area.

[0030] Furthermore, the step of adjusting the scanning posture or scanning path by the actuator of the driving 3D scanning system includes: The servo motor of the gimbal is controlled to rotate to change the laser emission direction of the laser scanning module, or the linear motor is controlled to drive the moving platform to move along the Bézier curve trajectory in three-dimensional space. At the same time, the scanning frequency and scanning range of the laser scanning module are adjusted to ensure the continuity and integrity of data acquisition.

[0031] In this embodiment, the operator wears augmented reality glasses, which overlay a fused virtual and real scene onto their field of vision. The operator tracks a specific pipe on the virtual model of the heat exchanger using eye-tracking and issues a voice command to inspect the pipe in detail. The interface recognizes this combination of inputs as a clear object selection instruction. The instruction parsing module, according to predefined rules, parses the instruction into a set of specific control parameters: including the 3D spatial bounding box coordinates of the target pipe requiring high-precision scanning in the global coordinate system, and an instruction to increase the laser scanner resolution to the highest level.

[0032] Furthermore, the step of integrating the updated data into the 3D reality model includes: The newly acquired point cloud data is matched with the point cloud of the existing 3D reality model using an octree-based spatial index matching. The overlapping areas are merged and new area point clouds are added using a point cloud fusion algorithm. Then, the triangular mesh vertices and texture images of the corresponding areas are updated, and the mesh normals are recalculated to maintain the lighting consistency of the model.

[0033] In this embodiment, control commands are issued to the robot platform. The actuators begin to operate: the gimbal servo motor drives the laser scanner to precisely steer, aligning the scanning line of sight with the target pipe; simultaneously, the robot's moving chassis is fine-tuned to obtain the optimal scanning angle. The scanner performs a local rescan of the pipe at a higher resolution, acquiring updated and more detailed point cloud and texture data. The new data is transmitted back to the processing unit in real time. The update data integration step is initiated: the system integrates the newly acquired local high-precision point cloud with the existing full-scene 3D reality model. The updated area is quickly located using spatial indexing, the old data is replaced with the new point cloud data, and the triangular mesh and texture map of that area are regenerated. At this point, the geometric details and surface conditions of the target pipe in the 3D reality model, such as rust and scale, are updated and enhanced in real time.

[0034] Furthermore, the step of generating a combined virtual and real inspection view through fusion rendering includes: In the graphics rendering pipeline of the display terminal, the 3D real scene model is used as the base layer for real-time lighting rendering, and the virtual inspection model is used as the overlay layer for highlighting and contour enhancement rendering. The two images are superimposed through transparency blending technology, and interactive controls and status information are superimposed on the view to form an interactive virtual-real fusion visualization interface.

[0035] In this embodiment, the graphics rendering engine performs simultaneous fusion rendering. It overlays the updated 3D reality model as a background base onto the highlighted outline of the virtual design model. The target pipe is colored with a warning color in the virtual model, while its latest surface condition is displayed on the reality model. This combined virtual and real view is presented in real time on the operator's augmented reality glasses and the large monitoring screen in the background. The operator can clearly see the precise combination of virtual identifiers such as pipe numbers and design parameters with the actual equipment status, and can rotate and zoom the fusion view through gestures to examine it from various angles.

[0036] Furthermore, the method also includes a mapping maintenance step: During the inspection process, verification point cloud data of key areas is periodically collected through a 3D scanning system and compared with the virtual inspection model for local features. When the deviation in the mapping relationship is detected to exceed the dynamic threshold, the local remapping subprocess is automatically triggered to update the virtual-real mapping relationship matrix and write the adjustment record to the log file for traceability analysis.

[0037] In this embodiment, the graphics rendering engine performs simultaneous fusion rendering. It overlays the updated 3D reality model as a background base onto the highlighted outline of the virtual design model. The target pipe is colored with a warning color in the virtual model, while its latest surface condition is displayed on the reality model. This combined virtual and real view is presented in real time on the operator's augmented reality glasses and the large monitoring screen in the background. The operator can clearly see the precise combination of virtual identifiers such as pipe numbers and design parameters with the actual equipment status, and can rotate and zoom the fusion view through gestures to examine it from various angles.

[0038] Throughout the inspection mission, the system runs a mapping relationship maintenance process in the background. At regular intervals, the system automatically controls the scanner to perform a rapid verification scan of several pre-defined key feature areas, such as main flange connections. The acquired verification point cloud is then locally compared with the virtual model transformed according to the current mapping relationship matrix. In the experimental test, a minor offset caused by slight thermal deformation of the equipment was simulated. The system successfully detected this deviation and automatically triggered a local remapping subprocess, fine-tuning the mapping relationship only for the affected area and updating the mapping relationship matrix, thus ensuring the continuous accuracy of the virtual-real mapping throughout the long-cycle inspection. All adjustment actions are recorded in the log.

[0039] To verify the effectiveness of this method, a comparative test was designed. Traditional methods involve handheld point cloud scanning followed by offline modeling and then comparison with drawings, a process that is time-consuming and lacks interactivity. After applying this method, the tests show that: Efficiency improvement: The time cycle from on-site data collection to the generation of an interactive virtual-real fusion scene has been greatly shortened, achieving near real-time model building and mapping.

[0040] Interaction accuracy: The interactive commands issued by the operator, such as selecting a specific tube bundle, can be accurately parsed and executed by the system. After the collected data is fused after adjustment by the scanning system, the positioning deviation between the virtual marker and the physical target remains at an extremely low level, meeting the requirements for precise guidance.

[0041] Model fidelity and dynamism: The constructed 3D reality model is rich in detail and has realistic textures. After the system rescans key local areas, the model can be updated immediately, and the detail resolution of the updated area is significantly improved, truly reflecting the current state of the device.

[0042] Mapping robustness: Under simulated environmental vibration and equipment temperature rise scenarios, the system successfully maintained the stability of virtual-real space alignment through periodic mapping maintenance, and no mapping failure problem caused by cumulative errors occurred.

[0043] Inspection experience: Operators reported that inspections conducted through the augmented reality interface were intuitive and easy to control. They were able to quickly locate points of interest and obtain a comprehensive view that integrated design information and real-time status, significantly enhancing their decision support capabilities.

[0044] This embodiment fully demonstrates a closed-loop process from physical space to digital space, then controlling the physical scanning device through interactive command feedback, and achieving dynamic updates of the digital model. This method deeply integrates 3D real-scene modeling, high-precision spatial mapping, natural human-computer interaction, and real-time equipment control, providing an efficient, intuitive, and precise interactive control solution for industrial inspection.

[0045] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0046] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units. The above are merely embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made based on the description and drawings of this application, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

[0047] The specific embodiments of the invention have been described in detail above, but they are only examples, and this application is not limited to the specific embodiments described above. For those skilled in the art, any equivalent modifications or substitutions to the invention are also within the scope of this application. Therefore, all equivalent changes, modifications, and improvements made without departing from the spirit and principles of this application should be covered within the scope of this application.

Claims

1. A patrol interaction control method based on three-dimensional real-scene virtual-real mapping, characterized in that, Includes the following steps: S1 provides a three-dimensional scanning system, which integrates a laser scanning module and a positioning module for collecting three-dimensional geometric data and spatial location data of the target inspection area; S2, Based on the three-dimensional geometric data and spatial location data, a three-dimensional real-scene model of the target inspection area is constructed using a point cloud processing algorithm. The three-dimensional real-scene model includes geometric structure and surface texture. S3, Load the virtual inspection model corresponding to the target inspection area. The virtual inspection model is a digital model created based on design drawings or historical data. S4, execute the virtual-real mapping process, align the three-dimensional real scene model and the virtual inspection model in space through feature matching and coordinate transformation, and establish a one-to-one mapping relationship between the real scene and the virtual scene; S5, on the inspection interaction platform, receive inspection control commands input through the interactive interface, the inspection control commands including view control, path navigation or object selection; S6, based on the virtual-real mapping relationship, the inspection control command is parsed into a control parameter adjustment command for the three-dimensional scanning system; S7. According to the control parameter adjustment command, drive the actuator of the three-dimensional scanning system to adjust the scanning posture or scanning path to change the data acquisition area. S8: Real-time acquisition of updated data collected by the 3D scanning system after adjustment, and integration of the updated data into the 3D real-scene model to achieve dynamic updating of the model.

2. The method according to claim 1, characterized in that, The laser scanning module of the three-dimensional scanning system is a line laser scanner or an area array lidar, and the positioning module is a combined positioning unit of an inertial measurement unit and a global navigation satellite system. The steps for collecting three-dimensional geometric data and spatial position data of the target inspection area include: controlling the laser scanning module to emit laser beams at a preset sampling frequency and receiving echoes to generate discrete point cloud data; at the same time, the positioning module records the spatial coordinates and attitude angles corresponding to each point cloud data in real time, and binds and stores the point cloud data with the spatial coordinates and attitude angles.

3. The method according to claim 1, characterized in that, The steps for constructing a 3D reality model based on 3D geometric data and spatial location data include: The three-dimensional geometric data collected by the laser scanning module and the spatial location data recorded by the positioning module are spatiotemporally fused to generate a point cloud dataset with global coordinate system labels, in which each point contains three-dimensional coordinates and reflection intensity information. Spatial sorting and redundancy removal are performed on the point cloud dataset. Point cloud fragments collected at different times are stitched together into a continuous and complete point cloud surface representation through a point cloud registration algorithm to ensure the consistency of point clouds in overlapping areas. A surface reconstruction algorithm based on the Poisson equation is applied to triangulate the point cloud surface to construct an initial geometric mesh model, which consists of vertices, edges, and triangular faces, while preserving the topological structure. From the color image sequence synchronously acquired by the laser scanning module, the image pixels are mapped onto the corresponding triangular facets of the geometric mesh model through camera calibration and extrinsic parameter calibration. Texture mapping is performed using ultraviolet coordinates to form a preliminary textured 3D model. The initial 3D model is geometrically refined and smoothed, and the density and shape of the triangular facets are adjusted to eliminate holes and distortions. At the same time, the color differences between texture images are integrated to output the optimized final 3D reality model, which supports real-time rendering and interactive operation.

4. The method according to claim 1, characterized in that, The steps for performing the virtual-real mapping process include: Feature elements with significant geometric or textural features, including corner points, edge contours, and planar regions, are extracted from the 3D real-world model and the virtual inspection model, respectively. A feature descriptor describing the location and attributes of each feature element is generated. The similarity between the feature descriptors of the 3D real scene model and the feature descriptors of the virtual inspection model is calculated. The correspondence between feature elements is established by nearest neighbor search, and incorrect matching pairs are eliminated to improve the matching reliability. Based on the correspondence pair, the iterative nearest point optimization algorithm is used to calculate the spatial transformation parameters from the virtual inspection model to the 3D real scene model. The spatial transformation parameters include rotation matrix, translation vector and scaling factor to ensure that feature alignment error is minimized. The spatial transformation parameters are applied to the coordinates of all vertices of the virtual inspection model to perform an overall rigid transformation or non-rigid deformation on the virtual inspection model, so that it is precisely aligned with the key feature areas of the three-dimensional real scene model in three-dimensional space. The mapping accuracy is evaluated by calculating the distance between the sampled points of the aligned model and analyzing the visual overlap. If the accuracy does not meet the preset tolerance standard, the feature extraction sub-step is automatically returned for iterative optimization until the mapping accuracy meets the requirements, and the final virtual-real mapping relationship matrix is ​​solidified.

5. The method according to claim 1, characterized in that, The interactive interface is a gesture recognition interface in a virtual reality headset or an eye-tracking interface in augmented reality glasses, and the inspection control commands are generated through gesture, eye movement, or voice input. The step of receiving inspection control commands input by the operator through the interactive interface includes: The system captures the operator's hand gestures, eye movements, or voice commands through sensors, converts them into digital signals, and identifies them as specific view control commands, path navigation commands, or object selection commands through the command parsing module.

6. The method according to claim 1, characterized in that, The step of parsing the inspection control command into control parameter adjustment commands for the 3D scanning system includes: Based on the type and parameters of the inspection control commands, the predefined command mapping rule library is queried to parse the view control commands into the pitch and yaw angle adjustment values ​​of the laser scanning module, the path navigation commands into the speed and direction control sequence of the mobile platform, and the object selection commands into the boundary coordinates and resolution settings of the scanning area.

7. The method according to claim 1, characterized in that, The steps for adjusting the scanning posture or scanning path by the actuator of the driving 3D scanning system include: The servo motor of the gimbal is controlled to rotate to change the laser emission direction of the laser scanning module, or the linear motor is controlled to drive the moving platform to move along the Bézier curve trajectory in three-dimensional space. At the same time, the scanning frequency and scanning range of the laser scanning module are adjusted to ensure the continuity and integrity of data acquisition.

8. The method according to claim 1, characterized in that, The steps for integrating the updated data into the 3D reality model include: The newly acquired point cloud data is matched with the point cloud of the existing 3D reality model using an octree-based spatial index matching. The overlapping areas are merged and new area point clouds are added using a point cloud fusion algorithm. Then, the triangular mesh vertices and texture images of the corresponding areas are updated, and the mesh normals are recalculated to maintain the lighting consistency of the model.

9. The method according to claim 1, characterized in that, The steps for generating a combined virtual and real inspection view through fusion rendering include: In the graphics rendering pipeline of the display terminal, the 3D real scene model is used as the base layer for real-time lighting rendering, and the virtual inspection model is used as the overlay layer for highlighting and contour enhancement rendering. The two images are superimposed through transparency blending technology, and interactive controls and status information are superimposed on the view to form an interactive virtual-real fusion visualization interface.

10. The method according to claim 1, characterized in that, It also includes the mapping relationship maintenance step: During the inspection process, verification point cloud data of key areas is periodically collected through a 3D scanning system and compared with the virtual inspection model for local features. When the deviation in the mapping relationship is detected to exceed the dynamic threshold, the local remapping subprocess is automatically triggered to update the virtual-real mapping relationship matrix and write the adjustment record to the log file for traceability analysis.

Images