An augmented reality occlusion rendering method and system based on dynamic projection of regions of interest and propagation of asynchronous mask with check

Abstract

The application discloses an augmented reality (AR) occlusion rendering method and system based on dynamic projection of a region of interest (ROI) and asynchronous mask propagation with verification, and belongs to the field of computer vision and AR technology. In view of the problems of sparse point cloud penetration, monocular depth scale ambiguity and mask propagation failure under the mobile terminal AR thermal power constraint, the ROI is intercepted to compress the calculation domain, asynchronous reasoning is triggered through the multi-source joint mechanism of real-time positioning and mapping feature points, priori map and neural network processor, the mask and depth map are output through the shared feature multi-task network, the mask is propagated and the occlusion hole is repaired through depth-guided pose distortion in continuous non-key frames, and a three-level verification state machine based on inertial measurement unit mutation, visual tracking quality and propagation frame number life cycle is run to ensure robustness. Under the premise of ensuring the occlusion accuracy, the running power consumption is controlled within the mobile terminal thermal budget, and the application is suitable for mobile terminal AR navigation scenes.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, augmented reality, and real-time localization and mapping, specifically to a method and system for augmented reality occlusion rendering in scenarios where computing power and thermal power consumption are limited on mobile devices. Background Technology

[0002] In augmented reality navigation and interaction systems, correctly handling the depth occlusion relationship between virtual markers and the real physical environment is a core technical aspect for achieving immersive virtual-real fusion. Mobile augmented reality (AR) devices face the following engineering challenges due to limitations in on-chip system power consumption, heat dissipation capabilities, and the physical characteristics of embedded visual sensors.

[0003] The sparse feature point cloud output by mobile real-time localization and mapping (SLAM) systems lacks continuous physical boundary information. Directly using it as the basis for occlusion determination will produce jagged penetration artifacts at the edges of virtual objects, i.e., clipping. When deploying a continuous dense depth estimation network on a mobile device to solve this problem, the device needs to maintain high-frequency visual-inertial odometry tracking while running a 60FPS rendering pipeline. If network inference calculations are superimposed on each full-frame image, the device will trigger the on-chip system thermal protection mechanism after a few minutes of actual operation, causing the processor to be forced to reduce its frequency, and the rendering frame rate and tracking accuracy will decrease simultaneously.

[0004] Schemes using pure image optical flow or homography matrices for inter-frame mask propagation face the problem of lacking geometric constraints in non-planar obstacle scenes. Optical flow fields are constructed based on the assumption of constant planar brightness, which leads to geometric distortions in areas of depth discontinuity or obstacle edges. Furthermore, the lack of coupling with the absolute coordinate system of SLAM means mask drift cannot be corrected at a physical scale. When the camera moves rapidly or there are independent moving objects in the scene, the tracking assumptions upon which these schemes rely are broken, and existing technologies generally lack multi-level verification and exit mechanisms to address this situation.

[0005] Monocular depth estimation networks output relative depth, lacking an absolute physical scale. Directly overlaying the relative depth map with the absolute scale coordinate system of SLAM introduces scale ambiguity, causing occlusion determination boundaries to deviate from the real physical surface. When the viewpoint changes, previously occluded background areas are exposed as rendering holes. If these holes are not filled or resampling is not triggered, visual discontinuities will occur.

[0006] The aforementioned technical problems are not independent of each other. The computational cost contradiction between sparse point cloud and dense depth estimation, the alignment gap between relative depth and absolute scale, and the trade-off between geometric robustness and real-time performance in inter-frame propagation, constitute coupled technical obstacles under the thermal power consumption constraints of mobile devices. Summary of the Invention

[0007] The technical problem this invention aims to solve is that, under the strict constraints of computing power and thermal power consumption in mobile devices, existing occlusion rendering solutions cannot achieve an engineering-feasible balance between pixel-level precision occlusion detection, low-power continuous operation, and robustness in scenarios with intense motion. To address this technical problem, this invention provides a mobile AR occlusion rendering method, comprising the following steps:

[0008] Step 1: Obtain the camera's six-DOF pose and the spatial coordinates of the virtual identifier in the world coordinate system for the current frame. Use the camera intrinsic parameter matrix to project the virtual identifier onto the two-dimensional image plane. Expand the projection area outward according to a preset buffer ratio to construct a dynamic bounding box for the region of interest (ROI).

[0009] Step 2: Count the number N of sparse SLAM feature points in the region of interest. When N is greater than or equal to a preset threshold, extract the physical depth value of each feature point and compare it with the virtual rendering depth value point by point to determine whether there is occlusion.

[0010] Step 3: When N is less than the preset threshold and the prior environment map has been loaded, ray projection is performed along the direction of the region of interest based on the current pose, and the depth of the intersection point of the ray and the prior map is used as a supplementary physical depth reference value to participate in the occlusion comparison.

[0011] Step 4: Synchronously invoke the lightweight object detection model pre-deployed on the neural network processor (NPU) to determine whether there are intrusive dynamic objects in the region of interest;

[0012] Step 5: When any of the verification paths in Step 2, Step 3 or Step 4 determines that there is a real occlusion, generate an asynchronous occlusion trigger signal;

[0013] Step 6: Respond to the trigger signal, wake up the background asynchronous artificial intelligence (AI) inference thread, and extract the local image patch corresponding to the region of interest in the key frame as the network input;

[0014] Step 7: Employ a multi-task lightweight network structure with a shared feature extraction layer to output local two-dimensional occlusion masks and local relative depth maps in parallel;

[0015] Step 8: Extract the absolute depth values ​​of existing SLAM sparse feature points in the region of interest as anchor points, and adaptively select a hierarchical scale alignment strategy according to the number of anchor points to align the relative depth map to the absolute scale of SLAM and construct a 3D mask patch.

[0016] Step 9: In consecutive non-key frames, obtain the inter-frame pose transformation matrix of the current frame relative to the key frame, and perform depth-guided pose projection distortion on the 3D mask patch.

[0017] Step 10: Calculate the percentage of pixel area in the occluded hole region caused by the change of viewpoint during the distortion process. When the percentage of the hole is lower than the preset threshold, perform morphological repair. When the percentage of the hole is higher than the preset threshold, trigger the re-execution of steps 2 to 6.

[0018] Step 11: Run the real-time verification module in parallel as an independent state machine and set three priority interrupt conditions: the first priority is the detection of sudden changes in the angular velocity or linear acceleration of the inertial measurement unit (IMU), the second priority is the detection of the confidence of optical flow tracing or feature reprojection error in the edge region of the mask, and the third priority is the detection of timeout of the number of continuous propagation frames of the mask.

[0019] Step 12: When any priority interruption condition is met, the current mask is forcibly discarded and the occlusion check and asynchronous inference are retried.

[0020] The specific process of constructing the bounding box of the dynamic local region of interest in step 1 above is as follows: Obtain the six-degree-of-freedom pose matrix of the camera at the current moment and the spatial coordinates of the virtual navigation mark in the world coordinate system. Use the camera's pinhole intrinsic parameter matrix to project the virtual navigation mark onto the current two-dimensional image frame plane to obtain the smallest two-dimensional bounding rectangle containing the projection of the virtual mark. Based on this rectangle, expand it outward by 20% to 25% pixels in each of the four directions (up, down, left, and right) to construct the bounding box of the region of interest. This buffer area is used to accommodate the occlusion effect diffusion area caused by the change in the depth of adjacent pixels during actual rendering of the virtual mark.

[0021] Steps 2 to 5 above constitute a multi-source joint deep testing mechanism. The specific execution methods of the three testing paths are as follows:

[0022] For the depth comparison path based on sparse feature points in step 2: using a 100×100 pixel region as the baseline density detection unit, count the number N of effective sparse feature points output by SLAM in the region of interest of the current frame; when N≥4, extract the three-dimensional spatial coordinates of each feature point and calculate its physical depth value to the camera optical center, and compare it point by point with the depth value of the corresponding pixel in the virtual identifier rendering depth buffer. When the physical depth value of any feature point is less than the virtual rendering depth value, it is determined that there is occlusion; when N<4, the region is determined to be a weak texture or low feature region, and the sparse point cloud is insufficient to provide reliable occlusion determination, and the path is automatically downgraded.

[0023] For the ray casting path based on the prior map in step 3: when the system has preloaded the environmental prior map of the scene and N<4, one or more ray castings are performed along the direction of the region of interest based on the current camera pose, and the intersection depth of the ray and the prior map is taken as a supplementary physical depth reference value to participate in occlusion comparison; when the system is in a scene without a prior map, this path does not perform any calculation;

[0024] For the dynamic object perception path based on target detection in step 4: the lightweight real-time target detection model pre-deployed on the NPU receives the scaled input of the local image patch of the region of interest, and outputs the detection result of whether there is a dynamic object in the region and its two-dimensional bounding box; if the dynamic object bounding box overlaps with the projection area of the virtual marker, it is determined that dynamic intrusion occlusion occurs;

[0025] The above three inspection paths are executed in parallel logically and complement each other. When any inspection path determines that occlusion exists, an asynchronous occlusion trigger signal is generated and written into the message queue.

[0026] The multi-task lightweight network structure adopted in the above step 7 is as follows: the shared feature extraction layer adopts the front-end feature extraction part of the lightweight semantic segmentation network; the segmentation head performs upsampling and pixel-by-pixel classification on the basis of the shared features, and outputs the accurate two-dimensional occlusion mask of the region of interest; the depth regression head adopts a lightweight monocular depth regression structure, and outputs the pixel-by-pixel relative depth map of the region; the shared feature extraction layer enables the two tasks to reuse the same set of intermediate feature representations, and compared with executing two models independently, the total calculation amount and memory access amount are reduced.

[0027] The specific process of the hierarchical scale alignment strategy in the above step 8 is as follows: extract an anchor point set from the existing SLAM sparse feature points in the region of interest, and each anchor point provides the relative depth prediction value corresponding to its pixel coordinate and the absolute physical depth value in the SLAM coordinate system ; when the number of anchor points N≥4, the random sample consensus (RANSAC) algorithm is used to filter outlier points, then least squares fitting is performed based on the inlier point set to solve the affine transformation parameter

[0028] ,

[0029] where s is the scale factor and t is the translation offset. The solved parameters are applied to each pixel of the entire local relative depth map to obtain a depth map with the absolute SLAM scale, and a three-dimensional mask patch is constructed by combining the two-dimensional mask boundary; when 0<N<4, it degenerates to a median scaling strategy, which calculates the ratio of absolute depth to relative depth for each anchor point and takes the median as the global scale factor , using

[0030]

[0031] Linear scaling is performed, where t is zero. When N=0, the system dynamically closes the depth regression branch before entering network inference, and only performs segmentation head inference to output a two-dimensional mask. In the subsequent rendering stages of steps 9 to 10, this two-dimensional mask is directly used to cull the screen space pixels of the current frame.

[0032] The specific process of morphological repair and forced resampling in step 10 above is as follows: Calculate the proportion of the occlusion hole pixel area to the total pixel area of ​​the mask region; when the hole proportion is less than 5%, call the morphological dilation operation to post-process the binary mask, and use 3×3 or 5×5 square structural elements to grow the filling depth value from the existing effective pixels at the hole edge inward; when the hole proportion reaches or exceeds 5%, it is determined that the viewpoint deviation of the current frame relative to the key frame has exceeded the geometric tolerance of mask propagation, and trigger re-inference with the current frame as the key frame through the message queue.

[0033] The specific settings for the three-level priority interrupt conditions in step 11 above are as follows: The first priority directly reads the IMU output register, calculates the sliding window variance or instantaneous gradient of the angular velocity amplitude and linear acceleration amplitude of the continuous sampling points, and sets the interrupt flag with zero delay when the preset mutation threshold is exceeded. This path does not depend on image frame data, and the response delay is on the order of microseconds. The second priority is executed in the frame that is not triggered by the first priority. It extracts sparse optical flow tracking points in the edge region of the current mask, calculates the pixel displacement and tracking confidence of the corresponding points between adjacent frames based on the optical flow algorithm, or uses the visual feature reprojection error of the current frame by the SLAM system as a tracking quality metric. When the confidence is lower than the threshold or the reprojection error exceeds the threshold, an interrupt is triggered. The third priority records the number of consecutive propagation frames after the current mask is generated by the frame counter. When the count value reaches 15 frames, an interrupt is triggered. 15 frames correspond to an effective duration of about 250 milliseconds at a rendering rate of 60Hz. Resampling is forcibly started before the perspective distortion accumulates to an unacceptable level.

[0034] After any priority condition in step 12 above triggers an interrupt, the system executes the following takeover sequence: mark the current mask as invalid and release it from shared memory; clear the pending propagation tasks related to the mask in the message queue; send a new inference request with the current frame as the keyframe to the asynchronous inference thread, and re-execute the multi-source joint verification and asynchronous inference process.

[0035] Corresponding to the above method, the present invention also provides a mobile AR occlusion rendering system, including a dynamic region of interest capture module, a multi-source joint depth verification and triggering module, an asynchronous multi-task inference and scale alignment module, a depth-guided distortion and occlusion hole repair module, and a hardware priority mask verification and forced exit module. Each module performs steps 1 to 12 above respectively. The modules communicate across threads through locked shared memory and message queues. The system runs on a mobile device equipped with a heterogeneous computing unit (CPU), GPU (Graphics Processing Unit), and NPU.

[0036] The beneficial effects achieved by this invention are as follows: By dynamically extracting ROIs, the inference computation is limited to the local neighborhood of the virtual object projection. Combined with a multi-source joint triggering mechanism, high-computing-power inference is executed only at low frequency and asynchronously under real occlusion conditions. A multi-task network with a shared feature extraction backbone network is used to achieve joint computation compression of segmentation and depth estimation. While ensuring pixel-level occlusion accuracy, the continuous operating power consumption is controlled within the mobile terminal thermal budget. Through a hierarchical scale alignment strategy with SLAM sparse point clouds as anchors, adaptive fusion between relative depth prediction and absolute coordinate system is achieved, decoupling the depth scale coupling between monocular depth estimation and SLAM localization system. By replacing pure image domain mask propagation with depth-guided pose projection distortion, supplemented by morphological repair and forced resampling linkage strategies based on the occlusion hole area ratio, and a three-level forced exit state machine based on IMU hardware priority, visual verification, and propagation frame lifecycle, a complete abnormal takeover path is provided in the case of drastic camera movement, dynamic scene changes, and weak texture environments. Attached Figure Description

[0037] Figure 1 : System overall dual-thread architecture and occlusion rendering workflow diagram.

[0038] Figure 2 : Detailed schematic diagram of asynchronous joint inference and hierarchical scale alignment module.

[0039] Figure 3 State machine diagram for mask verification and forced exit based on hardware priority. Detailed Implementation

[0040] This system runs on a smart mobile device equipped with heterogeneous computing units including CPU, GPU, and NPU. The main rendering thread runs at 60Hz on the CPU / GPU, responsible for ROI capture, multi-source joint verification, pose projection distortion, and final culling rendering for each frame. The asynchronous AI inference thread runs on the NPU, only being awakened to perform low-frequency inference when an occlusion trigger signal is received. The two threads communicate across threads via locked shared memory and a message queue. After writing the trigger signal and related metadata to the message queue, the main thread continues to execute the rendering loop of the current frame without blocking and waiting for the inference result. After the NPU thread completes inference and scale alignment, it writes the processed mask data back to shared memory and sets the completion flag, which the main thread reads at the verification node of the next frame.

[0041] In the dynamic ROI construction in step 1, the system obtains the six-DOF pose matrix of the camera at the current moment and the spatial coordinates of the virtual navigation marker in the world coordinate system. Using the camera's pinhole intrinsic parameter matrix, the virtual navigation marker is projected onto the current two-dimensional image frame plane, obtaining the smallest two-dimensional bounding rectangle containing the projection of the virtual marker. Based on this rectangle, it expands outward by 20% to 25% pixels in each of the four directions (up, down, left, and right) to construct the region of interest bounding box. This buffer area is used to accommodate the occlusion effect diffusion area caused by the virtual marker due to changes in the depth of adjacent pixels during actual rendering.

[0042] In the multi-source joint depth verification process from steps 2 to 5, the system uses a 100×100 pixel region as the baseline density detection unit. The number of effective sparse feature points output by the SLAM simultaneous localization and mapping system within the region of interest in the current frame is counted and denoted as N. When N is greater than or equal to 4, the region is considered to have sufficient texture features. The 3D spatial coordinates of each feature point are extracted, and its physical depth value to the camera's optical center is calculated. This value is then compared point-by-point with the depth value of the corresponding pixel in the virtual identifier rendering depth buffer. If the physical depth value of any feature point is less than the virtual rendering depth value, it is determined that a physical surface is occluding the virtual identifier. When N is less than 4, the region is determined to be a weak texture or low-feature region, and the sparse point cloud is insufficient to provide reliable occlusion determination. If the system has pre-loaded a priori environmental map of the scene—including a semi-dense point cloud map or a semantic 3D model—then, based on the current six-DOF pose of the camera, one or more ray projections are performed along the direction of the region of interest. The depth of the intersection point between the ray and the prior map is used as a supplementary physical depth reference value for occlusion comparison. When the system is in a completely new scene without a prior map, this path does not perform any computation. Meanwhile, a lightweight real-time object detection model pre-deployed on the NPU—such as MobileNet-SSD or an equivalent edge-side inference model—receives scaled input from a local image patch of the region of interest and outputs the detection results and their 2D bounding boxes for the presence of pedestrians, vehicles, or other dynamic objects within that region. If the object detection model reports an overlap between the bounding box of a dynamic object and the virtual sign projection area, dynamic intrusion occlusion is determined to have occurred. The three verification paths—feature point depth comparison, prior map ray projection, and dynamic object detection—are logically executed in parallel. When any verification path determines that occlusion exists, an asynchronous occlusion trigger signal is generated and written to a message queue.

[0043] In the asynchronous inference and scale-aligned execution steps 6 to 8, the NPU thread retrieves the trigger signal from the message queue, obtains the corresponding keyframe timestamp and region of interest coordinates, and extracts the local image patch from the frame buffer. After size normalization, the image patch is fed into a multi-task lightweight network with a shared feature extraction layer. As a specific network structure example, the shared feature extraction layer can adopt the front-end feature extraction part of Fast-SCNN or a similar lightweight semantic segmentation network; the segmentation head outputs an accurate two-dimensional occlusion mask for the region of interest based on the shared features through upsampling and pixel-by-pixel classification; the depth regression head adopts a lightweight monocular depth regression structure similar to MiDaS-Small, outputting a pixel-by-pixel relative depth map of the region. The shared feature extraction layer allows the two tasks to reuse the same set of intermediate feature representations, compressing the total computation and memory access compared to executing two models independently.

[0044] During the scale alignment stage, the system extracts a set of anchor points from existing SLAM sparse feature points within the region of interest. Each anchor point provides a relative depth prediction value corresponding to its pixel coordinates. and the absolute physical depth value of the feature point in the SLAM coordinate system. The system performs hierarchical alignment based on the number of anchor points N. When N is greater than or equal to 4, the number of anchor points is sufficient to support robust statistical fitting: first, outliers are filtered using the RANSAC algorithm—outlier sources include depth outliers of SLAM feature points and local failures in network prediction at that pixel location—then least-squares fitting is performed based on the set of inliers to solve for the affine transformation parameters.

[0045] ,

[0046] Where s is the scale factor and t is the translation bias. The solved parameters are applied to each pixel of the entire local relative depth map to obtain a depth map with SLAM absolute scale, and a 3D mask patch is constructed by combining the 2D mask boundary. When N is greater than 0 and less than 4, the number of anchor points is insufficient to support RANSAC outlier filtering and robust fitting, and the system degenerates into a median scaling strategy: for each anchor point, the ratio of its absolute depth to its relative depth is calculated, and the median of the ratios is taken as the global scale factor. ,by

[0047]

[0048] The relative depth map is linearly scaled, and t is set to zero. When N equals 0—corresponding to extremely weak textures or featureless scenes such as white walls—the system has already predicted this state in step 2. The asynchronous inference thread dynamically closes the depth regression branch before entering network inference, and only performs segmentation head inference to output a two-dimensional mask. In the subsequent rendering stages of steps 9 and 10, this two-dimensional mask is directly used to perform screen space pixel culling rendering of the current frame, without involving depth-guided pose distortion calculations.

[0049] During the inter-frame propagation phase in steps 9 and 10, for consecutive non-keyframes after the current mask keyframe is generated, the main thread obtains the six-DOF pose transformation matrix of the current frame relative to the keyframe from the SLAM system. For each valid pixel in the 3D mask patch generated in step 8, using the pinhole camera model and the absolute depth value of that point, it is first back-projected from the keyframe pixel coordinate system to a 3D spatial point in the keyframe camera coordinate system, then transformed to the current frame camera coordinate system via the pose transformation matrix, and finally reprojected to the current frame pixel coordinate system via the camera intrinsic parameter matrix. During this projection process, the background area that was originally occluded by the foreground object in the keyframe viewpoint is exposed as an image hole—i.e., an occlusion hole—in the current frame viewpoint. The system calculates the proportion of the occlusion hole pixel area to the total pixel area of ​​the mask region. When the hole percentage is less than 5%, the viewpoint change is limited, and the propagated mask remains geometrically valid. The system calls a morphological dilation operation to post-process the binary mask, using 3×3 or 5×5 square structuring elements to grow the filling depth value inward from the existing valid pixels at the hole edge. The computational cost of this operation is negligible on the CPU and does not affect the 60Hz frame rate target of the main rendering thread. When the hole percentage is greater than or equal to 5%, the system determines that the viewpoint deflection of the current frame relative to the keyframe has exceeded the geometric tolerance of mask propagation, and the current propagation result cannot be used for reliable occlusion rendering. The system sends a forced exit signal to the verification module of steps 11 to 12 through the message queue and requests a re-inference with the current frame as the keyframe from the asynchronous inference thread.

[0050] In the real-time verification execution of steps 11 to 12, the three priority conditions run in parallel as state machines independent of steps 2 to 10. The first priority—IMU mutation detection—directly reads the output register of the inertial measurement unit, calculates the sliding window variance or instantaneous gradient of the angular velocity amplitude and linear acceleration amplitude of several consecutive sampling points, and immediately sets an interrupt flag when the preset mutation threshold is exceeded. The signal link of this path only involves reading the IMU hardware register and comparing the threshold, without relying on any image frame data or visual algorithm output, with a response latency on the order of microseconds. The second priority—visual verification—executed within frames where the first priority is not triggered: extracts sparse optical flow tracking points in the edge region of the current mask, calculates the pixel displacement of corresponding points between adjacent frames based on the Lucas-Kanade or equivalent optical flow algorithm, and uses the deviation between the optical flow prediction position and the feature matching position as a negative correlation index of tracking confidence; or uses the reprojection error between the visual feature points extracted by the SLAM system in the current frame and the map points—that is, the Euclidean distance between the feature points in the current frame image coordinates and the coordinates of the map points projected onto the image plane by the estimated pose—as a measure of tracking quality. When the optical flow confidence level is lower than a preset threshold or the reprojection error exceeds a preset threshold, the mask propagation is deemed to have failed due to scene changes or tracking degradation, triggering an interruption. The third priority—lifecycle management—uses a frame counter to record the number of consecutive propagation frames since the current mask was generated. The counter increments after each non-key frame is processed. An interruption is triggered when the count reaches 15 frames. 15 frames correspond to approximately 250 milliseconds of effective duration at a 60Hz rendering rate. This parameter is set based on the typical camera motion rate and scene structure change frequency of mobile walking navigation scenarios, forcibly initiating resampling before perspective distortion accumulates to an unacceptable level. After any priority condition triggers an interruption, the system executes the following takeover sequence: marking the current mask as invalid and releasing its occupied space from shared memory; clearing the pending propagation tasks related to the mask in the message queue; sending a new inference request with the current frame as the keyframe to the asynchronous inference thread, and re-executing the multi-source joint verification in steps 2 to 5 and the asynchronous inference process in steps 6 to 8.

[0051] The method and system described in this invention are applicable to application areas requiring augmented reality occlusion rendering on mobile devices, including but not limited to tourist attraction guides, museum artifact displays, and science and technology education and professional training venues such as water conservancy project demonstrations, fire emergency training, earthquake disaster simulations, and power facility inspections. Virtual navigation markers are arrows, path guidance labels, or points of interest (POIs), superimposed on the real-world scene captured by the device's camera. In this scenario, the method, through the ROI mechanism in step 1, limits inference computation to hundreds to thousands of pixels within the projection neighborhood of each virtual marker; through multi-source joint triggering in steps 2 to 5, NPU inference is activated only when physical occlusion actually occurs—in open corridors with sufficient feature points and no obstructions, the system can run continuously for tens of minutes without triggering inference once; through pose propagation in steps 9 to 10, matrix operations replace network inference in consecutive non-keyframes, with the additional computational cost per frame being far lower than full-frame image inference; and through three-level verification in steps 11 to 12, immediate takeover occurs when the user suddenly turns around, the device falls, or dense crowds pass through the navigation markers. For reference, on devices equipped with mid-range mobile SoCs, the continuous power consumption of monocular depth inference for full-frame images is in the range of several watts, and the SoC junction temperature rises to the threshold that triggers frequency reduction protection after running for several minutes; the continuous average power consumption of the embodiments of the present invention can be controlled in the range of hundreds of milliwatts, which meets the thermal budget constraints of long-term navigation sessions on mobile devices.

[0052] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A mobile augmented reality occlusion rendering method, characterized in that... The steps of this occlusion rendering method are as follows: Step 1: Obtain the camera's six-DOF pose and the spatial coordinates of the virtual identifier in the world coordinate system for the current frame. Use the camera intrinsic parameter matrix to project the virtual identifier onto the two-dimensional image plane. Expand the projection area outward according to a preset buffer ratio to construct a dynamic bounding box for the local region of interest. Step 2: Count the number N of sparse feature points in the real-time localization and mapping (SLAM) region of interest. When N is greater than or equal to a preset threshold, extract the physical depth value of each feature point and compare it with the virtual rendering depth value point by point to determine whether there is occlusion. Step 3: When N is less than the preset threshold and the prior environment map has been loaded, ray projection is performed along the direction of the region of interest based on the current pose, and the depth of the intersection point of the ray and the prior map is used as a supplementary physical depth reference value to participate in the occlusion comparison. Step 4: Synchronously invoke the lightweight object detection model pre-deployed on the neural network processor to determine whether there are intrusive dynamic objects in the region of interest; Step 5: When any of the verification paths in Step 2, Step 3 or Step 4 determines that there is a real occlusion, generate an asynchronous occlusion trigger signal; Step 6: Respond to the trigger signal, wake up the background asynchronous AI inference thread, and extract the local image patch corresponding to the region of interest in the key frame as the network input; Step 7: Employ a multi-task lightweight network structure with a shared feature extraction layer to output local two-dimensional occlusion masks and local relative depth maps in parallel; Step 8: Extract the absolute depth values ​​of existing SLAM sparse feature points in the region of interest as anchor points, and adaptively select a hierarchical scale alignment strategy according to the number of anchor points to align the relative depth map to the absolute scale of SLAM and construct a 3D mask patch. Step 9: In consecutive non-key frames, obtain the inter-frame pose transformation matrix of the current frame relative to the key frame, and perform depth-guided pose projection distortion on the 3D mask patch. Step 10: Calculate the percentage of pixel area in the occluded hole region caused by the change of viewpoint during the distortion process. When the percentage of the hole is lower than the preset threshold, perform morphological repair. When the percentage of the hole is higher than the preset threshold, trigger the re-execution of steps 2 to 6. Step 11: Run the real-time verification module in parallel as an independent state machine and set three priority interrupt conditions: the first priority is the detection of sudden changes in the angular velocity or linear acceleration of the inertial measurement unit, the second priority is the detection of the confidence of optical flow tracing or feature reprojection error in the mask edge region, and the third priority is the detection of timeout of the continuous propagation frame of the mask. Step 12: When any priority interruption condition is met, the current mask is forcibly discarded and the occlusion check and asynchronous inference are retried.

2. The method as described in claim 1, characterized in that: Steps 2 to 5 above constitute a three-level parallel verification mechanism. The first level is feature point level verification, which counts the number N of sparse SLAM feature points in the region of interest and performs point-by-point comparison between physical depth and virtual rendering depth when N is greater than or equal to a preset threshold to determine occlusion. The second level is the prior structure level check, which performs ray casting comparison based on the current pose when N is less than a preset threshold and an environmental prior map has been loaded; the third level is the dynamic object perception fallback check, which calls the lightweight real-time object detection model pre-deployed on the neural network processor to identify whether there are intrusive dynamic objects in the region of interest.

3. The method as described in claim 1, characterized in that: In steps 6 and 7 above, the asynchronous AI inference thread runs on the device's neural network processor and communicates with the main rendering thread running on the central processing unit / graphics processor (CPU / GPU) through locked shared memory and message queues. The multi-task lightweight network structure includes a shared feature extraction layer, a segmentation head for outputting the local two-dimensional occlusion mask, and a depth regression head for outputting the local relative depth map.

4. The method as described in claim 1, characterized in that: In step 8 above, the hierarchical scale alignment strategy includes: when the number of SLAM sparse feature points N in the region of interest is greater than or equal to 4, using the Random Sample Consensus Algorithm (RANSAC) to filter outgoing points and fitting affine transformation parameters based on the ingoing points. The relative depth of the network output is aligned to the absolute scale of SLAM, where s is the scale factor and t is the translation amount; when N is greater than 0 and less than 4, the scale alignment is degraded to a median scaling strategy based on existing anchor points; when N equals 0, the scale alignment is degraded to screen space pixel culling rendering in the current frame using only the local two-dimensional occlusion mask.

5. The method as described in claim 1, characterized in that: In step 10 above, the preset threshold for the percentage of the occluded hole pixel area is 5%; when the percentage of the hole is less than 5%, a morphological dilation operation with a 3×3 or 5×5 kernel is used to grow the filling depth value from the edge pixel of the hole inward; when the percentage of the hole is greater than or equal to 5%, it is determined that the viewing angle has been drastically deflected, triggering the re-execution of steps 2 to 6 and requesting asynchronous inference with the current frame as the keyframe.

6. The method as described in claim 1, characterized in that: In step 11 above, the specific settings of the three-level priority interruption conditions are as follows: the first priority directly reads the output register of the inertial measurement unit, calculates the sliding window variance or instantaneous gradient of the angular velocity amplitude and linear acceleration amplitude of the continuous sampling points, and sets the interruption flag with zero delay when the preset sudden change threshold is exceeded. This path does not depend on image frame data. The second priority is executed within frames where the first priority is not triggered. It extracts sparse optical flow tracking points in the edge region of the current mask, calculates the pixel displacement and tracking confidence of corresponding points between adjacent frames based on the optical flow algorithm, or uses the reprojection error of the visual features of the current frame by the SLAM system as a tracking quality metric. An interrupt is triggered when the confidence is lower than the threshold or the reprojection error exceeds the threshold. The third priority uses a frame counter to record the number of consecutive propagation frames since the current mask is generated. An interrupt is triggered when the count value reaches the preset maximum temporal threshold.

7. The method as described in claim 6, characterized in that: In step 11 above, the preset maximum timing threshold is 15 frames, which corresponds to a mask duration of approximately 250 milliseconds at a rendering rate of 60Hz. In step 12 above, after any priority condition triggers an interrupt, the system executes the following takeover sequence: marking the current mask as invalid and releasing it from shared memory, clearing the pending propagation tasks related to the mask in the message queue, sending a new inference request with the current frame as the keyframe to the asynchronous inference thread, and re-executing the occlusion verification and asynchronous inference process from step 2 to step 8.

8. A mobile augmented reality occlusion rendering system, characterized in that... include: The dynamic region of interest (ROI) capture module is used to obtain the six-DOF pose and spatial coordinates of the virtual identifier of the current frame based on camera projection, and to project them onto the two-dimensional image plane using the camera intrinsic parameter matrix and expand the projection area outward according to a preset buffer ratio to construct a dynamic local region of interest bounding box. The multi-source joint depth inspection and triggering module runs on the main rendering thread and is used to perform the multi-source joint depth inspection mechanism within the bounding box of the region of interest and generate an asynchronous occlusion trigger signal. The multi-source joint depth inspection mechanism integrates SLAM sparse feature point density comparison, environmental prior map ray projection downgrade fallback, and lightweight dynamic target detection on the neural network processor side to determine occlusion. The asynchronous multi-task inference and scale alignment module runs in the background asynchronous artificial intelligence inference thread. It is used to extract local image blocks of the region of interest of the key frame in response to the trigger signal. It outputs local two-dimensional occlusion mask and local relative depth map in parallel through a multi-task lightweight network with a shared feature extraction layer. It also adaptively selects a hierarchical scale alignment strategy based on the number of SLAM sparse anchor points in the region of interest to align the relative depth to the absolute scale of SLAM. The depth-guided warp and occlusion hole repair module is used to obtain the inter-frame relative pose transformation matrix in consecutive non-key frames, perform depth-guided pose projection warp on the scale-aligned 3D mask patch, calculate the pixel area ratio of occlusion holes and perform morphological repair when it is below the threshold, and issue a forced exit command when it is above the threshold. The processed binarized mask is used as a clipping mask to perform culling rendering. The hardware priority mask verification and forced exit module is used to set a three-level state machine in parallel: a first priority interrupt based on inertial measurement unit mutation, a second priority interrupt based on visual verification, and a third priority interrupt based on the lifetime of propagation frames. The first priority is based on direct reading of the inertial measurement unit hardware registers, the second priority is based on visual tracking quality, and the third priority is based on the propagation frame count. If any condition is met, the current mask is forcibly discarded and the multi-source joint depth verification and triggering module and the asynchronous multi-task inference and scale alignment module are retried.

9. The system as described in claim 8, characterized in that: The system runs on a mobile device with a heterogeneous computing unit consisting of a central processing unit / graphics processor (CPU / GPU) and a neural network processor; the main rendering thread runs on the CPU / GPU, and the asynchronous artificial intelligence inference thread runs on the neural network processor. The two communicate across threads through locked shared memory and message queues.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images