Virtual object pose estimation method and system for optical head-mounted display calibration

Abstract

The invention discloses a virtual object pose estimation method and system for optical head-mounted display calibration. The method comprises the following steps: establishing a cube vertex identity coding system based on color combination; a target instance segmentation framework model based on a Mask RCNN framework is constructed, through fixed six-channel output, an attention mechanism and deformable convolution are introduced, and a loss function including color consistency constraint is adopted for optimization; during training, transfer learning and a multi-scale data enhancement strategy are adopted, and a final model can accurately output independent masks of six color surfaces of a cube; during application, an input image is segmented, vertex 2D coordinates are calculated through geometric fitting and reasoning, and vertex identity recognition and verification are carried out according to a color coding system; and then matching the 2D vertex with the 3D vertex, solving an initial pose by using an EPnP algorithm, and finally outputting high-precision six-degree-of-freedom camera pose parameters through nonlinear optimization integrating reprojection errors and geometric constraints.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, augmented reality and artificial intelligence, in particular to a virtual object pose estimation method and system for optical head-mounted display calibration. BACKGROUND

[0002] In the fields of augmented reality, robot vision guidance and optical head-mounted display calibration, accurately estimating the position and pose (i.e. 6DoF pose) of a virtual or real object relative to a camera is a key technology. In particular, in an optical head-mounted display calibration system, a virtual calibration object with a known geometric structure often needs to be projected in the user's field of view, and the pose of the virtual object is calculated by capturing the image of the virtual object through the camera to complete the system calibration. However, the virtual calibration object has its unique challenges: first, it is digitally generated and lacks the rich natural texture of real objects; second, to meet specific optical calibration needs, its structure is often designed with hollow or transparent areas, which further increases the complexity of visual features.

[0003] Currently, existing object pose estimation mainly includes the following: 1. Pose estimation method based on traditional feature points and PnP This method is the most classic pose solving framework. It first detects specific feature points (such as corner points) in the image, then establishes the correspondence between these image 2D points and the known 3D model points of the object, and finally solves the pose using the perspective n-point algorithm. This scheme performs excellently when dealing with real calibration boards with high contrast and clear corner points (such as chessboard, ArUco code). However, when applied to virtual, especially hollow-structured cubes, this method faces fundamental challenges: Image quality defects: virtual projection has pixel aliasing and blurring effects, resulting in inaccurate positioning of real physical corner points in the image and easy generation of false corner points.

[0004] Insufficient feature points and occlusion: under single-view observation, part of the corner points of the cube will be occluded, resulting in insufficient number of detectable feature points, making the PnP algorithm unable to stably solve due to insufficient input data.

[0005] Severe interference of hollow structure: the circular hollow structure on the virtual cube produces a large number of arc edges similar to the visual features of real corner points, making it difficult for traditional corner point detection algorithms to effectively distinguish between true and false corner points. More importantly, this method cannot identify the identity information of each corner point, and under the combined action of hollow interference and the symmetry of the cube itself, it is difficult to establish a correct and accurate 2D-3D point correspondence relationship, and any erroneous matching will directly lead to the complete failure of pose estimation.

[0006] 2. End-to-end pose estimation network based on deep learning This type of method utilizes convolutional neural networks to attempt to directly regress the 6-DOF pose parameters of an object from one or more input images. It avoids explicit feature detection and matching steps. However, its core problem lies in treating pose estimation as a purely "end-to-end" regression task, failing to effectively utilize the inherent structured geometry of the virtual cube, thus leading to the following drawbacks: Lack of geometric constraints: The model learns the mapping from image appearance features to pose parameters only from the data, ignoring the strong geometric constraints inherent in the cube (such as planar structure, parallel and perpendicular relationships, etc.). This "black box" regression makes it difficult to guarantee the geometric rationality of the output pose.

[0007] Poor robustness to complex structures: When faced with complex situations such as hollowing out or transparency of virtual objects, the model has difficulty distinguishing between the edges of real structures and interference features due to the lack of explicit geometric reasoning process, resulting in a decrease in estimation accuracy.

[0008] Insufficient data dependence and generalization: This method requires training on a large dataset containing precisely labeled poses of the target object. The trained model is extremely sensitive to changes in the appearance of the virtual object (such as projection brightness and background changes), resulting in weak generalization ability. Furthermore, its black-box nature makes it difficult to diagnose and systematically correct estimation biases.

[0009] 3. Zero-shot visual localization method based on large models This is an emerging technological approach that utilizes large-scale multimodal pre-trained models to understand image content and describe or locate objects through natural language commands. This method demonstrates strong zero-shot generalization capabilities. However, its fundamental limitation lies in: Lack of measurement accuracy: The model's output is usually a natural language description or non-metric, qualitative positional information, and it is completely unable to output accurate 6-DOF pose parameters (i.e. translation vectors and rotation matrices) with physical scale significance.

[0010] Output format incompatibility: Its unstructured output format is completely incompatible with the structured, digital input interface required for downstream calibration calculations, and cannot be directly integrated into the existing calibration system pipeline.

[0011] Therefore, existing technologies face an irreconcilable contradiction between accuracy, generalization, and feasibility when dealing with pose estimation of virtual calibration objects. Traditional geometric methods, while highly accurate under ideal conditions, suffer from low feasibility when dealing with textureless and highly perturbed virtual objects. Deep learning methods attempt to address generalization issues through data-driven approaches but are hampered by accuracy and generalization bottlenecks caused by data dependence and the lack of geometric constraints. Large model methods, while possessing strong zero-shot generalization, lack metric precision and are format-incompatible, failing to meet the application requirements of high-precision calibration. Therefore, there is an urgent need in this field for a novel pose estimation technique that can balance high accuracy, strong generalization ability, and effectively address the unique characteristics of virtual objects. Summary of the Invention

[0012] To overcome the shortcomings of existing technologies, this invention proposes a virtual object pose estimation method and system for optical head-mounted display calibration, which is particularly suitable for calibration processes in mixed reality systems. The aim is to achieve accurate 6DoF pose estimation for textureless, hollow, and transparent virtual objects without relying on local corner points or texture features. This reduces the dependence on large amounts of labeled data for specific virtual objects, enabling them to quickly adapt to new virtual objects. This constructs a pose estimation system with high robustness to local occlusion and image noise, and establishes accurate 2D-3D point correspondences to resolve vertex identity ambiguity and symmetric pose ambiguity problems in pose solving.

[0013] The first aspect of this invention is to provide a virtual object pose estimation method for optical head-mounted display calibration, comprising: Step 1: Assign different colors to the six faces of the cube; define each vertex as a unique combination of the three colored faces that intersect it, forming a vertex identity coding system based on color combinations; Step 2: Build a Mask-based R The target instance segmentation framework model of CNN is improved by adopting a ResNet50 backbone network with integrated feature pyramids; introducing an attention module in the middle of the network, fixing the output channels of the segmentation head to 6 channels and integrating deformable convolutions to adapt to the deformation of the cube surface; and adopting a composite loss function with color consistency constraints. The model is trained using a transfer learning strategy, including: loading only the pre-trained weights of the backbone network during initialization; freezing the backbone network and training only the improvement modules during training; training using a pre-built hybrid dataset and applying multi-scale training and data augmentation techniques; and iteratively optimizing by monitoring the loss curve and cross-validation to obtain a model that can directly output the independent masks corresponding to the visible parts of the six color faces; each mask precisely corresponds to a specific color face of the cube. Step 3: Input a single RGB image, and use the trained target instance segmentation framework model to segment and obtain independent masks for six color faces; perform mask post-processing and geometric fitting on each segmented mask, and determine the optimal 2D coordinates of the vertices through geometric inference; determine the unique identity of each vertex according to the vertex identity encoding system based on color features established in Step 1, verify its geometric relationship, and output a complete vertex-face relationship graph containing 2D vertex coordinates and information of the color face to which it belongs; Step 4: Match the 2D vertex coordinates identified in Step 3 with the corresponding 3D vertices in the preset cube model; use the EPnP algorithm combined with RANSAC to solve the initial camera pose; construct an objective function that includes weighted projection error and cube geometric constraints, use the LM algorithm for iterative optimization, and finally output high-precision 6DoF camera pose parameters.

[0014] Furthermore, the construction of the hybrid dataset used in the training of the target instance segmentation framework model in this invention includes: generating synthetic sample images containing virtual cubes to form a synthetic sample dataset; acquiring images displayed through a head-mounted display using an industrial camera and a preset cube model, preprocessing the real images and calculating reference poses, and forming a real sample image dataset with the processed images and corresponding reference poses; and mixing the synthetic sample dataset and the real sample image dataset to construct a hybrid dataset.

[0015] Preferably, the construction of the hybrid dataset specifically includes: generating synthetic sample images using 3D model generation software and a preset cube model to form a synthetic sample dataset; acquiring images displayed through a head-mounted display using an industrial camera and a preset cube model, performing color enhancement and normalization processing on the acquired images, and calculating the cube pose of the acquired images using the PNP algorithm as a reference pose, with the processed images and the corresponding reference pose forming a real sample image dataset; and mixing the synthetic sample dataset and the real sample image dataset to construct a hybrid dataset.

[0016] Furthermore, step two specifically includes: A target instance segmentation framework model is constructed, based on Mask R-CNN architecture. Its core tasks are instance classification, localization, and pixel-level mask segmentation. For the unique structure of the virtual cube surface with large areas of uniform color blocks and circular cutouts, the model employs ResNet-50 combined with a feature pyramid network as its backbone to extract and fuse multi-scale features. A convolutional block attention module is introduced in the mid-stage of the backbone network. The model's segmentation head has a fixed output of six channels, corresponding to the six surfaces of the cube, and includes deformable convolutional layers to adapt to geometric deformations caused by perspective projection. During training, a composite loss function is used, adding a color consistency constraint loss to the basic segmentation loss. This color consistency constraint loss includes an internal uniformity loss to minimize the variance of pixel colors within each surface, and an interval discreteness loss to maximize the difference in average color vectors between different surfaces. Pre-trained weights are loaded only on the ResNet-50 portion of the constructed target instance segmentation framework model, and the weights of the model's improvement modules are randomly initialized. In subsequent training, the backbone network is frozen, and only the convolutional block attention module, region proposal network, detection head, and redesigned segmentation head are trained. 80% of the images in the pre-constructed mixed dataset are selected as the training set, and 20% as the validation set. The training set is input into the constructed target instance segmentation framework model for training, and multi-scale training and image augmentation processing are applied to the training set. During training, the model's fit is judged by monitoring the changes in the loss function value. When the loss function value stabilizes and no longer decreases, cross-validation is used to evaluate the model's segmentation accuracy and the model parameters are corrected until the final trained target instance segmentation framework model is obtained.

[0017] Furthermore, the convolutional block attention module in step two includes a parallel channel attention mechanism and a spatial attention mechanism; wherein the channel attention mechanism enhances the sensitivity to highly saturated colors, and the spatial attention mechanism guides the network to focus on large color block areas, suppressing background interference.

[0018] Furthermore, step two involves applying multi-scale training and image enhancement processing to the training set, including: the multi-scale training randomly scaling the input image to a resolution of 256×256 to 1024×1024; and the image enhancement processing includes random rotation, color adjustment, adding noise, and simulating occlusion blocks.

[0019] Furthermore, the mask post-processing and geometric fitting in step three specifically include: Morphological operations are performed on the six color face masks obtained from segmentation to fill the holes, and the effective mask with the largest area for each color is retained through connected component analysis. The maximum outer contour of each effective mask is extracted, and the mask face with the largest area is selected as the reference face. The reference face is directly fitted with the RANSAC algorithm to obtain an accurate quadrilateral. For the remaining auxiliary faces, RANSAC fitting is performed first, and then the cube geometric prior is introduced to optimize through parallel face verification and adjacent edge constraints. Finally, the optimal 2D coordinates of the vertices are determined through geometric reasoning.

[0020] More preferably, the steps for determining the optimal 2D coordinates of a vertex through geometric reasoning include: A differentiated strategy is adopted for RANSAC fitting: when all three relevant surfaces are perfectly fitted, the intersection of the quadrilateral edges is directly obtained by the least squares method; otherwise, the reference surface is used as the spatial anchor point. First, the possible position range of the target vertex on the reference surface quadrilateral is determined. Then, the geometric information of another well-fitted auxiliary surface is combined to calculate the line-surface intersection. Finally, the direction or region information of the third surface is used for constraint optimization. The optimal 2D coordinates of the vertex are determined by geometric reasoning.

[0021] Furthermore, the method also includes a verification step. After obtaining the 6DoF pose parameters in step four, the pose parameters obtained in step four are compared with the reference pose labeled in step one using a verification set. The rotation and translation errors of all 3D vertices involved in the solution are calculated after reprojection onto the image plane. The mean and maximum values ​​are statistically analyzed as internal consistency evaluation indicators. The degree of consistency between the cube representation size calculated by the pose and the known physical size is used to exclude serious scale errors. The user further filters and excludes unusable data based on the visualization results. Finally, the optimized 6DoF pose parameters are output, and a visualization comparison result containing the original image, segmentation mask, fitted contour, identified vertices, and reprojected wireframe model is generated.

[0022] A second aspect of the present invention is to provide a virtual object pose estimation system for optical head-mounted display calibration, comprising: A vertex identity encoding system construction module is used to establish a vertex identity encoding system for a virtual cube; wherein the vertex identity encoding system assigns different colors to the six faces of the cube, so that each vertex is uniquely identified by the three colored faces that intersect with it. The instance segmentation module includes a target instance segmentation framework model built on the Mask R-CNN framework. The model adopts a ResNet-50 backbone network with integrated feature pyramid network. An attention module is introduced in the middle stage of the backbone network. The number of output channels of the segmentation head is fixed at 6 and deformable convolutional layers are integrated. A composite loss function with color consistency constraints is adopted. The training adopts a transfer learning strategy. During initialization, pre-trained weights are loaded only in the ResNet-50 backbone network. During training, the backbone network is frozen and optimized using a pre-established mixed dataset. The geometric feature extraction module is used to input the received single RGB image into the target instance segmentation framework model trained by the instance segmentation module for image segmentation, obtain independent masks of the visible parts of the six color faces, and perform mask post-processing and geometric fitting on each segmented mask, determine the optimal 2D coordinates of the vertices through geometric inference; and determine the unique identity of each vertex based on the vertex identity encoding system construction module, and output a vertex-face topology graph containing 2D vertex coordinates and their association information with the color faces; The vertex calculation and pose solving module is used to match the 2D vertex coordinates obtained by the geometric feature extraction module with the corresponding 3D vertices in the preset cube model; solve the initial camera pose by combining the EPnP algorithm with RANSAC; construct an objective function that includes weighted projection error and cube geometric constraints, iteratively optimize it using the LM algorithm, and finally output high-precision 6DoF camera pose parameters.

[0023] The beneficial effects of this invention are as follows: The virtual object pose estimation method described in this invention designs a unique color coding mechanism and a phased training strategy that integrates real data. By first identifying abstract geometric surfaces, then calculating the intersection points of the surfaces, and then calculating the pose, it realizes the leap from easily disturbed local feature perception to global structural reasoning, ensuring that the model can achieve accurate virtual calibration object pose estimation in both synthetic and real scenes. Attached Figure Description

[0024] Figure 1 This is a flowchart of the virtual object pose estimation method for optical head-mounted display calibration described in this invention. Figure 2 These are the multiple cube images synthesized using Blender software in step two; Figure 3 The virtual cube image generated in step two by projection through an optical head-mounted display; Figure 4 It is a real cube image projected through an optical head-mounted display. Detailed Implementation

[0025] To make the objectives, technical solutions, beneficial effects, and significant advancements of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings provided in the examples of the present invention. Obviously, all the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] In the description of this application, unless otherwise expressly specified and limited, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance; the term "multiple" refers to two or more; unless otherwise specified or explained, the terms "connected," "fixed," etc., should be interpreted broadly. For example, "connected" can be a fixed connection, a detachable connection, an integral connection, or an electrical connection; "connected" can be a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0027] like Figure 1 As shown, a virtual object pose estimation method for optical head-mounted display calibration includes: Step 1: Establish a vertex identity coding system based on color features The six faces of the cube are assigned six colors: red, green, blue, yellow, purple, and pink. The eight vertices of the cube are each defined as a unique combination of three intersecting faces of the same color. For example, vertex V1 is the intersection of the red, green, and blue faces; vertex V2 is the intersection of the red, green, and yellow faces; and so on. This encoding method ensures that each vertex has a unique color characteristic. Even if a vertex is invisible due to circular cutouts or is occluded, as long as its three adjacent colored faces can be partially identified, the vertex's existence and identity can be uniquely determined through reasoning. This encoding provides a robust semantic foundation for subsequent vertex matching and is the source of the method's anti-interference capability.

[0028] Step 2: Construct a hybrid training set S21: Construction of the virtual sample training dataset Using Blender 3D scripting software and a preset cube model, 1000 composite sample images were generated and physically accurate rendered to produce a large-scale synthetic sample dataset. The following parameters were precisely controlled during the compositing process: Figure 2As shown, the preset cube model is a 3D cube model. Each face of the cube is given a different color, and each face uses a highly saturated pure color (fixed RGB value). The diameter of the circular cutout is controlled to be 2 / 3 of the cube's face width, and the material transparency is set to the range of 0.6-1.0. The image covers lighting variations of 200-1000 lux, pitch angle variations of ±60°, yaw angle rotation of 360°, random occlusion of 0-30%, and complex and varied background environments.

[0029] During the rendering process, the system synchronously generates the corresponding ground truth segmentation mask and the ground truth pose of the synthesized image (generated by the script). The ground truth mask accurately marks the pixel-level region of each color face and automatically handles the influence of the hollowed-out region, ensuring a high degree of consistency between the training data and the real application scenario.

[0030] S22: Construction of a Real Sample Image Dataset Two hundred real images, including varying brightness, glare, color deviation, and blur, were systematically acquired from multiple angles and distances using a high-resolution industrial camera and displayed on a head-mounted display, forming a real sample image dataset. Like the synthetic images, the acquired images were generated based on a pre-defined cube model, with each face of the cube using highly saturated solid colors. Color enhancement and normalization were performed on the acquired images, and manual data annotation was performed using annotation tools. The corner points were used to calculate the pose of the entire cube (the reference pose of the head-mounted display projected cube under the real image) using the PNP algorithm, which served as the reference pose for subsequent pose estimation models.

[0031] A cube projected in a dark environment via an optical head-mounted display, such as Figure 3 As shown.

[0032] A cube projected through an optical head-mounted display under normal lighting conditions, such as... Figure 4 As shown.

[0033] S23: A hybrid dataset is constructed by using a synthetic sample dataset and a real sample image dataset in an 8:2 ratio.

[0034] Step 3: S31: Construct a target instance segmentation framework model for instance segmentation of a virtual cube. The target instance segmentation framework model is based on a lightweight Mask R-CNN architecture, and its core task is to achieve instance classification, localization, and pixel-level mask segmentation. For the special structure of the virtual cube's surface with large areas of uniform color blocks and circular cutouts, the basic model was comprehensively optimized as follows.

[0035] The core architecture employs ResNet-50 combined with a feature pyramid network as its backbone. This backbone network begins with basic Stem layers for initial convolution and downsampling, followed by four progressively deeper feature extraction stages, named Res2, Res3, Res4, and Res5. This approach specifically selects the feature outputs from Res2, Res3, Res4, and Res5 stages, which possess varying characteristics from high resolution and rich detail to low resolution and strong semantic information. Through the top-down path of the feature pyramid network, these multi-scale features are fused and enhanced, generating unified feature maps at five levels: P2, P3, P4, P5, and P6, providing comprehensive multi-scale information for subsequent processing. The network extensively utilizes residual blocks, incorporating 1×1 convolutional dimensionality reduction, 3×3 convolutions, and 1×1 convolutional dimensionality enhancement, combined with residual connections, to ensure training stability and the effectiveness of feature representation.

[0036] To address the issue of insufficient feature extraction efficiency of general backbone networks for this task, four key optimizations were implemented. First, a CBAM attention module (convolutional block attention module) was introduced in the Res2 and Res3 intermediate stages of ResNet. This module combines channel attention and spatial attention mechanisms in parallel: channel attention learns the importance weights of each channel by analyzing global information to enhance the network's sensitivity to highly saturated colors in the cube; spatial attention learns a spatial weight map by fusing max and average pooling information and using large 11×11 convolutions, guiding the network to actively focus on large areas of uniform color blocks while suppressing interference from the cutouts and background, thereby better capturing continuous color blocks and sharpening edge geometric features.

[0037] Secondly, the segmentation head structure was redesigned, fixing its output channels to six, directly corresponding to the six color faces of the virtual cube, thus clearly defining the task as a six-class pixel classification problem. This segmentation head employs a five-layer convolutional architecture: the first two layers use standard 3×3 convolutions for basic feature extraction; the third layer inserts deformable convolutions, whose kernel offsets are dynamically learned by the network to adapt to the trapezoidal deformation of the cube faces under perspective projection; the last two layers use standard convolutions for feature refinement; finally, transposed convolutions are used to upsample to a 28×28 resolution output. Simultaneously, a multi-component composite loss function is used for optimization. In addition to the basic segmentation loss, a color consistency constraint loss is added, comprising two parts: internal uniformity loss encourages minimizing the pixel color variance within each color face mask; and interval discreteness loss encourages maximizing the difference between the average color vectors of different color faces. This constraint utilizes the physical property that the cube's faces are uniformly and distinctly colored, guiding the model to learn the correct spectral features.

[0038] S32: Only load COCO pre-trained weights into the ResNet-50 part of the backbone network, retaining the general feature extraction function. The weights of other improved regions are randomly initialized. When training with our own dataset, freeze the parts with loaded COCO weights and train only the CBAM module, RPN, detection head, and new segmentation head.

[0039] During training, the model is fine-tuned on the established hybrid dataset by leveraging its learned general object representation capabilities to adapt the model to the feature distribution of the virtual cube. For the mixed dataset constructed in step two, 80% of the images are selected as the training set and 20% as the validation set. The training set is then input into the target instance segmentation framework model constructed in S31 for training. During training, a multi-scale training strategy is implemented to allow the model to learn feature representations at multiple resolutions from 256×256 to 1024×1024, thereby enhancing its robustness to scale changes. OpenCV image processing technology is used to randomly rotate and slightly adjust the hue and saturation of the training set images to simulate color deviations under different lighting and display devices. Gaussian noise is added to randomly generate noise blocks with shapes similar to the target shape and superimpose them on the image to simulate additional occlusion, forcing the model to focus on the overall shape rather than local texture. By comprehensively using data augmentation techniques such as color jittering, random rotation, and Gaussian noise, the generalization ability of the model in complex real-world environments is significantly improved.

[0040] S33: During training, monitor the loss function value to determine if the model is overfitting or underfitting, and make appropriate adjustments. When the loss function no longer decreases, use cross-validation to evaluate the model's accuracy and correct the model, thus completing the training of the network framework and obtaining a target instance segmentation framework model capable of accurately segmenting the six color faces of a cube. The framework model can directly output six independent and accurate binary mask images, each mask corresponding to a specific color face of the cube.

[0041] Step 4: Vertex calculation and construction of vertex relationships S41: Input a single RGB image; use OpenCV image processing technology to normalize and enhance the image; use the target instance segmentation framework model trained in step three to perform color surface segmentation on the input image; S42: Perform quadrilateral fitting on the six segmented color face masks, construct a topological relationship graph between all fitted quadrilaterals, analyze the spatial adjacency, parallelism, perpendicularity and other relationships between each color face, deduce the possible spatial structure of the cube, and identify possible occlusion and missing situations. In the mask post-processing and geometric fitting stages, the six color face masks obtained from segmentation are first cleaned by filling small holes through morphological operations and performing connected component analysis to retain the effective masks with the largest area for each color to eliminate noise interference. Then, the maximum outer contour (i.e., face boundary contour) of each effective mask is extracted, and the face corresponding to the largest mask is selected as the reference face based on area comparison. The RANSAC algorithm is then used to directly fit the accurate quadrilateral. For the remaining auxiliary faces, RANSAC fitting is first attempted, and then a cube geometric prior is introduced for verification: the quadrilateral fitting result is optimized through parallel face verification and adjacent edge constraints. The fitting quality is judged using contour fitting error, mask region coverage, and geometric rationality as metrics. Contour fitting error is directly calculated by the average distance from the mask boundary points to each side of the fitted quadrilateral; the smaller the distance, the higher the fit. Mask region coverage is measured by the intersection-union ratio (IU), with a high IU ensuring the fitted shape completely covers the target area. Geometric rationality is verified by calculating the included angles of corresponding edge directions and comparing the position and direction of shared edges to check parallelism and geometric consistency. Each indicator is compared with a preset threshold. If it is lower than the preset threshold, the fitting quality is considered insufficient, triggering a downgrade process. Only the edge segments or main direction information are recorded for subsequent inference.

[0042] During the vertex calculation stage, a differentiated strategy is adopted for different fitting conditions: when all three relevant surfaces are perfectly fitted, the intersection of the quadrilateral edges is directly obtained by the least squares method; in the more common non-ideal case, the reference surface is used as the spatial anchor point. First, the possible position range of the target vertex on the reference surface quadrilateral is determined. Then, the geometric information of another well-fitted auxiliary surface is combined to perform line-surface intersection calculation. Finally, the direction or region information of the third surface is used for constraint optimization, and the optimal 2D coordinates of the vertex are determined by geometric reasoning.

[0043] S43: Based on the vertex identity encoding system established in step one, determine the corresponding color combination according to the color of the quadrilateral where the intersection point is located (or nearby), thereby determining the unique vertex; verify whether the relationship between all identified vertices conforms to the cube geometric constraints, thereby outputting a complete vertex-face relationship graph containing 2D vertex position and its corresponding color face information.

[0044] Step 5: Progressively optimize pose solution S51: Match the 2D vertices identified in step four with the corresponding vertices in the preset cube model; S52: Use the EPnP algorithm to solve the initial camera pose, combine it with the RANSAC mechanism to eliminate erroneous corresponding point pairs, and solve the preliminary camera rotation and translation relationship (6DoF pose) from the 2D-3D point pairs; at the same time, evaluate the rationality of the initial pose. S53: Construct an optimization objective function that includes a reprojection error term and a geometric constraint term. The reprojection error term uses a weighted Huber loss function to minimize the positional deviation between the detected 2D vertices and their corresponding 3D model points projected onto the current pose, and assigns differentiated weights to vertices with different reliability. Simultaneously, a geometric constraint term is introduced, utilizing the inherent orthogonality, parallelism, and equal-length side length priors of the cube to penalize pose solutions that violate the physical geometry. The Levenberg-Marquardt (LM) algorithm is used to iteratively optimize this objective function, and the convergence stability is monitored in real time during the optimization process to obtain high-precision 6DoF pose parameters (including rotation matrices, quaternions, and translation vectors). Convergence is monitored during the optimization process to avoid getting trapped in local optima.

[0045] S54: Before actual deployment, performance is evaluated using an independent validation set. On the validation set, the 6DoF pose parameters obtained from optimization in step five are compared with the reference pose labeled in step one. Pixel-level rotation and translation errors of all 3D vertices involved in the solution are calculated after reprojection onto the image plane, and their mean and maximum values ​​are statistically analyzed as internal consistency evaluation metrics for this estimation. Subsequently, the 6DoF pose parameters (including rotation matrices, quaternions, and translation vectors) are output. Simultaneously, the system automatically generates a visual comparison result including the original image, a color face segmentation result mask, the fitted quadrilateral contour, identified vertex markers, and a wireframe diagram of the complete cube model after reprojection based on the solved pose.

[0046] In the actual calibration process, multiple images are acquired as input, resulting in multiple pose results. To ensure the reliability of the output poses, the system implements a real-time verification mechanism. First, a physical plausibility check is performed to verify that the translation vector ensures the cube is in front of the camera (with a positive Z-coordinate) and within a preset reasonable observation distance. The cube's apparent size in the camera coordinate system is then calculated based on the pose, and this estimated value must roughly match the known physical size to eliminate serious scale errors. Subsequently, from the filtered data, the system automatically generates a visual diagnostic image, overlaying the original image, the segmented color mask, the fitted quadrilateral contour, the identified vertices, and the complete cube wireframe model reprojected based on the current pose. Finally, the user further eliminates unusable data.

[0047] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style of the specification is merely for clarity. Those skilled in the art should regard the specification as a whole, and the technical solutions in the embodiments can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A method for estimating the pose of a virtual object for optical head-mounted display calibration, characterized in that... include: Step 1: Assign different colors to the six faces of the cube; define each vertex as a unique combination of the three colored faces that intersect it, forming a vertex identity coding system based on color combinations; Step 2: Build a Mask R-based system The target instance segmentation framework model of CNN framework has been improved by adopting a ResNet50 backbone network with integrated feature pyramids; an attention module is introduced in the middle of the network; the output channels of the segmentation head are fixed to 6 channels and deformable convolution is integrated to adapt to the deformation of the cube surface. And a composite loss function including color consistency constraints is adopted; The model is trained using a transfer learning strategy, including: loading only the pre-trained weights of the backbone network during initialization; freezing the backbone network and training only the improvement modules during training; training using a pre-built hybrid dataset and applying multi-scale training and data augmentation techniques; and iteratively optimizing by monitoring the loss curve and cross-validation to obtain a model that can directly output the independent masks corresponding to the visible parts of the six color faces; each mask precisely corresponds to a specific color face of the cube. Step 3: Input a single RGB image, and use the trained target instance segmentation framework model to segment and obtain independent masks for six color faces; perform mask post-processing and geometric fitting on each segmented mask, and determine the optimal 2D coordinates of the vertices through geometric inference; determine the unique identity of each vertex according to the vertex identity encoding system based on color features established in Step 1, verify its geometric relationship, and output a complete vertex-face relationship graph containing 2D vertex coordinates and information of the color face to which it belongs; Step 4: Match the 2D vertex coordinates identified in Step 3 with the corresponding 3D vertices in the preset cube model; use the EPnP algorithm combined with RANSAC to solve the initial camera pose; construct an objective function that includes weighted projection error and cube geometric constraints, use the LM algorithm for iterative optimization, and finally output high-precision 6DoF camera pose parameters.

2. The virtual object pose estimation method for optical head-mounted display calibration according to claim 1, characterized in that, The construction of the hybrid dataset used during the training of the target instance segmentation framework model includes: generating synthetic sample images containing virtual cubes to form a synthetic sample dataset; acquiring images displayed through a head-mounted display using an industrial camera and a preset cube model, preprocessing the acquired images and calculating reference poses, and forming a real sample image dataset with the processed images and corresponding reference poses; and mixing the synthetic sample dataset and the real sample image dataset to construct a hybrid dataset.

3. The virtual object pose estimation method for optical head-mounted display calibration according to claim 1, characterized in that, Step two specifically includes: A target instance segmentation framework model is constructed, based on Mask R-CNN architecture. Its core tasks are instance classification, localization, and pixel-level mask segmentation. For the unique structure of the virtual cube surface with large areas of uniform color blocks and circular cutouts, the model employs ResNet-50 combined with a feature pyramid network as its backbone to extract and fuse multi-scale features. A convolutional block attention module is introduced in the middle stage of the backbone network. The model's segmentation head has a fixed output of six channels, corresponding to the six surfaces of the cube, and includes deformable convolutional layers to adapt to geometric deformations caused by perspective projection. During training, a composite loss function is used, which, in addition to the basic segmentation loss, adds a color consistency constraint loss. This color consistency constraint loss includes an internal uniformity loss to minimize the variance of pixel colors within each surface, and an interval discreteness loss to maximize the difference in average color vectors between different surfaces. Pre-trained weights are loaded only on the ResNet-50 portion of the constructed target instance segmentation framework model, and the weights of the model's improvement modules are randomly initialized. In subsequent training, the backbone network is frozen, and only the convolutional block attention module, region proposal network, detection head, and redesigned segmentation head are trained. 80% of the images in the pre-constructed mixed dataset are selected as the training set, and 20% as the validation set. The training set is input into the constructed target instance segmentation framework model for training, and multi-scale training and image augmentation processing are applied to the training set. During training, the model's fit is judged by monitoring the changes in the loss function value. When the loss function value stabilizes and no longer decreases, cross-validation is used to evaluate the model's segmentation accuracy and the model parameters are corrected until the final trained target instance segmentation framework model is obtained.

4. The virtual object pose estimation method for optical head-mounted display calibration according to claim 3, characterized in that, The convolutional block attention module in step two includes parallel channel attention mechanism and spatial attention mechanism; the channel attention mechanism enhances the sensitivity to high-saturation colors, and the spatial attention mechanism guides the network to focus on large color block areas and suppresses background interference.

5. The virtual object pose estimation method for optical head-mounted display calibration according to claim 3, characterized in that, Step two involves applying multi-scale training and image enhancement processing to the training set, including: the multi-scale training randomly scaling the input image to a resolution of 256×256 to 1024×1024; and the image enhancement processing includes random rotation, color adjustment, adding noise, and simulating occlusion blocks.

6. The virtual object pose estimation method for optical head-mounted display calibration according to claim 1, characterized in that, Step three, mask post-processing and geometric fitting, specifically includes: Morphological operations are performed on the six color face masks obtained from segmentation to fill the holes, and the effective mask with the largest area for each color is retained through connected component analysis. The maximum outer contour of each effective mask is extracted, and the mask face with the largest area is selected as the reference face. The reference face is directly fitted with the RANSAC algorithm to obtain an accurate quadrilateral. For the remaining auxiliary faces, RANSAC fitting is performed first, and then the cube geometric prior is introduced to optimize through parallel face verification and adjacent edge constraints. Finally, the optimal 2D coordinates of the vertices are determined through geometric reasoning.

7. The virtual object pose estimation method for optical head-mounted display calibration according to claim 6, characterized in that, The steps to determine the optimal 2D coordinates of a vertex through geometric reasoning include: A differentiated strategy is adopted for RANSAC fitting: when all three relevant surfaces are perfectly fitted, the intersection of the quadrilateral edges is directly obtained by the least squares method; otherwise, the reference surface is used as the spatial anchor point. First, the possible position range of the target vertex on the reference surface quadrilateral is determined. Then, the geometric information of another well-fitted auxiliary surface is combined to calculate the line-surface intersection. Finally, the direction or region information of the third surface is used for constraint optimization. The optimal 2D coordinates of the vertex are determined by geometric reasoning.

8. The virtual object pose estimation method for optical head-mounted display calibration according to claim 1, characterized in that, The method also includes a verification step. After obtaining the 6DoF pose parameters in step four, the pose parameters obtained in step four are compared with the reference pose labeled in step one using a verification set. The rotation and translation errors of all 3D vertices involved in the solution are calculated after reprojection onto the image plane. The mean and maximum values ​​are statistically analyzed as internal consistency evaluation indicators. The degree of consistency between the cube representation size calculated by the pose and the known physical size is used to exclude serious scale errors. The user further filters and excludes unusable data based on the visualization results. Finally, the optimized 6DoF pose parameters are output, and a visualization comparison result containing the original image, segmentation mask, fitted contour, identified vertices, and reprojected wireframe model is generated.

9. A virtual object pose estimation system for optical head-mounted display calibration, characterized in that... include: A vertex identity encoding system construction module is used to establish a vertex identity encoding system for a virtual cube; wherein the vertex identity encoding system assigns different colors to the six faces of the cube, so that each vertex is uniquely identified by the three colored faces that intersect with it. The instance segmentation module includes a target instance segmentation framework model built on the Mask R-CNN framework. The model adopts a ResNet-50 backbone network with integrated feature pyramid network. An attention module is introduced in the middle stage of the backbone network. The number of output channels of the segmentation head is fixed at 6 and deformable convolutional layers are integrated. A composite loss function with color consistency constraints is adopted. The target instance segmentation framework model is trained using a transfer learning strategy. During initialization, pre-trained weights are loaded only in the ResNet-50 backbone network. During training, the backbone network is frozen and optimized using a pre-established hybrid dataset. The geometric feature extraction module is used to input the received single RGB image into the target instance segmentation framework model trained by the instance segmentation module for image segmentation, obtain independent masks of the visible parts of the six color faces, and perform mask post-processing and geometric fitting on each segmented mask, determine the optimal 2D coordinates of the vertices through geometric inference; and determine the unique identity of each vertex based on the vertex identity encoding system construction module, and output a vertex-face topology graph containing 2D vertex coordinates and their association information with the color faces; The vertex calculation and pose solving module is used to match the 2D vertex coordinates obtained by the geometric feature extraction module with the corresponding 3D vertices in the preset cube model; solve the initial camera pose by combining the EPnP algorithm with RANSAC; construct an objective function that includes weighted projection error and cube geometric constraints, iteratively optimize it using the LM algorithm, and finally output high-precision 6DoF camera pose parameters.

Images